Anyone tried the new alpha release yet?

LLogan M

Anyone tried the new alpha release yet? Definitely open to comments on any of the change made.

My favourite new feature is the new IngestionPipeline + cache

Plain Text

client = qdrant_client.QdrantClient(location=":memory:")
vector_store = QdrantVectorStore(client=client, collection_name="test_store")

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(chunk_size=25, chunk_overlap=0),
        TitleExtractor(),
        OpenAIEmbedding(),
    ],
    cache=IngestionCache(cache=RedisCache(), collection="test_cache"),
    vector_store=vector_store,
)

# Ingest directly into a vector db
pipeline.run(documents=[Document.example()])

# Create your index
from llama_index import VectorStoreIndex
index = VectorStoreIndex.from_vector_store(vector_store)

19 comments

KKaveen

Does this mean that you can dynamically build an index as your application operates?

KKaveen

e.g start with some set of documents, user uploads more, you run pipeline.run to add those documents to the vector store?

LLogan M

yessir

LLogan M

That was a big motivation for this -- separating ingestion from normal querying.

So now, you could technically deploy a microservice API just for ingestion

LLogan M

also the transformations thing is cool too -- makes it very clear what's happening to your data

KKaveen

this is HUGE!

KKaveen

This completely changes how my indexing functionalities work

KKaveen

Before, when people added documents I would create a separate index and then hook the agent up to that index by re-initializing the agent with that new index as a separate tool

KKaveen

now it can just query the same index!

LLogan M

yea as long as you are using some vector db backend (like qdrant, pinecone, etc.) this should work for that!

KKaveen

Oh it doesn't work for in-memory vector stores?

LLogan M

It should work for chroma and qdrant too. I suspect to for base vector store 🤔

kkrieger_

This is sick! Just wondering though, were you previously not be able to run just the ingestion inside of something like an AWS lambda function?

LLogan M

You could, but it wasn't exactly clear how to do this

This has clearer intentions, and is a lot more customizable

kkrieger_

Heck yes, love that! Def gonna play with this today

LLogan M

let me know if you run into any issues or have questions! A full release + proper docs is coming later this week 🙂

kkrieger_

Hey quick question, off the top of your head - would this work with pgvector/supabase?

LLogan M

It would! Or at least it should lol

kkrieger_

Lol sick, I'll try this out later today! I'm not married to pgvector but figured I'd ask before going that route

Add a reply

Find answers from the community

Anyone tried the new alpha release yet?