As we saw in my last blog post, there is a shape for stories.
The next ask we need to ask is, if the stories have a shape, can we engage that adviseation to produce RAG better. Turns out we can.
The shape of the stories helps us in chunking, which is a presentant problem area in RAG.
Chunking is one problem with RAG. Too huge a chunk, you neglect definiteity. Too petite a chunk, you neglect context. What’s the right size. This is the problem we try to mend by benevolent the “shape of stories”. Use the alter in rescheduleednt space to determine context alter.
Semantic chunking has been tried before and is useable in structuretoils enjoy LlamaIndex. But I wanted to get a better sense for the chunking.
Lets have a watch at some stories:
Here we can evidently see we can have two chunks.
This one is alittle challenginger, but if we zoom in, we can sustain some threshanciaccesss and discover a jump. Whenever there is a jump in the rescheduleednt space, we can chunk.
The presentant part is remending “what”, the jump is.
It can be as basic as the distance meacertain. Or we can watch at the slope etc.
We have experimented with this and are releasing an API that you can examine to see if this chunking strategy toils for you.
A comparison of the semantic chunking in Llamaindex to our chunking is given below
The chunking was done on Paul Grahams essay.
All the chunks for the essay can be set up in this gist.
As you can see, our chunking has much more immacurescheduleeder topics whereas the default semantic chunking bunches a bunch of topics into one chunk. Read the essay and chunk it manupartner and appraise 🙂
