Chunk with a recursive splitter tuned for prose
Chunking is where most RAG pipelines silently lose quality. Too small, and you lose context around the answer. Too large, and irrelevant text dilutes the retrieval score. The recursive splitter tries a list of separators in order, so it prefers paragraph breaks, then line breaks, then spaces, only falling back to character-level cuts when needed.
from langchain_text_splitters import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
separators=['\n\n', '\n', ' ', ''],
)
splits = text_splitter.split_documents(documents)
print(f'Created {len(splits)} chunks')chunk_size=1000 characters is a good default for prose, roughly 150 to 250 words. chunk_overlap=200 means each chunk repeats the last 200 characters of the previous one, so an idea that straddles a boundary is still retrievable as a whole.
How overlap rescues straddling ideas
A sentence split by a hard boundary is still captured whole, thanks to the overlap window.
Ordering exercise: Order the separators from most preferred to last resort
Loading practice…
It is a reasonable default for prose, but the right number depends on your content. Dense technical text with definitions that span paragraphs may want 30 percent overlap. Very structured content like code or reference manuals can go down to 10 percent without losing retrieval quality. The way to tune it is to watch failure cases: if relevant chunks consistently miss, raise overlap.
Quiz: Quiz
Loading practice…
Checkpoint: Loader and chunker checkpoint
Loading practice…