What is the best chunking strategy for RAG?

A recursive split inside each heading with tables kept whole; test the others against it on your golden set (chapters 4 and 5).

Is semantic chunking worth it for RAG?

Usually not: a 2024 study found fixed-size chunking best on 3 of 5 evidence retrieval datasets (chapter 4).

How much overlap should RAG chunks have?

Start at zero and keep overlap only if recall rises more than IoU falls (chapter 5).

What is late chunking in RAG?

A long-context embedder reads a window of the document, then pools each chunk's tokens, so chunks keep their context (chapter 4).

How do I chunk PDFs for RAG?

Parse with a layout-aware parser, then chunk the elements with chunk_by_title or Docling's HybridChunker (chapters 3 and 4).

How do chunk size and the embedding model relate?

Every chunk with its context line must fit the embedder's token limit, counted with its own tokenizer (chapter 7).

Do I need RAG at all for a small knowledge base?

No, if it fits in the prompt; Anthropic puts the line at about 200,000 tokens (chapter 1).

How do I measure whether a chunking change helped?

Score recall at k and token IoU on exact excerpts, and keep a change only if recall rises more than IoU falls (chapters 2 and 9).

A free ebook, as a PDF and an EPUB

Chunking strategies for RAG: from raw documents to chunks

Parse, split, size, add context, embed and index your documents, and prove each change on your own golden set.

Look insidePages from the PDF, as they print.

Page 4: The answer was in the manual, split across two chunks.
Page 8: Store each golden answer as an exact excerpt, so the labels survive any rechunking.
Page 14: A parse error is a retrieval error before the first split.
Page 28: Proposition chunking turns each passage into one-fact statements, at a model call per passage.
Page 34: Small chunks are where overlap still earns its place.
Page 38: Context lines and BM25 cut failed retrievals, and a reranker cuts them further.
Page 48: A metadata filter applied after the vector scan can return almost nothing.

A free ebook on chunking strategies for RAG: clean PDFs and HTML, pick a chunker, size and overlap, add context, embed, index, and prove it on recall.

1The support assistant's path from raw documents to an answer3
2Measure recall on the split you have today7
3Clean the documents before you split them14
4Split by structure first, then test the clever chunkers21
5Pick chunk size and overlap from your questions, not a default32
6Give each chunk its source, section path and a context line37
7Fit the chunk to the embedding model43
8Index the chunks for hybrid search and metadata filters48
9Rerun the golden set and keep only what earned its cost53
10Keep the index in sync as the manuals change57
11Cheatsheet63
12Questions engineers ask about chunking64

What the ebook covers

1
The support assistant's path from raw documents to an answer
Acme Inc is a made up company. Its support docs assistant answers from a help center and a few hundred PDF manuals, and a customer's "Does the X200 warranty cover water damage?" comes back unanswered though the manual has it. This book walks the pipeline one step per chapter, with that question riding along.
2
Measure recall on the split you have today
Acme is at step 1. Before it changes anything, it records what the assistant retrieves today: a golden set of real questions with answer excerpts, a recall at k script and a baseline row to beat.
3
Clean the documents before you split them
Acme is at step 2. Recall is lowest on manual questions, and the parsed text is why. Keep headings and tables, then clean the text before the first split.
4
Split by structure first, then test the clever chunkers
Acme is at step 3 with clean, structured text. A recursive split inside each heading is the default that keeps the warranty table whole. Semantic, late and hierarchical chunking each get a fair test on the golden set, and each has to earn its cost.
5
Pick chunk size and overlap from your questions, not a default
Acme is at step 4. The split follows the manual's structure, but the size and overlap are still guesses. The evidence on overlap points both ways, so this chapter prints both sides and lets Acme's golden set decide.
6
Give each chunk its source, section path and a context line
Acme is at step 5. The chunks are the right size, but some still read as orphans: a maintenance step with no model name could belong to any product. Give each chunk its source, its section path and a short context line, and retrieve small chunks with their parents.
7
Fit the chunk to the embedding model
Acme is at step 6. A chunk with its context line in front is longer than the chunk alone, and an embedding model quietly drops whatever runs past its limit. Check every chunk against the model's tokenizer, then choose dimensions for the index.
8
Index the chunks for hybrid search and metadata filters
Acme is at step 7, storing chunks in Postgres with pgvector, with Qdrant shown as the alternative. The right chunk can still be lost here: filtered out after the vector scan, or missed by vectors when the question is a product code. HNSW settings, iterative scans and hybrid BM25 close those gaps.
9
Rerun the golden set and keep only what earned its cost
Acme is at step 8, the end of the map. Every change from chapters 3 to 8 is tested one at a time and stays only if recall rises more than IoU falls. Parser, chunker, size and context go through one script; an embedder or index change swaps evaluate's embed and search. Each remaining miss goes back to the step that caused it.
10
Keep the index in sync as the manuals change
Manuals get revised, retired and rechunked. Hash each one, replace only its chunks, stamp every row with the chunker and model that made it, and rebuild into a new index when either changes.
11
Cheatsheet
The defaults Acme ended with and the order to change them, on one page to keep beside the pipeline.

See the lab this ebook comes fromBuild a Second Brain with Claude Code: A second brain over your notes, docs and Linear that answers with sources and serves the same memory to any agent over MCP.

See the lab Get the ebook

Chunking strategies for RAG: from raw documents to chunks

The ebook is open in a new tab

Build it live, with an engineer reviewing your choices

Look insidePages from the PDF, as they print.

What the ebook covers

The support assistant's path from raw documents to an answer

Measure recall on the split you have today

Clean the documents before you split them

Split by structure first, then test the clever chunkers

Pick chunk size and overlap from your questions, not a default

Give each chunk its source, section path and a context line

Fit the chunk to the embedding model

Index the chunks for hybrid search and metadata filters

Rerun the golden set and keep only what earned its cost

Keep the index in sync as the manuals change

Cheatsheet

See the lab this ebook comes fromBuild a Second Brain with Claude Code: A second brain over your notes, docs and Linear that answers with sources and serves the same memory to any agent over MCP.