Chunking documents

Chunking is the most under-appreciated step in RAG. Chunks that are too big dilute the embedding. Chunks that are too small lose context. Naive fixed-size splitters cut mid-sentence. The fix is a two-stage splitter: header-aware first, recursive within each section.

document_utils.py
python
from langchain_text_splitters import (
    RecursiveCharacterTextSplitter,
    MarkdownHeaderTextSplitter,
)

class DocumentProcessor:
    def __init__(self):
        # Stage 1: Split by markdown headers first
        self.md_splitter = MarkdownHeaderTextSplitter(
            headers_to_split_on=[
                ("#", "Header 1"),
                ("##", "Header 2"),
                ("###", "Header 3"),
            ]
        )
        # Stage 2: Recursively split each section to size
        self.recursive_splitter = RecursiveCharacterTextSplitter(
            chunk_size=500,
            chunk_overlap=50,
            separators=["\n\n", "\n", ". ", " ", ""],
        )

Two splitters stacked. The markdown splitter respects document structure. The recursive splitter handles sections that are still too long, trying natural break points before character counts.

Two-stage chunking flow

Header splits preserve structure. Recursive splits keep chunks in the target size.

document_utils.py
python
def chunk_document(self, document_content, chunk_size=500, chunk_overlap=50):
    content = document_content["content"]

    # 1. Split by markdown headers first
    md_splits = self.md_splitter.split_text(content)

    final_chunks = []
    chunk_idx = 0

    # 2. Recursively split each section
    for split in md_splits:
        headers = split.metadata  # header names become metadata
        sub_splits = self.recursive_splitter.split_text(split.page_content)

        for text in sub_splits:
            final_chunks.append({
                "content": text,
                "chunk_index": chunk_idx,
                "metadata": headers,
            })
            chunk_idx += 1

    # Fallback for plain text with no markdown structure
    if not final_chunks:
        for i, text in enumerate(self.recursive_splitter.split_text(content)):
            final_chunks.append({
                "content": text,
                "chunk_index": i,
                "metadata": {},
            })

    return final_chunks

Header names follow each chunk as metadata. Later, this becomes useful context for reranking and for answer citations.

Overlap protects against ideas that sit on the boundary between two chunks. A sentence that introduces a concept at the end of one chunk often explains it at the start of the next. Without overlap, the retriever can hit one chunk without the context that makes it meaningful. Fifty characters is a reasonable default for a five hundred character chunk. For technical content with dense definitions, raise both.

Matching exercise: Match the splitter to what it does

Loading practice…

Checkpoint: Document processing checkpoint

Loading practice…