Why chunking matters

Imagine trying to find a specific cooking technique in Green Bites' 50-page kitchen manual. You wouldn't embed the entire manual as one vector because that would be like trying to describe a whole book in a single sentence. Chunking breaks documents into smaller, focused pieces.

Problem 1: Context window limits. Even modern LLMs have a maximum number of tokens they can process. A 50-page manual exceeds that limit, so you cannot send it all at once.

Problem 2: Cost and irrelevance. Even if the manual fits in the context window, sending all 50 pages when the user only asks about "oven temperature for sourdough" wastes tokens and money on irrelevant content.

Individual sentences are often too small and lose surrounding context. A sentence like "This applies to all orders" is meaningless without knowing what "this" refers to. The sweet spot is usually 200-500 characters: large enough to carry context, small enough to stay focused on one topic. We will explore the exact trade-offs shortly.

Problem 4: Blurry embeddings. A single embedding for 50 pages tries to capture everything and ends up capturing nothing well. Smaller chunks produce sharper, more focused embeddings that match specific queries.