Where vector RAG breaks

Most RAG tutorials start the same way: chunk the document, embed the chunks, store them in a vector database, retrieve by cosine similarity. This works for some problems, but it quietly fails on structured documents like research papers, legal filings, and technical manuals. Before we build the alternative, we need to understand exactly why.

Failure one: context splits across chunks. Your chunker slices a section about replication right before the sentence that defines the consistency model. The embedding for the replication chunk scores high, but the answer lives in the next chunk, which never gets retrieved.

Failure two: near matches beat right matches. Cosine similarity rewards passages that sound like the query. A question about "how Bigtable handles failures" matches paragraphs that mention failure handling, but the actual answer sits under a section called Recovery, which never uses the word failure in its body.

Rerankers help when the right chunk is in the candidate set. They do nothing when the chunker split the answer in half or the right section never matched in the first place. You are bolting a second model onto a retrieval strategy that was the wrong shape for the document.