The vectorless idea

Here is the core idea of vectorless RAG. A PDF already has structure: title, abstract, sections, subsections. Instead of throwing that away by chunking, you preserve it as a tree. Then an LLM walks the tree one node at a time, deciding at each step whether to descend, retrieve, or back off.

A large language model reading a section title and a short summary is surprisingly good at routing. Better than cosine similarity for most structured documents. You spend a few cheap LLM calls to navigate, then one call to answer. Total cost is often less than hosting a vector index.

Per query it is more expensive. Per system it often is not. Vector RAG has a constant tax: embedding on every ingest, hosting the index, re-embedding when documents change. Vectorless has zero infrastructure cost. For a small to medium corpus with infrequent queries, the math usually favors vectorless. You will measure this yourself later in the course.