Debugging what your retriever returns

When an answer feels wrong, the fastest fix is to look at the chunks that went into it. If retrieval is returning junk, no prompt rewrite can save you. A few lines of logging turn retrieval from a black box into something you can actually reason about.

What to log on every query

A retrieval trace records each stage so you can see where quality was lost.

rag_utils.py
python
async def retrieve_with_trace(self, query: str, filters=None):
    query_embedding = (await self.embedding_provider.generate_embeddings([query]))[0]

    candidates = self.vector_store.query(
        query_embedding=query_embedding,
        n_results=20,
        where=filters,
    )

    # Log the vector search results before reranking
    for rank, doc in enumerate(candidates, start=1):
        print(f"[candidate {rank}] url={doc.url} preview={doc.content[:80]!r}")

    reranked = self.reranker.rerank(query, candidates, top_k=5)

    # Log what survived reranking
    for rank, doc in enumerate(reranked, start=1):
        print(f"[reranked {rank}] url={doc.url} preview={doc.content[:80]!r}")

    return reranked

A traced retriever prints both stages so you can see exactly which chunks were considered and which survived reranking. When answers look off, this is the first place to check.

Check the reranked chunks first. If the right passage is in there and the model still got it wrong, your prompt is weak. If the right passage is missing, widen the candidate pool or inspect your chunking. If the passage never made it into the index at all, the problem is ingestion, not retrieval.

Matching exercise: Match the symptom to the real root cause

Loading practice…

Quiz: Quiz

Loading practice…