Top-k in RAG: Tune retriever k on your own data

Every RAG tutorial uses k=5. You shipped with k=5 because the tutorial did. Some answers come back wrong because the chunk that holds the answer ranked 6th and never reached the model. You try k=20 to be safe. Now answers are slower, cost more, and the model gets distracted by chunks that do not matter.

Top-k in RAG is the retriever setting that decides how many chunks reach the LLM, by keeping only the k chunks that score highest against the question. The right k depends on your chunk size, your embedding model, your questions and your latency budget, so you measure it on your own data. This guide is the 3-step tuning process: a labeled eval set, a recall@k curve, and the knee of that curve.

Key takeaways

• Top-k is the number of retrieved chunks passed to the LLM. k=5 is a tutorial default, not a tuned value.
• Tune k with a labeled eval set of 50 to 100 real questions, then measure hit rate and recall at k=3, 5, 10 and 20.
• Pick the lowest k past the knee of the curve, where more chunks add little recall but still add tokens.
• Retrieve wide and keep narrow: fetch 50 or more candidates, rerank them, then pass the top k to the model.
• Re-tune k when you change the embedding model, the chunk size or the corpus, and test on a holdout set so k does not overfit.

What is top-k in RAG?

Top-k is how many chunks the retriever returns for one question. The vector store scores every chunk against the question embedding and sorts by score. The retriever keeps the first k and the prompt carries them to the model.

Do not confuse it with top-k sampling. Sampling top-k limits which next tokens the model may pick while it generates text. Retriever top-k limits which documents the model gets to read. They share a name and nothing else.

Why does k matter more than tutorials admit?

Because k sets the trade-off between recall and noise. Recall asks whether the retriever found the chunks that hold the answer. Noise is every chunk the model reads that does not help. Three failure modes show up in production:

1. k too low (k=3): recall drops on questions whose answer spans more than one chunk. The answer sits at rank 4 or 5 and never reaches the LLM.
2. k too high (k=20 with no reranker): token cost and latency grow with every chunk. Models also use the middle of a long context worst, so the right chunk at rank 11 can be ignored (Liu et al., 2023).
3. One k for every query type: a lookup question needs one chunk, a comparison needs both sides. One number fits neither.

How do you tune k in three steps?

Step 1: build an eval set with ground truth

Collect 50 to 100 real questions from production logs. For each one, record which documents hold the answer. This is the slow step, but you do it once and every later change reuses it.

# filename: eval/retrieval_eval_set.py
# description: Labeled eval set for retriever tuning.
EVAL_SET = [
    {
        'question': 'What is our refund policy?',
        'relevant_doc_ids': ['policy/refunds.md'],
    },
    {
        'question': 'How do I configure the API rate limiter?',
        'relevant_doc_ids': ['docs/rate-limiting.md', 'examples/rate-limiter.py'],
    },
    # ... 50+ more
]

Step 2: measure hit rate and recall at several k values

Run the retriever once per question at the largest k, then slice the results. Track two numbers, because they answer different questions:

• Hit rate@k: the share of questions with at least one relevant document in the top k. It tells you whether a single-chunk answer is reachable.
• Recall@k: the share of all relevant documents that made the top k, averaged over questions. It tells you whether multi-chunk answers have every piece they need.

# filename: eval/tune_k.py
# description: Hit rate and recall at several k values for retriever tuning.
async def measure_at_k(eval_set, retriever, ks=(3, 5, 10, 20)):
    hits = {k: 0 for k in ks}
    recall = {k: 0.0 for k in ks}
    for item in eval_set:
        retrieved = await retriever.search(item['question'], k=max(ks))
        relevant = set(item['relevant_doc_ids'])
        for k in ks:
            found = relevant & {r.doc_id for r in retrieved[:k]}
            hits[k] += bool(found)
            recall[k] += len(found) / len(relevant)
    n = len(eval_set)
    return {k: {'hit_rate': hits[k] / n, 'recall': recall[k] / n} for k in ks}

A typical recall curve for a well-built retriever looks like this:

• recall@3 = 0.72
• recall@5 = 0.85
• recall@10 = 0.93
• recall@20 = 0.97

Step 3: find the knee of the curve

The knee is where the curve flattens. In the example, going from 5 to 10 adds 8 points of recall. Going from 10 to 20 adds only 4 points and doubles the retrieved tokens. The knee is near 10.

Pick the lowest k that keeps about 95 percent of the recall you get at k=20. That is your starting point. In the RAG systems I have tuned, it lands between 5 and 10 without a reranker.

Should you retrieve more chunks and rerank?

Yes, when recall at your chosen k is still too low. A reranker is a cross-encoder model that reads the question and each candidate together and scores how well they match. It is slower than vector search per chunk, so you run it on a candidate pool, not on the whole corpus.

The pattern is retrieve wide, keep narrow. Fetch 50 to 150 candidates with vector search, or hybrid vector plus BM25 search. Rerank them. Pass the top k to the model. Anthropic’s Contextual Retrieval study retrieved 150 candidates and kept 20 after reranking. In that setup, 20 chunks beat 10 or 5, and reranking cut the retrieval failure rate from 5.7 percent to 1.9 percent.

So the right k also depends on what sits in front of it. With a reranker, a larger k is often cheap in quality terms because the ranks are better. Measure both setups on the same eval set before you choose.

How does k affect cost and latency?

Retrieved tokens grow linearly with k. If each chunk is 400 tokens, k=10 adds about 4,000 input tokens per question, and k=20 adds 8,000. The fixed parts of the prompt (system prompt, history, question) do not change, so total cost grows a little slower than k.

Rule of thumb: if going from k=5 to k=10 gains less than 5 points of recall, stay at k=5. If it gains 10 points or more, move to k=10. Prompt caching does not help here, because retrieved chunks change with every question. For tracing what each answer cost, see Langfuse and Grafana monitoring.

Should k change by query type?

Yes, if you can classify queries. Three buckets cover most traffic:

1. Lookup questions (“What is the refund policy?”): k=3 is enough. One chunk usually holds the answer.
2. Comparison questions (“How does X differ from Y?”): k=8 to k=12. You need content from both sides.
3. Multi-hop questions (“What led to X and who approved it?”): k=10 to k=15, or split the question into sub-queries as in multi-hop RAG with query decomposition.

Run a small, fast classifier on the question first, then set k. It adds one short model call. On lookup-heavy traffic it removes most of the retrieved tokens from the common case.

Another option is a score threshold: keep chunks above a similarity cutoff, capped at a maximum k. A threshold adapts per question, but raw similarity scores shift when you change the embedding model, so re-tune it with the same eval set.

How do you avoid overfitting k to your eval set?

Split the eval set 70/30 into tuning and holdout. Tune k on the 70 percent and check it on the 30 percent. If recall at the chosen k drops a lot on the holdout, the eval set is not representative and you need more varied questions.

Refresh the eval set every quarter. Production questions drift, and a stale set hides that drift. Add new questions from real logs and retire ones that no longer match the workload. The evals lab builds this loop end to end.

What to do next

1. Build a labeled eval set of 50 to 100 question and document pairs from real production queries.
2. Measure hit rate and recall at k=3, 5, 10 and 20. Plot the curve and find the knee.
3. Pick the lowest k that keeps about 95 percent of the recall at k=20. Ship it.
4. If recall is still short, add a reranker: retrieve 50 or more, keep the top k.
5. If you can classify queries, set k per query type.
6. Alert when recall@k drops below your baseline. A drop means the corpus or the model changed and k needs re-tuning.

Frequently asked questions

What is a good default top-k for RAG?

Start with k=5 to k=10 without a reranker, then measure. With a reranker over 50 or more candidates, k=10 to k=20 is common. Anthropic’s Contextual Retrieval study found 20 reranked chunks beat 10 or 5. The only reliable default is the one you measure: compute recall@k on a labeled eval set from your own documents and pick the knee of the curve.

What is the difference between top-k retrieval and top-k sampling?

Top-k retrieval decides how many document chunks the retriever passes to the LLM. Top-k sampling is a decoding setting that limits the model to the k most likely next tokens while it writes. They are independent. Changing retrieval k changes what the model reads. Changing sampling k changes how varied its wording is.

What is recall@k and why does it matter?

Recall@k is the share of relevant documents that appear in the top k results, averaged over your questions. Hit rate@k is the share of questions with at least one relevant document in the top k. Together they show whether the retriever finds the content an answer needs. If the right chunk is not in the top k, no prompt can fix the answer.

Does a bigger context window make k irrelevant?

No. A long context window lets you send more chunks, but every chunk still costs tokens and latency. Models also use information in the middle of a long context less reliably than information at the start or end. A tuned k, or a reranker that puts the best chunks first, still beats sending everything.

How often should I re-tune k?

Re-tune whenever you change the embedding model, the chunk size, the reranker or a large part of the corpus. Otherwise, check quarterly. Production questions drift, so refresh the eval set on the same schedule. An alert on recall@k against your baseline tells you when drift has started between scheduled checks.

Sources

• Introducing Contextual Retrieval, Anthropic, September 2024: the 150-candidate rerank to top 20 setup and the failure rate numbers.
• Lost in the Middle: How Language Models Use Long Contexts, Liu et al., TACL 2024: models use the start and end of a long context better than the middle.
• Evaluation in information retrieval, Manning, Raghavan and Schütze, Introduction to Information Retrieval: the textbook treatment of recall and precision.