Cross-encoder reranking

RRF gives you a strong set of candidates, but it treats the query and document as two independent encodings. A cross-encoder runs the query and candidate together through one model, so every attention head can compare them directly. It is slower per pair but far more precise. You only run it on a small candidate set, which makes the cost manageable.

Bi-encoder vs cross-encoder

Why reranking fixes precision at the top.

document_utils.py
python
from sentence_transformers import CrossEncoder

# Load once at module level. Tiny, CPU-friendly.
_RERANKER_MODEL = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

class RAGPipeline:
    def __init__(self, llm_provider):
        self.reranker = _RERANKER_MODEL

    def rerank(self, query_text, candidates, max_chunks=5):
        if not candidates:
            return []

        # Build (query, document) pairs
        pairs = [[query_text, doc.content] for doc in candidates]

        # One forward pass per pair. Small batch, manageable latency.
        scores = self.reranker.predict(pairs)

        # Sort by score and return the top K
        doc_scores = list(zip(candidates, scores))
        doc_scores.sort(key=lambda x: x[1], reverse=True)
        return [doc for doc, _ in doc_scores[:max_chunks]]

The reranker takes each query-document pair, scores it in one forward pass, and the top results move to the front. You run it on the fused RRF candidates, not the full corpus, which keeps latency acceptable.

document_utils.py
python
def retrieve_relevant_chunks(
    self,
    query_text,
    query_vector,
    query_sparse_vector,
    document_id,
    max_chunks=5,
):
    limit = max(20, max_chunks * 4)

    # Stage 1: Hybrid RRF fetch
    search_result = self.client.query_points(
        collection_name=f"doc_{document_id}",
        prefetch=[
            models.Prefetch(query=query_vector, using="dense", limit=limit),
            models.Prefetch(query=query_sparse_vector, using="sparse", limit=limit),
        ],
        query=models.FusionQuery(fusion=models.Fusion.RRF),
        limit=limit,
        with_payload=True,
    )
    candidates = [
        DocumentChunk(**point.payload, metadata=point.payload)
        for point in search_result.points
    ]

    # Stage 2: Cross-encoder reranking
    return self.rerank(query_text, candidates, max_chunks)

The full retrieval path in one function. Fetch a wide set with RRF, then narrow it to the strongest candidates with the reranker. The LLM only sees the final top K.

Hybrid retrieval is fast but imprecise at the very top. If you fetch only 5, the right answer might be at position 8 and you never see it. Fetching 20 gives the reranker room to find the best answer inside a realistic candidate pool. The reranker is slower per pair, but 20 pairs is still tens of milliseconds on CPU for a small cross-encoder. You trade a little latency for much better precision at position one.

Quiz: Quiz

Loading practice…

Validation checklist: End-to-end retrieval checklist

Loading practice…

Checkpoint: Hybrid retrieval checkpoint

Loading practice…