BM25 keyword index with rank-bm25

Vector search is great at meaning, but it struggles with rare tokens. If a user asks about "Karpathy" or a specific library name, cosine similarity may not put the right chunk at the top. BM25 solves this: it is a classic IR ranking that rewards exact keyword matches while accounting for term frequency and document length.

keyword_index.py
python
from rank_bm25 import BM25Okapi
import re


class KeywordIndex:
    """Build and search a BM25 keyword index from transcript chunks."""

    def __init__(self, chunks: list[dict]):
        self.chunks = chunks
        self.corpus = [chunk["text"] for chunk in chunks]
        self.tokenized_corpus = [self._tokenize(text) for text in self.corpus]
        self.bm25 = BM25Okapi(self.tokenized_corpus)

    @staticmethod
    def _tokenize(text: str) -> list[str]:
        """Lowercase and split on word boundaries."""
        return re.findall(r"\w+", text.lower())

    def search(self, query: str, top_k: int = 5) -> list[dict]:
        query_tokens = self._tokenize(query)
        scores = self.bm25.get_scores(query_tokens)
        top_indices = sorted(
            range(len(scores)),
            key=lambda i: scores[i],
            reverse=True,
        )[:top_k]
        results = []
        for idx in top_indices:
            if idx < len(self.chunks):
                chunk = self.chunks[idx].copy()
                chunk["bm25_score"] = float(scores[idx])
                results.append(chunk)
        return results

BM25Okapi from rank-bm25 does the heavy lifting. Tokenization is deliberately simple: lowercase plus word-boundary regex. Matching the tokenization between corpus and query is the only rule that matters.

keyword_index.py
python
def build_keyword_index(chunks: list[dict]) -> KeywordIndex:
    """Convenience function to build a BM25 index from chunks."""
    return KeywordIndex(chunks)

A tiny wrapper so callers never touch the class directly. app.py imports build_keyword_index, hands it the chunks, and gets back a ready-to-search KeywordIndex.

Yes, and the gap is bigger than most people expect. Semantic search collapses to nearest-meaning, which can drown out exact names, acronyms, and numeric values. BM25 catches those with surgical precision. The two signals compensate for each other, and fusion gets you the best of both.

Quiz: Quiz

Loading practice…