Dense and sparse embeddings

Dense vectors capture meaning. Sparse vectors capture exact words. A user searching for "error E1050" wants the sparse retriever to hit on the token E1050. A user searching for "how do I cancel" wants the dense retriever to match "end your subscription". You will generate both for every chunk.

Dense vs sparse vectors

Two vector types, two different retrieval strengths.

document_utils.py
python
from sentence_transformers import SentenceTransformer

# Load once at module level. 384-dim, fast on CPU.
_EMBEDDING_MODEL = SentenceTransformer("all-MiniLM-L6-v2")

class DocumentProcessor:
    def __init__(self):
        self.embedding_model = _EMBEDDING_MODEL

    async def generate_embeddings(self, texts):
        """Generate dense embeddings for a batch of chunks."""
        if not texts:
            return []

        loop = asyncio.get_event_loop()
        embeddings = await loop.run_in_executor(
            None,
            lambda: self.embedding_model.encode(texts, convert_to_tensor=False),
        )
        return embeddings.tolist()

all-MiniLM-L6-v2 is a solid default: 384 dimensions, runs comfortably on CPU, and has strong retrieval quality for its size. The encode call is offloaded to a thread so the event loop stays free.

document_utils.py
python
from collections import Counter
import re
from qdrant_client import models

class SimpleSparseEmbedder:
    """Hash-based sparse embedder, BM25-style."""

    def __init__(self, vocab_size: int = 30000):
        self.vocab_size = vocab_size

    def _tokenize(self, text: str):
        return re.findall(r"\w+", text.lower())

    def _hash_token(self, token: str) -> int:
        return int(abs(hash(token))) % self.vocab_size

    def compute_vector(self, text: str) -> models.SparseVector:
        tokens = self._tokenize(text)
        if not tokens:
            return models.SparseVector(indices=[], values=[])

        counts = Counter(tokens)
        indices, values = [], []
        for token, count in counts.items():
            indices.append(self._hash_token(token))
            values.append(float(count))

        return models.SparseVector(indices=indices, values=values)

The sparse embedder hashes each token into a 30k-dim space and records term frequency. For production use SPLADE or a corpus-tuned BM25. The shape of what we send to Qdrant is identical either way.

Not directly. BM25 adds inverse document frequency so common words like "the" contribute less, and it saturates term frequency. This simple hasher only uses term counts. For teaching and small corpora it works. For production, swap in SPLADE or FastEmbed BM25. The Qdrant interface stays exactly the same: a SparseVector of indices and values.

Quiz: Quiz

Loading practice…