Local embeddings with sentence transformers

Embeddings turn each chunk into a dense vector. Semantically similar chunks land near each other in that vector space. We could call a hosted embedding API, but that costs money per call and adds latency. Running sentence-transformers locally is free, fast, and offline.

embedder.py
python
import numpy as np
from sentence_transformers import SentenceTransformer


EMBEDDING_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
_model: SentenceTransformer | None = None


def _get_model() -> SentenceTransformer:
    global _model
    if _model is None:
        _model = SentenceTransformer(EMBEDDING_MODEL)
    return _model


def get_embeddings(texts: list[str], client=None) -> np.ndarray:
    """Embed a list of texts locally."""
    model = _get_model()
    embeddings = model.encode(texts, convert_to_numpy=True,
                               show_progress_bar=False)
    return embeddings.astype(np.float32)

The model loads once and stays cached in the _model global. MiniLM-L6-v2 produces 384-dimensional vectors and is small enough to run on any laptop.

MiniLM-L6-v2 is a sweet spot: small (around 80 MB), fast on CPU, and good enough quality for most retrieval tasks. For higher quality, swap in BAAI/bge-small-en or all-mpnet-base-v2. The API is identical, so you only change one constant.

Fill in the blanks: Swap the embedding model

Loading practice…

One heads-up about where this code will live: the app is a Streamlit script, and Streamlit reruns the whole script from top to bottom on every user interaction. Any expensive setup, like loading an embedding model, has to be cached so it happens once per process instead of once per click. That is exactly the job of the module-level _model cache above.

Quiz: Quiz

Loading practice…