Embedding cache key design

Re-ingesting the same document should not pay the embedding cost twice. A small content-hash cache in front of the embedder turns repeated work into a dictionary lookup. The trick is keying on something stable across runs and across pods.

pipeline/cache.py
python
import hashlib
from typing import Dict, List

EMBED_DIM = 384


def key_for(text: str, model_name: str) -> str:
    """Stable cache key from model name + sha256 of normalised text."""
    norm = text.strip().lower()
    digest = hashlib.sha256(norm.encode('utf-8')).hexdigest()
    return f'{model_name}::{digest}'


class EmbedCache:
    def __init__(self) -> None:
        self._store: Dict[str, List[float]] = {}

    def get(self, key: str) -> List[float] | None:
        return self._store.get(key)

    def put(self, key: str, vec: List[float]) -> None:
        self._store[key] = vec

Two ingredients in every cache key: the model name (so a model swap busts the cache) and a sha256 of the normalised text. Same text plus same model gives the same key on every pod.

One discipline keeps this safe across pod restarts. The cache must be content-addressed, never id-addressed. If your input text changes but the document id stays the same, an id-keyed cache returns stale vectors. A content hash forces a re-embed exactly when it should.

pipeline/cache.py
python
from typing import List, Optional

_cache = EmbedCache()


def encode_cached(embedder, texts: List[str]) -> List[List[float]]:
    """Look up by content hash first, run the model only on misses."""
    keys = [key_for(t, embedder.model_name) for t in texts]
    vecs: List[Optional[List[float]]] = [_cache.get(k) for k in keys]

    miss_idx = [i for i, v in enumerate(vecs) if v is None]
    if miss_idx:
        fresh = embedder.encode([texts[i] for i in miss_idx])
        for i, vec in zip(miss_idx, fresh):
            _cache.put(keys[i], vec)
            vecs[i] = vec
    return vecs

The wiring is one small function in front of the embedder. Cache hits skip the model entirely, misses run through the normal encode path and get stored for the future. The ingest path calls encode_cached instead of encode, so the rest of the service never knows a cache exists.

Content-hash cache plus idempotent upsert

Together the content-hash cache and ChromaDB upsert make repeated ingests both fast and safe.

Quiz: Quiz

Loading practice…