Model caching inside the image

SentenceTransformer downloads the model on first use and caches it under HF_HOME. Without intervention, every new pod cold-starts that download. Across a scale event, that is minutes of latency the user can feel.

Two workable strategies. Strategy one bakes the model into the image by running a single import at build time. The image gets bigger but pods start instantly. Strategy two mounts HF_HOME on a shared volume so the first pod downloads and subsequent pods reuse it. Pick based on your image registry cost and your cluster pattern.

Dockerfile (bake strategy)

dockerfile

# After uv sync, warm the HuggingFace cache in the image
RUN uv run python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('all-MiniLM-L6-v2')"

# Now the cache lives under /root/.cache/huggingface inside the image
ENV HF_HOME=/root/.cache/huggingface

This runs once at build time so the compressed model weights land in the image. Trade-off: image size grows by the size of the model. For all-MiniLM-L6-v2 that is roughly 90 MB, an easy trade for instant pod starts.

k8s/deployment.yaml (volume strategy)

yaml

volumeMounts:
  - name: chroma-data
    mountPath: /data/chroma
  - name: hf-cache
    mountPath: /root/.cache/huggingface
volumes:
  - name: chroma-data
    emptyDir: {}
  - name: hf-cache
    emptyDir: {}

This is a preview snippet from the Deployment manifest you will build in full when the service lands on the cluster; for now, focus on the volume shape. An emptyDir survives container restarts within a pod but not rescheduling. For cross-pod reuse, swap to a PersistentVolumeClaim backed by a read-many volume in your cluster.

Quiz: Quiz

Loading practice…