Indexing items in Qdrant

The trained item tower turns each article into a 16-dimensional vector. We push those vectors into a Qdrant collection so that, at serving time, retrieving the top hundred candidates for a customer is a single ANN query.

scripts/seed_qdrant.py
python
store = PostgresStore()
vectors = VectorStore()
registry = ModelRegistry()

item_model = registry.load_item_model()
articles = store.read_table('articles', columns=['article_id','garment_group_name','index_group_name']).to_pandas()

ds = tf.data.Dataset.from_tensor_slices({
    'article_id': articles['article_id'].astype(str).values,
    'garment_group_name': articles['garment_group_name'].astype(str).values,
    'index_group_name': articles['index_group_name'].astype(str).values,
})
embeddings = []
for batch in ds.batch(2048):
    embeddings.extend(v.tolist() for v in item_model(batch).numpy())

vectors.ensure_collection(settings.QDRANT_ITEM_COLLECTION, vector_size=settings.TWO_TOWER_MODEL_EMBEDDING_SIZE, recreate=True)
vectors.upsert(settings.QDRANT_ITEM_COLLECTION, ids=articles['article_id'].astype(str).tolist(), vectors=embeddings)

Embed the catalogue with the saved item tower, ensure a fresh Qdrant collection exists, upsert in batches.

recsys/data/vector_store.py
python
class VectorStore:
    def search(self, name, query_vector, top_k=100):
        if hasattr(self._client, 'query_points'):
            response = self._client.query_points(
                collection_name=name,
                query=list(query_vector),
                limit=top_k,
                with_payload=True,
            )
            hits = response.points
        else:
            hits = self._client.search(
                collection_name=name,
                query_vector=list(query_vector),
                limit=top_k,
                with_payload=True,
            )
        return [{'id': str(h.id), 'score': float(h.score), 'payload': h.payload or {}} for h in hits]

Both the running Qdrant container and the embedded local-mode store implement the same search interface.

terminal
bash
make embeddings
curl http://localhost:6333/collections/articles_embeddings | jq .result

Seed and inspect the collection. Qdrant ships a dashboard at localhost:6333.

Quiz: Quiz

Loading practice…

Checkpoint: Retrieval checkpoint

Loading practice…