Scaling to many source sites

One index can hold content from many sites at once. The trick is metadata filtering: tag every chunk with its source URL, and filter the query so the retriever only considers chunks from the sites you care about for a given question.

Multi-source retrieval with filters

One collection, many sources, filtered queries.

rag_utils.py
python
# Query a single source by filtering on URL metadata
results = vector_store.query(
    query_embedding=query_embedding,
    n_results=20,
    where={"url": "https://example-store.com/pricing"},
)

# Or query across a whitelist of sources
results = vector_store.query(
    query_embedding=query_embedding,
    n_results=20,
    where={"url": {"$in": [
        "https://example-store.com/pricing",
        "https://example-store.com/refund-policy",
    ]}},
)

ChromaDB's where clause runs before similarity search, so filtered queries are both correct and fast. The same index can serve any subset of your sources on demand.

router.py
python
@router.post("/add-url")
async def add_url(request: URLRequest, background_tasks: BackgroundTasks):
    """Queue a URL for background ingestion"""
    url_str = str(request.url)

    if url_str not in processing_status:
        processing_status[url_str] = {
            "url": url_str,
            "status": "processing",
            "progress": 0,
            "documents_count": 0,
        }
        background_tasks.add_task(
            process_url_background,
            url_str,
            request.chunk_size,
            request.chunk_overlap,
        )

    return {"url": url_str, "status": "processing"}

Ingestion runs in the background so the request returns instantly. Users can check progress via a status endpoint while chunks flow into ChromaDB.

Start with one collection and metadata filters. It is simpler to manage, easier to back up, and lets you run cross-site queries when that makes sense. Split into multiple collections only when one collection starts hurting performance or when sources have truly different schemas. Premature partitioning adds complexity without payoff.

Quiz: Quiz

Loading practice…

Validation checklist: End-to-end smoke test

Loading practice…