Retrieve then synthesize with citations

A grounded answer is a retrieve-then-synthesize path. Embed the question with the same model used at ingest, fetch the most similar documents from ChromaDB, and hand them to the LLM as context. No context means no answer.

utils/llm_provider.py
python
from abc import ABC, abstractmethod
from typing import AsyncGenerator


class LLMProvider(ABC):
    @abstractmethod
    async def generate_stream(self, prompt: str, **kwargs) -> AsyncGenerator[str, None]:
        """Yield the answer text chunk by chunk."""


def get_llm_provider() -> LLMProvider:
    # Reads LLM_PROVIDER from env and returns the matching
    # implementation: OpenRouter by default, with Fireworks,
    # Gemini, and OpenAI variants behind the same interface.
    ...

The synthesis step talks to an LLM through this provider abstraction. Every provider implements the same async generate_stream contract, so the query path never knows which vendor sits behind it. get_llm_provider is the factory RAGService calls at startup, which is why switching providers is a config change rather than a code change.

pipeline/query.py
python
async def answer_question(
    question: str,
    embedder: RayEmbedder,
    index: ChromaIndex,
    llm_provider,
    top_k: int = 4,
) -> dict:
    # 1. Embed query
    query_vec = embedder.embed_batch([question])[0]

    # 2. Retrieve
    docs = index.query(query_vec, top_k=top_k)

    if not docs:
        return {
            'answer': "I couldn't find any indexed documents to answer your question.",
            'sources': [],
        }

    # 3. Synthesize
    prompt = PROMPT_TEMPLATE.format(
        context=_format_context(docs),
        question=question,
    )

    answer_chunks: List[str] = []
    async for chunk in llm_provider.generate_stream(prompt, temperature=0.2, max_tokens=600):
        answer_chunks.append(chunk)
    answer = ''.join(answer_chunks).strip() or '(no answer)'

    return {
        'answer': answer,
        'sources': [
            {'id': d.id, 'score': round(d.score, 4), 'preview': d.text[:200]}
            for d in docs
        ],
    }

Empty retrieval returns an honest no-answer rather than hallucinating. Sources are returned alongside the answer so the UI can surface citations and debugging teams can trace what the model read.

Quiz: Quiz

Loading practice…