Batch runner

The batch runner lives in the service layer. It iterates over QA pairs, uses provided contexts when they exist or retrieves through the live pipeline when they do not, synthesizes if needed, and hands the result to the RAGAS scorer. The aggregate function averages the per-pair scores so you get both fine-grained and rolled-up numbers.

service.py
python
async def evaluate(request: EvaluateRequest) -> EvaluateResponse:
    scores: list[EvalScore] = []
    for pair in request.pairs:
        if pair.contexts:
            contexts = pair.contexts
            answer = ""
        else:
            graph_result = await rag_graph.run_graph(
                question=pair.question,
                rewrite_n=request.rewrite_n,
                top_k=request.top_k,
                anonymize=False,
                with_evaluation=False,
            )
            contexts = [s["content"] for s in graph_result.get("sources", [])]
            answer = graph_result.get("answer", "")

        if not answer:
            from rag_graph import _synthesize_node
            synth_state = {"question": pair.question, "contexts": contexts, "pii_mapping": {}}
            synth_result = await _synthesize_node(synth_state)
            answer = synth_result.get("answer", "")

        scores.append(score_answer(
            question=pair.question,
            answer=answer,
            contexts=contexts,
            ground_truth=pair.ground_truth,
        ))

    return EvaluateResponse(scores=scores, aggregate=aggregate_scores(scores))

Note that evaluate runs with anonymize=False and with_evaluation=False on the inner graph, then scores separately. That avoids double-evaluation and keeps the PII layer from interfering with ground-truth comparisons.

ragas_utils.py
python
def aggregate_scores(scores: list[EvalScore]) -> EvalScore:
    """Average a list of EvalScore into a single aggregate."""
    if not scores:
        return EvalScore()

    def _avg(key: str) -> float | None:
        vals = [getattr(s, key) for s in scores if getattr(s, key) is not None]
        if not vals:
            return None
        return round(sum(vals) / len(vals), 3)

    return EvalScore(
        faithfulness=_avg("faithfulness"),
        answer_relevance=_avg("answer_relevance"),
        context_precision=_avg("context_precision"),
        context_recall=_avg("context_recall"),
    )

Simple mean across pairs. The None-skipping matters because context_recall is only present when a ground truth exists. Missing values are skipped rather than counted as zero, which would bias the aggregate.

terminal
bash
curl -X POST http://localhost:8000/advanced-rag/evaluate \
  -H "Content-Type: application/json" \
  -d @gold_set.json | jq '.aggregate'

One curl call runs the entire gold set through the pipeline, scores each answer, and returns per-pair scores plus an aggregate. This is the primitive your CI job will call.