Build a gold set

A gold set is twenty to fifty question-and-answer pairs that represent the real shapes of user questions. Not a thousand random queries. Not three cherry-picked examples. A tight, realistic set is what makes regression thresholds meaningful. If the set covers your failure modes, catching a regression is a one-line diff.

models.py
python
class QAPair(BaseModel):
    """A question with ground-truth answer for offline evaluation."""
    question: str
    ground_truth: str
    contexts: list[str] | None = None  # optional pre-provided contexts

class EvaluateRequest(BaseModel):
    """Batch offline evaluation request."""
    pairs: list[QAPair]
    rewrite_n: int = 3
    top_k: int = 5

class EvaluateResponse(BaseModel):
    """Response for batch evaluation."""
    scores: list[EvalScore]
    aggregate: EvalScore

Contexts are optional. If you omit them, the harness retrieves through the live pipeline. If you include them, the harness scores only the synthesis step, which is useful for isolating LLM quality changes from retrieval changes.

A good gold set covers four shapes: easy lookups the baseline must pass, multi-hop questions that exercise the sub-graph, paraphrased questions that exercise the rewriter, and privacy-sensitive questions that exercise the anonymizer. Each shape gets a handful of examples. Together they tell you which layer broke when a metric drops.

gold_set.json (example)
json
{
  "pairs": [
    {
      "question": "What embedding model does the pipeline use?",
      "ground_truth": "sentence-transformers all-MiniLM-L6-v2"
    },
    {
      "question": "which node runs before the synthesizer and how are embedding errors handled?",
      "ground_truth": "sub_retrieve runs before synthesize. Per-rewrite failures are logged and skipped without failing the request."
    },
    {
      "question": "how do we mask emails before sending to the model?",
      "ground_truth": "PIIAnonymizer uses Presidio to detect entities and replaces each with a placeholder like <EMAIL_1>, storing a reversible mapping."
    }
  ],
  "rewrite_n": 3,
  "top_k": 5
}

Keep ground truths short and specific. They drive context recall and give the scorer a concrete target to compare against.