Regression thresholds in CI

A gold set that nobody runs is decoration. The point is to wire the aggregate into CI so a regression fails the build. Pick thresholds you actually hit today on the current pipeline, then ratchet them up only when the pipeline improves. Never set aspirational thresholds. They just get ignored when they fire.

scripts/check_thresholds.py
python
import json, sys, httpx

THRESHOLDS = {
    "faithfulness": 0.80,
    "answer_relevance": 0.75,
    "context_precision": 0.60,
}

with open("gold_set.json") as f:
    payload = json.load(f)

resp = httpx.post("http://localhost:8000/advanced-rag/evaluate", json=payload, timeout=300)
resp.raise_for_status()
agg = resp.json()["aggregate"]

failures = []
for metric, floor in THRESHOLDS.items():
    score = agg.get(metric)
    if score is None:
        continue
    if score < floor:
        failures.append(f"{metric}={score:.3f} < {floor:.2f}")

if failures:
    print("REGRESSION:", "; ".join(failures))
    sys.exit(1)
print("All thresholds met:", agg)

Exit codes drive CI. A non-zero exit blocks the deploy. Metrics with None values are skipped so the script stays useful even when context_recall is not populated.

.github/workflows/eval.yml
yaml
name: RAG eval
on: pull_request
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: make setup
      - run: nohup make run &
      - run: sleep 10
      - run: python scripts/check_thresholds.py
        env:
          FIREWORKS_API_KEY: ${{ secrets.FIREWORKS_API_KEY }}

Every pull request runs the gold set against the new code. If aggregate scores fall below the thresholds, the PR is blocked until someone explains why. Regressions fail the build, not the customer.

Then the gold set is missing a question shape. Add examples that reflect the real failures, re-baseline, and keep iterating. A gold set is a living artifact, not a one-time deliverable. The right instinct when you find a new failure in production is to write it into the gold set, then fix it. That way the same regression cannot sneak back in silently.