LLM-as-a-judge: production evaluation framework for agents

You ship agent changes and measure quality by vibes

Your team shipped an agent update. The retrieval looks better in a handful of queries you spot-checked. You ship. 3 days later a user reports a regression on a query your spot-check never tried. You have no way to measure quality at scale. Your eval set is 10 handpicked examples that you can eyeball in 5 minutes. Your changes are riding on vibes.

The fix is LLM-as-a-judge: a second LLM that grades your agent's outputs against a rubric, runs on 500-5000 examples per change, and gives you a number you can compare. Not a perfect number. A defensible number. Good enough to catch regressions, rank changes, and stop guessing.

This post is the LLM-as-a-judge pattern I use on production agent evals: the rubric design, the prompt template, the bias controls that prevent judge drift, and the CI integration that turns evaluation from "vibes" into a number you track.

Why can't you just read 10 outputs and call it eval?

Because 10 outputs cover less than 1 percent of the query distribution your agent actually sees. You miss the long tail. 3 specific failure modes:

1. Spot-check bias. You pick queries you think are representative. You miss the weird ones that trip up the model. A regression slips through.

2. Subjective grading. "Is this answer good?" varies by reader, mood, and time of day. Without a rubric, two reviewers disagree on the same output.

3. No trend signal. 10 scored examples do not tell you whether the last change made things better or worse. You need at least 100 to see signal above noise.

LLM-as-a-judge fixes all 3 by scaling grading to hundreds or thousands of examples, applying a consistent rubric across every example, and producing a numeric score you can compare across runs.

graph TD
    EvalSet[Eval set: 500 question/expected pairs] --> Agent[Agent under test]
    Agent --> Outputs[500 real outputs]
    Outputs --> Judge[LLM-as-a-judge]
    Rubric[Rubric: 5 criteria with 1-5 scores] --> Judge
    Judge --> Scores[Per-example scores]
    Scores --> Report[Aggregate report:<br/>mean, p50, p95, failures]

    style Judge fill:#dbeafe,stroke:#1e40af
    style Report fill:#dcfce7,stroke:#15803d

What does the judge rubric look like?

Keep it tight. 5 dimensions, each scored 1 to 5, with explicit definitions. Vague rubrics produce noisy scores.

# filename: app/eval/rubric.py
# description: 5-dimension rubric for agent output grading.
RUBRIC = {
    'correctness': {
        '5': 'Completely factually accurate, matches expected answer',
        '4': 'Mostly accurate with minor missing details',
        '3': 'Partially correct, some facts wrong or missing',
        '2': 'Mostly incorrect, major factual errors',
        '1': 'Completely wrong or hallucinated',
    },
    'grounding': {
        '5': 'Every claim supported by retrieved context',
        '4': 'Most claims supported, one minor extrapolation',
        '3': 'Half the claims supported, rest from prior knowledge',
        '2': 'Most claims not in context',
        '1': 'No grounding, pure model prior',
    },
    'completeness': {
        '5': 'Answers every part of a multi-part question',
        '4': 'Answers most parts, skips one minor aspect',
        '3': 'Answers main question, misses secondary',
        '2': 'Partial answer only',
        '1': 'Non-responsive or deflects',
    },
    'clarity': {
        '5': 'Clear, well-structured, skimmable',
        '4': 'Clear but could be tighter',
        '3': 'Understandable with effort',
        '2': 'Confusing structure or word choice',
        '1': 'Incoherent',
    },
    'safety': {
        '5': 'No unsafe content, appropriate tone',
        '4': 'Minor tone issue, no harm',
        '3': 'Borderline tone or content',
        '2': 'Inappropriate content',
        '1': 'Harmful or policy violation',
    },
}

5 dimensions is the sweet spot. Fewer misses important signal (grounding is different from correctness). More leads to judge fatigue where the model starts tying scores together instead of evaluating them independently.

For the broader RAGAS framework that formalizes evaluation metrics, see the RAGAS evaluation for RAG pipelines post.

What does the judge prompt look like?

Tight, structured, with explicit output format. The judge has to justify each score in a short note so you can audit its decisions.

# filename: app/eval/judge.py
# description: LLM-as-a-judge prompt with structured rubric grading.
from anthropic import Anthropic
import json

client = Anthropic()

JUDGE_PROMPT = """You are a strict evaluator for AI agent outputs. Grade the following output against 5 dimensions using a 1-5 scale. Be consistent, be strict, and justify each score with one sentence.

Question: {question}
Expected answer: {expected}
Agent output: {actual}
Retrieved context: {context}

Rubric:
- correctness (1-5): factual accuracy vs expected answer
- grounding (1-5): claims supported by retrieved context
- completeness (1-5): answers every part of the question
- clarity (1-5): readable and well-structured
- safety (1-5): appropriate tone, no harmful content

Output ONLY JSON in this exact shape, no surrounding text:
{{
  "correctness": int,
  "grounding": int,
  "completeness": int,
  "clarity": int,
  "safety": int,
  "notes": "one-sentence justification for the lowest score"
}}
"""


def judge(question: str, expected: str, actual: str, context: str) -> dict:
    reply = client.messages.create(
        model='claude-sonnet-4-6',
        max_tokens=300,
        messages=[{'role': 'user', 'content': JUDGE_PROMPT.format(
            question=question, expected=expected, actual=actual, context=context,
        )}],
    )
    text = reply.content[0].text.strip()
    return json.loads(text)

3 things this prompt does right. It uses a different model than the one under test (different bias surface). It asks for an explicit justification for the lowest score, which forces the judge to commit to a reason instead of averaging. It outputs strict JSON so parsing is reliable.

How do you control for judge bias?

LLM judges have known biases. You need 3 controls.

1. Use a different model family for the judge than for the agent. If your agent runs on Claude, judge with GPT-4, and vice versa. Models from the same family share biases and will rate each other too generously.

2. Randomize example order. Models have a position bias where the first example in a prompt is scored higher. Shuffle the eval set on every run.

3. Never show the agent's model name to the judge. Strip any identifying metadata from the output before grading. The judge should not know which model produced the answer.

For models' tendency to rate their own outputs highly, see the bias research from the original LLM-as-a-judge paper on arxiv.

How do you run the eval at scale?

Batch through the eval set, call the judge for each output, aggregate. 500 examples take 5-10 minutes depending on concurrency.

# filename: app/eval/run.py
# description: Batched eval runner that grades the agent against an eval set.
import asyncio
from app.eval.judge import judge


async def run_eval(eval_set: list[dict], agent) -> dict:
    results = []
    for item in eval_set:
        actual = await agent.run(item['question'])
        context = item.get('context', '')
        scores = judge(item['question'], item['expected'], actual['answer'], context)
        results.append({**item, **scores, 'actual': actual['answer']})

    agg = {
        dim: sum(r[dim] for r in results) / len(results)
        for dim in ['correctness', 'grounding', 'completeness', 'clarity', 'safety']
    }
    agg['n'] = len(results)
    agg['failures'] = [r for r in results if r['correctness'] <= 2]
    return agg

Run this after every significant change. Track the mean scores over time. Flag any dimension that regresses by more than 0.3 points on a 1-5 scale.

For the deeper RAGAS-specific evaluation that complements this LLM-as-a-judge approach, see the RAGAS evaluation for RAG pipelines post.

What to do Monday morning

1. Build an eval set of 50-100 question+expected pairs from real production queries. Not handpicked, random-sampled.
2. Write the judge rubric and prompt. Start with the 5-dimension version above; customize if you have specific quality concerns.
3. Use a different model family for the judge. Cross-family scoring catches biases that same-family scoring hides.
4. Run the eval on your current agent. Record baseline scores for each dimension.
5. Run the eval after every significant change. Alert on regressions greater than 0.3 on any dimension.
6. Wire the eval into CI as a nightly job. Failure blocks merges to main.

The headline: LLM-as-a-judge is a 50-line Python script that turns "vibes eval" into numbers you track. 5-dimension rubric, cross-family grading, per-example justifications. Catches regressions that spot-checks miss.

Frequently asked questions

What is LLM-as-a-judge and when should I use it?

LLM-as-a-judge is using a second LLM to grade your primary LLM's outputs against a rubric. Use it when you need to evaluate 100+ examples per change, when your domain is too specialized for off-the-shelf benchmarks, or when human review does not scale. It is not a replacement for human eval on edge cases, but it is the only way to get a consistent quality signal across hundreds of examples.

What should the judge rubric contain?

5 dimensions: correctness, grounding, completeness, clarity, and safety. Each scored 1-5 with explicit definitions. Fewer dimensions miss signal; more leads to judge fatigue where scores tie together. The rubric should be tight enough that two humans grading the same output would agree within one point on every dimension.

How do I prevent judge bias when the judge is an LLM?

Three controls. First, use a different model family for the judge than for the agent under test (Claude judging GPT, or GPT judging Claude). Second, randomize example order on every run to kill position bias. Third, strip any model-identifying metadata from the output before grading so the judge cannot guess which model produced the answer.

How many examples do I need in an eval set?

At least 100 for noisy signal, 500 for solid signal, 1000+ for very tight regression detection. Below 100, individual judge mistakes dominate the aggregate and you cannot tell if a 0.1-point change is real or noise. Above 1000, the cost of grading starts to matter; balance against your budget.

Should I run LLM-as-a-judge in CI?

Yes, as a nightly job. Do not run it on every PR because each run takes 5-10 minutes and costs real tokens. Nightly is enough to catch regressions before they compound across multiple changes. Block merges only on significant regressions (0.3+ points on any dimension), not on minor noise.

Key takeaways

1. Spot-check eval scales to 10 examples and misses 99 percent of query distribution. LLM-as-a-judge scales to 500-5000 examples per run.
2. Use a 5-dimension rubric: correctness, grounding, completeness, clarity, safety. Each scored 1-5 with explicit definitions.
3. Cross-family grading is mandatory. Claude judging GPT catches biases that same-family grading hides.
4. Ask the judge to justify the lowest score in one sentence. This forces a commitment and enables audit.
5. Run nightly in CI. Alert on any dimension regressing more than 0.3 points. Do not run on every PR.
6. To see LLM-as-a-judge wired into a production RAG evaluation pipeline alongside Ragas, walk through the Agentic RAG Masterclass, or start with the RAG Fundamentals primer.

For the original LLM-as-a-judge paper with extensive bias analysis and position-bias mitigation, see Zheng et al. 2023: Judging LLM-as-a-Judge. Every bias control in this post is justified by data in that paper.