Prompt evaluation pipelines

You have learned to write great prompts. But how do you know which prompt is best? In production, you need to measure prompt quality systematically, not just eyeball it. This lesson covers evaluation pipelines: how to score, compare, and optimize prompts with data.

Consider two prompts for summarization: Prompt A: "Summarize this article." Prompt B: "You are a news editor. Summarize this article in 2-3 sentences for a busy executive. Focus on key facts, skip background." You can tell B is better by reading one output. But what about across 100 articles? You need automated evaluation: run both prompts on the same inputs and score the outputs on criteria like accuracy, conciseness, and format compliance.

For a quick check, yes. But human judgment is inconsistent and does not scale. When you have 5 prompt variants running across 50 test cases, you need automated scoring. Evaluation pipelines give you reproducible, objective numbers you can track over time, which is essential for production systems where prompt quality directly affects user experience.

14_prompt_optimization.py
python
# Simple A/B evaluation pipeline
def evaluate_prompt(prompt_template, test_cases):
    scores = []
    for case in test_cases:
        prompt = prompt_template.format(**case["input"])
        output = get_completion(prompt)
        score = score_output(output, case["expected"])
        scores.append(score)
    return sum(scores) / len(scores)

# Test cases with expected outputs
test_cases = [
    {"input": {"article": "..."}, "expected": "key facts only"},
    {"input": {"article": "..."}, "expected": "2-3 sentences"},
]

score_a = evaluate_prompt("Summarize: {article}", test_cases)
score_b = evaluate_prompt(
    "Summarize in 2-3 sentences for an executive: {article}",
    test_cases
)
print(f"Prompt A: {score_a:.1%}, Prompt B: {score_b:.1%}")

An evaluation pipeline runs each prompt against test cases and scores the outputs. This gives you objective data to compare prompt variants.

How do you score outputs? Three approaches: 1. Exact match: Does the output match the expected answer? (Good for classification) 2. LLM-as-judge: Use another LLM call to score quality on a 1-5 scale (Good for open-ended tasks) 3. Heuristic checks: Does it meet length limits? Is it valid JSON? Does it contain required keywords? (Good for format compliance) Production systems combine all three for a comprehensive score.

14_prompt_optimization.py
python
# LLM-as-judge scoring
def llm_judge(output, criteria):
    judge_prompt = f"""Score this output on a scale of 1-5.
Criteria: {criteria}

Output to evaluate:
{output}

Respond with only a number 1-5."""
    score = get_completion(judge_prompt, temperature=0)
    return int(score.strip())

# Score on multiple criteria
def multi_criteria_score(output):
    scores = {
        "accuracy": llm_judge(output, "factual accuracy"),
        "conciseness": llm_judge(output, "brevity and clarity"),
        "format": llm_judge(output, "follows requested format"),
    }
    return scores

LLM-as-judge uses a separate LLM call to score quality. Score on multiple criteria to get a nuanced view of prompt performance.

For initial development, 10-20 diverse test cases are enough to spot major differences between prompts. For production decisions, aim for 50-100 cases covering edge cases and common scenarios. The key is diversity: include easy cases, hard cases, edge cases, and adversarial inputs. A small but diverse set beats a large homogeneous one.

Quiz: Quiz

Loading practice…