Evaluation (LLM-as-Judge)

The evaluation pattern uses an LLM as an automated judge to score responses across multiple dimensions: accuracy, completeness, clarity, and safety. This enables consistent, scalable quality assessment without manual review of every response.

Multi-dimension evaluation

It can, which is why you use structured rubrics with specific criteria rather than open-ended "is this good?" prompts. Multi-dimensional scoring reduces bias by forcing the judge to evaluate concrete attributes independently.

patterns/19a_evaluation.py
python
class ResponseEvaluator:
    def __init__(self):
        self.llm = get_llm()

    def evaluate_response_quality(self, question, response):
        prompt = f"""Rate this response 1-10 on each dimension:
        Question: {question}
        Response: {response}

        Score: accuracy, completeness, clarity, relevance
        Format: ACCURACY: X, COMPLETENESS: X, CLARITY: X, RELEVANCE: X
        """
        return self.llm.generate(prompt).content

    def comprehensive_evaluation(self, question, response):
        quality = self.evaluate_response_quality(question, response)
        safety = self.evaluate_response_safety(response)
        accuracy = self.evaluate_factual_accuracy(question, response)

        overall = (quality_score + safety_score + accuracy_score) / 3
        recommendation = (
            "APPROVE" if overall >= 7
            else "REVIEW" if overall >= 5
            else "REJECT"
        )
        return {"overall": overall, "recommendation": recommendation}

ResponseEvaluator that scores quality, safety, and accuracy.

Quiz: Quiz

Loading practice…

Matching exercise: Match evaluation concepts

Loading practice…

Flashcards: Flashcards

Loading practice…

You can now use LLMs to evaluate other LLM outputs automatically. Next, we will build monitoring systems that track your AI agent health in real-time.