Reading scores without being misled
RAGAS is LLM-as-judge under the hood. That has consequences. Scores have variance across runs. They can be gamed by wordy answers. And a perfect faithfulness score on empty context still looks perfect. Knowing how to read the scores is part of shipping them.
Flashcards: Flashcards
Loading practice…
def _lightweight_scores(question, answer, contexts, ground_truth):
"""Heuristic fallback when RAGAS isn't available."""
if not answer or not contexts:
return EvalScore(faithfulness=0.0, answer_relevance=0.0, context_precision=0.0, context_recall=0.0)
joined_ctx = " ".join(contexts).lower()
answer_words = {w for w in answer.lower().split() if len(w) > 3}
ctx_words = {w for w in joined_ctx.split() if len(w) > 3}
q_words = {w for w in question.lower().split() if len(w) > 3}
faithfulness = len(answer_words & ctx_words) / max(len(answer_words), 1)
answer_relevance = len(answer_words & q_words) / max(len(q_words), 1)
context_precision = len(ctx_words & q_words) / max(len(ctx_words), 1)
return EvalScore(
faithfulness=round(min(faithfulness, 1.0), 3),
answer_relevance=round(min(answer_relevance, 1.0), 3),
context_precision=round(min(context_precision, 1.0), 3),
)The lightweight scorer is a sanity check, not a quality bar. Use it to make tests pass without the RAGAS install. Do not set regression thresholds against it.
Quiz: Quiz
Loading practice…
Checkpoint: RAGAS checkpoint
Loading practice…