Relevancy check

Not all retrieved documents are useful. The relevancy check pattern scores each document for relevance, checks factual grounding, assesses content quality, and recommends ACCEPT or REJECT to prevent irrelevant information from polluting your RAG responses.

patterns/30_relevancy_check.py
python
class RelevancyChecker:
    def score_relevancy(self, query, content):
        prompt = f"""Rate relevance 0-10:
        Query: {query}
        Content: {content}
        Score and list matching concepts."""
        return self.llm.generate(prompt).content

    def check_fact_grounding(self, content):
        prompt = f"""Assess factual accuracy:
        Content: {content}
        Return: accuracy score, grounding level (high/medium/low)"""
        return self.llm.generate(prompt).content

class RelevancyPipeline:
    def process_content(self, query, content):
        relevancy = self.checker.score_relevancy(query, content)
        grounding = self.checker.check_fact_grounding(content)
        quality = self.checker.assess_content_quality(content)

        # Weighted score: 0.3*relevancy + 0.3*grounding + 0.2*quality + 0.2*verification
        overall = (relevancy * 0.3 + grounding * 0.3
                  + quality * 0.2 + verification * 0.2)
        return {
            "score": overall,
            "recommendation": "ACCEPT" if overall >= 6 else "REJECT"
        }

RelevancyChecker scores documents and recommends accept/reject.

Quiz: Quiz

Loading practice…

Matching exercise: Match relevancy check Components

Loading practice…

Flashcards: Flashcards

Loading practice…

Start with a threshold of 6 out of 10 and tune based on your use case. A higher threshold (7-8) means stricter filtering but you may lose some relevant documents. A lower threshold (4-5) lets more through but increases noise. Monitor the reject rate and user satisfaction to find the sweet spot.

With relevancy checks in place, your RAG system filters out noise before it reaches the LLM. Next, we tackle data processing and anonymization for production pipelines.