Human-in-the-Loop

The human-in-the-loop pattern lets your AI handle routine tasks autonomously while escalating edge cases and sensitive decisions to humans. The key mechanism is a confidence threshold: if the AI is confident enough, it proceeds automatically; if not, it asks for human review.

Confidence-based escalation

Start conservative (high threshold like 0.9) so most decisions go to humans, then gradually lower it as you build trust in the system. Monitor the human override rate to calibrate.

patterns/13_human_in_loop.py
python
class HumanInTheLoop:
    def __init__(self, confidence_threshold=0.8):
        self.confidence_threshold = confidence_threshold
        self.pending_reviews = []
        self.llm = get_llm()

    def process_request(self, request):
        ai_response, confidence = self._ai_process(request)

        if confidence >= self.confidence_threshold:
            return self._auto_approve(ai_response)
        else:
            return self._escalate_to_human(
                request, ai_response, confidence
            )

    def _escalate_to_human(self, request, ai_response, confidence):
        review_item = {
            "id": f"review_{len(self.pending_reviews) + 1}",
            "request": request,
            "ai_response": ai_response,
            "confidence": confidence,
        }
        self.pending_reviews.append(review_item)
        # In production: notify human reviewer
        return {"status": "pending_review", "review_id": review_item["id"]}

Confidence-based routing between auto-approve and human review.

Add human review for high-stakes decisions: financial transactions, content moderation, medical advice, or any action that is hard to reverse. For low-risk, reversible actions like search queries or formatting text, letting the AI proceed autonomously is fine. Start with more human oversight and reduce it as you build confidence in the system.

Quiz: Quiz

Loading practice…

Flashcards: Flashcards

Loading practice…

Human-in-the-loop is the safety net that makes AI systems trustworthy in the real world. Combined with MCP, goal monitoring, and exception handling, you now have a complete integration toolkit. These four patterns connect your agents to external services while keeping humans in control where it matters most.