Cost-aware routing

The provider abstraction makes a second pattern cheap: start with the cheapest capable model, and only escalate to a premium one if the first answer is bad. For many workloads, eighty percent of traffic stays on the cheap tier and your bill drops accordingly.

service.py
python
from providers.base import Message, ChatResult
from providers.chain import chat_with_fallback

def is_low_quality(result: ChatResult) -> bool:
    text = result.text.strip()
    if len(text) < 20:
        return True
    if text.lower().startswith("i'm sorry") or 'cannot help' in text.lower():
        return True
    return False

async def chat_with_escalation(messages: list[Message]) -> ChatResult:
    cheap = await chat_with_fallback(messages, ['openrouter/google/gemini-2.5-flash'])
    if not is_low_quality(cheap):
        return cheap
    return await chat_with_fallback(messages, ['gpt-4o', 'openrouter/anthropic/claude-3.5-sonnet'])

A tiny quality gate decides whether to escalate. Real systems use a grader model, a regex for expected structure, or a confidence score. The point is that the escalation decision lives next to the provider seam, not sprinkled across the app.

For many tasks, yes. If you expect valid JSON, a parse attempt is a perfect quality gate. If you expect a category label, a match against a known enum works. Reach for an LLM grader only when structural checks cannot express quality, and remember the grader itself costs money and latency.

Ordering exercise: Order the request flow

Loading practice…

Checkpoint: Resilience and routing checkpoint

Loading practice…