A fallback chain that survives outages

Sooner or later the primary provider returns a 429, a 503, or a connection timeout. You do not want the user to see it. A fallback chain tries the primary, and on failure, tries a second provider that answers the same contract.

Primary, then fallback

A single failure shifts traffic to the next provider in the chain without the caller noticing.

service.py
python
import logging
from providers.base import Message, ChatResult
from providers.registry import resolve

log = logging.getLogger(__name__)

class LLMUnavailable(Exception):
    pass

async def chat_with_fallback(
    messages: list[Message],
    chain: list[str],
) -> ChatResult:
    last_error: Exception | None = None
    for model in chain:
        provider, resolved = resolve(model)
        try:
            return await provider.chat(messages, resolved)
        except Exception as error:
            log.warning('provider_failed', extra={'model': model, 'error': str(error)})
            last_error = error
    raise LLMUnavailable(f'all providers failed: {last_error}')

The chain is just a list of model strings. Iterate, try each, log the failure, and fail loudly only when every option is exhausted. Application code still calls a single function.

Short is better. Each fallback adds latency on the failure path, and a long chain hides bugs behind endless retries. Two or three entries is plenty: a primary, a backup from a different vendor, and optionally a local model for severe outages. Pick providers that fail independently so you do not chain two models behind the same upstream.

Quiz: Quiz

Loading practice…