Exponential backoff with tenacity

Most 429 and 5xx errors clear in a second or two. Falling over to a new provider on the first failure wastes both latency and money. Retry the primary a couple of times with exponential backoff, and only then move on.

utils/llm_provider.py
python
from tenacity import (
    retry,
    stop_after_attempt,
    wait_random_exponential,
    retry_if_exception_type,
)
from openai import APIStatusError, APIConnectionError, RateLimitError

RETRYABLE = (RateLimitError, APIConnectionError)

def is_retryable_status(error: BaseException) -> bool:
    if isinstance(error, RETRYABLE):
        return True
    if isinstance(error, APIStatusError):
        return error.status_code in {408, 409, 500, 502, 503, 504}
    return False

retry_openai = retry(
    retry=retry_if_exception_type(Exception),
    stop=stop_after_attempt(3),
    wait=wait_random_exponential(multiplier=0.5, max=8),
    reraise=True,
)

Three attempts, jittered exponential wait up to eight seconds, and reraise the original error so the caller can decide whether to fall back. Jitter matters because it prevents thousands of clients from retrying in lockstep after a spike.

utils/llm_provider.py
python
from providers.retry import retry_openai, is_retryable_status

class OpenAICompatibleProvider(LLMProvider):
    async def chat(self, messages: list[Message], model: str) -> ChatResult:
        @retry_openai
        async def _call():
            response = await self.client.chat.completions.create(
                model=model,
                messages=[m.model_dump() for m in messages],
            )
            return response

        try:
            response = await _call()
        except Exception as error:
            if not is_retryable_status(error):
                raise
            raise
        return self._to_result(response, model)

The retry decorator wraps a single call inside the adapter. Callers never see 429 hiccups that clear in two seconds. They only see genuine failures that deserve a fallback.

Quiz: Quiz

Loading practice…