Retry and timeout policy

Local Ollama on a laptop is fast for short prompts and slow for long ones. vLLM with a cold queue can stall for ten seconds before the first token. Picking timeouts is the difference between a smooth UX and frozen requests no one ever cancels.

Three timeouts, three failure modes

Connect timeout catches DNS or TCP issues. Read timeout catches stalled streams. Total timeout caps the worst case for a user request.

utils/llm_provider.py

python

import httpx
from openai import AsyncOpenAI

# Tight connect, generous read for slow first-token, hard total cap.
_HTTP_CLIENT = httpx.AsyncClient(
    timeout=httpx.Timeout(
        connect=2.0,
        read=30.0,
        write=10.0,
        pool=2.0,
    ),
)

client = AsyncOpenAI(
    api_key=os.environ['OPENAI_API_KEY'],
    http_client=_HTTP_CLIENT,
    max_retries=0,  # we own retry policy ourselves
)

Pass an httpx client with explicit timeouts. Disable the SDKs built-in retry so your tenacity layer is the only retry code in the stack.

Quiz: Quiz

Loading practice…