Observability: model, latency, tokens

A provider abstraction is where observability belongs. You already see every call, you already know which backend served it, and you already have token usage in ChatResult. Adding structured logs here gives you per-provider latency and cost for free.

Logs at the seam

Every path through the application lands in one logging point, so per-provider comparisons are easy.

service.py

python

import time
import logging
from providers.base import Message, ChatResult
from providers.registry import resolve

log = logging.getLogger('llm')

async def chat(messages: list[Message], model: str) -> ChatResult:
    provider, resolved = resolve(model)
    start = time.perf_counter()
    try:
        result = await provider.chat(messages, resolved)
        return result
    finally:
        latency_ms = (time.perf_counter() - start) * 1000
        log.info(
            'llm_call',
            extra={
                'model': model,
                'latency_ms': round(latency_ms, 1),
                'prompt_tokens': getattr(result, 'prompt_tokens', None) if 'result' in locals() else None,
                'completion_tokens': getattr(result, 'completion_tokens', None) if 'result' in locals() else None,
            },
        )

One log line per call with model, latency, and token counts. Point this at any observability backend and you get per-provider dashboards for free.

Quiz: Quiz

Loading practice…