Idempotent retry boundary

Networks drop responses. Clients retry. The orchestrator must treat the retry as the same operation, not a brand new one. The standard fix is an idempotency key the client mints once, the server stores once, and any retry returns the same stored result without re-running tools or charging the LLM call again.

layers/orchestrator.py
python
from typing import Any
from datetime import timedelta


class IdempotencyStore:
    """In-memory for the workshop, swap for Redis in production."""
    def __init__(self) -> None:
        self._cache: dict[str, dict] = {}

    def get(self, key: str) -> dict | None:
        return self._cache.get(key)

    def put(self, key: str, value: dict, ttl: timedelta = timedelta(hours=24)) -> None:
        self._cache[key] = value  # add expiry in real impl


async def run_orchestrator(envelope: RequestEnvelope, store: IdempotencyStore) -> dict:
    key = envelope.payload.get('idempotency_key')
    if key:
        cached = store.get(key)
        if cached:
            return cached  # short-circuit retry
    result = await _run_real_pipeline(envelope)
    if key:
        store.put(key, result)
    return result

Cache the response by idempotency_key. Retries hit the cache instead of re-running tools, retrieval, and the LLM. Production swaps the dict for Redis with TTL and atomic SETNX.

Two design rules. The client mints the idempotency key, not the server: a retry must use the same key as the original. And the cached value is the entire response, including any tool side-effect receipts, so the client cannot replay a half-completed operation.

Quiz: Quiz

Loading practice…