SIGTERM and in-flight drain

Docker stops a container by sending SIGTERM and waiting a grace period. If your process does nothing, any background jobs running die mid-flight. Users see failed generations. Idempotency keys help them retry, but a clean drain avoids the retry entirely.

job_store.py
python
import asyncio
import logging

logger = logging.getLogger(__name__)

async def drain(self, timeout: float = 10.0) -> None:
    """
    Wait for in-flight tasks to finish, up to timeout seconds.
    Called from the FastAPI lifespan on shutdown so Docker's SIGTERM
    does not cut running jobs off mid-flight.
    """
    async with self._lock:
        pending = [t for t in self._tasks.values() if not t.done()]

    if not pending:
        return

    logger.info("draining_jobs", extra={"pending": len(pending), "timeout": timeout})
    try:
        await asyncio.wait_for(
            asyncio.gather(*pending, return_exceptions=True),
            timeout=timeout,
        )
    except asyncio.TimeoutError:
        logger.warning("drain_timeout", extra={"pending": len(pending)})

Gather all in-flight tasks and wait for them, with a ceiling so we do not block shutdown forever. return_exceptions=True means a failing task does not short-circuit the wait on everyone else. If we hit the timeout, we log and move on.

Drain sequence on SIGTERM

Docker sends SIGTERM, lifespan runs drain, jobs finish or time out, container exits.

uvicorn stops accepting new connections on SIGTERM and then waits for in-flight requests to finish, up to its own shutdown timeout. Long SSE streams should emit a terminal event and break out of the generator when cancellation is raised. The client sees a clean end-of-stream instead of a TCP reset, and your logs close out the request properly.

Quiz: Quiz

Loading practice…