Failure recovery

Jobs fail. The LLM provider rate-limits you, the model returns a bad schema, the container gets a SIGTERM, a dependency explodes. The goal is not zero failures. The goal is every failure shows up in the job store with a useful error, and the worker cleans up on cancel instead of corrupting state.

service.py
python
async def run_background_job(job_id: str, payload: JobStart) -> None:
    await job_store.set_state(job_id, "running")
    logger.info("job_started", extra={"job_id": job_id})
    try:
        if payload.delay_seconds:
            await asyncio.sleep(payload.delay_seconds)

        buffer = []
        async for chunk in llm_provider.generate_stream(payload.prompt):
            buffer.append(chunk)
        result = "".join(buffer).strip() or "(empty response)"

        await job_store.set_state(job_id, "done", result=result)
        logger.info("job_finished", extra={"job_id": job_id})
    except asyncio.CancelledError:
        await job_store.set_state(job_id, "failed", error="cancelled")
        logger.warning("job_cancelled", extra={"job_id": job_id})
        raise
    except Exception as e:
        await job_store.set_state(job_id, "failed", error=str(e))
        logger.exception("job_failed", extra={"job_id": job_id})

Three states: running on start, done on success, failed on error or cancel. The CancelledError branch re-raises after writing state so asyncio can finish cancelling the task. Without the re-raise, you swallow cancellation and asyncio thinks the task finished normally.

Job state machine

Every job walks one of four paths from pending to a terminal state.

Because asyncio relies on CancelledError propagating to complete the cancellation protocol. If you swallow it, the task looks like it completed normally, gather in the shutdown drain never signals cancellation to callers, and you get lingering tasks on shutdown. Write your state, then re-raise.

Quiz: Quiz

Loading practice…

Checkpoint: Background jobs checkpoint

Loading practice…