Async health checks that degrade gracefully

Health checks that throw when one dependency is down are a footgun. Kubernetes restarts the pod, the dependency is still down, the new pod also fails. Your health route should answer per-dependency, never raise, and let /health always return 200 with details in the body.

healthchecks/postgres.py
python
import asyncio
import asyncpg

async def check_postgres(opts) -> dict:
    if not opts.postgres:
        return {'status': 'unavailable', 'detail': 'DB_POSTGRES not configured'}
    try:
        conn = await asyncio.wait_for(
            asyncpg.connect(opts.postgres), timeout=opts.command_timeout_seconds
        )
        try:
            await conn.execute('SELECT 1')
        finally:
            await conn.close()
        return {'status': 'healthy'}
    except Exception as exc:
        return {'status': 'unavailable', 'detail': str(exc)[:200]}

A single async check that tries a cheap SELECT 1 on Postgres with a timeout. On success it returns healthy. On any failure it returns unavailable with the error message trimmed. It never raises.

service.py
python
import asyncio

async def health(self) -> dict:
    checks = await asyncio.gather(
        check_postgres(self.db),
        check_minio(self.minio),
        check_airflow(self.airflow),
        return_exceptions=False,
    )
    named = dict(zip(['postgres', 'minio', 'airflow'], checks))
    overall = 'ok' if all(c['status'] == 'healthy' for c in checks) else 'degraded'
    return {'status': overall, 'checks': named}

The aggregator runs the per-dependency checks concurrently and produces the summary FastAPI returns on /health. The overall status is healthy only when every check is healthy. Partial success produces degraded.

Prefer 200 with a body that says "degraded". Kubernetes readiness probes read the body for detail, and operators read the body for triage. A 500 makes Kubernetes restart the pod, which does not fix a downstream outage. A 503 tells Kubernetes to stop sending traffic, which is what you want on a cold start or critical dep failure, never for a minor blip.

Graceful degradation policy

Pod stays alive. Only routes that need the failing dep return their own errors.

Different routes degrade differently based on what they need.

Quiz: Quiz

Loading practice…