Prometheus and Grafana for oncall

Prometheus scrapes metrics. Grafana renders them. Together they form the oncall command center. The trick is designing SLIs that predict user impact and filtering out the noisy metrics that fire false alerts. A dashboard that pages oncall for green signals is worse than no dashboard.

main.py
python
from prometheus_client import Counter, make_asgi_app

pipeline_triggers = Counter('pipeline_triggers_total', 'Count of /pipeline/trigger calls', ['status'])

metrics_app = make_asgi_app()
app.mount('/metrics', metrics_app)

@router.post('/pipeline/trigger')
async def trigger(req: BatchRequest):
    try:
        result = await service.trigger_batch(req)
        pipeline_triggers.labels(status='accepted').inc()
        return result
    except Exception:
        pipeline_triggers.labels(status='error').inc()
        raise

Expose Prometheus metrics from the FastAPI control plane. The prometheus-client library registers default metrics plus a custom counter for pipeline-trigger calls. Prometheus scrapes /metrics every 15 seconds.

Good SLIs predict user impact. For a batch platform: DAG freshness (minutes since last successful run), p95 task duration, and error rate on the control plane. Bad SLIs measure implementation details: CPU utilization, memory pressure, or internal queue depth. The former fire when users feel pain. The latter fire on cosmetic changes.

Three rules. One, every alert names a specific user-visible impact. Two, every alert has a runbook link. Three, every alert fires only when a threshold is held for a multi-minute window. Alerts that fire on a single sample are noise. Alerts without runbooks teach oncall to ignore them. Alerts without user impact teach oncall to ignore the page.

Quiz: Quiz

Loading practice…

Checkpoint: Checkpoint: governance and observability are first class

Loading practice…