Alerting on span-level SLOs

Traces are for debugging. SLOs are for sleeping at night. Once duration_ms and error attributes live on every span, you can aggregate them and set alerts. The goal is to catch a regression before a customer does.

From spans to SLOs

observability/slos.yaml

yaml

slos:
  - name: sql_tool_latency
    span_kind: TOOL
    span_name: run_sql_tool
    metric: p95_duration_ms
    window: 15m
    target: 800
    alert_when: greater_than

  - name: rag_tool_error_rate
    span_kind: RETRIEVER
    span_name: run_rag_tool
    metric: error_rate
    window: 10m
    target: 0.01
    alert_when: greater_than

  - name: synthesis_latency
    span_kind: CHAIN
    span_name: synthesise_node
    metric: p95_duration_ms
    window: 15m
    target: 2500
    alert_when: greater_than

A plain YAML file describes every SLO. Each one points to a span kind and name, a metric, a window, and a threshold. Wire it into Phoenix evals or your own aggregator.

Timed quiz: SLO rapid fire

Loading practice…

Every SLO deserves a one-page runbook: the signal that fires, the span to open first, the attribute to read, the rollback if it stays red. Writing the runbook is where observability turns from a toy into a team habit.