Newsletter, guides
FastAPI Prometheus metrics: Monitor AI agent services
Add Prometheus metrics to a FastAPI agent with prometheus-fastapi-instrumentator, LLM token and latency histograms, multiprocess mode, panels and alerts.
A user says the agent got slower over the last week. You check your logs and grep for latency. You find nothing useful, because logs are not the right shape for latency analysis. You end up asking the user “how slow?” and guessing at the cause.
FastAPI Prometheus metrics are a monitoring layer that records requests, latency and tokens as time series, by exposing a /metrics endpoint that Prometheus scrapes. Traces answer “why did this request fail?” Metrics answer “is anything trending badly?” This guide covers the metrics side: HTTP metrics with prometheus-fastapi-instrumentator, the LLM metrics to add by hand, the histogram bucket trap, multiprocess mode, the dashboard and the alerts.
Key takeaways
- An agent service needs four metric types: a request counter, an error counter by type, a latency histogram and a token histogram.
prometheus-fastapi-instrumentatoradds HTTP request counts and latency in one line, labeled by route template, so path parameters do not explode cardinality.- Default histogram buckets stop at 10 seconds in the Python client and at 1 second in the instrumentator’s labeled histogram. Agent calls need buckets up to 60 or 120 seconds.
- A Prometheus percentile from a classic histogram is an estimate. Its error is bounded by the width of the bucket the percentile falls in.
- Uvicorn or Gunicorn with several worker processes needs
PROMETHEUS_MULTIPROC_DIRset, or each scrape sees only one worker’s numbers.
Why do agent services need Prometheus metrics?
Because aggregate trends show things single traces cannot. A trace tells you why one request failed. Metrics tell you whether p95 latency has crept up over seven days, across every request.
Three questions only metrics answer well:
- Is p95 latency climbing over the last day or week, compared with last week?
- Is the error rate higher on one endpoint or model than another, across all requests?
- Are tokens per request growing, which shows up in the bill a few days later?
All three are about distributions over time. Prometheus stores numeric time series cheaply and queries them fast. Trace stores can answer the same questions, usually at a higher query cost. For the trace side, see Langfuse and Grafana monitoring.
How do you add Prometheus to FastAPI?
Install prometheus-fastapi-instrumentator and call it once on your app. It adds middleware and a /metrics endpoint.
# filename: app/main.py
# description: HTTP metrics for every route, with agent-sized latency buckets.
from fastapi import FastAPI
from prometheus_fastapi_instrumentator import Instrumentator
app = FastAPI()
Instrumentator(excluded_handlers=['/metrics']).instrument(
app,
latency_lowr_buckets=(0.5, 1, 2.5, 5, 10, 30, 60, 120),
).expose(app, include_in_schema=False)
The defaults give you http_requests_total with handler, method and status labels, and http_request_duration_seconds with handler and method. Status codes are grouped into 2xx, 4xx and 5xx. The handler label is the route template, such as /threads/{thread_id}, and unmatched paths are grouped as none.
Route templates matter. A label taken from the raw URL path creates one time series per thread ID or user ID. The Prometheus naming guide warns against labels with unbounded values like user IDs, because each value is a separate series.
What are the four metrics that matter for agent services?
The instrumentator covers HTTP traffic. An agent also spends most of its time in model calls and tools, which HTTP metrics cannot see. Add those by hand with prometheus_client.
# filename: app/metrics.py
# description: LLM-specific metrics next to the instrumentator's HTTP metrics.
from prometheus_client import Counter, Histogram
LLM_ERRORS = Counter(
'agent_llm_errors_total',
'LLM and tool call errors by type',
['model', 'error_type'],
)
LLM_LATENCY = Histogram(
'agent_llm_call_duration_seconds',
'Duration of one model call',
['model'],
buckets=(0.25, 0.5, 1, 2.5, 5, 10, 20, 30, 60, 120),
)
LLM_TOKENS = Histogram(
'agent_llm_tokens',
'Tokens per model call',
['model', 'direction'],
buckets=(100, 250, 500, 1000, 2500, 5000, 10000, 25000, 50000, 100000),
)
1. Request counter
http_requests_total from the instrumentator. Rate queries, error ratios and traffic splits all start here. It is labeled by route, method and status class.
2. Error counter
Count errors by type: timeout, provider error, rate limit, tool failure. Without the type label you see that “errors are up” but not which cause. Keep error_type to a short fixed list, not raw exception messages.
3. Latency histogram
One histogram per model call, plus the HTTP histogram for the whole request. The gap between the two is time spent in tools and your own code.
4. Token histogram
Agent requests vary a lot in token use. A lookup might use a few hundred tokens and a long multi-turn session tens of thousands. The distribution shows when requests shift toward the expensive end. The name has no _total suffix, because Prometheus reserves _total for counters.
How do you record LLM calls?
Wrap every model call in one helper so no call path skips the metrics. This example uses the OpenAI Python SDK, whose responses carry usage.prompt_tokens and usage.completion_tokens.
# filename: app/llm.py
# description: One wrapper that times, counts and labels every model call.
import time
from openai import APITimeoutError, AsyncOpenAI, RateLimitError
from app.metrics import LLM_ERRORS, LLM_LATENCY, LLM_TOKENS
client = AsyncOpenAI()
async def chat(model: str, messages: list[dict]):
start = time.perf_counter()
try:
response = await client.chat.completions.create(model=model, messages=messages)
except APITimeoutError:
LLM_ERRORS.labels(model=model, error_type='timeout').inc()
raise
except RateLimitError:
LLM_ERRORS.labels(model=model, error_type='rate_limit').inc()
raise
except Exception:
LLM_ERRORS.labels(model=model, error_type='provider_error').inc()
raise
finally:
LLM_LATENCY.labels(model=model).observe(time.perf_counter() - start)
LLM_TOKENS.labels(model=model, direction='input').observe(response.usage.prompt_tokens)
LLM_TOKENS.labels(model=model, direction='output').observe(response.usage.completion_tokens)
return response
The same wrapper is the right place for retries and the Langfuse span. One wrapper keeps the three in step.
What is the histogram bucket trap?
Buckets decide how precise your percentiles are. histogram_quantile finds the bucket the percentile falls in and interpolates inside it, assuming values spread evenly across the bucket. The Prometheus docs state that the error is limited by the width of that bucket.
Example: with buckets 0.1, 0.5, 1 and a real p95 of 3.5 seconds, every slow request lands in +Inf. Prometheus cannot interpolate into an infinite bucket, so it reports the highest finite bound, 1 second. The dashboard shows a flat line while users wait longer every day. That is exactly the instrumentator’s default labeled histogram.
The Python client’s default buckets run from 5 milliseconds to 10 seconds. That fits web APIs. An agent turn that takes 30 seconds lands in +Inf there too.
Three rules for bucket choice:
- Cover the expected range. If agent calls take 1 to 60 seconds, the buckets must reach past 60.
- Space them roughly by doubling. Values like 0.5, 1, 2.5, 5, 10, 30, 60 keep resolution at both ends.
- Keep the count small. Every bucket is one more series per label combination.
How do you run Prometheus metrics with several Uvicorn workers?
Set PROMETHEUS_MULTIPROC_DIR before the workers start. The Python client assumes one process with shared memory. With uvicorn --workers 4, each worker keeps its own counters, and a scrape reaches only the one worker that happens to answer.
In multiprocess mode, each worker writes its metrics to files in that directory, and the /metrics handler merges them. The instrumentator’s expose() does the merge for you when the variable is set.
# filename: start.sh
# description: Start four workers that share one metrics directory.
export PROMETHEUS_MULTIPROC_DIR=/tmp/prometheus
rm -rf "$PROMETHEUS_MULTIPROC_DIR" && mkdir -p "$PROMETHEUS_MULTIPROC_DIR"
exec uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 4
Set the variable in the start script, not in Python, and wipe the directory on each start. Multiprocess mode has limits: custom collectors such as process CPU and memory do not work, and exemplars are not supported. For the worker setup itself, see FastAPI and Uvicorn in production.
What should the dashboard show?
Four panels, each answering one question:
- Request rate by route. Shows traffic shape. A sudden drop means clients stopped calling.
- Error ratio by route. 5xx requests as a share of all requests.
- p50 and p95 latency, request and model call. Two lines on one chart. A widening gap means a long tail.
- p95 tokens per call by model. Token growth usually shows up here before it shows up in the bill.
# p95 model call latency by model
histogram_quantile(0.95, sum by (le, model) (rate(agent_llm_call_duration_seconds_bucket[5m])))
# error ratio by route
sum by (handler) (rate(http_requests_total{status="5xx"}[5m]))
/
sum by (handler) (rate(http_requests_total[5m]))
Add panels only when you have a question for them. A dashboard with a few focused panels gets read every morning.
What should you alert on?
Three alerts catch most production issues. The thresholds are examples: set yours from two weeks of baseline data.
- Error ratio above 2 percent for 5 minutes. Catches provider outages, bad deploys and bugs that hit part of the traffic.
- p95 latency above your target for 10 minutes. Set the target from what users expect, not from today’s median.
- Request rate below half the same hour last week for 15 minutes. Catches a client that started failing silently and stopped calling you.
Skip alerts on raw token volume. Token use varies by request type, so the alert fires too often to act on. Watch the token panel weekly instead, and alert only on a per-tenant jump against that tenant’s own history.
What to do next
- Add
prometheus-fastapi-instrumentatorwith agent-sizedlatency_lowr_bucketsand curl/metricsto see the output. - Add the three LLM metrics and route every model call through one wrapper.
- If you run more than one worker, set
PROMETHEUS_MULTIPROC_DIRin the start script and confirm counts add up across workers. - Point Prometheus, or a hosted Prometheus, at
/metricswith a 15 second scrape interval. - Build the four panels and the three alerts, then tune thresholds after two weeks of data.
- Add trace links next to the panels, as the Langfuse agent dashboard lab does end to end.
Frequently asked questions
What is prometheus-fastapi-instrumentator?
It is an open source Python package that adds Prometheus metrics to a FastAPI or Starlette app. One call, Instrumentator().instrument(app).expose(app), adds middleware and a /metrics endpoint. You get request counts and latency histograms labeled by route template, method and status class. It also supports multiprocess mode and custom metric functions.
What metrics should an agentic AI service expose to Prometheus?
Four: a request counter, an error counter by error type, a latency histogram for requests and model calls, and a token histogram per model call. Label them by route and model. Add tenant only when you have a small, fixed number of tenants. These four answer most trend questions about an agent in production.
What histogram buckets should I use for agent latency?
Buckets that span the full range your calls take, spaced roughly by doubling. For calls between half a second and two minutes, 0.5, 1, 2.5, 5, 10, 30, 60, 120 works. The Python client’s default buckets end at 10 seconds, and the instrumentator’s labeled histogram ends at 1 second. Both hide slow agent calls in the +Inf bucket.
Why do my Prometheus counters reset or jump with several Uvicorn workers?
Because each worker process keeps its own counters, and each scrape reaches one worker at random. Set PROMETHEUS_MULTIPROC_DIR to an empty directory in the start script before the workers launch. Each worker then writes to files there, and the /metrics handler merges them into one view of the whole service.
Should I use Prometheus or OpenTelemetry for agent metrics?
Either can work, and they combine. Prometheus with the Python client is the shortest path when you already run Prometheus and Grafana. OpenTelemetry fits when you want one SDK for traces and metrics and export to several backends. Prometheus accepts OpenTelemetry metrics too, so you can start with one and add the other later.
Sources
- Histograms and summaries, Prometheus, October 2026: how
histogram_quantileinterpolates and why the error is bounded by bucket width. - Metric and label naming, Prometheus, October 2026: the
_totalsuffix for counters, base units and the warning on unbounded label values. - Multiprocess mode, Prometheus Python client, October 2026:
PROMETHEUS_MULTIPROC_DIRand the limits of multiprocess mode. - prometheus-fastapi-instrumentator, trallnag on GitHub, July 2026: default metrics, labels, status grouping and route template handlers.
Build what you just read, with Param, live
You read the guide
- Free Live LabTrace AI Agents in ProductionEvery agent run traced in Langfuse, a dashboard of cost, latency and failures per agent and per customer, and an alert before one goes over budget.Free seat
- ProWatch the video coursesDesign a Software Factory for Your Coding Agents, Build Your First Software Factory Execution Harness and each new course as the sessions are cut. The first lesson of each is free after you sign in.$149 a year
- WorkshopAI Agents in Production for EngineersA multi-agent system deployed with an identity per agent, permissions, guardrails, an LLM gateway, traces and evals, plus a checklist for your own agents.$497
- BuildcampAgentic AI Buildcamp for Engineers & PMsDesign, build and deploy a production agentic AI system in four weeks$997