Replay a failing trace in Phoenix

A user asks "What does our return policy say about electronics?" The agent routes to web search instead of RAG, burns tokens on a Serper call, and returns an answer that does not match the PDF. This is the kind of bug that used to take an afternoon. With Phoenix open, it takes a few minutes.

The trace-driven debugging loop

Ordering exercise: Order the replay steps

Loading practice…

agent/router.py (routing rule tightened)
python
ROUTER_SYSTEM = """\
You are a routing agent for an e-commerce analytics assistant.
Given a user message, pick one of: sql, rag, web_search.

RULES:
  - If the message mentions policies, manuals, returns, warranty, or documents,
    choose rag. Even if no PDFs are loaded, still choose rag.
  - If the message asks about orders, customers, payments, revenue, or counts,
    choose sql.
  - Otherwise choose web_search.

Output only valid JSON: {"route": "...", "reason": "..."}
"""

The router now has an explicit rule that keeps the return-policy question on the RAG route. The trace told you exactly which sentence to add.

Same workflow, different span. Open the synthesis span, read the full prompt and completion, compare against the retrieval span to see which chunks made it into the context. If a chunk is missing or the prompt leads the model astray, fix the prompt or the retriever, not the router.

AI prompt: Try it: diagnose a trace with an LLM

Loading practice…