Reference architectures for AI apps
Six shapes most production AI systems take, when each one fits, what it is made of and how it fails first.
Use this as the first page of a design review. Find the shape closest to what you are building, check the failure it is known for, and plan for that before anything else. Each shape links to the Live Lab where we build it.
- 1
Workflow with one model step
Decides: Your code- Use it when
- The steps are known in advance and only one needs judgement: triage, extraction, drafting for approval.
- Made of
- A queue or trigger
- Deterministic steps
- One model call with a schema for its output
- An exception path to a person
- Fails first
- The model step returns something the next step cannot parse. Validate its output against a schema and route failures to a person.
- 2
Single agent with tools
Decides: The model- Use it when
- The path depends on what the agent finds: research, debugging, open-ended support.
- Made of
- A model in a loop
- A small set of well-described tools
- A step limit and a budget
- Traces for every run
- Fails first
- It loops or wanders. Cap steps and spend, and read traces of failed runs every week.
- 3
Supervisor with workers
Decides: A supervisor model- Use it when
- The work splits into parts that need different tools, permissions or can run in parallel.
- Made of
- A supervisor that plans and routes
- Workers with narrow tools
- A written contract for what each worker returns
- Fails first
- Context is lost at handoffs. Define what each worker returns, and test the handoff, not only the worker.
- 4
Retrieval over your documents
Decides: Your code, then the model- Use it when
- Answers must come from your own documents and cite them.
- Made of
- Parsing that keeps structure
- Chunking and embeddings
- A vector index with filters
- Answer generation with citations
- Fails first
- The right chunk is never retrieved. Measure recall on real questions before tuning the prompt.
- 5
Durable agent with human approval
Decides: The model, inside a workflow engine- Use it when
- Runs take minutes to days, call slow systems or need sign-off.
- Made of
- A durable workflow engine
- Tool calls as retryable steps
- Approval steps that can wait
- Idempotent side effects
- Fails first
- A retry repeats a side effect, such as sending an email twice. Make every side effect idempotent.
- 6
Tools behind a gateway
Decides: Policy at the gateway- Use it when
- Several teams and agents share tools through MCP.
- Made of
- MCP servers per system
- One gateway in front
- Access rules per team and agent
- An audit log of calls
- Fails first
- An agent can call a tool nobody meant it to. Default to deny and log every call.
Pick the simplest shape that does the job. You can move from a workflow to an agent later; moving back is harder.