Choose the right LLM for each task, not one for all

Your agent uses one model for everything. The same large model routes the question, grades the retrieved chunks, extracts JSON and writes the final answer. It works, but every step pays large-model prices and waits for large-model latency, including the steps that only output “yes” or “no”.

Choosing the right LLM for each task means giving every step of your app the smallest model that passes your evals for that step. Routing, grading, classification and extraction usually run well on a provider’s small tier. Planning, hard reasoning and the answer the user reads get the large tier. You decide with a per-step eval set, not with a leaderboard.

Key takeaways

• Split your app into steps and pick a model per step. One model for everything overpays on the simple steps.
• Every major provider sells three tiers: a small fast model, a balanced middle model and a large frontier model. Prices across tiers differ by an order of magnitude or more.
• Small models fit classification, routing, grading and extraction. Large models fit planning, multi-step reasoning, code and the final answer.
• Start each step on the small tier and move up only when that step fails its eval. Keep model names in config, not in code.
• Mixture of experts (MoE) is how a model is built, not a tier. It lowers compute per token, but the whole model still has to fit in memory.

Why is one model for every step a mistake?

Because the steps have different needs. Look at a typical RAG agent. It has at least three thinking steps:

1. Route: decide whether the question needs your documents or the web.
2. Grade: decide whether each retrieved chunk is relevant.
3. Answer: write the reply the user reads.

Route and grade produce one label. They run on every request, often several times, and the user waits for them before the answer starts. The answer step is the only one where writing quality shows. Paying frontier prices and latency for all three is the expensive default.

What are the model tiers in 2026?

Every major provider sells the same three shapes. Names change every few months, so learn the tiers and look up the current names when you build.

Tier	What it is for	Examples, October 2026
Small	High-volume, latency-sensitive work: classify, route, extract	Claude Haiku 5.5, GPT-6 Luna, Gemini 3.5 Flash-Lite
Middle	Balanced speed and quality: chat, summaries, most agent steps	Claude Sonnet 5.5, GPT-6.1 Sol, Gemini 3.8 Flash
Large	Hard reasoning, long agentic tasks, code	Claude Opus 5.5 and Fable 5.1, GPT-6 Astra, Gemini 3.1 Pro (preview)
Open weight	Self-hosted, data stays on your hardware	gpt-oss-20b and gpt-oss-120b, Gemma, Qwen

The price gap is the reason this matters. On Anthropic’s own pricing table, Claude Haiku 5.5 starts at $0.10 per million input tokens and Claude Opus 5.5 costs $4, a 40x spread. Anthropic describes Haiku as built “for high-volume, latency-sensitive tasks such as classification, extraction, and routing”. The other providers draw the same line between their small and large models.

Which tasks fit which tier?

Sort each step by two questions: is the output a label or structured data, and does a wrong answer reach the user?

• Small tier: intent classification, routing, relevance grading, guardrail checks, JSON extraction from text, short summaries for logs. The output is constrained, so a small model rarely has room to go wrong. Use structured output so the label is always valid.
• Middle tier: most chat turns, summaries the user reads, tool-calling agent steps with a handful of tools, drafting.
• Large tier: planning a multi-step task, debugging, writing or reviewing code, answers that combine many sources, anything where your eval shows the middle tier failing.

Thinking effort is a second dial. Many current models let you set how much they reason per request. A middle model at high effort can beat a large model at low effort on some tasks and cost less. Test both settings.

How do you route each step to its own model?

Put the mapping in one config, so a model change is a one-line diff and every step is easy to find. Here is a small router with the Anthropic SDK. The same shape works with any provider.

# filename: models.py
# description: One place that maps each step of the agent to a model.
import anthropic

client = anthropic.Anthropic()

MODEL_FOR = {
    'route': 'claude-haiku-5-5',
    'grade': 'claude-haiku-5-5',
    'extract': 'claude-haiku-5-5',
    'answer': 'claude-sonnet-5-5',
    'plan': 'claude-opus-5-5',
}


def ask(step: str, system: str, user: str, max_tokens: int = 1024) -> str:
    """Call the model configured for this step and return its text."""
    response = client.messages.create(
        model=MODEL_FOR[step],
        max_tokens=max_tokens,
        system=system,
        messages=[{'role': 'user', 'content': user}],
    )
    return next(b.text for b in response.content if b.type == 'text')


def route(question: str) -> str:
    label = ask(
        'route',
        system='Reply with exactly one word: docs or web.',
        user=question,
        max_tokens=64,
    )
    return 'web' if 'web' in label.lower() else 'docs'

If your steps live in an agent framework, the same mapping goes in each node or agent definition; see choosing an AI agent framework. Log the step name and model with every call. When a step fails, you can see which model made the call, and your cost dashboard can group spend by step.

How do you prove a smaller model is good enough?

With a small eval set per step. A leaderboard tells you which model is best on someone else’s task. You need to know whether the small model gets your routing right.

1. Collect 50 to 100 real inputs for the step from logs.
2. Label the correct output for each. For route and grade this is one word.
3. Run the small model and the model you use today on the same set.
4. If the small model matches on the cases that matter, switch. If not, try the middle tier.

Rerun the set whenever you change a prompt or a provider releases a new model. A new small model often passes a step that the previous one failed. The evals lab builds this loop end to end.

Where does mixture of experts fit?

Mixture of experts (MoE) is a way to build a model, not a tier you choose. An MoE model splits its feed-forward layers into many experts. A small router inside the model sends each token to a few of them, so only part of the weights does work per token.

That makes MoE models cheaper to run per token than a dense model of the same total size. OpenAI’s open-weight gpt-oss-120b has 117 billion parameters in total but uses about 5.1 billion per token. The catch is memory: every expert must be loaded, so you still need hardware that holds the full model. Many of the largest open-weight models now use MoE, gpt-oss among them. For hosted models you do not need to care, because the provider’s price per token already reflects it.

What else cuts cost besides a smaller model?

Model choice is one lever. Two others often matter as much:

• Prompt caching: a long, stable system prompt or document set is read from cache at a fraction of the input price on later calls. The prompt caching lab shows how.
• Batch processing: work that can wait, such as nightly classification or evals, runs at half price on Anthropic’s Batch API.

Also check failures. A small model that fails and retries can cost more than a middle model that succeeds the first time. See retry patterns for LLM API errors for handling those calls.

What to do next

1. List every LLM call in your app and name its step: route, grade, extract, answer, plan.
2. Move model names into one config keyed by step, like MODEL_FOR above.
3. Build a 50-question eval set for your two highest-volume steps.
4. Try the small tier on those steps. Switch each step that passes.
5. Log cost and latency per step, and rerun the evals when a provider ships a new model.

Frequently asked questions

Which LLM should I use for routing and classification?

Use the small tier of your provider, such as Claude Haiku, a GPT small model or Gemini Flash-Lite, and confirm it on 50 to 100 labeled examples from your logs. Routing and classification produce a single label, so small models usually match large ones there. Use structured output or a fixed label list so every reply parses, and move up a tier only if the eval shows misses.

Does using several models instead of one actually save money?

Usually, because the steps that run most often are the simplest. Route and grade calls happen on every request, sometimes several times, while the final answer happens once. Moving those calls to a small model that costs a tenth or less per token cuts most of their spend. Measure on your own traffic: log cost per step before and after the switch.

What is a mixture of experts model?

A mixture of experts model splits parts of the network into many experts, and a router inside the model sends each token to only a few of them. It does less compute per token than a dense model of the same total size, so it is cheaper to run. All experts still sit in memory, so hosting one needs hardware for the full parameter count.

Should I self-host an open-weight model instead?

Self-host when data must stay on your hardware or when your volume is high and steady enough to keep GPUs busy. Otherwise, hosted APIs are usually cheaper and simpler, and many providers also host open-weight models such as gpt-oss. Whichever you choose, keep the same per-step evals so you can compare quality directly.

How often should I revisit my model choices?

Whenever a provider releases a new model and at least once a quarter. Providers ship new small models often, and a new small model may pass a step that needed a middle one before. Keep model names in one config and rerun the per-step evals, so a switch is a tested one-line change.

Sources

• Models overview, Anthropic, October 2026: current Claude lineup, per-token prices and Haiku’s intended tasks.
• Models, OpenAI, October 2026: GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna positioning.
• Gemini models, Google, October 2026: current Flash and Flash-Lite models and Gemini 3.1 Pro preview.
• Introducing gpt-oss, OpenAI, August 2025: total and active parameters for gpt-oss-120b and gpt-oss-20b.
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, Shazeer et al., January 2017: the gated mixture of experts design.