LLM Model Routing: Send Easy Requests to a Cheaper Model

A model router picks a model per request. Easy requests such as classification, extraction and short rewrites go to a small, cheap model, and difficult ones go to a larger model. The saving comes from the price gap between models, and the risk is quality loss on requests the router misjudges. So build the simplest router that works, measure it against each model alone on a labeled eval set, and ship it only if the quality drop is inside a margin you chose in advance.

Who this is for and what you will have at the end

This guide is for developers running an LLM feature in production or about to, who see a growing bill and wonder whether every request needs the strongest model. At the end you will have a working Python router for the Claude API, a cost calculator that uses logged token usage, a cascade option, and an eval script that compares three setups: small model only, large model only and routed. If you have no eval set yet, start with getting started with LLM evals and golden datasets for regression testing.

The code uses the anthropic Python SDK and model IDs and prices from Anthropic's documentation as checked on 30 October 2026. It was not run against the live API while writing this article, so test it with your own key and a small sample first.

When routing is worth it

Routing pays off when three things are true:

1. You have a mix of request difficulty. If all your requests are hard, a router has nothing to save.
2. Volume is high enough. The engineering and evaluation effort has a fixed cost. At low volume, picking one model is cheaper than building a router.
3. Quality is measurable. Without an eval you cannot tell whether the cheap model is good enough on a given class of request.

Anthropic's models overview describes the smallest current model, Claude Haiku 5.5, as intended for high-volume, latency-sensitive tasks such as classification, extraction and routing, and describes Claude Sonnet 5.5 as the best combination of speed and intelligence. That matches how most routers divide work.

What the models cost

Prices from Anthropic's pricing page on 30 October 2026, in USD per million tokens (MTok):

Two details matter for routing. First, Haiku 5.5 has a higher price for prompts over 100,000 tokens, so very long inputs lose part of the advantage. Second, batch requests are 50 percent off and prompt cache reads cost less than base input, so routing is one of several levers. For caching, see prompt caching cost and latency. Prices change, so read the current table before you rely on these numbers.

Using these prices, a request with 2,000 input tokens and 500 output tokens costs:

• Haiku 5.5: 2,000 x $0.10 / 1,000,000 + 500 x $0.50 / 1,000,000 = $0.0002 + $0.00025 = $0.00045
• Sonnet 5.5: 2,000 x $2 / 1,000,000 + 500 x $10 / 1,000,000 = $0.004 + $0.005 = $0.009

In that example the large model costs 20 times as much. If a router correctly handles 60 percent of traffic on the small model, the blended cost falls substantially. The exact saving depends on your mix, so measure it.

Remember that the thinking and effort settings also change the number of output tokens a model uses, and the default effort differs by model. Check the effort and thinking behavior in the documentation and set it explicitly for fair comparisons.

Design decisions before code

Write down these four things first.

1. The quality metric. For example: the share of eval cases where the answer is judged correct, or where the JSON validates and matches the expected labels.
2. The tolerance. For example: the router may lose no more than two percentage points of pass rate against the large model alone. Choose it before you see results, so you do not move the goalposts afterward.
3. The route set. Two tiers (small and large) is enough to start. A third tier adds evaluation work.
4. The routing signal. What do you know about a request before you call a model? Task type, input length, presence of code, required output format, customer tier and whether a previous attempt failed are all cheap signals.

Step 1: A rule-based router

Start with rules. They cost nothing per request, are easy to explain and are easy to test.

from dataclasses import dataclass
 
SMALL = "claude-haiku-5-5"
LARGE = "claude-sonnet-5-5"
 
# Task types that your application already knows when it builds the request
SIMPLE_TASKS = {"classify", "extract", "tag", "short_rewrite"}
 
 
@dataclass
class Request:
    task: str
    prompt: str
    needs_reasoning: bool = False
 
 
def route(req: Request) -> str:
    # Hard signals first: anything that needs multi-step reasoning goes large
    if req.needs_reasoning:
        return LARGE
    # Very long prompts: small-model price advantage shrinks above 100k tokens,
    # and long context is where quality differences tend to show
    if len(req.prompt) > 200_000:  # characters, a rough proxy for tokens
        return LARGE
    if req.task in SIMPLE_TASKS:
        return SMALL
    return LARGE

Key points:

• The default is the large model. A request goes to the small model only when it matches a rule you trust. This keeps mistakes expensive in money, not in quality.
• The 200,000 character threshold is a rough proxy for tokens. Count real tokens if the threshold matters to you, and see token counting and context window limits.
• Task labels come from your application, not from the model. That is the cheapest and most reliable signal.

Step 2: Call the model and record usage

Every response reports token usage, which you use for cost tracking. The SDK call shape is the same for every model.

import time
import anthropic
 
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment
 
# USD per million tokens, from the Anthropic pricing page (30 Oct 2026)
PRICES = {
    "claude-haiku-5-5": {"in": 0.10, "out": 0.50},
    "claude-sonnet-5-5": {"in": 2.00, "out": 10.00},
}
 
 
def cost_usd(model: str, input_tokens: int, output_tokens: int) -> float:
    p = PRICES[model]
    return (input_tokens * p["in"] + output_tokens * p["out"]) / 1_000_000
 
 
def call(model: str, system: str, prompt: str, max_tokens: int = 1024) -> dict:
    start = time.perf_counter()
    msg = client.messages.create(
        model=model,
        max_tokens=max_tokens,
        system=system,
        messages=[{"role": "user", "content": prompt}],
    )
    text = "".join(b.text for b in msg.content if b.type == "text")
    return {
        "model": model,
        "text": text,
        "stop_reason": msg.stop_reason,
        "input_tokens": msg.usage.input_tokens,
        "output_tokens": msg.usage.output_tokens,
        "cost": cost_usd(model, msg.usage.input_tokens, msg.usage.output_tokens),
        "latency_s": time.perf_counter() - start,
    }

Two cautions. The cost function ignores cache pricing and the over-100,000-token Haiku price, so extend it if you use caching or long prompts. And always check stop_reason: a value of max_tokens means the answer was cut off, which is a failure you should count.

Step 3: An optional cascade

A cascade tries the small model and escalates if a cheap check fails. It works best when you can verify an output mechanically, for example that it is valid JSON with the required keys.

import json
 
 
def valid_ticket(text: str) -> bool:
    try:
        data = json.loads(text)
    except json.JSONDecodeError:
        return False
    return isinstance(data, dict) and {"category", "priority"} <= data.keys()
 
 
def cascade(system: str, prompt: str) -> dict:
    first = call(SMALL, system, prompt)
    if first["stop_reason"] != "max_tokens" and valid_ticket(first["text"]):
        first["escalated"] = False
        return first
    second = call(LARGE, system, prompt)
    second["escalated"] = True
    # Count both calls: the failed attempt still cost money and time
    second["cost"] += first["cost"]
    second["latency_s"] += first["latency_s"]
    return second

The cascade's total cost on an escalated request is the sum of both calls, and its latency is also the sum. Track the escalation rate. If more than a modest share of requests escalate, the small model is wrong for that task and a rule that sends it straight to the large model is cheaper.

A validity check does not prove the answer is correct, only well-formed. For tasks where correctness matters, pair the cascade with an eval that compares labels.

Step 4: A classifier router (only if rules are not enough)

If you cannot tell difficulty from task labels, you can ask the small model to judge it. This adds a call to every request, so the savings must outweigh it.

ROUTER_SYSTEM = (
    "You decide whether a request needs a strong reasoning model. "
    "Answer with exactly one word: SIMPLE or COMPLEX. "
    "COMPLEX means multi-step reasoning, math, code changes, or ambiguous "
    "instructions. SIMPLE means classification, extraction, or short rewriting."
)
 
 
def llm_route(prompt: str) -> str:
    r = call(SMALL, ROUTER_SYSTEM, prompt[:4000], max_tokens=256)
    return SMALL if r["text"].strip().upper().startswith("SIMPLE") else LARGE

Notes:

• The prompt is truncated for the router call to cap its cost.
• Any unclear answer falls to the large model, so errors lean toward quality.
• The router call has its own cost and latency, so add it to your totals when you evaluate.

Step 5: Evaluate the router

The evaluation compares three setups on the same labeled cases: small only, large only and routed. Build the dataset from real requests, with an expected answer for each, and include both easy and hard cases in realistic proportions.

import csv
import json
import statistics
 
 
def grade(case: dict, text: str) -> bool:
    # Replace with your own check: exact label match, schema validation,
    # or an LLM judge for open-ended answers
    return text.strip().lower() == case["expected"].strip().lower()
 
 
def run(setup: str, cases: list[dict], system: str) -> dict:
    passed, costs, lats = 0, [], []
    for case in cases:
        req = Request(task=case["task"], prompt=case["prompt"],
                      needs_reasoning=case.get("needs_reasoning", False))
        model = {"small": SMALL, "large": LARGE}.get(setup) or route(req)
        r = call(model, system, case["prompt"])
        passed += grade(case, r["text"])
        costs.append(r["cost"])
        lats.append(r["latency_s"])
    n = len(cases)
    return {
        "setup": setup,
        "pass_rate": passed / n,
        "cost_per_request": sum(costs) / n,
        "p50_latency_s": statistics.median(lats),
    }
 
 
if __name__ == "__main__":
    with open("eval_cases.jsonl") as f:
        cases = [json.loads(line) for line in f]
    system = "Answer with the label only."
    rows = [run(s, cases, system) for s in ("small", "large", "routed")]
    with open("routing_report.csv", "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=rows[0].keys())
        w.writeheader()
        w.writerows(rows)
    for row in rows:
        print(row)

Each line of eval_cases.jsonl is a JSON object like this:

{"task": "classify", "prompt": "Customer says the invoice total is wrong.", "expected": "billing"}

Reading the report

Break the results down by task type. An overall average can hide a task where the small model fails badly. Also look at the individual cases where the routed result fails but the large model passes. Those are the router's mistakes, and they tell you which rule to change.

Use enough cases that the difference is meaningful. A handful of cases cannot show a two-point gap, so expand the set before you rely on a tolerance that small. Because model output varies between runs, repeat the evaluation and look at the spread.

Step 6: Roll out and monitor

1. Log every request with the route chosen, model, token counts, cost, latency, stop reason and whether the output passed any automatic check.
2. Ship behind a flag and start with a small share of traffic, or route only one task type first.
3. Sample outputs for review. Periodically pull routed outputs, especially those from the small model, and grade them against the large model's output on the same input.
4. Watch the escalation rate if you use a cascade, and the share of requests going to each model.
5. Re-run the eval when you change a prompt, a rule or a model. Anthropic publishes retirement dates for each model in its models overview, so plan for updates. For example, the page lists Claude Haiku 5.5 as not retiring sooner than October 7, 2027.

Pitfalls

• Comparing models with different effort or thinking settings. The default effort differs by model. Set it explicitly so you compare like with like.
• Ignoring output tokens. Output is priced higher than input, and a model that writes longer answers can erase a price advantage. Compare cost per request, not price per token.
• Routing on a signal you cannot trust. If task labels are wrong or users can influence them, the router will send hard requests to the small model.
• Forgetting the router's own cost. A classifier call on every request adds to your bill and latency.
• Judging only on average. Check the worst task type, and any request class with high business impact.
• Treating a validity check as correctness. Valid JSON can still be the wrong label.

Summary checklist

1. Confirm there is a mix of easy and hard requests and enough volume to justify routing.
2. Define the quality metric and tolerance before you build.
3. Start with rules, default to the large model, and send only trusted request types to the small model.
4. Log tokens, cost, latency, stop reason and route for every call.
5. Add a cascade or classifier only if the numbers justify it.
6. Evaluate small only, large only and routed on the same labeled set, and break results down by task type.
7. Roll out gradually, sample outputs and re-run the eval when anything changes.

References

• Anthropic, Models overview: https://platform.claude.com/docs/en/about-claude/models/overview
• Anthropic, Pricing: https://platform.claude.com/docs/en/about-claude/pricing
• Anthropic, Messages API reference: https://platform.claude.com/docs/en/api/messages