# LLM Model Routing: Send Easy Requests to a Cheaper Model > Build a simple Python router that sends each request to a small or large Claude model, then measure cost and quality against an eval set before you ship it. Source: https://www.epifive.com/blog/llm-model-routing-cheap-vs-capable-model-cost-control · Published 2026-10-30 · EpiFive ## Key takeaways - Routing sends each request to a small or large model so you pay for the capable model only when the request needs it. - Anthropic's pricing page lists Claude Haiku 5.5 at $0.10 input and $0.50 output per million tokens for prompts up to 100,000 tokens, against $2 and $10 for Claude Sonnet 5.5. - Start with rule-based routing on task type and input features, and add a classifier or a cascade only if measurements justify it. - Decide quality and cost targets before you build, then compare the router with each model alone on the same labeled eval set. - Log the route, model, token usage and outcome for every request so you can find the requests the router gets wrong. A model router picks a model per request. Easy requests such as classification, extraction and short rewrites go to a small, cheap model, and difficult ones go to a larger model. The saving comes from the price gap between models, and the risk is quality loss on requests the router misjudges. So build the simplest router that works, measure it against each model alone on a labeled eval set, and ship it only if the quality drop is inside a margin you chose in advance. ## Who this is for and what you will have at the end This guide is for developers running an LLM feature in production or about to, who see a growing bill and wonder whether every request needs the strongest model. At the end you will have a working Python router for the Claude API, a cost calculator that uses logged token usage, a cascade option, and an eval script that compares three setups: small model only, large model only and routed. If you have no eval set yet, start with [getting started with LLM evals](https://www.epifive.com/blog/ai-engineering-llm-evals-small-team-getting-started) and [golden datasets for regression testing](https://www.epifive.com/blog/llm-golden-dataset-regression-testing). The code uses the `anthropic` Python SDK and model IDs and prices from Anthropic's documentation as checked on 30 October 2026. It was not run against the live API while writing this article, so test it with your own key and a small sample first. ## When routing is worth it Routing pays off when three things are true: 1. **You have a mix of request difficulty.** If all your requests are hard, a router has nothing to save. 2. **Volume is high enough.** The engineering and evaluation effort has a fixed cost. At low volume, picking one model is cheaper than building a router. 3. **Quality is measurable.** Without an eval you cannot tell whether the cheap model is good enough on a given class of request. Anthropic's models overview describes the smallest current model, Claude Haiku 5.5, as intended for high-volume, latency-sensitive tasks such as classification, extraction and routing, and describes Claude Sonnet 5.5 as the best combination of speed and intelligence. That matches how most routers divide work. ## What the models cost Prices from Anthropic's pricing page on 30 October 2026, in USD per million tokens (MTok): | Model | API ID | Input | Output | |---|---|---|---| | Claude Haiku 5.5 (prompts up to 100,000 tokens) | `claude-haiku-5-5` | $0.10 | $0.50 | | Claude Haiku 5.5 (prompts over 100,000 tokens) | `claude-haiku-5-5` | $0.50 | $2.50 | | Claude Sonnet 5.5 | `claude-sonnet-5-5` | $2 | $10 | | Claude Opus 5.5 | `claude-opus-5-5` | $4 | $20 | | Claude Fable 5.1 | `claude-fable-5-1` | $10 | $50 | Two details matter for routing. First, Haiku 5.5 has a higher price for prompts over 100,000 tokens, so very long inputs lose part of the advantage. Second, batch requests are 50 percent off and prompt cache reads cost less than base input, so routing is one of several levers. For caching, see [prompt caching cost and latency](https://www.epifive.com/blog/anthropic-prompt-caching-cost-latency). Prices change, so read the current table before you rely on these numbers. Using these prices, a request with 2,000 input tokens and 500 output tokens costs: - Haiku 5.5: 2,000 x $0.10 / 1,000,000 + 500 x $0.50 / 1,000,000 = $0.0002 + $0.00025 = $0.00045 - Sonnet 5.5: 2,000 x $2 / 1,000,000 + 500 x $10 / 1,000,000 = $0.004 + $0.005 = $0.009 In that example the large model costs 20 times as much. If a router correctly handles 60 percent of traffic on the small model, the blended cost falls substantially. The exact saving depends on your mix, so measure it. Remember that the thinking and effort settings also change the number of output tokens a model uses, and the default effort differs by model. Check the effort and thinking behavior in the documentation and set it explicitly for fair comparisons. ## Design decisions before code Write down these four things first. 1. **The quality metric.** For example: the share of eval cases where the answer is judged correct, or where the JSON validates and matches the expected labels. 2. **The tolerance.** For example: the router may lose no more than two percentage points of pass rate against the large model alone. Choose it before you see results, so you do not move the goalposts afterward. 3. **The route set.** Two tiers (small and large) is enough to start. A third tier adds evaluation work. 4. **The routing signal.** What do you know about a request before you call a model? Task type, input length, presence of code, required output format, customer tier and whether a previous attempt failed are all cheap signals. ## Step 1: A rule-based router Start with rules. They cost nothing per request, are easy to explain and are easy to test. ```python from dataclasses import dataclass SMALL = "claude-haiku-5-5" LARGE = "claude-sonnet-5-5" # Task types that your application already knows when it builds the request SIMPLE_TASKS = {"classify", "extract", "tag", "short_rewrite"} @dataclass class Request: task: str prompt: str needs_reasoning: bool = False def route(req: Request) -> str: # Hard signals first: anything that needs multi-step reasoning goes large if req.needs_reasoning: return LARGE # Very long prompts: small-model price advantage shrinks above 100k tokens, # and long context is where quality differences tend to show if len(req.prompt) > 200_000: # characters, a rough proxy for tokens return LARGE if req.task in SIMPLE_TASKS: return SMALL return LARGE ``` Key points: - The default is the large model. A request goes to the small model only when it matches a rule you trust. This keeps mistakes expensive in money, not in quality. - The 200,000 character threshold is a rough proxy for tokens. Count real tokens if the threshold matters to you, and see [token counting and context window limits](https://www.epifive.com/blog/llm-token-counting-context-window-limits). - Task labels come from your application, not from the model. That is the cheapest and most reliable signal. ## Step 2: Call the model and record usage Every response reports token usage, which you use for cost tracking. The SDK call shape is the same for every model. ```python import time import anthropic client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY from the environment # USD per million tokens, from the Anthropic pricing page (30 Oct 2026) PRICES = { "claude-haiku-5-5": {"in": 0.10, "out": 0.50}, "claude-sonnet-5-5": {"in": 2.00, "out": 10.00}, } def cost_usd(model: str, input_tokens: int, output_tokens: int) -> float: p = PRICES[model] return (input_tokens * p["in"] + output_tokens * p["out"]) / 1_000_000 def call(model: str, system: str, prompt: str, max_tokens: int = 1024) -> dict: start = time.perf_counter() msg = client.messages.create( model=model, max_tokens=max_tokens, system=system, messages=[{"role": "user", "content": prompt}], ) text = "".join(b.text for b in msg.content if b.type == "text") return { "model": model, "text": text, "stop_reason": msg.stop_reason, "input_tokens": msg.usage.input_tokens, "output_tokens": msg.usage.output_tokens, "cost": cost_usd(model, msg.usage.input_tokens, msg.usage.output_tokens), "latency_s": time.perf_counter() - start, } ``` Two cautions. The cost function ignores cache pricing and the over-100,000-token Haiku price, so extend it if you use caching or long prompts. And always check `stop_reason`: a value of `max_tokens` means the answer was cut off, which is a failure you should count. ## Step 3: An optional cascade A cascade tries the small model and escalates if a cheap check fails. It works best when you can verify an output mechanically, for example that it is valid JSON with the required keys. ```python import json def valid_ticket(text: str) -> bool: try: data = json.loads(text) except json.JSONDecodeError: return False return isinstance(data, dict) and {"category", "priority"} <= data.keys() def cascade(system: str, prompt: str) -> dict: first = call(SMALL, system, prompt) if first["stop_reason"] != "max_tokens" and valid_ticket(first["text"]): first["escalated"] = False return first second = call(LARGE, system, prompt) second["escalated"] = True # Count both calls: the failed attempt still cost money and time second["cost"] += first["cost"] second["latency_s"] += first["latency_s"] return second ``` The cascade's total cost on an escalated request is the sum of both calls, and its latency is also the sum. Track the escalation rate. If more than a modest share of requests escalate, the small model is wrong for that task and a rule that sends it straight to the large model is cheaper. A validity check does not prove the answer is correct, only well-formed. For tasks where correctness matters, pair the cascade with an eval that compares labels. ## Step 4: A classifier router (only if rules are not enough) If you cannot tell difficulty from task labels, you can ask the small model to judge it. This adds a call to every request, so the savings must outweigh it. ```python ROUTER_SYSTEM = ( "You decide whether a request needs a strong reasoning model. " "Answer with exactly one word: SIMPLE or COMPLEX. " "COMPLEX means multi-step reasoning, math, code changes, or ambiguous " "instructions. SIMPLE means classification, extraction, or short rewriting." ) def llm_route(prompt: str) -> str: r = call(SMALL, ROUTER_SYSTEM, prompt[:4000], max_tokens=256) return SMALL if r["text"].strip().upper().startswith("SIMPLE") else LARGE ``` Notes: - The prompt is truncated for the router call to cap its cost. - Any unclear answer falls to the large model, so errors lean toward quality. - The router call has its own cost and latency, so add it to your totals when you evaluate. ## Step 5: Evaluate the router The evaluation compares three setups on the same labeled cases: small only, large only and routed. Build the dataset from real requests, with an expected answer for each, and include both easy and hard cases in realistic proportions. ```python import csv import json import statistics def grade(case: dict, text: str) -> bool: # Replace with your own check: exact label match, schema validation, # or an LLM judge for open-ended answers return text.strip().lower() == case["expected"].strip().lower() def run(setup: str, cases: list[dict], system: str) -> dict: passed, costs, lats = 0, [], [] for case in cases: req = Request(task=case["task"], prompt=case["prompt"], needs_reasoning=case.get("needs_reasoning", False)) model = {"small": SMALL, "large": LARGE}.get(setup) or route(req) r = call(model, system, case["prompt"]) passed += grade(case, r["text"]) costs.append(r["cost"]) lats.append(r["latency_s"]) n = len(cases) return { "setup": setup, "pass_rate": passed / n, "cost_per_request": sum(costs) / n, "p50_latency_s": statistics.median(lats), } if __name__ == "__main__": with open("eval_cases.jsonl") as f: cases = [json.loads(line) for line in f] system = "Answer with the label only." rows = [run(s, cases, system) for s in ("small", "large", "routed")] with open("routing_report.csv", "w", newline="") as f: w = csv.DictWriter(f, fieldnames=rows[0].keys()) w.writeheader() w.writerows(rows) for row in rows: print(row) ``` Each line of `eval_cases.jsonl` is a JSON object like this: ```json {"task": "classify", "prompt": "Customer says the invoice total is wrong.", "expected": "billing"} ``` ### Reading the report | Result | What it means | What to do | |---|---|---| | Routed pass rate within tolerance of large, cost lower | The router works | Ship it behind a flag and monitor | | Routed pass rate well below large | Too many hard cases on the small model | Tighten the rules or add a cascade | | Small alone matches large on a task | That task does not need routing | Send it to small directly | | Routed cost close to large | Few requests qualify for the small model | Routing may not be worth the complexity | Break the results down by task type. An overall average can hide a task where the small model fails badly. Also look at the individual cases where the routed result fails but the large model passes. Those are the router's mistakes, and they tell you which rule to change. Use enough cases that the difference is meaningful. A handful of cases cannot show a two-point gap, so expand the set before you rely on a tolerance that small. Because model output varies between runs, repeat the evaluation and look at the spread. ## Step 6: Roll out and monitor 1. **Log every request** with the route chosen, model, token counts, cost, latency, stop reason and whether the output passed any automatic check. 2. **Ship behind a flag** and start with a small share of traffic, or route only one task type first. 3. **Sample outputs for review.** Periodically pull routed outputs, especially those from the small model, and grade them against the large model's output on the same input. 4. **Watch the escalation rate** if you use a cascade, and the share of requests going to each model. 5. **Re-run the eval** when you change a prompt, a rule or a model. Anthropic publishes retirement dates for each model in its models overview, so plan for updates. For example, the page lists Claude Haiku 5.5 as not retiring sooner than October 7, 2027. ## Pitfalls - **Comparing models with different effort or thinking settings.** The default effort differs by model. Set it explicitly so you compare like with like. - **Ignoring output tokens.** Output is priced higher than input, and a model that writes longer answers can erase a price advantage. Compare cost per request, not price per token. - **Routing on a signal you cannot trust.** If task labels are wrong or users can influence them, the router will send hard requests to the small model. - **Forgetting the router's own cost.** A classifier call on every request adds to your bill and latency. - **Judging only on average.** Check the worst task type, and any request class with high business impact. - **Treating a validity check as correctness.** Valid JSON can still be the wrong label. ## Summary checklist 1. Confirm there is a mix of easy and hard requests and enough volume to justify routing. 2. Define the quality metric and tolerance before you build. 3. Start with rules, default to the large model, and send only trusted request types to the small model. 4. Log tokens, cost, latency, stop reason and route for every call. 5. Add a cascade or classifier only if the numbers justify it. 6. Evaluate small only, large only and routed on the same labeled set, and break results down by task type. 7. Roll out gradually, sample outputs and re-run the eval when anything changes. ## References - Anthropic, Models overview: https://platform.claude.com/docs/en/about-claude/models/overview - Anthropic, Pricing: https://platform.claude.com/docs/en/about-claude/pricing - Anthropic, Messages API reference: https://platform.claude.com/docs/en/api/messages ## Frequently asked questions ### What is LLM model routing? Model routing means choosing a model for each request instead of using one model for everything. A router sends simple requests to a cheaper, faster model and difficult ones to a more capable model, so you pay the higher price only when it is likely to matter. ### How do I know the router is not hurting quality? Run the same labeled eval set through the router, through the small model alone and through the large model alone. Compare pass rate and cost per request. Ship the router only if its pass rate is within a margin you set in advance of the large model, at a lower cost. ### Should the router itself be an LLM call? It can be, but every routing call adds cost and latency. Start with simple rules such as task type, input length and required output format. Add a small-model classifier only if rules leave too much quality or cost on the table, and measure it like any other component. ### What about escalating when the small model fails? A cascade tries the small model first and escalates to the large model if a check fails, for example the output does not validate against a schema or the model reports low confidence. It protects quality on hard cases at the cost of paying for two calls on those requests.