TL;DR — Routing sends each request to the cheapest model that can handle it, escalating only what needs escalating. Done well it halves cost and cuts tail latency without a visible quality drop — but it is not free: in the worked example below, routing gives up 0.5 accuracy points to halve the bill, because easy requests now go to a weaker model. The real risk is not the tiers, it is the router: a classifier with a 15% error rate gave back 2.3 accuracy points, more than four times what routing cost in the first place. So the rule is that the router must be cheaper and more reliable than the decision it is replacing — and where a deterministic rule can decide, no model should be involved at all. It stops working when request difficulty is not predictable from the request, at which point you are guessing and should escalate on a verified failure instead.
1. Simple explanation
You have a cheap fast model and an expensive slow one. Most requests are easy and the cheap one handles them perfectly. A minority are hard and need the expensive one. Sending everything to the expensive model works and wastes money; sending everything to the cheap one saves money and is wrong too often.
Routing is the obvious middle: look at each request, guess how hard it is, and pick accordingly.
The catch is that the guess is itself a decision that can be wrong, and a wrong guess is worse than either extreme — you pay for the expensive model on an easy request, or you get a wrong answer on a hard one that the expensive model would have got right. So the quality of a routed system is bounded by the quality of its router, not by the quality of its best model.
Analogy — a hospital triage desk. Most arrivals need a nurse; a few need a surgeon. Triage exists because sending everyone to the surgeon is unaffordable and sending everyone to the nurse is dangerous. But triage is a skill in its own right, and a bad triage nurse is worse than no triage: they send appendicitis home and refer sprained ankles to theatre. The analogy carries the mechanism exactly — the value is in accurate sorting, the sorting is a separate fallible step, and you must measure the sorter separately from the specialists.
2. Diagram
ALL-LARGE ROUTED ALL-SMALL
every request ┌── router ──┐ every request
│ │ cheap + │ │
▼ │ reliable? │ ▼
┌─────────┐ └──┬──────┬──┘ ┌─────────┐
│ LARGE │ │ easy │ hard │ SMALL │
│ $10/1k │ ┌────▼──┐ ┌─▼──────┐ │ $0.2/1k │
│ 2600ms │ │ SMALL │ │ LARGE │ │ 400ms │
└─────────┘ └───────┘ └────────┘ └─────────┘
95.5% acc $5.10/1k · 1500ms 79.0% acc
$10.00/1k 95.0% acc $0.20/1k
correct but 2x cheaper, 1100ms faster, cheap but wrong
overpaying and 0.5 points WORSE 16 points too often
THE ROUTER IS THE WEAK LINK
router error rate accuracy lost vs a perfect router
0% 95.0% —
5% 94.2% -0.8 pp
15% 92.7% -2.3 pp <- worse than routing's own cost
30% 90.3% -4.6 pp
THE ORDER THAT MATTERS
1. deterministic rule? ──yes──▶ answer it. no model, no router. $0
2. cached identical Q? ──yes──▶ replay. $0
3. cheap tier $
4. escalate ONLY on a verified failure $$$
3. How it works
3.1 Tiering: naming the levels before routing between them
A tier is a (model, settings) pair with a measured cost, latency, and accuracy on your traffic. Vendor benchmarks are not a substitute — a model that leads a public leaderboard can be worse than a small one on your narrow task, and the only way to know is to score both.
In practice three tiers cover most systems:
| tier | typical use | what it must be |
|---|---|---|
| deterministic | known facts, lookups, format conversions | code, not a model |
| cheap | classification, extraction, short generation | the default |
| expensive | multi-step reasoning, ambiguity, high-stakes | the exception |
The first row is the one teams skip. If a fact is already in your database or already in the request, no model should be consulted — a conditional in a template is deterministic, free, and testable, while the same rule expressed as an instruction is probabilistic and billed per call.
3.2 Complexity-based routing, and what you can route on
The router needs a difficulty signal. In increasing order of cost:
- Deterministic features — request length, presence of a known keyword, whether a required field is missing, whether a retrieval step found anything above threshold. Free, and surprisingly strong.
- Retrieval score — if your knowledge base returned a confident match, the request is by definition easy. This reuses work you already did, which makes it the best-value signal in most RAG systems.
- A small classifier — a cheap model asked to emit a single label from a fixed list. Constrain the output to one token from a known set; anything else is invalid and should fail closed to escalation.
- The cheap model's own confidence — ask it, then escalate on low confidence. Beware: self-reported confidence is poorly calibrated and correlates weakly with correctness.
3.3 Cost-aware selection: the arithmetic that justifies the tier
Routing is worth it when the saving exceeds the value of the accuracy you give up. That means you need a price for being wrong. Teams resist naming one, and then make the decision implicitly and badly. A rough number beats no number.
worth_it iff (C_large - C_routed) > (A_large - A_routed) * value_per_correct
If you cannot estimate value_per_correct, you cannot evaluate a routing decision — you can only assert one.
3.4 Fallback: escalate on evidence, not on vibes
There are two different mechanisms people call "fallback" and conflating them causes outages.
Escalation is routing after the fact: the cheap tier answered, something checked the answer, the check failed, so re-run on the expensive tier. This improves quality because the check is evidence. Good checks are cheap and mechanical — did the JSON parse, is the cited fact actually in the source, is the number in range.
Failover is what you do when a provider is down. It must never silently degrade quality in a way the caller cannot detect. The failure mode to avoid: an exception handler that swallows the error and returns whatever the cheapest path produces, logged as a success. That converts an outage into a stream of confidently wrong answers, which is strictly worse than an error, because nobody gets paged.
Every fallback path needs its own status in the logs. If a degraded answer and a good answer look identical in your telemetry, you will not find out until a customer tells you.
3.5 A/B testing a routing change
Routing changes are quality changes and need the same rigour as a model swap: split traffic, run long enough for significance, and compare on the metric you care about rather than the one that is easy to measure. Two traps specific to routing. First, cost and quality move in opposite directions, so you need a decision rule agreed before the test — "ship if cost drops 40% and accuracy drops less than 0.5 points," not "look at the numbers and argue." Second, aggregate accuracy hides the group that got worse: slice by request type, because routing concentrates its damage on whatever the router misjudges.
3.6 Where routing stops being the right answer
Routing assumes difficulty is predictable from the request. When it is not — when a short innocuous question happens to require careful multi-step work — any router will misclassify it, and the fix is escalation on a verified failure rather than a better classifier. Routing also stops paying when the tiers are close in price, since the saving no longer covers the added complexity and the new failure mode. And if you have no scored evaluation set, routing is unmeasurable: you cannot tell a 0.5-point loss from a 5-point one, so you are not making a tradeoff, you are hoping.
4. The math
4.1 Blended cost and accuracy
With a fraction e of traffic escalated:
C_routed = (1 - e) * C_small + e * C_large
A_routed = (1 - e) * A_small|easy_traffic + e * A_large|hard_traffic
and with an imperfect router of error rate r, some easy requests land on
large (waste) and some hard requests land on small (mistakes):
A_actual = A_routed - r * (A_large|hard - A_small|hard) * fraction_hard
└──────────── the cost of a bad router ────────────┘
The subtracted term is what §3.2 is really about: the gap between tiers on hard traffic is the penalty multiplier on router error. A big quality gap between tiers makes accurate routing more valuable, not less.
4.2 Worked example
1000 requests, half of them genuinely hard. Two tiers, measured:
SMALL $0.0002/call p95 400 ms 97% on easy 61% on hard
LARGE $0.0100/call p95 2600 ms 98% on easy 93% on hard
Sweeping the escalation threshold:
threshold -> large $/1k accuracy mean ms
0.00 100% $ 10.00 95.5% 2600 (all large)
0.25 75% $ 7.55 95.2% 2050
0.50 50% $ 5.10 95.0% 1500
0.75 25% $ 2.65 87.0% 950
1.01 0% $ 0.20 79.0% 400 (all small)
Routing at the true boundary (0.50) is 2.0x cheaper and 1100 ms faster than all-large, and 0.5 accuracy points worse. It is 16 points better than all-small for $4.90 per thousand calls.
Note the shape between 0.50 and 0.75: accuracy falls off a cliff (95.0% → 87.0%) while cost only halves again. That knee is where the threshold starts sending genuinely hard requests to the small tier. Finding it requires a scored set; guessing at it is how routing gets a bad reputation.
And the router's own error rate:
error rate accuracy lost vs perfect
0% 95.0% 0.0pp
5% 94.2% -0.8pp
15% 92.7% -2.3pp
30% 90.3% -4.6pp
At 15% router error you have lost 2.3 points — more than four times the 0.5 points routing cost you in the first place. The router, not the tier choice, is the thing to measure.
5. Real code
"""Route by complexity across two model tiers, and find the threshold that wins."""
from dataclasses import dataclass
@dataclass
class Tier:
name: str
usd_per_call: float
p95_ms: int
acc_easy: float # accuracy on easy traffic
acc_hard: float # accuracy on hard traffic
SMALL = Tier("small", 0.0002, 400, acc_easy=0.97, acc_hard=0.61)
LARGE = Tier("large", 0.0100, 2600, acc_easy=0.98, acc_hard=0.93)
# 1000 requests: complexity score in [0,1]; >= 0.5 is genuinely hard.
TRAFFIC = [(i / 1000, (i / 1000) >= 0.5) for i in range(1000)]
FRACTION_HARD = sum(1 for _s, h in TRAFFIC if h) / len(TRAFFIC)
def route(threshold: float) -> dict:
"""Send a request to LARGE when its complexity score >= threshold."""
cost = correct = 0.0
latencies = []
escalated = 0
for score, is_hard in TRAFFIC:
tier = LARGE if score >= threshold else SMALL
escalated += tier is LARGE
cost += tier.usd_per_call
latencies.append(tier.p95_ms)
correct += tier.acc_hard if is_hard else tier.acc_easy
n = len(TRAFFIC)
return {
"threshold": threshold,
"escalated_pct": escalated / n * 100,
"usd_per_1k": cost,
"accuracy": correct / n,
"mean_ms": sum(latencies) / n,
}
print(f"traffic: {len(TRAFFIC)} requests, {FRACTION_HARD:.0%} genuinely hard\n")
print(f"{'threshold':>9} {'-> large':>9} {'$/1k':>8} {'accuracy':>9} {'mean ms':>8}")
rows = {}
for t in (0.0, 0.25, 0.5, 0.75, 1.01):
r = route(t)
rows[t] = r
label = {0.0: " (all large)", 1.01: " (all small)"}.get(t, "")
print(f"{t:>9.2f} {r['escalated_pct']:>8.0f}% ${r['usd_per_1k']:>7.2f} "
f"{r['accuracy']:>8.1%} {r['mean_ms']:>7.0f}{label}")
all_large, all_small, routed = rows[0.0], rows[1.01], rows[0.5]
print(f"\nrouting at the true complexity boundary (0.50):")
print(f" vs all-large : {all_large['usd_per_1k'] / routed['usd_per_1k']:.1f}x cheaper, "
f"{all_large['mean_ms'] - routed['mean_ms']:.0f} ms faster, "
f"{(routed['accuracy'] - all_large['accuracy']) * 100:+.1f} accuracy points")
print(f" vs all-small : {(routed['accuracy'] - all_small['accuracy']) * 100:+.1f} "
f"accuracy points for ${routed['usd_per_1k'] - all_small['usd_per_1k']:.2f} per 1k")
# What an imperfect classifier costs: mis-scored requests go to the wrong tier.
print("\nrouter misclassification is the real risk:")
print(f" {'error rate':>10} {'accuracy':>9} {'lost vs perfect':>16}")
for err in (0.0, 0.05, 0.15, 0.30):
correct = 0.0
for i, (score, is_hard) in enumerate(TRAFFIC):
seen_hard = (not is_hard) if (i % 100) < err * 100 else is_hard
tier = LARGE if seen_hard else SMALL
correct += tier.acc_hard if is_hard else tier.acc_easy
acc = correct / len(TRAFFIC)
print(f" {err:>10.0%} {acc:>8.1%} {(acc - routed['accuracy']) * 100:>15.1f}pp")
# Routing is a REAL trade, not a free lunch: it buys cost and latency, and it
# does cost accuracy, because easy requests now go to the weaker model.
assert routed["usd_per_1k"] < all_large["usd_per_1k"]
assert routed["accuracy"] > all_small["accuracy"]
assert routed["accuracy"] < all_large["accuracy"] # <- the price of routing
assert round(all_large["usd_per_1k"] / routed["usd_per_1k"], 1) == 2.0
lost = (all_large["accuracy"] - routed["accuracy"]) * 100
print(f"\nrouting is not free: it gives up {lost:.1f} accuracy points to halve cost")
print("decide whether that trade is acceptable BEFORE shipping it, on a scored set")
print("all assertions passed")
# Output:
# traffic: 1000 requests, 50% genuinely hard
#
# threshold -> large $/1k accuracy mean ms
# 0.00 100% $ 10.00 95.5% 2600 (all large)
# 0.25 75% $ 7.55 95.2% 2050
# 0.50 50% $ 5.10 95.0% 1500
# 0.75 25% $ 2.65 87.0% 950
# 1.01 0% $ 0.20 79.0% 400 (all small)
#
# routing at the true complexity boundary (0.50):
# vs all-large : 2.0x cheaper, 1100 ms faster, -0.5 accuracy points
# vs all-small : +16.0 accuracy points for $4.90 per 1k
#
# router misclassification is the real risk:
# error rate accuracy lost vs perfect
# 0% 95.0% 0.0pp
# 5% 94.2% -0.8pp
# 15% 92.7% -2.3pp
# 30% 90.3% -4.6pp
#
# routing is not free: it gives up 0.5 accuracy points to halve cost
# decide whether that trade is acceptable BEFORE shipping it, on a scored set
# all assertions passed
Replace acc_easy / acc_hard with numbers from your own scored set. Until you have those four numbers, the table is a shape, not a decision.
6. Real-world example
A document-processing service classified incoming forms and extracted a handful of fields. It ran everything through the largest available model because that was what the prototype used, and the bill was the single largest line item in the product's cost of goods.
The team added routing: a small classifier picked a tier based on document type and page count. Cost fell 60% in the first week and accuracy held on their dashboard.
Six weeks later a customer escalated. A whole category of documents — scanned forms with handwritten annotations — had been silently getting worse. The router keyed off document type, handwritten annotations did not change the type, and the small tier was much worse at them than the large one. The aggregate accuracy metric had barely moved because those documents were 4% of volume; within that 4%, correctness had dropped by more than twenty points.
Two design errors, neither about the tiers. The router keyed on a feature (document type) that was not the actual difficulty driver (presence of handwriting), which is the classic routing mistake. And quality was tracked only in aggregate, so a severe regression in a small slice was invisible — exactly the failure §3.5 warns about.
The fix was to add a cheap deterministic pre-check for handwriting and route on that, plus per-category quality tracking with alerts. Worth noting what they did not do: revert the routing. It was still saving most of that 60%. The problem was never routing, it was routing on the wrong signal and not slicing the metric.
7. Interview questions companies actually ask
Q1. How do you decide which requests go to the expensive model? Start with the signals you already have for free — request length, missing fields, whether retrieval returned a confident match — before adding a classifier, because a model that routes is another model that can be wrong. Then pick the threshold empirically against a scored set, looking for the knee where accuracy falls off faster than cost. And put a tier above both: if a deterministic rule can answer, no model should be involved.
Q2. What is the main risk of model routing? The router, not the models. It introduces a new decision that can be wrong in a way that is invisible in aggregate metrics — and because the penalty for a misroute scales with the quality gap between your tiers, a big gap makes router errors more expensive rather than less. In the worked example, a 15% router error rate cost four times more accuracy than routing itself saved.
Q3. What is the difference between fallback and escalation? Escalation is quality-driven: the cheap tier answered, a mechanical check failed, so you re-run on the expensive tier. That improves outcomes because the check is evidence. Failover is availability-driven: a provider is down, so you use another path. The dangerous version is a bare exception handler that returns whatever the cheapest path produced and logs it as success, which turns an outage into confidently wrong answers with nobody paged. Give every degraded path its own status.
Q4. How would you A/B test a routing change? Agree the decision rule before you start — for example ship if cost falls 40% and accuracy falls under half a point — because cost and quality move in opposite directions and post-hoc argument always favours whoever wants to ship. Split traffic, run to significance, then slice accuracy by request category, since routing concentrates its damage in whatever the router misjudges and an aggregate number will hide it.
Q5. When is routing not worth it? When difficulty is not predictable from the request, when the tiers are close in price so the saving does not cover the added failure mode, or when you have no scored evaluation set — in which case you cannot measure the tradeoff you are making. In the first case the right pattern is escalation on a verified failure rather than prediction.
Q6. Your routed system has the same average accuracy as before but customers complain. What happened? Almost certainly a slice regressed while the average held, because the regressed group is small. Break accuracy down by request type, source, language, and any other axis you have, and compare against the pre-routing baseline per slice. This is why per-category quality tracking is a prerequisite for routing rather than a nice-to-have.
Q7. How do you choose the tiers themselves? Measure candidates on your own scored set rather than trusting public leaderboards, because a model that leads a general benchmark can lose badly on a narrow task. Record cost, p95 latency, and accuracy separately for easy and hard traffic — those four numbers per tier are what every routing decision is computed from, and without them you are asserting rather than deciding.
8. When to use / tradeoffs
Reach for routing when:
- Traffic is genuinely mixed and difficulty is visible in the request
- The tiers differ by roughly an order of magnitude in cost
- You already have a scored evaluation set with easy and hard slices
- Tail latency matters and the expensive tier is slow
Reach for something else when:
- A deterministic rule can answer — then answer it, and skip the model entirely
- Difficulty is invisible until you attempt the task — escalate on a verified failure
- You cannot yet measure per-slice accuracy — build that first
| Situation | Why it breaks | Use instead |
|---|---|---|
| Difficulty not predictable from the request | Any router misclassifies | Escalate on a failed mechanical check |
| Router keys on a proxy, not the real driver | Whole slices silently degrade | Find the actual difficulty feature |
| Only aggregate accuracy tracked | Small-slice regressions invisible | Per-category metrics with alerts |
| Tiers within ~2x on price | Saving does not justify the new failure mode | One tier, tuned |
| Exception handler returns the cheap path | Outage becomes confident wrong answers | Distinct status per degraded path |
| No scored evaluation set | The tradeoff is unmeasurable | Build the set, then route |
Honest limits. The model in §4 is deliberately simple and flatters routing in three ways. It assumes a single scalar difficulty score with a clean boundary, when real difficulty is multi-dimensional and the boundary is fuzzy. It assumes tier accuracy is constant within easy and hard buckets, when in reality it varies continuously and the small tier degrades gradually rather than falling off a cliff. And it treats router error as uniform noise, when real misclassification is correlated — routers fail systematically on particular request types, which is exactly why the §6 failure looked like a 4% slice rather than scattered errors. Correlated failure is much worse than the table suggests. Finally, the accuracy numbers assume you have a scored set that reflects production traffic; if it was assembled from easy examples, every number here is optimistic and the knee is in the wrong place.
9. Summary + related articles
- Routing sends each request to the cheapest tier that can handle it. It is a real trade, not a free lunch: halving cost cost 0.5 accuracy points in the worked example.
- The tier above every model tier is deterministic code. If the answer is already known, no model should be consulted.
- The router is the weak link. A 15% router error rate cost 2.3 accuracy points — over four times what routing saved. Measure the router separately from the tiers.
- Route on the actual difficulty driver, not a convenient proxy. Keying on the wrong feature degrades whole slices silently.
- Escalation (quality-driven, after a failed check) and failover (availability-driven) are different mechanisms. Give every degraded path its own log status, or an outage becomes confident wrong answers.
- Find the threshold on a scored set — there is a knee where accuracy falls faster than cost, and guessing at it is how routing gets a bad name.
- Track accuracy per slice. An aggregate metric will hide a twenty-point regression in a small category.
- Routing stops applying when difficulty is not predictable from the request, when tiers are close in price, or when you cannot measure quality.
Related:
- Cost-Latency Tradeoffs — the curve routing moves you along, plus batching and caching
- RAG Cost Optimization: Find the Step That Runs Forty Times — §3.2 per-role routing and §3.5 on not making the call at all
- Reasoning Budgets: When Thinking Tokens Are Waste — tier choice when the difference is hidden deliberation
- Production Agents — cost per task with routing, and keeping decisions in code
- Agent Evaluation — building the scored, sliced set every number here depends on
- LLM Observability — per-route cost and latency attribution
- RAG at Scale — retrieval confidence as a free routing signal
Resources
- Chen, Zaharia & Zou (2023) — FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, arXiv:2305.05176 — the cascade-and-escalate pattern with measured cost/quality curves: https://arxiv.org/abs/2305.05176
- Ong et al. (2024) — RouteLLM: Learning to Route LLMs with Preference Data, arXiv:2406.18665 — training a router and the cost/quality frontier it traces: https://arxiv.org/abs/2406.18665
- Kleinberg & Tardos — Algorithm Design, Ch. 6 on dynamic programming — useful background for threshold optimisation over a scored set.
- OpenAI — production best practices, including cost management: https://platform.openai.com/docs/guides/production-best-practices
- Anthropic — model overview and selection guidance for choosing tiers: https://docs.claude.com/en/docs/about-claude/models
- Note on the original stub's links: the LangChain and Anthropic A/B-testing URLs it cited no longer resolve to the pages implied; the references above were checked instead.