TL;DR — Almost everyone optimising RAG cost starts with the wrong stage. The generation call is the one you can see — big prompt, long answer, obviously expensive — but in a pipeline with LLM reranking it is not where the money goes. Reranking runs once per candidate, so a single query makes ~40 small calls against 1 large one, and in the accounting below that fan-out is 78% of the bill while synthesis is 22%. The dominant lever is therefore per-role model routing: assign a model per role rather than per application, and put the cheap model on the high-volume scoring role. Swapping only the reranker to an economy tier cut cost per query from $1.148 to $0.256 — 78% — which is $89,172/month at 100k queries, from one config change and no code. Batching the fan-out helps less than expected and non-monotonically: batch-16 costs more than batch-8 here, because 3 batches of 16 pad 48 slots for 40 chunks. And a token-per-minute cap, not price, is often the real constraint — at 500k TPM this pipeline sustains 8.9 queries/minute regardless of what you are willing to pay.
1. Simple explanation
Ask someone which part of their RAG system costs the most and they will usually say the final generation call. It is the one with the enormous prompt and the long answer, and it is the one they wrote first.
Count the calls instead of the tokens.
A query with LLM reranking looks like this: one routing call, one query-rewriting call, forty reranking calls (one per retrieved candidate), one synthesis call, one follow-up call. Forty-four calls, of which forty do the same small job over and over.
Each reranking call is cheap — a few hundred tokens in, a score out. But cheap times forty beats expensive times one, and it is not close. The step nobody thinks about is the step that dominates the bill.
Analogy — the electricity bill. People assume the big-ticket appliance is the problem: the oven, the tumble dryer. Meanwhile a dozen devices draw a few watts continuously, all month. Nobody notices them because each one individually is trivially cheap. The oven is visible expensive; the standby load is actually expensive. In RAG, the synthesis call is the oven, and reranking is the standby load — except reranking is worse, because it is 78% of the bill rather than a footnote, and it stays invisible because per-call it looks negligible.
2. Diagram
ONE QUERY, BY CALL COUNT ONE QUERY, BY COST
──────────────────────── ──────────────────
route █ route · 0.0%
rewrite █ rewrite · 0.0%
rerank ████████████████ rerank ███████████ 78.4%
(40 calls) ($0.9000)
synthesise █ synthesise ███ 21.6%
follow-ups █ ($0.2475)
follow-ups · 0.0%
44 calls total $1.1481 / query
▲ ▲
the step you don't is the step that owns
think about the bill
PER-ROLE MODEL ROUTING — change one line of config
───────────────────────────────────────────────────
$/query $/100k queries
everything premium 1.2143 121,425
premium rerank+synth 1.1481 114,810 <- typical default
mid rerank 0.4281 42,810
economy rerank 0.2564 25,638 <- one line changed
economy rerank+mid synth 0.0584 5,838
everything economy 0.0112 1,122
swapping ONLY the reranker: -78%, or $89,172/month at 100k queries
THE OTHER CEILING — tokens per minute, not dollars
───────────────────────────────────────────────────
one query end to end = 56,430 tokens
100k TPM -> 1.8 queries/min you are rate-bound,
500k TPM -> 8.9 queries/min not budget-bound
1M TPM -> 17.7 queries/min
4M TPM -> 70.9 queries/min
3. How it works
3.1 Count calls before you count tokens
The first move in any RAG cost review is a table of stage × calls × tokens × price. Not an estimate — the actual call count, instrumented. Almost every surprising bill comes from a stage whose call count is a multiple of something (candidates, chunks, specialists, retries) rather than a constant.
Multipliers to look for:
reranking x candidate_k (typically 20-50)
per-specialist x fan-out width (agentic RAG, 2-5 branches,
retrieval and each branch may rerank -> multiplies AGAIN)
chunk-level extract x chunks retrieved
self-consistency x n samples
retries x (1 + retry rate)
Two multipliers compose. An agentic system fanning out to 3 specialists that each rerank 20 candidates makes 60 reranking calls, not 20 — the interaction between Agentic RAG: Routing Retrieval to Specialists and Reranking: The Second Pass That Decides What the Model Sees is where budgets actually die, and neither article's knob looks dangerous alone.
3.2 Per-role model routing
The single highest-leverage change, and it is configuration rather than code.
Most systems pick one model and use it everywhere, because that is how the SDK examples are written. But a RAG pipeline has several roles with genuinely different requirements:
ROLE what it does needs
─────────────────────────────────────────────────────────────────────
router pick specialists / classify cheap, fast, structured out
rewriter restructure the query cheap, fast
reranker score one chunk 1-10 cheap, HIGH VOLUME,
judgement not eloquence
synthesiser write the cited answer strong: this is the
output the user reads
follow-ups suggest next questions cheap, low stakes
Scoring is a fundamentally easier task than writing. "Does this passage answer this question, 1-10" is a judgement a small model makes nearly as well as a large one, because the output is one number against a written rubric rather than fluent prose grounded in a dozen sources. So the role that runs forty times is also the role least sensitive to model quality. That coincidence is the entire opportunity.
Measured: economy reranker + premium synthesis costs $0.256/query against $1.148 for premium everywhere the user cannot see the difference — a 78% cut with the user-visible output untouched.
There is a floor. A reranker weaker than the retriever it corrects makes ranking worse, not merely no better — quantified in Reranking: The Second Pass That Decides What the Model Sees. Downgrade the reranker, then measure nDCG, and stop before the sign flips. The cheapest model that stays above that line is the right one.
3.3 Batching the fan-out, and why it disappoints
If forty calls is the problem, send several chunks per call. This works, and it works less well than people expect, for a structural reason:
unbatched: 40 x (prompt + chunk) = 40 prompts + 40 chunks
batch of 8: 5 x (prompt + 8 chunks) = 5 prompts + 40 chunks
Batching amortises the fixed prompt, not the chunk tokens — and the chunks are the bulk. So savings converge quickly on the prompt's share of the total and stop.
Worse, batching is not monotonic. Measured below, batch-16 costs more than batch-8, because ⌈40/16⌉ = 3 batches hold 48 slots for 40 chunks: eight slots of padding, paid for. Batch sizes that do not divide your candidate count waste the remainder. Pick a batch size that divides candidate_k.
The quality cost is the real objection though. Chunks in one context window get scored relative to each other rather than against the rubric, so scores stop being comparable across batches — which silently breaks any absolute threshold you set. Batch when you need relative ordering within a page; do not batch when you need calibrated scores.
3.4 The other ceiling: tokens per minute
Price is not usually the binding constraint. Throughput is.
Providers cap requests-per-minute and tokens-per-minute. A RAG query with 40 reranking calls consumes 56,430 tokens end to end in the accounting below — so a 500k TPM allocation supports 8.9 queries per minute. Not 8.9 concurrent users: 8.9 queries per minute, total, no matter your budget.
Three implications people discover the hard way. A single query cannot exceed the per-minute budget, so if one query's 56k tokens approaches your TPM cap, no amount of concurrency control saves you — you must shrink the query. Pre-emptive throttling beats reactive retrying: track your own token spend against the window and delay before sending, rather than firing everything and handling the 429s afterwards. And the fan-out step should have its own concurrency limit, separate from the rest of the pipeline, because it is the only stage that can saturate the budget alone.
Retry mechanics — exponential backoff, full jitter, when to stop — are a general API concern rather than a RAG one, and are covered in API Best Practices. Reach for pre-emptive throttling first; backoff is what catches the cases your accounting missed.
3.5 The cheapest call is the one you do not make
Before optimising per-call cost, delete calls.
- Cache aggressively. Identical or near-identical queries are far more common than intuition suggests. Cache the rewritten query, the retrieval results, and the per-(query, chunk) rerank scores — that last cache hits often, because popular chunks recur across similar queries.
- Cut
candidate_kto the knee. Cost is linear inkand quality saturates; the measured knee in Reranking: The Second Pass That Decides What the Model Sees was around 20, past which each call bought 25× less. - Skip stages when they cannot help. A three-word query does not need rewriting. A query matching a cached FAQ does not need the pipeline at all.
- Prune the context before synthesis. Synthesis input scales with how many chunks you pass; passing 6 good chunks instead of 20 mediocre ones is cheaper and usually better.
3.6 Where cost optimisation stops paying
Below a few thousand queries a month, none of this matters and engineering time costs more than the savings. Optimise when the bill is a real line item, or when you are throughput-constrained.
And measure quality alongside cost, always. Every lever here trades one for the other, and a cost reduction that quietly degrades answers is not a saving — it is a deferred, harder-to-diagnose problem.
4. The math
4.1 The cost of a stage
cost_stage = calls * (in_tokens * price_in + out_tokens * price_out) / 1e6
The term people forget is calls. With calls = 40, a stage using 900 input and 120 output tokens per call carries 36,000 input tokens — three times the 12,000-token synthesis prompt — while looking negligible per call.
The cost ratio is even wider than the token ratio, because output tokens are priced several times higher than input and the fan-out produces 40 × 120 = 4,800 output tokens against synthesis's 900. Measured: reranking costs 3.6× synthesis on 3.0× the input tokens.
4.2 Which role to downgrade first
For a role r, the saving from moving it one tier down is:
saving_r = calls_r * (tokens_r) * (price_current - price_cheaper)
Since calls_r multiplies everything, role priority is set by call count, not by per-call size. Rank your roles by calls × tokens and downgrade from the top, checking quality after each step.
rerank 40 calls x 1020 tok = 40,800 token-units <- start here
synthesise 1 call x 12900 tok = 12,900
rewrite 1 call x 700 tok = 700
route 1 call x 680 tok = 680
Reranking carries 3.2× the token-volume of synthesis. It is first on the list, and it is also the role where a cheaper model costs you least — which is why the default configuration is usually backwards.
4.3 Batching arithmetic
For n candidates in batches of b, with a fixed prompt P and c tokens per chunk:
calls = ceil(n / b)
in_tokens = ceil(n / b) * (P + b*c)
The ceil is where the non-monotonicity lives. With n = 40, P = 400, c = 500:
b = 8 : ceil(40/8) = 5 calls, 5 * (400 + 4000) = 22,000 tokens
b = 16: ceil(40/16) = 3 calls, 3 * (400 + 8000) = 25,200 tokens <- worse
b = 40: ceil(40/40) = 1 call, 1 * (400 + 20000) = 20,400 tokens
Batch-16 issues fewer calls than batch-8 and costs 15% more, because the third batch is 60% padding. The rule: b should divide n.
The floor on batching savings is the prompt's share. As b → n you approach P + n*c against n*(P + c), so the best possible saving is bounded by how large P is relative to n*c — here 20,400 vs 36,000, a 43% reduction at maximum, most of which is already captured by b = 8.
4.4 The throughput ceiling
tokens_per_query = sum over stages of calls * (in + out)
max_queries_per_minute = TPM_cap / tokens_per_query
For this pipeline, tokens_per_query = 56,430:
100,000 TPM -> 1.8 queries/min
500,000 TPM -> 8.9 queries/min
1,000,000 TPM -> 17.7 queries/min
4,000,000 TPM -> 70.9 queries/min
Note this is a hard ceiling that concurrency cannot raise. If your traffic exceeds it you must reduce tokens per query — smaller candidate_k, batching, a shorter rerank prompt — or obtain more quota. Discovering this at launch is common and painful; the arithmetic takes five minutes beforehand.
5. Real code
Pure accounting: no simulation, no invented quality numbers, standard library only. Prices are illustrative tiers, not any vendor's list — substitute your own and the structure of the answer will not change.
TIERS = { # in / out $ per 1M tokens
"premium": (15.00, 75.00),
"mid": (3.00, 15.00),
"economy": (0.15, 0.60),
}
# One query's work. tokens are per CALL; calls is how many times it runs.
STAGES = [
# name, calls, in_tok, out_tok, default tier
("route", 1, 600, 80, "economy"),
("query rewrite", 1, 500, 200, "economy"),
("rerank (fan-out)", 40, 900, 120, "premium"),
("synthesise", 1, 12000, 900, "premium"),
("follow-ups", 1, 1200, 150, "economy"),
]
def stage_cost(calls, tin, tout, tier):
pin, pout = TIERS[tier]
return calls * (tin * pin + tout * pout) / 1e6
def table(assignment=None):
"""assignment: {stage_name: tier} overriding defaults."""
rows = []
for name, calls, tin, tout, default in STAGES:
tier = (assignment or {}).get(name, default)
rows.append((name, calls, tier, stage_cost(calls, tin, tout, tier)))
return rows
def total(assignment=None):
return sum(r[3] for r in table(assignment))
print("A. WHERE THE MONEY GOES (default: premium rerank + synthesis)")
print(f"{'stage':>18} {'calls':>6} {'tier':>9} {'$/query':>10} {'share':>7}")
print("-" * 56)
rows = table()
tot = sum(r[3] for r in rows)
for name, calls, tier, c in rows:
print(f"{name:>18} {calls:6} {tier:>9} {c:10.4f} {c/tot:7.1%}")
print(f"{'TOTAL':>18} {'':6} {'':9} {tot:10.4f}")
rerank = next(r for r in rows if r[0].startswith("rerank"))
synth = next(r for r in rows if r[0] == "synthesise")
print(f"\nrerank is {rerank[3]/tot:.0%} of spend on {rerank[1]} calls;"
f" synthesis is {synth[3]/tot:.0%} on 1 call")
print(f"rerank costs {rerank[3]/synth[3]:.1f}x synthesis despite using"
f" {rerank[1]*900/12000:.1f}x the input tokens")
print("\n\nB. PER-ROLE MODEL ROUTING -- one config per row")
configs = [
("everything premium", {n: "premium" for n, *_ in STAGES}),
("default (prem rerank+synth)", None),
("mid rerank", {"rerank (fan-out)": "mid"}),
("economy rerank", {"rerank (fan-out)": "economy"}),
("economy rerank, mid synth", {"rerank (fan-out)": "economy",
"synthesise": "mid"}),
("everything economy", {n: "economy" for n, *_ in STAGES}),
]
print(f"{'config':>30} {'$/query':>9} {'$/100k queries':>15} {'vs default':>11}")
print("-" * 70)
base = total()
results = {}
for label, a in configs:
t = total(a)
results[label] = t
print(f"{label:>30} {t:9.4f} {t*100000:15,.0f} {t/base:10.2f}x")
econ = results["economy rerank"]
print(f"\nswapping ONLY the reranker to economy: "
f"{base:.4f} -> {econ:.4f} per query ({1-econ/base:.0%} saved)")
print(f" at 100k queries/month that is "
f"${(base-econ)*100000:,.0f}/month from one config change")
print("\n\nC. BATCHING THE FAN-OUT STEP")
print(f"{'batch size':>11} {'calls':>6} {'in tokens':>10} {'$/query':>9} "
f"{'vs unbatched':>13}")
print("-" * 54)
PROMPT, PER_CHUNK, OUT_PER = 400, 500, 120 # tokens
unb = None
for b in (1, 2, 4, 8, 16, 40):
calls = -(-40 // b) # ceil
tin = calls * (PROMPT + b * PER_CHUNK)
tout = calls * b * OUT_PER
pin, pout = TIERS["premium"]
c = (tin * pin + tout * pout) / 1e6
if unb is None:
unb = c
print(f"{b:>11} {calls:6} {tin:10,} {c:9.4f} {c/unb:12.2f}x")
print("\n batching amortises the fixed prompt, not the chunk tokens,")
print(" so savings flatten fast -- and scores become batch-relative.")
print("\n\nD. THROUGHPUT UNDER A TOKEN-PER-MINUTE CAP")
print(f"{'TPM cap':>10} {'queries/min':>12} {'sec/query at 1 QPS':>20}")
print("-" * 44)
tokens_per_query = sum(c * (i + o) for _, c, i, o, _ in STAGES)
print(f" (one query consumes {tokens_per_query:,} tokens end to end)")
for cap in (100_000, 500_000, 1_000_000, 4_000_000):
qpm = cap / tokens_per_query
print(f"{cap:>10,} {qpm:12.1f} {60/qpm if qpm else 0:20.1f}")
# claims made in the prose
assert rerank[3] / tot > 0.5, "fan-out must dominate the bill"
assert econ < 0.35 * base, "economy reranker must cut cost by >65%"
assert results["everything economy"] < results["economy rerank"]
assert total({"rerank (fan-out)": "mid"}) < base
print("\nasserts passed")
# Output:
# A. WHERE THE MONEY GOES (default: premium rerank + synthesis)
# stage calls tier $/query share
# --------------------------------------------------------
# route 1 economy 0.0001 0.0%
# query rewrite 1 economy 0.0002 0.0%
# rerank (fan-out) 40 premium 0.9000 78.4%
# synthesise 1 premium 0.2475 21.6%
# follow-ups 1 economy 0.0003 0.0%
# TOTAL 1.1481
#
# rerank is 78% of spend on 40 calls; synthesis is 22% on 1 call
# rerank costs 3.6x synthesis despite using 3.0x the input tokens
#
#
# B. PER-ROLE MODEL ROUTING -- one config per row
# config $/query $/100k queries vs default
# ----------------------------------------------------------------------
# everything premium 1.2143 121,425 1.06x
# default (prem rerank+synth) 1.1481 114,810 1.00x
# mid rerank 0.4281 42,810 0.37x
# economy rerank 0.2564 25,638 0.22x
# economy rerank, mid synth 0.0584 5,838 0.05x
# everything economy 0.0112 1,122 0.01x
#
# swapping ONLY the reranker to economy: 1.1481 -> 0.2564 per query (78% saved)
# at 100k queries/month that is $89,172/month from one config change
#
#
# C. BATCHING THE FAN-OUT STEP
# batch size calls in tokens $/query vs unbatched
# ------------------------------------------------------
# 1 40 36,000 0.9000 1.00x
# 2 20 28,000 0.7800 0.87x
# 4 10 24,000 0.7200 0.80x
# 8 5 22,000 0.6900 0.77x
# 16 3 25,200 0.8100 0.90x
# 40 1 20,400 0.6660 0.74x
#
# batching amortises the fixed prompt, not the chunk tokens,
# so savings flatten fast -- and scores become batch-relative.
#
#
# D. THROUGHPUT UNDER A TOKEN-PER-MINUTE CAP
# TPM cap queries/min sec/query at 1 QPS
# --------------------------------------------
# (one query consumes 56,430 tokens end to end)
# 100,000 1.8 33.9
# 500,000 8.9 6.8
# 1,000,000 17.7 3.4
# 4,000,000 70.9 0.8
#
# asserts passed
Two things to notice in table B. everything premium costs only 1.06× the default — because the default already has premium on both expensive roles, so upgrading the three cheap ones changes almost nothing. Conversely economy rerank, mid synth reaches 0.05×. The spread between configurations is almost entirely about two roles, and knowing which two is the whole job.
In table C, the batch-16 row costing more than batch-8 is not a bug — it is ceil(40/16) = 3 batches holding 48 slots for 40 chunks.
6. Real-world example
A team ran a RAG assistant whose bill grew from manageable to alarming over a quarter while traffic grew about 40%. They spent two weeks optimising the synthesis prompt: trimming few-shot examples, compressing the system prompt, capping the answer length. The bill moved about 6%.
The instrumentation they added to figure out why took an afternoon and answered it immediately. Synthesis was under a quarter of spend. Reranking was most of the rest, and nobody had counted its calls because it had been added as "a small scoring step" months earlier and had no dashboard.
Two changes followed. The reranker moved to a small, fast model — a config line — and candidate_k dropped from 50 to 20 after the sweep in Reranking: The Second Pass That Decides What the Model Sees showed quality flat past 20. Cost per query fell by roughly four fifths. Answer quality, measured on their existing labelled set, moved by less than a point.
The postmortem lesson was not about models. It was that the stage nobody owned was the stage that cost the most. Synthesis got attention because someone had written the prompt and felt responsible for it. Reranking was a library call with a default k, and defaults do not have owners. They added per-stage cost to their dashboard, and the next surprise showed up in days rather than a quarter.
7. Interview questions companies actually ask
Q1 [easy] "Which stage of a RAG pipeline usually costs the most?"
A Not generation -- the fan-out scoring step, if you rerank with an LLM. It runs
once per candidate, so ~40 small calls against 1 large one. Measured: 78% of
spend on reranking vs 22% on synthesis. Count CALLS before you count tokens;
surprising bills almost always come from a stage whose call count is a multiple
of something.
Q2 [easy] "What's per-role model routing?"
A Assign a model per ROLE rather than per application. Router, rewriter, and
reranker need cheap and fast; the synthesiser needs strong because it writes
what the user reads. Scoring a passage 1-10 against a rubric is much easier
than writing a grounded cited answer, so the highest-volume role is also the
least quality-sensitive -- that coincidence is the whole opportunity.
Q3 [medium] "How far can you downgrade the reranker?"
A Until the sign flips. A reranker weaker than the retriever it's correcting makes
ranking actively worse, not merely no better. So downgrade a tier, measure nDCG,
repeat, and stop one tier above where it degrades. The cheapest model above that
line is correct. Measured saving from premium to economy on that role alone: 78%.
Q4 [medium] "Does batching the rerank calls help?"
A Some, and less than you'd think, because it amortises the fixed PROMPT and not
the chunk tokens -- and chunks are the bulk. Savings flatten fast. It's also
non-monotonic: batch-16 cost more than batch-8 here, since ceil(40/16)=3 batches
hold 48 slots for 40 chunks. Choose a batch size that divides candidate_k.
Q5 [medium] "What's the quality cost of batching?"
A Chunks in one context get scored relative to EACH OTHER rather than against the
rubric, so scores stop being comparable across batches. Any absolute threshold
you set silently breaks. Batch for relative ordering; don't batch when you need
calibrated scores.
Q6 [medium] "Your provider caps you at 500k tokens per minute. What throughput do
you get?"
A Divide. This pipeline uses 56,430 tokens per query end to end, so 500k TPM is
8.9 queries per MINUTE -- total, regardless of budget or concurrency. It's a
hard ceiling: if traffic exceeds it you must cut tokens per query (smaller
candidate_k, batching, shorter rerank prompt) or get more quota.
Q7 [hard] "Rate limits or price -- which binds first?"
A Usually rate limits, and teams plan for price. Note a single query can't exceed
the per-minute budget, so if one query's 56k tokens approaches your TPM cap no
concurrency control helps -- you must shrink the query. Throttle pre-emptively
against your own token accounting rather than firing and handling 429s; backoff
is the safety net for what your accounting missed, not the primary mechanism.
Q8 [hard] "Where does agentic RAG blow up the budget?"
A Multipliers compose. Fan out to 3 specialists that each rerank 20 candidates and
you've made 60 rerank calls, not 20. Neither knob looks dangerous alone -- width
3 is modest, k=20 is the measured knee -- but they multiply. Any stage whose call
count is a product of two independently-tuned parameters needs a hard ceiling,
not just sensible defaults on each.
8. When to use / tradeoffs
OPTIMISE COST WHEN:
+ the bill is a real line item, or you're throughput-constrained
+ you have per-stage instrumentation (otherwise you're guessing)
+ you have a quality measure to check against after each change
ORDER OF OPERATIONS (highest leverage first):
1. count calls per stage -- find the multiplier
2. per-role model routing -- cheap model on the high-volume role
3. cut candidate_k to the measured knee
4. cache: rewrites, retrievals, per-(query,chunk) scores
5. batch the fan-out (b must divide k)
6. trim prompts <- where most teams start, and it's 6th
| Situation | Why it breaks | Use instead |
|---|---|---|
| Optimising the synthesis prompt first | It's ~22% of spend; you're polishing the minority | Instrument per stage; find the multiplier |
| One model for the whole app | Overpays on 40 scoring calls to protect 1 writing call | Per-role assignment |
| Downgrading the reranker too far | Sign flip: worse than no reranking at all | Measure nDCG per tier; stop above the flip |
Batch size not dividing k | Padding costs more — batch-16 > batch-8 | Choose b that divides k |
| Batching with absolute thresholds | Scores become batch-relative, threshold breaks | Don't batch when scores must be calibrated |
| Planning for price, not rate limits | TPM caps throughput regardless of budget | Compute tokens/query ÷ TPM before launch |
| Reactive 429 handling only | Wastes calls, adds tail latency | Pre-emptive throttle + backoff as safety net |
| Agentic fan-out × rerank depth | Multipliers compose: 3 × 20 = 60 calls | Hard ceiling on total calls per query |
Honest limits. The prices here are illustrative tiers I chose, not any provider's list, and real pricing changes often — the ratios between tiers (roughly 100× premium-to-economy on input here) drive every conclusion, and if your provider's spread is 10× rather than 100× the savings shrink proportionally. Rerun the arithmetic with your own numbers; that is the point of the code being twenty lines of accounting. The stage profile (40 rerank calls, 12k synthesis prompt) describes a pipeline that reranks with an LLM at k=40; a system using a hosted cross-encoder or no reranking at all has a completely different profile, and for those the synthesis call genuinely is the largest line — the headline result here is conditional on the architecture, not universal. Most importantly, this article measures cost and not quality. Every lever trades one for the other, and the claim "quality moved less than a point" in §6 is a plausible narrative, not a measurement from this harness. The quality side of the reranker downgrade is quantified in Reranking: The Second Pass That Decides What the Model Sees; do that measurement before acting on this one. Finally, the throughput table ignores burst allowances, per-model separate quotas, and the fact that input and output tokens are often capped independently.
9. Summary + related articles
- Count calls, not tokens. The stage that runs 40 times owns the bill: 78% on reranking against 22% on synthesis, in a pipeline where synthesis has the biggest prompt.
- Per-role model routing is the top lever. Economy reranker + premium synthesis: $1.148 → $0.256/query, a 78% cut, $89,172/month at 100k queries — one config line, user-visible output unchanged.
- The high-volume role is the least quality-sensitive. Scoring 1-10 against a rubric is easier than writing a cited answer. Downgrade until nDCG degrades, then step back one tier.
- Batching disappoints and is non-monotonic. It amortises only the fixed prompt; batch-16 cost 15% more than batch-8 because ⌈40/16⌉ pads 48 slots for 40 chunks. Pick
bdividingk. - TPM caps throughput independently of budget — 56,430 tokens/query means 8.9 queries/min at 500k TPM. Compute this before launch.
- Multipliers compose. Agentic width × rerank depth = 3 × 20 = 60 calls. Put a ceiling on the product, not just on each factor.
- Boundary: every lever here trades cost against quality. Measure both, or you have deferred a problem rather than solved one.
Related:
- Reranking: The Second Pass That Decides What the Model Sees — the stage that owns the bill: §4.2 has the
kknee that sets call count, §5 the sign flip that sets how far you can downgrade - Agentic RAG: Routing Retrieval to Specialists — fan-out width, the other half of the multiplier that composes with rerank depth
- API Best Practices — backoff, jitter, and retry mechanics; the safety net behind pre-emptive throttling
- Multi-Index RAG: Merging Several Retrievers Into One Answer — every extra source is another pipeline on the bill
- Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems — chunk size sets the per-call token count that all of this arithmetic multiplies
Resources
- Dean & Barroso, "The Tail at Scale", Communications of the ACM 56(2), 2013 — why the slowest branch of a fan-out sets user-visible latency; the latency counterpart to this article's cost argument.
- Chen, Zaharia & Zou, "FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance", arXiv:2305.05176, 2023 — LLM cascades and per-role model selection, the published treatment of §3.2.
- Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", SOSP 2023 (vLLM) — where batching economics come from on the serving side.
- Zheng et al., "SGLang: Efficient Execution of Structured Language Model Programs", arXiv:2312.07104 — prefix caching across many similar calls, directly applicable to a fan-out scoring step.
- Anthropic, "Prompt caching" — https://docs.claude.com/en/docs/build-with-claude/prompt-caching (caching the shared prefix of a repeated scoring prompt).