TL;DR — Cost and latency are not a single dial: batching, caching, and async processing each move a different one, and confusing them is how teams optimise the wrong thing. Batch tiers are typically ~50% cheaper and structurally cannot serve an interactive request. Caching cuts cost linearly with hit rate but barely touches tail latency, because p95 is set by the misses — a 40% hit rate still shows a 1.6 s p95 against a 2.6 s origin. So the honest framing is a feasibility question first (which modes can meet the SLA at all?) and an optimisation question second (which is cheapest among those?). It stops working as a framework when your SLA is stated as an average, because an average SLA is unenforceable and every technique here will appear to satisfy it while users suffer.
1. Simple explanation
Every serving decision trades money against waiting. But there are three genuinely different ways to trade, and they are not interchangeable.
Batching means collecting work and processing it later in bulk. Providers charge substantially less for this because they can schedule it when capacity is free. The cost is that the answer arrives hours later, so it is only available for work nobody is waiting on.
Caching means remembering answers you have already computed. Repeats become free and instant. The catch is that only the repeats improve; anything new pays full price and full latency.
Async processing means accepting the request, returning immediately, and delivering the result later. This does not make anything cheaper or faster — it changes who waits, which is often exactly what you need.
The mistake is treating these as three sizes of the same lever. They are three different levers, and only one of them helps with a given problem.
Analogy — a print shop. Overnight jobs are cheapest because the shop runs them when the machines are idle (batching). Reprinting a poster they already have on file is instant and nearly free (caching). And handing you a collection ticket instead of making you stand at the counter does not speed up the printing at all — it just means you can go and do something else (async). A customer who needs one poster in the next five minutes is not helped by the overnight discount, no matter how good it is.
2. Diagram
THREE LEVERS, THREE DIFFERENT EFFECTS
cost p95 latency who waits
batching ↓↓ ↑↑↑↑ nobody (offline work)
caching ↓ (linear ↓ (only caller
in hit on hits)
rate)
async — — a background worker
(caller is released)
FEASIBILITY BEFORE OPTIMISATION
SLA p95 = 1 s SLA p95 = 4 h
┌──────────────────┐ ┌──────────────────────────────┐
│ real-time+cache75│ │ batch $5.00 /1k ◀── cheapest
│ $2.50 /1k │ │ real-time+c75 $2.50 /1k │
└──────────────────┘ │ real-time+c40 $6.00 /1k │
batch: IMPOSSIBLE │ real-time $10.00 /1k │
cache40: too slow └──────────────────────────────┘
all feasible -> now optimise
WHY CACHING DOES NOT FIX THE TAIL
origin p95 = 2600 ms, hit = 5 ms
hit rate cost/1k p95
0% $10.00 2600 ms
40% $ 6.00 1560 ms <- cost -40%, p95 only -40%
75% $ 2.50 653 ms
99% $ 0.10 31 ms
the MISSES set the tail. a cache is a cost lever that
happens to help the median, not a latency lever.
3. How it works
3.1 Batch versus real-time is a feasibility question, not a preference
A batch tier is not "slower real-time." It is a different service with a different contract: submit now, collect within a window measured in hours. The discount is real and typically around 50%, but no amount of discount makes it eligible for a request a user is waiting on.
So the first question is never "which is cheaper." It is which modes can meet the SLA at all. Once you have that shortlist, cost sorts it. Reversing the order is how a team ends up architecting around a batch discount they cannot use.
Work that genuinely batches: overnight enrichment, backfills, embedding a corpus, scoring a dataset, generating summaries nobody has asked for yet. Work that does not: anything with a human waiting.
3.2 Caching: what it actually moves
A read-through cache reduces cost in direct proportion to hit rate — a 40% hit rate is a 40% cost reduction, near enough. That is the strongest and most reliable cost lever available, because it removes the call entirely rather than making it cheaper.
Its effect on latency is where intuition goes wrong. Hits return in milliseconds, so the median improves dramatically at even a modest hit rate. But p95 and p99 are determined by the slowest requests, and the slowest requests are misses, which pay full origin latency. A cache does not make your tail fast; it makes fewer people experience the tail.
Two consequences. If your problem is cost, cache aggressively. If your problem is p95, caching helps far less than it appears to in a demo — you must make the origin faster.
Cache keys are where correctness lives. The key must include everything the answer depends on: the normalised request, the tenant or user scope if answers differ, and a version of the underlying knowledge, so that editing a source fact invalidates dependent entries automatically. Getting this wrong produces the worst class of bug — confidently serving a stale answer, with no error anywhere.
3.3 Async processing changes who waits, not how long
Accepting a request, returning a job id, and delivering the result via callback, poll, or push does not reduce cost or latency by a single millisecond. What it does is release the caller, which matters enormously for perceived experience and for capacity: a connection you are not holding open is a connection you can use for someone else.
The related and much more common mistake is the opposite — code that looks asynchronous but blocks. A synchronous call inside an async handler occupies the event loop for its full duration, so concurrent requests serialise behind it. This is covered with a runnable demonstration in Async Programming; the symptom worth memorising here is that p50 looks fine while p95 collapses under load.
3.4 Quality as the third axis
Cost and latency are easy to measure, so teams optimise them and let quality drift silently. Any decision in this article can be restated as a three-way trade, and the discipline is to name a price for quality before you start:
acceptable iff saving > quality_lost * value_per_correct_answer
If nobody will name value_per_correct_answer, the decision still gets made — implicitly, by whoever ships. A rough figure argued about openly beats a precise one nobody stated.
3.5 SLAs: state them as percentiles or do not state them
An SLA on the average is unenforceable and actively misleading. In a bimodal system — cheap cached path plus expensive escalation path — the mean can sit in a region where almost no individual request lands. Every technique in this article will appear to satisfy an average SLA while a meaningful minority of users have a bad time.
State latency SLAs at p95 or p99, name the measurement point (server-side, or including network?), and define what happens on breach. Then the feasibility table in §2 becomes computable rather than rhetorical.
3.6 Where this framework stops applying
It assumes each request is independent and the load is roughly stationary. Under bursty traffic, queueing dominates and latency becomes a function of utilisation rather than of your mode choice — past roughly 70–80% utilisation, waiting time rises sharply and no caching strategy compensates. It also assumes cost is per-call; with reserved or provisioned capacity the marginal call is free and the entire optimisation inverts toward maximising utilisation instead. And it says nothing about correctness under partial failure, which is usually a bigger operational risk than either cost or latency.
4. The math
4.1 The three levers, formally
batching: C_batch = C_realtime * (1 - discount)
L_batch = window (hours, not ms)
caching: C_cached = C_origin * (1 - h) h = hit rate
L_p95 ≈ L_origin * (1 - h) + L_hit * h
└─ still dominated by L_origin at modest h ─┘
async: C unchanged, L unchanged, caller_wait -> ~0
4.2 Cache break-even against a batch discount
When both are feasible — that is, for offline work — which is cheaper?
cache beats batch iff (1 - h) < (1 - discount)
iff h > discount
with a 50% batch discount, the cache must exceed a 50% hit rate
4.3 Worked example
Origin: $0.0100 per call, p95 2600 ms. Batch tier: 50% off, 4-hour window.
mode $/1k calls p95
real-time $ 10.00 2.6 s
batch $ 5.00 4.0 h
real-time + cache@40% $ 6.00 1.6 s
real-time + cache@75% $ 2.50 0.7 s
Now the feasibility question, per SLA:
SLA (p95) feasible modes
1 s real-time + cache@75%
3 s real-time, real-time + cache@40%, real-time + cache@75%
30 s real-time, real-time + cache@40%, real-time + cache@75%
4 h real-time, batch, real-time + cache@40%, real-time + cache@75%
Read the top row: at a 1-second p95 target, only one of four options qualifies, and the cheapest option overall (batch, $5.00) is not among them. Read the bottom row: once you can wait four hours, batch becomes eligible — but a 75% cache is still cheaper at $2.50 and 5500x faster, so the batch discount is only the right answer when your hit rate is low.
And the cautionary line: cache@40% has a p95 of 1.56 s, not 5 ms. Caching cut cost by 40% and the tail by 40%, not to nothing. The misses set the tail.
5. Real code
"""Batch vs real-time vs cached serving: cost, latency, and which SLAs are reachable."""
from dataclasses import dataclass
USD_PER_CALL = 0.0100
BATCH_DISCOUNT = 0.50 # typical published batch-tier discount
CACHE_HIT_MS = 5
REALTIME_P95_MS = 2600
BATCH_P95_MS = 4 * 60 * 60 * 1000 # a 4-hour window
@dataclass
class Mode:
name: str
usd_per_call: float
p95_ms: float
def cached(hit_rate: float, base: Mode) -> Mode:
"""A read-through cache: hits cost nothing and return immediately."""
return Mode(
f"{base.name} + cache@{hit_rate:.0%}",
base.usd_per_call * (1 - hit_rate),
base.p95_ms * (1 - hit_rate) + CACHE_HIT_MS * hit_rate,
)
REALTIME = Mode("real-time", USD_PER_CALL, REALTIME_P95_MS)
BATCH = Mode("batch", USD_PER_CALL * BATCH_DISCOUNT, BATCH_P95_MS)
modes = [REALTIME, BATCH, cached(0.40, REALTIME), cached(0.75, REALTIME)]
print(f"{'mode':<26} {'$/1k calls':>11} {'p95':>12}")
for m in modes:
p95 = f"{m.p95_ms/1000:.1f} s" if m.p95_ms < 60_000 else f"{m.p95_ms/3.6e6:.1f} h"
print(f"{m.name:<26} ${m.usd_per_call*1000:>10.2f} {p95:>12}")
# Which SLA can each mode actually meet?
print(f"\n{'SLA (p95)':>12} feasible modes")
for sla_ms in (1000, 3000, 30_000, BATCH_P95_MS):
ok = [m.name for m in modes if m.p95_ms <= sla_ms]
label = f"{sla_ms/1000:.0f} s" if sla_ms < 60_000 else f"{sla_ms/3.6e6:.0f} h"
print(f"{label:>12} {', '.join(ok) if ok else '(none)'}")
# Cache break-even: how high must the hit rate be to beat the batch discount?
target = BATCH.usd_per_call
need = 1 - target / USD_PER_CALL
print(f"\nbatch tier is {1-BATCH_DISCOUNT:.0%} off, so a cache must exceed a "
f"{need:.0%} hit rate\nto be cheaper than batching -- while staying "
f"{BATCH.p95_ms/REALTIME.p95_ms:.0f}x faster.")
# A 40% cache does NOT reach a 1s SLA: 60% of traffic still pays full latency.
c40 = cached(0.40, REALTIME)
print(f"\nbut note: cache@40% has p95 {c40.p95_ms/1000:.2f}s, NOT 5 ms --")
print("p95 is set by the MISSES. Caching cuts cost long before it cuts tail latency.")
assert BATCH.usd_per_call * 1000 == 5.0
assert round(need, 2) == 0.50
assert c40.p95_ms > 1000 # a 40% hit rate misses a 1 s SLA
assert cached(0.75, REALTIME).p95_ms < 1000 # 75% is enough to reach it
assert BATCH.p95_ms > 30_000 # batch can never meet an online SLA
# Cost falls linearly with the hit rate, but so does p95 -- there is no point at
# which caching makes the tail fast, only points where it makes it acceptable.
assert round(cached(0.50, REALTIME).usd_per_call / USD_PER_CALL, 2) == 0.50
print("\nall assertions passed")
# Output:
# mode $/1k calls p95
# real-time $ 10.00 2.6 s
# batch $ 5.00 4.0 h
# real-time + cache@40% $ 6.00 1.6 s
# real-time + cache@75% $ 2.50 0.7 s
#
# SLA (p95) feasible modes
# 1 s real-time + cache@75%
# 3 s real-time, real-time + cache@40%, real-time + cache@75%
# 30 s real-time, real-time + cache@40%, real-time + cache@75%
# 4 h real-time, batch, real-time + cache@40%, real-time + cache@75%
#
# batch tier is 50% off, so a cache must exceed a 50% hit rate
# to be cheaper than batching -- while staying 5538x faster.
#
# but note: cache@40% has p95 1.56s, NOT 5 ms --
# p95 is set by the MISSES. Caching cuts cost long before it cuts tail latency.
#
# all assertions passed
The p95 model for caching is deliberately crude — it interpolates rather than computing a true percentile over a mixed distribution. It is right about the direction and the order of magnitude, which is what the decision needs; compute the real percentile from your own logs before quoting a number to anyone.
6. Real-world example
A team ran a nightly enrichment job over roughly two million records and a small interactive endpoint for on-demand lookups of the same kind. Both called the same model through the same client library.
They discovered the batch tier and its discount, and moved everything to it — including the interactive endpoint, on the reasoning that the code was shared and the discount applied to all of it. The nightly job got 50% cheaper, which was real and worth having. The interactive endpoint started returning results hours later, and because it had been built to return a job id already, nothing errored. Requests were accepted, jobs were queued, and results appeared eventually. Monitoring was green: no exceptions, no timeouts, throughput unchanged.
It took two weeks and a support ticket to notice, because there was no p95 latency SLA on that endpoint — only an availability target, which was being met perfectly. The endpoint was available. It was just useless.
The lesson is the ordering in §3.1. The team optimised before checking feasibility, and the absence of a percentile SLA meant nothing in their monitoring could express "this is now too slow to be worth anything." The eventual fix was trivial — route the interactive path to real-time, keep the nightly job on batch, add a p95 SLA with an alert — but the diagnosis was slow precisely because every dashboard said the system was healthy.
7. Interview questions companies actually ask
Q1. How do you decide between batch and real-time inference? It is a feasibility decision before it is a cost decision: ask which modes can meet the latency SLA at all, then choose the cheapest among the survivors. A batch tier is not slower real-time, it is a different contract with a window measured in hours, so it is eligible only for work nobody is waiting on. Getting the order backwards is how teams architect around a discount they cannot use.
Q2. How much does caching help latency? Much less than it helps cost, and the distinction matters. Cost falls roughly linearly with hit rate because you remove the call entirely. But p95 is set by the misses, which still pay full origin latency — a 40% hit rate against a 2.6-second origin still shows a p95 of about 1.6 seconds. Caching makes fewer users experience the tail; it does not make the tail shorter.
Q3. What goes in a cache key? Everything the answer depends on: the normalised request, any user or tenant scope where answers differ, and a version of the underlying knowledge or prompt so that editing a source fact invalidates dependent entries automatically. Omitting the version produces the worst failure class available — a confidently served stale answer with no error anywhere in the system.
Q4. Does async processing make things faster? No. It changes who waits. Cost and latency are unchanged; the caller is released, which improves perceived experience and frees connections for other work. The related trap is code that looks async but blocks — a synchronous call inside an async handler holds the event loop for its full duration, which serialises concurrent requests and shows up as a healthy p50 with a collapsing p95.
Q5. Why is an average-latency SLA a bad idea? Because it is unenforceable and hides bimodal behaviour. With a fast cached path and a slow escalation path, the mean can land in a range where almost no individual request actually falls, so the SLA is satisfied while a substantial minority of users have a bad experience. State latency at p95 or p99, name the measurement point, and define the consequence of a breach.
Q6. Where does this whole framework break down? Under bursty load, where queueing dominates and latency becomes a function of utilisation rather than mode choice — past roughly 70–80% utilisation, waiting time climbs steeply and no cache compensates. It also inverts under reserved or provisioned capacity, where the marginal call is free and the goal becomes maximising utilisation rather than minimising calls.
Q7. How do you keep quality from silently degrading while you optimise? Name a value per correct answer before you start, so every cost saving can be checked against the quality it costs. Then track quality per slice rather than in aggregate, because optimisation damage concentrates in whatever subset your new fast path handles worst, and an average will hide a severe regression in a small category.
8. When to use / tradeoffs
Reach for batching when:
- Nobody is waiting: backfills, enrichment, corpus embedding, dataset scoring
- Volume is high enough that ~50% is material
- The work can tolerate a multi-hour window and partial-failure retries
Reach for caching when:
- Requests repeat, even moderately — cost falls linearly with hit rate
- You can construct a key that includes a version of the underlying knowledge
- Cost is the problem you are trying to solve
Reach for async when:
- The work genuinely takes long enough that holding a connection is wasteful
- You want to free capacity rather than reduce work
- The client can handle a job id and a later result
| Situation | Why it breaks | Use instead |
|---|---|---|
| Interactive request on a batch tier | Window is hours; nothing errors, it is just useless | Real-time, with a p95 SLA |
| "Caching will fix our p95" | Misses set the tail | Make the origin faster; cache for cost |
| Cache key omits a knowledge version | Stale answers served confidently, no error | Version in the key |
| Average-latency SLA | Unenforceable, hides bimodality | p95/p99 with a named measurement point |
| Bursty traffic near saturation | Queueing dominates; mode choice is irrelevant | Capacity and admission control |
| Reserved capacity | Marginal call is free; the optimisation inverts | Maximise utilisation |
Honest limits. The latency arithmetic in §4 is an interpolation, not a percentile computation — mixing a 5 ms distribution with a 2600 ms one does not produce a p95 you can get by weighted averaging, and the true value depends on the shape of both. It is directionally right and numerically approximate; use it to choose what to measure, not as the measurement. The model also assumes independent requests at stationary load, ignores queueing entirely, and treats hit rate as a constant when in practice it varies by time of day and drops exactly when traffic patterns shift. The 50% batch discount and the cost figures are illustrative and change with vendor pricing. And the whole framing assumes cost is per-call: under provisioned capacity none of the conclusions hold.
9. Summary + related articles
- Batching, caching, and async are three different levers, not three sizes of one. Only one of them helps any given problem.
- Ask feasibility first — which modes can meet the SLA at all — then optimise cost among the survivors. Reversing this is the most common architectural error here.
- Batch tiers are ~50% cheaper and structurally ineligible for anything a user is waiting on.
- Caching cuts cost linearly with hit rate. It barely helps the tail, because p95 is set by the misses: 40% hit rate against a 2.6 s origin still gives ~1.6 s p95.
- A cache beats a batch discount only when hit rate exceeds the discount — above 50% for a 50% discount.
- Cache keys must include a version of the underlying knowledge, or you will serve stale answers with no error.
- Async changes who waits, not how long. Code that looks async but blocks gives you a fine p50 and a collapsing p95.
- State latency SLAs at p95/p99. An average SLA is unenforceable and hides bimodal pain.
- The framework breaks under bursty load near saturation, and inverts entirely under reserved capacity.
Related:
- Model Routing Patterns — choosing a cheaper tier per request, the other main cost lever
- RAG Cost Optimization: Find the Step That Runs Forty Times — §3.5 on not making the call at all, and the throughput ceiling
- LLM Observability — measuring the percentiles this article insists you state
- Async Programming — the blocking-in-async failure, with a runnable demonstration
- ML Inference Systems — serving architecture, batching mechanics, and caching layers
- Production Agents — caching break-even and bounded worst-case cost
- RAG at Scale — where these tradeoffs land in a retrieval system
Resources
- Little, J. D. C. (1961) — A Proof for the Queuing Formula: L = λW, Operations Research 9(3) — Little's Law, the relationship behind why utilisation governs latency once queueing dominates.
- Dean & Barroso (2013) — The Tail at Scale, Communications of the ACM 56(2) — the definitive treatment of why tail latency behaves differently from the mean, and why averages mislead: https://research.google/pubs/pub40801/
- Gregg, B. — Systems Performance (2nd ed.), chapters on methodology and latency analysis — how to measure percentiles honestly.
- Beyer et al. — Site Reliability Engineering, Ch. 4 "Service Level Objectives" — the standard reference for stating SLOs as percentiles: https://sre.google/sre-book/service-level-objectives/
- Python
asynciodocumentation — the async execution model referenced in §3.3: https://docs.python.org/3/library/asyncio.html - Provider batch-tier documentation is the authority on discounts and window guarantees; the ~50% figure used here is illustrative and should be re-checked before use in a decision.