ML Inference Systems

TL;DR — Training produces a model once; inference runs it millions of times a day, and that is where the real engineering lives. This article is about serving ML at scale: batch vs online (real-time) inference, model servers (REST/gRPC, TorchServe/Triton/KServe/SageMaker), the latency-vs-throughput tension, dynamic batching, caching, GPU utilisation, autoscaling, a model registry with versioning/rollback, and safe rollouts (canary/shadow/A-B) plus monitoring for latency, drift, and data quality. The one number to remember: optimise p99 latency at a target throughput within a cost budget — not the average.

---

1. Simple explanation

An inference system is the part of an ML product that takes a live request ("here's a user + a candidate item, score it") and returns a prediction fast enough and cheap enough to matter. Training is a batch job you run occasionally on a fixed dataset; inference is a 24/7 online service with SLAs, traffic spikes, and a pager.

Analogy — a restaurant kitchen. Training is writing and testing the recipes. Inference is the dinner rush: orders (requests) stream in, the chef (GPU) is the expensive bottleneck, and you win by batching similar orders, pre-plating popular dishes (caching), and hiring more cooks when the line gets long (autoscaling) — all while keeping the slowest customer's wait (p99) acceptable, not just the average.

---

2. The landscape (diagram)

                         CLIENT (app / another service)
                                    │  request
                                    ▼
        ┌──────────────── API GATEWAY / LOAD BALANCER ───────────────┐
        │        auth · rate-limit · routing · canary split          │
        └───────────────────────────┬───────────────────────────────┘
                                     ▼
   ┌──────────── MODEL SERVER (always-on container, REST/gRPC) ───────────┐
   │                                                                      │
   │   [1] prediction CACHE  ──hit?──►  return cached score               │
   │            │ miss                                                    │
   │   [2] FEATURE fetch  ◄────────  Feature Store (online: Redis/DynamoDB)│
   │            ▼                                                          │
   │   [3] DYNAMIC BATCHER  → collect requests up to Bmax or T ms         │
   │            ▼                                                          │
   │   [4] MODEL (GPU/CPU)  → forward pass on the batch                   │
   │            ▼                                                          │
   │   [5] post-process → cache → respond                                 │
   └──────────────────────────────────────────────────────────────────────┘
        ▲                    ▲                      ▲
        │                    │                      │
   MODEL REGISTRY      AUTOSCALER (HPA)       MONITORING
   versions·rollback   scale on QPS/GPU%      latency·drift·quality  →  ALERT

Two request styles sit on top of this:

   BATCH (offline)     nightly/hourly job → score millions of rows → write to a table/warehouse
   ONLINE (real-time)  one request → one (or a few) predictions → return in ms, under an SLA

---

3. Batch vs Online (real-time) inference

This is the first fork in any inference design and a guaranteed interview question.

Rule of thumb: if the prediction can be precomputed before the user needs it, batch it (cheaper, simpler). If it depends on live context (this session, this transaction, this query), you need online.

Hybrid is the common production answer. Precompute the heavy part offline (candidate embeddings, user features), then do a light online step at request time. Recommenders do exactly this: batch-generate candidates, then rank online. A group trip-recommendation app precomputes destination preference-vector embeddings into a vector index (batch) and only runs the cheap cosine + group-aggregation online.

---

4. Model serving: how a request actually gets a prediction

4.1 REST vs gRPC

   REST/JSON   → human-readable, easy to debug, ubiquitous. Slower serialisation.
   gRPC/proto  → binary, HTTP/2, streaming, ~2–5× lower overhead. Standard for
                 service-to-service, high-QPS, and Triton/TF-Serving internals.

External/public endpoint → REST is fine. Internal microservice mesh at high QPS → gRPC.

4.2 Serving frameworks (know the tradeoffs, not just the names)

Key mental model: Triton/TorchServe/TF-Serving serve a model; KServe orchestrates servers (it doesn't run the forward pass itself — it wraps Triton/TorchServe/etc. with K8s autoscaling, canary, and a standard inference protocol).

4.3 The request pipeline (numbered in the diagram)

1. Cache check — has this exact input been scored recently?
2. Feature fetch — pull online features from a feature store (see §8).
3. Batching — accumulate concurrent requests into one tensor (see §6).
4. Model forward pass — CPU or GPU.
5. Post-process, cache, respond.

---

5. Latency vs throughput — the core math

These two are in tension, and articulating the tradeoff is what separates a mid from a senior answer.

   Latency    = time for ONE request to complete           (want LOW)     unit: ms
   Throughput = requests completed per second (QPS/RPS)     (want HIGH)    unit: req/s

5.1 Percentiles, not averages

Never report a mean latency. Tail latency is what users feel and what SLAs are written on.

   p50 (median) = half of requests are faster than this
   p95          = 95% are faster; 1 in 20 is slower
   p99          = 99% are faster; the "tail" your worst users hit

Why the tail matters: a page that fans out to 100 backend calls will, on average, hit a p99-slow response on ~63% of page loads (1 − 0.99¹⁰⁰). Tail latency dominates at fan-out. Target and alert on p99, budget for p50.

5.2 Little's Law (the one formula interviewers love)

   L = λ · W

   L = average number of requests in the system (concurrency / in-flight)
   λ = arrival rate (throughput, req/s)
   W = average time a request spends in the system (latency, s)

Read it as: the number of requests "in flight" equals how fast they arrive times how long each one stays. It's an identity (always true in steady state), and it's the sizing tool:

• Capacity planning: if each request takes W = 50 ms and one replica handles L = 8 concurrent requests, one replica sustains λ = L/W = 8 / 0.05 = 160 req/s. Need 3,200 req/s? → 3200 / 160 = 20 replicas.
• The trap: pushing throughput (bigger batches, more concurrency) raises L, which — once the server saturates — raises W (latency). You cannot maximise both past the knee of the curve.

5.3 The utilisation → latency wall

   As utilisation ρ → 1, queueing delay blows up:   W_queue ∝ ρ / (1 − ρ)

Run a latency-sensitive service at ~100% GPU and the queue explodes — p99 goes vertical. That's why autoscalers target ~60–70% utilisation, not 95%: you're buying headroom for bursts and to keep the tail flat.

---

6. Dynamic batching — the highest-leverage trick on GPUs

A GPU processing 1 request vs 32 requests takes almost the same wall-clock time (the matrix multiply is bandwidth-bound). So batching multiplies throughput almost for free — at the cost of a little latency (each request waits for the batch to fill).

   Requests arriving:  r1 . r2 . . r3 r4 . . . r5   →  wait up to  T_max ms  OR  B_max items
                        └─────── collect into one batch ───────┘
                                        ▼
                                 single GPU forward pass
                                        ▼
                        split results → reply to r1..r5

   The batching tradeoff:
     bigger batch  → higher THROUGHPUT, better GPU utilisation, HIGHER tail latency
     smaller batch → lower latency, worse GPU utilisation
   Two knobs: max_batch_size (B_max) and max queue delay (T_max, e.g. 5–20 ms).

Latency accounting for one request: total = queue_wait + batch_forward_pass. You trade up to T_max of queue wait for a big throughput win. Triton/TorchServe/vLLM do this server-side, invisibly to the client. For LLMs, continuous (in-flight) batching goes further — new sequences join the batch as others finish generating, keeping the GPU full token-by-token.

---

7. Caching

Cheapest latency is the request you never compute.

   PREDICTION cache   key = hash(features)      → skip the model entirely on repeats
                      great when inputs repeat (popular query, same user+item)
   FEATURE cache      precomputed features in Redis/DynamoDB → skip recomputation
   EMBEDDING cache    store item/user vectors   → reuse across requests
   SEMANTIC cache     (LLM/RAG) cache by embedding-similarity of the prompt, not exact match

   Invalidation: TTL (time-based) · event-based (model version bump, feature update).
   Watch the classic bug: caching a prediction across a MODEL VERSION change → stale scores.
   Always include the model version in the cache key.

A RAG/LLM system leans on semantic caching ("what are DIEP flap risks?" ≈ "risks of DIEP flap reconstruction?") to dodge repeat LLM calls — a large cost win because LLM tokens are the expensive part.

---

8. Feature stores & training-serving skew

   OFFLINE store (training)   warehouse/parquet — big historical features for training
   ONLINE store (serving)     Redis/DynamoDB    — low-latency lookup at request time
                    ▲
                    └── SAME transformation code feeds both  (this is the whole point)

Training-serving skew is the #1 silent ML-serving bug: features computed one way in the training pipeline and a slightly different way at serving time. The model was trained on a distribution it never sees in production, and accuracy quietly rots. A feature store exists largely to guarantee one definition, two access paths, plus point-in-time correctness (no label leakage from the future during training).

---

9. GPU utilisation & autoscaling

Getting value from expensive GPUs:

   Dynamic batching          keep the GPU fed (§6)
   Concurrent model instances multiple copies per GPU to overlap compute + I/O
   Quantisation (INT8/FP8)   smaller, faster, more throughput per GPU
   Multi-model serving       pack several models onto one GPU (Triton) to raise avg util
   Right-size hardware        don't put a 2-layer model on an A100

Autoscaling (Kubernetes HPA / KServe / SageMaker):

   scale on:  QPS · GPU utilisation · queue depth · p99 latency  (not just CPU%)
   target ~60–70% util → headroom for bursts & flat tail (see §5.3)
   scale-to-zero for spiky/rare models (KServe) — but beware COLD START (model load)

Cold start is the reason ML serving usually uses always-on containers, not serverless (Lambda):

   Why NOT Lambda / plain serverless for most ML inference:
     ✗ COLD START   loading a multi-GB model + CUDA context = seconds → blows the SLA
     ✗ NO GPU       most FaaS has no/limited GPU; DL inference needs it
     ✗ TIMEOUTS     hard execution limits (e.g. 15 min) kill long generations
     ✗ SIZE LIMITS  deployment package / memory caps vs large model weights
   Why always-on CONTAINERS win:
     ✓ model stays warm in memory (load once, serve millions)
     ✓ GPU attached, dynamic batching across concurrent requests
     ✓ autoscale replicas up/down, keep a warm floor (min replicas ≥ 1)
   Serverless IS fine for: tiny CPU models, spiky low-traffic, or a thin pre/post-processing layer.

---

10. Model registry, versioning, rollback

   MODEL REGISTRY  (MLflow / SageMaker Model Registry / Vertex)
     model_name : fraud-ranker
       v3  stage=Production   metrics{auc:0.94}  artifact=s3://…  ← serving now
       v4  stage=Staging      metrics{auc:0.95}  artifact=s3://…  ← candidate
     each version pins: weights + code + training data hash + metrics + who/when

• Versioning every model (weights and the preprocessing code) makes deploys reproducible and lets serving load a specific version.
• Rollback = repoint the "Production" alias to the last-good version. Because artifacts are immutable and versioned, rollback is instant and safe — no rebuild.
• Reproducibility needs the triple: code + data snapshot + config, not just weights.

---

11. Safe rollouts: shadow, canary, A/B

Never flip 100% of traffic to a new model. Three escalating techniques:

   SHADOW (dark launch)
     copy each request → send to v4 too, but DISCARD its response (serve v3 to the user)
     purpose: measure v4's latency & predictions on real traffic with ZERO user risk

   CANARY
     route a SMALL % (1→5→25→100) of live traffic to v4, SERVE its responses, watch metrics
     purpose: limited-blast-radius real test; auto-rollback if error/latency/business metric dips

   A/B TEST
     split users into buckets, serve v3 vs v4, compare a BUSINESS metric (CTR, conversion)
     with statistical significance → decide which model actually wins on outcomes

   shadow → canary → A/B → full rollout      (increasing user exposure, increasing evidence)

Shadow vs canary — the exact distinction interviewers probe: in shadow, the new model's output is used only for monitoring (users never see it); in canary, the new model's output is actually served to the canary slice. Shadow tests safety/latency; canary and A/B test quality/business impact.

---

12. Monitoring: latency, drift, data quality

A deployed model degrades silently — no exception is thrown when it just gets wrong. Monitor three layers:

   [1] SYSTEM / SLA        p50/p95/p99 latency · QPS · error rate · GPU% · queue depth · cost
   [2] DATA QUALITY        nulls/missing features · schema changes · out-of-range values
                           · feature freshness (stale online features) · training-serving skew
   [3] MODEL QUALITY
        DATA DRIFT     input distribution shifts (P(x) changes)   → PSI, KL-divergence, KS-test
        CONCEPT DRIFT  x→y relationship changes (P(y|x) changes)  → the world moved on
        PREDICTION DRIFT output distribution shifts               → early warning w/o labels
        PERFORMANCE    accuracy/AUC/precision-recall once labels arrive (may be delayed)

Data drift vs concept drift (memorise): data drift = the inputs changed distribution (new user demographics); concept drift = the relationship changed (fraud tactics evolve, so the same features now mean something different). Both silently drop accuracy; drift alarms trigger retraining. Labels are often delayed (you learn a transaction was fraud weeks later), so lean on prediction drift and input drift as early proxies. Tools: Evidently, Prometheus/Grafana, cloud model monitors.

---

13. Real code — a minimal, faithful online server with batching + cache

# FastAPI online inference server: prediction cache + micro-batching + versioned model.
# Mirrors the shape of a typical LLM/RAG FastAPI service (always-on container).
import asyncio, hashlib, time
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()
MODEL_VERSION = "v4"                       # part of every cache key — avoids stale scores
_cache: dict[str, float] = {}              # prediction cache (use Redis in prod)

class Req(BaseModel):
    features: list[float]

def _key(features): 
    raw = f"{MODEL_VERSION}:{features}"
    return hashlib.sha1(raw.encode()).hexdigest()

# ---- the model (swap for Torch/Triton client). Batched forward pass. ----
def model_forward(batch: list[list[float]]) -> list[float]:
    # one GPU call for the WHOLE batch — near-constant cost vs a single item
    return [sum(f) / (len(f) or 1) for f in batch]     # stand-in for a real net

# ---- dynamic micro-batcher: collect up to B_MAX items or wait T_MAX ms ----
B_MAX, T_MAX = 32, 0.01
_queue: list[tuple[list[float], asyncio.Future]] = []
_lock = asyncio.Lock()

async def _batch_worker():
    while True:
        await asyncio.sleep(T_MAX)                      # T_MAX window (queue wait)
        async with _lock:
            if not _queue: 
                continue
            batch, futures = zip(*_queue[:B_MAX])
            del _queue[:B_MAX]
        for fut, score in zip(futures, model_forward(list(batch))):
            fut.set_result(score)                       # fan results back to callers

@app.on_event("startup")
async def _start():
    asyncio.create_task(_batch_worker())                # warm, always-on background loop

@app.post("/predict")
async def predict(req: Req):
    t0 = time.perf_counter()
    k = _key(req.features)
    if k in _cache:                                     # [1] cache hit → skip the model
        return {"score": _cache[k], "cached": True, "version": MODEL_VERSION}
    fut = asyncio.get_event_loop().create_future()
    async with _lock:
        _queue.append((req.features, fut))              # [3] join the dynamic batch
    score = await fut                                   # [4] awaited batched forward pass
    _cache[k] = score
    return {"score": score, "cached": False, "version": MODEL_VERSION,
            "latency_ms": round((time.perf_counter() - t0) * 1000, 2)}

The load-bearing ideas: model loaded once at startup (warm), version in the cache key, micro-batching across concurrent requests, and latency measured per request so p50/p99 can be scraped by monitoring.

---

14. Real-world example — LLM/RAG inference on containers

Two representative LLM/RAG systems built as FastAPI + Uvicorn services in always-on containers on AWS illustrate why ML serving looks the way it does:

   A GROUP TRIP-RECOMMENDATION APP  (FastAPI)
     startup: load sentence-transformers embedding model + Groq LLM client ONCE (warm)
     online path per request:
        embed query → ANN vector search (candidates) → cosine satisfaction
        → group-aggregation selector → optional Groq LLM call for the chat reply
     batch/offline: destination preference-vector embeddings precomputed into the index
     why not Lambda: the embedding model + warm LLM client can't pay cold-start per request

   A CLINICAL RAG SYSTEM  (FastAPI + SSE streaming)
     multi-agent RAG: structure question (PICOT) → a supervisor routes to specialist retrievers
        → Pinecone retrieval per department index → rank evidence → LLM synthesises cited answer
     serving traits: STREAMING responses (SSE) → long generations blow serverless timeouts
        → always-on container; per-department vector indexes = model/feature versioning by dataset

Why ML uses always-on containers, not Lambda, restated with these systems: the embedding model and LLM clients are loaded once at startup and kept warm (cold start would add seconds to every call); RAG answers stream token-by-token and can run long (serverless timeouts would cut them off); and retrieval needs low-latency vector DB connections held open. Serverless would pay model-load and connection cost on every invocation — fatal to the SLA. Containers load once and serve millions.

---

15. Interview questions companies actually ask

Q [easy]   "Batch vs online inference — when do you use each?"
  A Batch when predictions can be precomputed before they're needed (nightly recs, churn
    scores) — cheap, high-throughput, simple. Online when the prediction needs live context
    (fraud at checkout, search ranking, chat) — ms-latency SLA, always-on, autoscaled.
    Most real systems are HYBRID: precompute the heavy part, do a light step online.

Q [easy]   "Why measure p99 latency, not average?"
  A The average hides the tail. Users feel the slow requests, and with fan-out (a page
    calling many services) the odds of hitting a p99-slow response per page are high
    (1 − 0.99^N). SLAs are written on p95/p99; you budget on p50 but alert on p99.

Q [medium] "Explain dynamic batching and its tradeoff."
  A Accumulate concurrent requests into one tensor and do a single GPU forward pass, since
    a GPU costs almost the same for 1 vs 32 items. Bigger batch → higher throughput & GPU
    utilisation but higher tail latency (requests wait for the batch to fill). Tune with
    max_batch_size and a max queue delay (T_max). LLMs use continuous/in-flight batching.

Q [medium] "Use Little's Law to size a service."
  A L = λ·W. If each request takes W=50ms and a replica holds L=8 in flight, one replica
    does λ=L/W=160 req/s; for 3,200 req/s you need ~20 replicas. Pushing throughput raises
    L, which past saturation raises W — you can't max both, so autoscale at ~60–70% util.

Q [medium] "Why not serve ML models on AWS Lambda / serverless?"
  A Cold start (loading a multi-GB model + CUDA context = seconds → blows the SLA), no/limited
    GPU, hard timeouts that kill long generations, and package/memory limits. Always-on
    containers load the model once (warm), attach a GPU, batch across requests, and autoscale
    with a warm floor. Serverless is fine only for tiny CPU models or thin pre/post-processing.

Q [medium] "Shadow vs canary vs A/B — differences?"
  A Shadow: copy traffic to the new model but DISCARD its output (users see the old model) —
    tests latency/behaviour with zero risk. Canary: serve the new model to a small % of real
    traffic, watch metrics, auto-rollback on regression. A/B: bucket users, compare a BUSINESS
    metric with statistical significance. Order: shadow → canary → A/B → full rollout.

Q [hard]   "How do you monitor a model in production and know it's degrading?"
  A Three layers: (1) system/SLA (p99 latency, QPS, errors, GPU%, cost); (2) data quality
    (nulls, schema drift, feature freshness, training-serving skew); (3) model quality —
    data drift (PSI/KL/KS on inputs), concept drift (x→y changed), prediction drift as an
    early label-free signal, and accuracy/AUC once (often delayed) labels arrive. Drift alarms
    → retrain. No exception fires when a model is merely wrong, so you must instrument it.

Q [hard]   "What is training-serving skew and how do you prevent it?"
  A Features computed differently in training vs serving, so the model sees a distribution it
    never trained on and silently loses accuracy. Prevent with a FEATURE STORE that shares one
    transformation definition across offline (training) and online (serving) stores, plus
    point-in-time-correct joins to avoid label leakage. Log serving features and compare.

Q [hard]   "How do you get high GPU utilisation without wrecking latency?"
  A Dynamic batching, multiple concurrent model instances per GPU, quantisation (INT8/FP8),
    and multi-model packing (Triton). But cap utilisation at ~60–70%: queueing delay scales
    as ρ/(1−ρ), so running near 100% sends p99 vertical. Autoscale on QPS/queue-depth/p99, not CPU%.

Q [medium] "How do you version and roll back models safely?"
  A A model registry pins each version's weights + preprocessing code + data hash + metrics,
    with immutable artifacts and a Production alias. Deploy = repoint the alias (via canary);
    rollback = repoint to the last-good version — instant, no rebuild. Include the model
    version in every cache key so rollout/rollback doesn't serve stale cached predictions.

Sources: Meta ML System Design Guide (2026) · Serving ML Models at Scale (Sealos) · Model Serving with NVIDIA Triton (AIOZ) · vLLM vs Triton vs KServe (Kubenatives) · ML Model Deployment Strategies (TDS) · Canary & Shadow Testing (apxml) · Model Drift (Aerospike)

---

16. When to use / tradeoffs

   Batch inference     → precomputable predictions; cheapest per prediction; freshness lag OK.
   Online inference    → live context needed; pay for always-on + autoscale; guard the tail.
   FastAPI + model     → custom logic, LLM/RAG glue, small/medium scale.
   Triton/TorchServe   → squeeze max GPU throughput, multi-model, server-side batching.
   KServe/SageMaker    → want autoscaling + canary + monitoring handed to you on K8s/cloud.
   Serverless (Lambda) → ONLY tiny CPU models / spiky low traffic / thin pre-post steps.
   Dynamic batching    → GPU-bound & bursty traffic; not worth it for ultra-low-latency single calls.
   Caching             → repeated inputs (popular queries/users) or expensive LLM calls; watch invalidation.
   Shadow/canary/A-B   → always, for any model change touching users.

---

17. Summary + related articles

• Inference ≠ training: it's a 24/7 online service with SLAs, cost budgets, and drift.
• Batch vs online is the first fork; hybrid (precompute heavy, light online step) is the usual answer.
• Serve via REST/gRPC on TorchServe/Triton/TF-Serving, orchestrated by KServe/SageMaker.
• Latency vs throughput: report p50/p95/p99, size with Little's Law (L = λW), autoscale at ~60–70% util because queueing scales as ρ/(1−ρ).
• Dynamic batching is the top GPU throughput lever; caching (prediction/feature/semantic) is the cheapest latency.
• Registry + versioning make rollback instant; ship via shadow → canary → A/B.
• Monitor SLA, data quality, and data/concept/prediction drift — models fail silently.
• ML uses always-on containers, not Lambda (cold start, GPU, timeouts, size).

Related: Recommendation Systems · Feed Ranking · Search Systems · Fraud Detection · Common ML System Design Interview Questions · (scalability: ../11-1/scalability-basics.md) · (distributed systems: ../11-1/distributed-systems.md) · (API design: ../11-1/api-design.md)

Resources

• Serving ML at scale (inference optimisation) — https://sealos.io/blog/serving-machine-learning-models-at-scale-a-guide-to-inference-optimization/
• NVIDIA Triton Inference Server — https://ai.aioz.io/guides/ml-ops/model-serving-triton/
• vLLM vs Triton vs KServe — https://www.kubenatives.com/p/vllm-vs-triton-vs-kserve-kubernetes
• ML deployment strategies (canary/shadow/A-B) — https://medium.com/data-science/ml-model-deployment-strategies-72044b3c1410
• Model drift & monitoring — https://aerospike.com/blog/model-drift-machine-learning/
• Feature stores — https://www.featurestore.org/