LLM Observability

TL;DR — Log four things per call — route, tokens, cost, latency — and almost every LLM cost and performance question becomes answerable in an afternoon. Skip them and every optimisation is a guess. The two facts that make this urgent: the mean latency of a bimodal LLM system describes nobody (in the worked example the mean is 381 ms while p50 is 8 ms and p99 is 3.6 s), and a small slice of traffic usually owns most of the bill (9% of calls, 90% of cost). Neither is visible without a route label on each call. Drift detection is then just comparing a recent window against a baseline per route, because aggregate comparison hides a regression inside a minority path. It stops working when you log only aggregates or only failures — a degraded answer that returns HTTP 200 is invisible to every dashboard that watches error rates.

---

1. Simple explanation

Traditional service monitoring asks: is it up, is it fast, is it erroring? LLM systems break that model, because the most expensive failure returns a perfectly successful response containing a wrong answer, and the most expensive cost problem is a request that succeeded but took a path you did not intend.

So LLM observability adds two questions ordinary monitoring never asks. What did this call cost? — because unlike a database query, cost varies by more than an order of magnitude between calls to the same endpoint. And which path did it take? — because a system with a cache, a cheap tier, and an expensive tier is really three services wearing one URL, and their aggregate is meaningless.

Everything else follows from getting those two labels onto every call.

Analogy — an itemised phone bill versus a total. A total tells you the bill went up. An itemised bill tells you it was three international calls, which is actionable in seconds. The route label is the itemisation: without it you know your spend rose and nothing about why. The analogy holds mechanically — the aggregate is a sum over categories with wildly different unit costs, so the sum alone cannot identify the cause.

---

2. Diagram

THE FOUR FIELDS  (per call, not per minute)

  { route: "small",          <- WHICH PATH: cache | rule | small | large
    tokens_in: 1800,
    tokens_out: 40,
    cached_tokens: 1200,     <- prove your prompt cache is working
    cost_usd: 0.00019,
    latency_ms: 420,
    model_id: "...",
    outcome: "ok" }          <- ok | degraded | escalated | failed
                                        ^^^^^^^^ the one everyone omits


WHY THE MEAN IS USELESS HERE

  60% cache hits      ~6 ms   ┐
  30% small tier    ~420 ms   ├─ three populations, one endpoint
  10% large tier   ~2700 ms   ┘

     mean   381 ms   <- NO USER EXPERIENCES THIS
     p50      8 ms   <- a cache hit
     p95   2701 ms   <- the expensive tier
     p99   3607 ms

  a single "avg latency" chart would show 381 ms and look healthy


WHERE THE MONEY IS

  route    calls  share   p95 ms     $/1k   % of $
  cache     1240    62%       9   $ 0.000      0%
  small      574    29%     565   $ 0.196     10%
  large      186     9%    3808   $ 5.250     90%
                    ▲                          ▲
                    └── 9% of calls ───────────┘ 90% of cost


DRIFT = compare a window to a baseline, PER ROUTE

           baseline   recent    change
  p95 ms      3833      5986     +56%  ALERT
  $/call   0.00525   0.00785     +50%  ALERT

---

3. How it works

3.1 The four fields, and why route is first among them

route — which path served this call: a cache hit, a deterministic rule, the cheap tier, the expensive tier. This is the single highest-value field because every other metric is meaningless when averaged across paths, and adding it later is expensive.

tokens_in / tokens_out / cached_tokens — read from the provider's usage response rather than estimated. cached_tokens deserves its own field because prompt-cache hit rate is otherwise unobservable, and a cache that silently stopped working shows up as a slightly larger bill and nothing else.

cost_usd — computed at log time from the token counts and the rate for that specific model. Do not defer this to a dashboard query: rates change, and a stored cost is what makes historical comparison honest.

latency_ms — measured from a consistent point. Say which point, in the field name if necessary, because server-side and client-observed latency differ by the network and everyone eventually conflates them.

outcome — ok, degraded, escalated, failed. This is the field teams omit and then wish they had. A fallback that returns a lower-quality answer must be distinguishable from a good one; if both log as success, a silent degradation is invisible to every alert you have.

3.2 Percentiles, and agreeing on the definition

Report p50, p95, p99 and max — never the mean alone. In a system with a cache and multiple tiers, latency is genuinely multi-modal, and the mean lands in a valley between populations where few requests actually fall.

Two practical warnings. First, agree on a percentile definition — nearest-rank, linear interpolation, and the various sketch algorithms give different answers on small samples, and two teams comparing "p95" computed differently will argue for a week. Second, percentiles do not average. You cannot take the p95 of five servers and average them to get a fleet p95; you need the merged distribution, which is why sketch structures that support merging exist.

3.3 Token and cost attribution

Once route and cost_usd are on every call, attribution is a group-by. The characteristic finding is extreme concentration: a small fraction of calls dominates the bill, because the expensive tier costs orders of magnitude more per call than the cheap one.

This concentration is good news. It means cost optimisation has a small, identifiable target rather than requiring system-wide effort — and it means a change to the cheap path, however clever, cannot move the total much. Without per-call attribution, teams routinely optimise the 10% and wonder why the invoice does not move.

Attribute along whatever dimensions you will later want to slice: route, model, tenant, feature, and prompt version. Prompt version in particular pays for itself the first time a prompt change increases output length by 40% and nobody can say when it started.

3.4 Quality metrics, and the honest limits of proxies

Quality is the hardest axis because there is usually no label at request time. What is actually available:

Use the cheap ones continuously and the expensive ones on a sample. The critical discipline for LLM-as-judge is calibration: measure agreement against human labels before trusting it, and re-measure when you change the judge model, or you have replaced an unknown with a confident unknown.

3.5 Drift detection

Drift is a comparison, not a measurement: a recent window against a baseline, on the metrics you already log, per route. The mechanics are straightforward — pick a window, pick a baseline, alert on relative change beyond a threshold.

Three things worth doing. Compare per route, because a 50% cost increase confined to the expensive path is invisible in an aggregate where that path is 9% of calls. Watch token counts as well as cost and latency, since rising output length is the earliest signal of a prompt or model change. And alert on relative change rather than absolute thresholds, so the alert survives traffic growth.

What drifts in practice: providers update models behind a stable name, a prompt edit lengthens responses, retrieved context grows as a corpus grows, and user behaviour shifts so the mix of routes changes. All four are invisible without per-route baselines.

3.6 Where this stops working

Everything above is instrumentation of your calls, and cannot see inside the model. It will not tell you why a model changed its behaviour, only that it did. It also assumes you sample enough to have stable percentiles — at low volume, p99 over an hour is one request and alerting on it produces noise. And the whole approach presumes failures are detectable from what you log: a fluent, plausible, wrong answer that parses correctly and returns quickly is invisible to every metric here, which is why sampled human review never fully goes away.

---

4. The math

4.1 Cost per call, and why it must be stored

  cost = (tokens_in_fresh * rate_in
          + tokens_cached * rate_in * cache_discount
          + tokens_out * rate_out) / 1e6

  store the RESULT, not just the token counts -- rates change, and a recomputed
  historical cost is a different number from what you actually paid

4.2 Nearest-rank percentile

  p(values, q):  s = sorted(values)
                 k = round(q/100 * (len(s) - 1))
                 return s[k]

  and the property that trips people up:
      p95(A ∪ B)  ≠  mean(p95(A), p95(B))

4.3 Concentration and drift

  cost_share(route) = sum(cost | route) / sum(cost)
  drift(metric)     = metric_recent / metric_baseline - 1     -> alert if > θ

4.4 Worked example

2000 calls across three paths: 62% cache hits, 29% cheap tier, 9% expensive tier.

LATENCY — the mean is the least useful number here
  mean     381 ms   <- no user experiences this
  p50        8 ms
  p90      582 ms
  p95     2701 ms
  p99     3607 ms
  max     4357 ms

The mean of 381 ms sits between the cache population (~6 ms) and the cheap-tier population (~420 ms). A dashboard showing only that number would look healthy while 9% of users wait nearly three seconds.

BY ROUTE — aggregates hide which path is slow
  route     calls   share   p95 ms     $/1k   % of $
  cache      1240    62%        9 $  0.000      0%
  small       574    29%      565 $  0.196     10%
  large       186     9%     3808 $  5.250     90%

  total $1.0890 over 2000 calls = $0.545 per 1k

9% of calls are 90% of the cost. That single line is the entire argument for the route field: it makes the optimisation target obvious, and it is uncomputable without per-call labels.

Now drift, simulating the expensive tier starting to generate longer outputs:

  p95 ms   baseline    3833  recent    5986     +56%   ALERT
  $/call   baseline 0.00525  recent 0.00785     +50%   ALERT

Both breach a 25% threshold. Note this comparison was done within the large route — in the aggregate, a 50% cost rise on 9% of calls is a 4.5% total change, comfortably inside normal variance and invisible to any sensible alert.

---

5. Real code

"""The four numbers to log per LLM call, and why the mean hides the problem."""
import random

random.seed(7)

# 2000 calls. Most are fast cache hits or cheap routed answers; a tail escalates
# to a slow expensive tier. This bimodal shape is what real traffic looks like.
CALLS = []
for i in range(2000):
    r = random.random()
    if r < 0.60:
        route, ms, tin, tout = "cache", random.gauss(6, 2), 0, 0
    elif r < 0.90:
        route, ms, tin, tout = "small", random.gauss(420, 90), 1800, 40
    else:
        route, ms, tin, tout = "large", random.gauss(2700, 700), 1800, 300
    CALLS.append({"route": route, "ms": max(1.0, ms),
                  "tok_in": tin, "tok_out": tout})

RATES = {"cache": (0.0, 0.0), "small": (0.10, 0.40), "large": (1.25, 10.00)}


def cost(c):
    r_in, r_out = RATES[c["route"]]
    return (c["tok_in"] * r_in + c["tok_out"] * r_out) / 1e6


def pct(values, p):
    """Nearest-rank percentile -- the definition to agree on before comparing."""
    s = sorted(values)
    k = max(0, min(len(s) - 1, int(round(p / 100 * (len(s) - 1)))))
    return s[k]


lat = [c["ms"] for c in CALLS]
print("LATENCY — the mean is the least useful number here")
print(f"  mean {sum(lat)/len(lat):>7.0f} ms   <- no user experiences this")
for p in (50, 90, 95, 99):
    print(f"  p{p:<3}{pct(lat, p):>8.0f} ms")
print(f"  max  {max(lat):>7.0f} ms")

print("\nBY ROUTE — aggregates hide which path is slow")
print(f"  {'route':<8} {'calls':>6} {'share':>7} {'p95 ms':>8} {'$/1k':>8} {'% of $':>8}")
total_cost = sum(cost(c) for c in CALLS)
for route in ("cache", "small", "large"):
    sub = [c for c in CALLS if c["route"] == route]
    csum = sum(cost(c) for c in sub)
    print(f"  {route:<8} {len(sub):>6} {len(sub)/len(CALLS):>6.0%} "
          f"{pct([c['ms'] for c in sub], 95):>8.0f} "
          f"${csum/len(sub)*1000:>7.3f} {csum/total_cost:>7.0%}")

print(f"\n  total ${total_cost:.4f} over {len(CALLS)} calls "
      f"= ${total_cost/len(CALLS)*1000:.3f} per 1k")

# The headline: a small slice of traffic owns most of the bill AND the tail.
large = [c for c in CALLS if c["route"] == "large"]
share_calls = len(large) / len(CALLS)
share_cost = sum(cost(c) for c in large) / total_cost
print(f"\n  {share_calls:.0%} of calls are {share_cost:.0%} of the cost "
      f"-- you cannot see this without per-call `route`")

print("\nDRIFT — compare a window against a baseline, per route")
base = [c for c in CALLS[:1000] if c["route"] == "large"]
# Simulate a regression: the large tier starts thinking more.
recent = [{**c, "tok_out": c["tok_out"] + 260, "ms": c["ms"] + 2300}
          for c in CALLS[1000:] if c["route"] == "large"]
b_ms, r_ms = pct([c["ms"] for c in base], 95), pct([c["ms"] for c in recent], 95)
b_c = sum(cost(c) for c in base) / len(base)
r_c = sum(cost(c) for c in recent) / len(recent)
print(f"  p95 ms   baseline {b_ms:>7.0f}  recent {r_ms:>7.0f}  "
      f"{(r_ms/b_ms - 1)*100:>+6.0f}%")
print(f"  $/call   baseline {b_c:>7.5f}  recent {r_c:>7.5f}  "
      f"{(r_c/b_c - 1)*100:>+6.0f}%")
ALERT = 0.25
for name, ratio in (("latency", r_ms/b_ms - 1), ("cost", r_c/b_c - 1)):
    print(f"  {name:<8} {'ALERT' if ratio > ALERT else 'ok':<6} "
          f"(threshold {ALERT:.0%})")

assert pct(lat, 50) < 20                    # p50 is a cache hit
assert pct(lat, 99) > 2000                  # p99 is the expensive tier
assert sum(lat)/len(lat) > pct(lat, 50)     # mean sits above the median
assert share_calls < 0.15 and share_cost > 0.85
assert r_ms / b_ms - 1 > ALERT and r_c / b_c - 1 > ALERT
print("\nall assertions passed")

# Output:
#   LATENCY — the mean is the least useful number here
#     mean     381 ms   <- no user experiences this
#     p50        8 ms
#     p90      582 ms
#     p95     2701 ms
#     p99     3607 ms
#     max     4357 ms
#
#   BY ROUTE — aggregates hide which path is slow
#     route     calls   share   p95 ms     $/1k   % of $
#     cache      1240    62%        9 $  0.000      0%
#     small       574    29%      565 $  0.196     10%
#     large       186     9%     3808 $  5.250     90%
#
#     total $1.0890 over 2000 calls = $0.545 per 1k
#
#     9% of calls are 90% of the cost -- you cannot see this without per-call `route`
#
#   DRIFT — compare a window against a baseline, per route
#     p95 ms   baseline    3833  recent    5986     +56%
#     $/call   baseline 0.00525  recent 0.00785     +50%
#     latency  ALERT  (threshold 25%)
#     cost     ALERT  (threshold 25%)
#
#   all assertions passed

The generated traffic is synthetic but the shape — a fast majority, a slow expensive minority — is what production looks like once you have a cache and more than one tier.

---

6. Real-world example

A team ran an assistant feature for several months with standard service monitoring: request rate, error rate, average latency, and a monthly total from the provider's billing console. All green, all month.

Two things were happening that nothing on the dashboard could express.

The provider had updated the model behind a stable version alias, and the new revision produced noticeably longer answers. Output tokens per call rose by around a third. Cost rose with it, but the monthly total also included a growing user base, so the per-call increase was hidden inside overall growth — nobody could separate "more users" from "more expensive per user" because per-call cost was never stored.

Separately, an exception handler around the model call had been catching timeouts and returning a generic templated response. It logged at info level and returned HTTP 200. On the dashboard those were successes. Error rate stayed flat at a fraction of a percent while a small but growing share of users received a canned non-answer.

Both were found in one afternoon after adding four fields. Storing cost_usd and tokens_out per call made the model change appear immediately as a step in output length, dated precisely. Adding outcome with a degraded value turned the invisible fallback into a chart, and it was running at roughly 3% of calls — small enough to never trip an error alert, large enough to matter.

The instructive part is that neither problem was exotic and neither needed a tracing platform. They needed four fields written next to each call, which is an hour of work that had been deferred for months because everything looked fine.

---

7. Interview questions companies actually ask

Q1. What do you log per LLM call that you would not log for a normal API call? Cost and route. Cost, because unlike a database query the price varies by more than an order of magnitude between calls to the same endpoint, so an aggregate spend figure cannot be attributed. Route, because a system with a cache and multiple model tiers is several services behind one URL and their combined metrics describe none of them. Then token counts including cached tokens, and an outcome field that distinguishes a degraded answer from a good one.

Q2. Why report percentiles instead of average latency? Because LLM traffic is genuinely multi-modal — cache hits in milliseconds, cheap-tier calls in hundreds of milliseconds, escalations in seconds — and the mean lands in a gap where few requests actually fall. In the worked example the mean was 381 ms while p50 was 8 ms and p99 was 3.6 s; a single average chart would have looked healthy while a tenth of users waited three seconds.

Q3. Can you average percentiles across servers? No. Percentiles are not additive, so the p95 of a fleet is not the mean of per-server p95s — you need the merged distribution. That is why mergeable sketch structures like t-digest or HDR histograms exist, and why you should store histograms rather than pre-computed percentiles if you will ever aggregate across dimensions.

Q4. How do you detect that a model changed under you? Watch token counts and per-call cost per route against a baseline, not just latency. A provider revision behind a stable alias typically shows up first as a shift in output length, which moves cost before anyone notices a quality change. Compare within route, because a large relative change confined to a minority path is a small change in aggregate and will not alert.

Q5. Your error rate is flat but users complain about bad answers. What is missing? Almost certainly an outcome dimension. A fallback path that catches an exception and returns a templated or lower-quality response typically returns HTTP 200 and logs as a success, so error-rate monitoring cannot see it. Give every degraded path its own status value and chart it; the rate is usually low enough to never trip an error alert and high enough to generate complaints.

Q6. How do you monitor quality when you have no labels? In layers by cost. Free and continuous: schema validity, refusal rate, empty-answer rate, and user behaviour like retries, rephrasings, and escalation to a human. Cheap and mechanical: check that a cited fact appears in the source, that numbers are in range. Then LLM-as-judge on a sample, but only after calibrating its agreement against human labels — and re-calibrating whenever you change the judge, or you have swapped an unknown for a confident unknown.

Q7. What is the limitation of all of this? It observes your calls, not the model's internals, so it tells you that behaviour changed and not why. It also needs volume for stable tail percentiles — at low traffic, p99 over an hour is a single request and alerting on it is noise. And it cannot detect a fluent, plausible, wrong answer that parses correctly and returns quickly, which is precisely the LLM-specific failure mode. Sampled human review is the only real defence and does not go away.

---

8. When to use / tradeoffs

Instrument at this level when:

• You are about to change a model, a prompt, or a routing rule and need a before/after
• Model spend is material enough that attribution matters
• You have more than one path (cache, tiers, fallbacks) behind one endpoint
• You want any of the optimisations in the related articles, all of which need a baseline

Keep it lighter when:

• Pre-launch prototype with no users — log the four fields and skip the rest
• Volume too low for stable percentiles — record raw calls, defer alerting

Honest limits. The traffic in §5 is synthetic, and its clean separation between routes flatters the analysis — real distributions overlap, so route boundaries in a latency histogram are rarely as obvious. Nearest-rank percentiles on 2000 samples are noisy in the tail: p99 is effectively the 20th-slowest request, and it will move between runs even with no change to the system. The drift comparison uses a fixed 25% threshold, which is a placeholder — a real threshold has to be set from observed variance, or you will either miss regressions or page constantly. And the cost model assumes you know the rate for the exact model version serving each call, which is harder than it sounds when providers move revisions behind stable aliases. None of this instrumentation detects the failure that matters most: a confident wrong answer.

---

9. Summary + related articles

• Log four fields per call: route, tokens, cost, latency — plus an outcome that distinguishes degraded from good. That is most of LLM observability.
• route is the highest-value field. Without it, every aggregate metric averages populations that have nothing in common.
• Never report the mean alone. In the example the mean was 381 ms, p50 was 8 ms, p99 was 3.6 s. The mean described no actual user.
• Cost concentrates: 9% of calls were 90% of the bill. Good news — the optimisation target is small and identifiable, and invisible without labels.
• Percentiles do not average. Merge distributions, do not mean the percentiles.
• Store cost_usd at log time. Recomputing history after a rate change gives you a number you never paid.
• A fallback returning HTTP 200 is invisible to error-rate alerting. Give it a status.
• Drift is a windowed comparison per route — a 50% rise on 9% of traffic is 4.5% in aggregate and will never alert.
• Quality needs layers: free proxies continuously, mechanical checks cheaply, judge models on a calibrated sample, humans on a smaller one.
• None of this sees inside the model, needs volume for stable tails, and cannot catch a fluent wrong answer.

Related:

• Cost-Latency Tradeoffs — the SLA percentiles this article tells you how to measure
• Model Routing Patterns — per-route attribution is what makes routing decisions checkable
• Reasoning Budgets: When Thinking Tokens Are Waste — hidden token counts, and why cached_tokens needs its own field
• RAG Cost Optimization: Find the Step That Runs Forty Times — what to do once attribution shows where the money goes
• Agent Evaluation — the scored set behind the quality layer in §3.4
• Production Agents — observability and tracing in a wider production context
• ML Inference Systems — serving-side latency percentiles and capacity

Resources

• Dean & Barroso (2013) — The Tail at Scale, Communications of the ACM 56(2) — why tail latency behaves unlike the mean, and why averages mislead in exactly this way: https://research.google/pubs/pub40801/
• Dunning & Ertl — Computing Extremely Accurate Quantiles Using t-Digests, arXiv:1902.04023 — the mergeable sketch behind §3.2's warning that percentiles do not average: https://arxiv.org/abs/1902.04023
• Beyer et al. — Site Reliability Engineering, Ch. 4 "Service Level Objectives" and Ch. 6 "Monitoring Distributed Systems": https://sre.google/sre-book/service-level-objectives/
• Majors, Fong-Jones & Miranda — Observability Engineering — the high-cardinality, event-per-request model that per-call LLM logging is an instance of.
• OpenTelemetry — GenAI semantic conventions, an emerging standard for naming exactly these attributes; worth adopting rather than inventing field names: https://opentelemetry.io/docs/specs/semconv/gen-ai/
• Provider usage-response documentation is the authority on token accounting, including how cached and reasoning tokens are reported; read it before computing cost yourself.