TL;DR — RAG degrades by returning a confident, well-formatted, wrong answer with a 200 status code. Standard application monitoring is therefore nearly blind to it: of eight realistic failure modes below, six move neither error rate nor latency, and HTTP 5xx catches 1 of 8. The signals that work are retrieval-specific — score distribution catches 5 of 8, corpus size 4, empty-result rate 2 — and they are all things you must deliberately emit, because no framework does it for you. Alerting on them is a threshold problem with an explicit trade: at 500 queries/day a 0.05 score drop takes 4.2 days to detect at one false alarm per month, 1.8 days at five, and 18.1 days at one per ten months — while a 0.20 drop is caught in ~1 day regardless. Volume buys speed directly: the same 0.05 drop takes 14.4 days at 50 queries/day and 1.0 day at 10,000. And keep per-run artifacts so a complaint six weeks later is diagnosable — sampling successes is fine (1% costs 0.8 GB/year at 5,000 queries/day), but never sample failures, since diagnosing those is the entire point.
1. Simple explanation
Ask most teams how they know their RAG system is healthy and you get: error rate, p95 latency, maybe throughput. All three can be perfect while the system is returning nonsense.
That is not a hypothetical. Consider what happens when someone deploys a new embedding model to the query path while the index still holds vectors from the old one (Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For). Every request succeeds. Latency is unchanged. The response is well-formed, has citations, and reads fluently. It is about the wrong documents. Every dashboard is green.
This is the defining property of RAG monitoring: the system's characteristic failure is semantic, and semantic failures do not raise exceptions. You have to instrument the specific things that move when retrieval quality moves — and none of them are in your APM tool by default.
Analogy — a restaurant with a perfect kitchen dashboard. Tickets in, tickets out, average time to plate, zero dropped orders. Every metric green, every night. What none of them measure is whether the food tastes good. You would find that out from the one thing the dashboard cannot see: someone tasting the food, and someone counting how many plates come back. Retrieval score distributions are the tasting; empty-result and citation-resolve rates are the plates coming back.
2. Diagram
WHAT APM SEES WHAT ACTUALLY BROKE
───────────── ───────────────────
200 OK ✓ answer is about the wrong documents
p95 1.8s ✓ index holds old-model vectors
0 exceptions ✓ nobody will notice for weeks
throughput ✓ │
│ │
└──────────── the gap ───────────────┘
SIGNAL x FAILURE COVERAGE (§5)
http laten empty score citat corpus
5xx cy rate dist resolve size
embedding/index mismatch - - - YES - -
index partially rebuilt - - YES YES - YES
upstream source stopped - - - - - YES
reranker model swapped - YES - YES - -
prompt regression - - - - YES -
provider quota throttling YES YES - - - -
chunker config changed - - - YES - YES
corpus drift (new topics) - - YES YES - YES
─── ─── ─── ─── ─── ───
catches 1/8 2/8 2/8 5/8 1/8 4/8
▲▲▲ ▲▲▲ ▲▲▲
what you have now what you need
6 of 8 failures move NEITHER error rate nor latency
DETECTION DELAY (500 queries/day, sigma 1.0)
false alarms/mo drop 0.02 0.05 0.10 0.20
0.1 85.2d 18.1d 3.2d 1.0d
1.0 12.1d 4.2d 1.5d 1.0d
5.0 3.3d 1.8d 1.1d 1.0d
15.0 1.5d 1.2d 1.0d 1.0d
▲▲▲ ▲▲▲
budget matters here and not here
3. How it works
3.1 Emit retrieval-specific signals
Nothing in a web framework knows what a good retrieval looks like. These have to be emitted deliberately, per query:
SCORE DISTRIBUTION mean / p10 / p90 of the top-k relevance scores.
The broadest single signal -- 5 of 8 failures move
it -- because almost anything that breaks retrieval
changes how well the top results match.
EMPTY-RESULT RATE queries returning nothing above threshold. Rises on
corpus gaps, index problems, and topic drift.
CITATION RESOLVE RATE fraction of emitted citations pointing at a real
retrieved chunk. Catches generation-side regressions
that retrieval metrics cannot see.
CORPUS SIZE / FRESHNESS document and chunk counts, newest document age.
Catches ingestion stopping -- which otherwise has
NO symptom until someone asks about recent material.
PER-STAGE LATENCY + CALL COUNTS not for speed, but because a change in
call count means the pipeline took a different path.
Two things follow from the coverage table. Score distribution is the highest-value single addition — if you add one thing, add that. And no single signal catches everything: the widest covers 5 of 8, so you need three or four, chosen to overlap different failure classes.
3.2 Watch distributions, not averages
A mean hides the failures that matter. If half your queries get worse and half get better, the mean does not move. If a single topic breaks, the mean barely moves at all.
Track percentiles of the score distribution, and track them segmented — by query type, by department or tenant, by source. A tenant-specific index rebuild shows up instantly in that tenant's p50 and is invisible in the global mean. This is the same trap as reporting only the mean in an evaluation (RAG Evaluation: Attributing Failure and Sizing the Eval Set): aggregates hide the localised failures that are the most actionable.
3.3 Alert thresholds are an explicit trade
Given a noisy daily metric, an alert threshold trades detection speed against false alarms. Measured on a daily mean score at 500 queries/day:
false alarms/month drop 0.02 drop 0.05 drop 0.10 drop 0.20
0.1 85.2d 18.1d 3.2d 1.0d
1.0 12.1d 4.2d 1.5d 1.0d
5.0 3.3d 1.8d 1.1d 1.0d
15.0 1.5d 1.2d 1.0d 1.0d
Two readings. Large regressions are caught in about a day whatever budget you pick, so the alarm budget is not a decision about catastrophic failures — those are easy. It is entirely a decision about small ones: at a 0.02 drop, tightening from 5 to 0.1 false alarms per month costs you 3.3 days → 85.2 days.
So set the budget from the smallest regression you would actually act on. If a 0.02 drop would not change your behaviour, do not pay 85 days of latency trying to detect it — alert on 0.05 and accept that smaller drifts are found by periodic evaluation rather than by paging someone.
Volume buys detection speed directly:
queries/day days to detect a 0.05 drop (1 false alarm/month)
50 14.4
200 7.7
500 4.2
2,000 1.5
10,000 1.0
A low-traffic system cannot detect small regressions quickly at any threshold — the data does not exist yet. For those, lean on a scheduled evaluation run against a fixed set (RAG Evaluation: Attributing Failure and Sizing the Eval Set) rather than on production monitoring, because the eval set gives you a controlled comparison that traffic cannot.
3.4 Per-run artifacts
Monitoring tells you that something changed. Artifacts let you find out what. A complaint arrives six weeks after the fact — "this answer was wrong" — and without a stored run you cannot reconstruct what the system saw.
Persist per run:
manifest query, rewritten query, config version, model ids, index name,
feature flags, timings, per-stage call counts
retrieved the candidate documents with their scores at each stage
answer the final text, citations, and any structured block
Two properties matter more than completeness. Give the run an id and surface it in the UI, so a complaint arrives already attached to the run rather than to a vague description. And record the config version, because the most common answer to "why was this wrong" is that the run happened under different settings than the ones you are looking at now.
Retention is cheap if you sample sensibly. At 5,000 queries/day and 45 KB per bundle:
keep everything 6.8 GB/month 81.0 GB/year
sample 10% 0.7 GB/month 8.1 GB/year
sample 1% 0.1 GB/month 0.8 GB/year
all failures + 1% of successes 0.3 GB/month 4.1 GB/year
The last row is the right shape. Sampling successes is safe; sampling failures is not — the entire purpose of the bundle is diagnosing the run that went wrong, and a 1% sample of failures means 99% of your incidents are undiagnosable. Keep every run that errored, returned empty, scored below threshold, or was thumbs-downed, and sample the rest.
Watch what goes into the bundle: it contains the user's query and retrieved document text, so it inherits whatever data-handling obligations those carry. Retention limits and redaction belong in the design, not bolted on later.
3.5 Leading and lagging indicators
LEADING (minutes) score distribution, empty rate, call counts,
corpus size, citation resolve rate
-> move at deploy time, before users notice
LAGGING (days-weeks) thumbs-down rate, escalations to humans, task
completion, repeat-query rate
-> the ground truth, far too slow to alert on
You need both, and for different jobs. Leading indicators tell you something changed now; only lagging ones tell you whether it mattered. A change that moves a leading indicator and nothing lagging was probably harmless. A change that moves nothing leading but degrades the lagging metric means your instrumentation has a gap — and that gap is the more valuable finding.
The most useful lagging signal is usually the cheapest: what users do next. Rephrasing immediately, or escalating to a human, is a stronger quality signal than an explicit thumbs-down, because almost nobody clicks thumbs-down.
3.6 Where this is overkill
A prototype with a handful of daily queries does not need alert thresholds — you will notice. But run artifacts are worth having from day one: they cost nearly nothing at low volume and they are the difference between debugging a report and shrugging at it. Instrument artifacts early, add alerting when traffic makes it statistically meaningful.
4. The math
4.1 Detection is a power calculation
Monitoring a daily mean of n queries with per-query standard deviation σ, the standard error is:
SE = σ / sqrt(n)
For a one-sided alert with false-alarm probability α per day, the threshold sits z_α · SE below the baseline. The chance of catching a true drop δ on any single day is:
P(detect per day) = Φ( δ/SE − z_α )
expected days to detect = 1 / P(detect per day)
With σ = 1.0, n = 500, one false alarm per month (α = 1/30, z_α = 1.83):
SE = 1.0 / sqrt(500) = 0.0447
δ = 0.05 -> Φ(1.118 − 1.83) = Φ(−0.71) = 0.239 -> 4.2 days
δ = 0.20 -> Φ(4.472 − 1.83) = Φ(2.64) = 0.996 -> 1.0 days
Detection time is dominated by δ/SE. Since SE shrinks with √n, quadrupling traffic halves the detectable effect — the same square-root relationship that governs eval-set sizing.
4.2 The false-alarm budget only matters for small effects
Notice the asymmetry in the table. When δ/SE is large, Φ(δ/SE − z_α) is near 1 for any sane z_α, so the budget is irrelevant. When δ/SE is near z_α, the term is on the steep part of the normal CDF and small changes in z_α swing detection time enormously:
δ = 0.02: 0.1 alarms/month -> 85.2 days 5/month -> 3.3 days (26x)
δ = 0.20: 0.1 alarms/month -> 1.0 days 5/month -> 1.0 days (1x)
Practical consequence: pick the alarm budget by asking what the smallest regression worth paging someone about is. Optimising the budget for large failures is wasted effort, and chasing tiny ones costs weeks of detection latency on everything.
4.3 Coverage arithmetic
With signals catching overlapping subsets of failures, what matters is the union, not the best individual signal:
best single signal (score distribution) 5/8 = 63%
score dist + corpus size 7/8 = 88%
+ empty rate 7/8
+ latency (adds quota throttling) 8/8
Two well-chosen signals cover most of it; the last failure needs a signal you probably already have. Choose signals to cover different failure classes, not to maximise individual coverage — adding a second signal correlated with the first buys nothing, the same independence argument that governs Multi-Index RAG: Merging Several Retrievers Into One Answer.
4.4 Retention
GB/month = queries/day × 30 × sample_rate × KB_per_run / 1e6
Linear in every term, so a sample rate is a direct dial. The asymmetric policy — all failures, 1% of successes — costs 0.3 GB/month against 6.8 for everything: 95% of the storage saved while keeping 100% of the runs you would ever want to read.
5. Real code
Standard library only, deterministic, runs instantly. Part A is a coverage table (a judgement about which signals move, made explicit so you can disagree with it); B and C are exact arithmetic.
import math
from statistics import NormalDist
_ND = NormalDist()
# failure -> which signals move. The point of the table is how EMPTY the
# first two columns are.
FAILURES = [
# name, http_error, latency, empty_rate,
# score_dist, citation_resolve, corpus_size
("embedding/index mismatch", False, False, False, True, False, False),
("index partially rebuilt", False, False, True, True, False, True),
("upstream source stopped", False, False, False, False, False, True),
("reranker model swapped", False, True, False, True, False, False),
("prompt regression", False, False, False, False, True, False),
("provider quota throttling", True, True, False, False, False, False),
("chunker config changed", False, False, False, True, False, True),
("corpus drift (new topics)", False, False, True, True, False, True),
]
SIGNALS = ["http 5xx", "latency", "empty-result rate",
"score distribution", "citation resolve rate", "corpus size"]
def coverage():
print("A. WHICH SIGNAL CATCHES WHICH FAILURE")
head = f"{'failure':>28} " + " ".join(f"{s[:9]:>10}" for s in SIGNALS)
print(head)
print("-" * len(head))
caught = [0] * len(SIGNALS)
silent_to_apm = 0
for row in FAILURES:
name, flags = row[0], row[1:]
print(f"{name:>28} " + " ".join(
f"{(' YES' if f else ' -'):>10}" for f in flags))
for i, f in enumerate(flags):
caught[i] += f
if not (flags[0] or flags[1]):
silent_to_apm += 1
print()
for i, s in enumerate(SIGNALS):
print(f" {s:>22} catches {caught[i]}/{len(FAILURES)}")
print(f"\n {silent_to_apm}/{len(FAILURES)} failures move NEITHER error rate "
f"nor latency --")
print(" invisible to standard application monitoring.")
return caught, silent_to_apm
def detection(delta, daily_n, sigma=1.0, fp_per_month=1.0):
"""Days to detect a mean shift of `delta` with a given false-alarm budget.
One-sided test on the daily mean; the threshold is set so that the
expected number of false alarms per 30 days matches fp_per_month.
"""
alpha = fp_per_month / 30.0
z_alpha = _ND.inv_cdf(1 - alpha)
se = sigma / math.sqrt(daily_n)
z_beta = delta / se - z_alpha # power on a single day
p_detect_daily = _ND.cdf(z_beta)
if p_detect_daily <= 1e-9:
return float("inf"), 0.0
return 1.0 / p_detect_daily, p_detect_daily
def thresholds():
print("\n\nB. DETECTION DELAY vs FALSE ALARMS")
print(" monitoring a daily mean score, sigma=1.0, 500 queries/day")
print(f"{'false alarms/month':>20} " + "".join(
f"{f'drop {d}':>12}" for d in ("0.02", "0.05", "0.10", "0.20")))
print("-" * 68)
table = {}
for fp in (0.1, 1.0, 5.0, 15.0):
cells = []
for d in (0.02, 0.05, 0.10, 0.20):
days, _ = detection(d, 500, fp_per_month=fp)
table[(fp, d)] = days
cells.append("never" if days == float("inf")
else f"{days:.1f}d" if days < 100 else ">100d")
print(f"{fp:20.1f} " + "".join(f"{c:>12}" for c in cells))
print("\n a tighter alarm budget costs detection time on SMALL regressions")
print(" and costs almost nothing on large ones -- so tune the budget for")
print(" the smallest regression you actually intend to act on.")
print("\n volume buys detection speed (0.05 drop, 1 false alarm/month):")
print(f"{'queries/day':>14} {'days to detect':>16}")
print("-" * 32)
vols = {}
for n in (50, 200, 500, 2000, 10000):
days, _ = detection(0.05, n, fp_per_month=1.0)
vols[n] = days
print(f"{n:14,} {('never' if days == float('inf') else f'{days:.1f}'):>16}")
return table, vols
def retention():
print("\n\nC. WHAT RUN ARTIFACTS COST")
QPS_DAY = 5000
KB_PER_RUN = 45 # manifest + retrieved docs + answer + timings
print(f" {QPS_DAY:,} queries/day, {KB_PER_RUN} KB per full run bundle")
print(f"{'policy':>34} {'GB/month':>10} {'GB/year':>9} {'runs kept':>12}")
print("-" * 68)
plans = [("keep everything", 1.0),
("sample 10%", 0.10),
("sample 1%", 0.01),
("all failures + 1% of successes", 0.01 + 0.04)]
out = {}
for name, frac in plans:
per_month = QPS_DAY * 30 * frac * KB_PER_RUN / 1e6
out[name] = per_month
print(f"{name:>34} {per_month:9.1f}G {per_month*12:8.1f}G "
f"{int(QPS_DAY*30*frac):12,}")
print("\n sampling successes is safe; sampling FAILURES is not -- the whole")
print(" point of the bundle is diagnosing the run that went wrong.")
return out
caught, silent = coverage()
table, vols = thresholds()
out = retention()
# claims made in the prose
assert silent >= 6, "most RAG failures must be invisible to APM"
assert caught[0] <= 1, "http errors must catch almost nothing"
assert caught[3] >= 5, "score distribution must be the broadest signal"
assert table[(1.0, 0.20)] < table[(1.0, 0.02)], "big drops detect faster"
assert vols[10000] < vols[50], "more volume detects faster"
assert out["sample 1%"] < out["keep everything"] / 50
print("\nasserts passed")
# Output:
# A. WHICH SIGNAL CATCHES WHICH FAILURE
# failure http 5xx latency empty-res score dis citation corpus si
# ----------------------------------------------------------------------------------------------
# embedding/index mismatch - - - YES - -
# index partially rebuilt - - YES YES - YES
# upstream source stopped - - - - - YES
# reranker model swapped - YES - YES - -
# prompt regression - - - - YES -
# provider quota throttling YES YES - - - -
# chunker config changed - - - YES - YES
# corpus drift (new topics) - - YES YES - YES
#
# http 5xx catches 1/8
# latency catches 2/8
# empty-result rate catches 2/8
# score distribution catches 5/8
# citation resolve rate catches 1/8
# corpus size catches 4/8
#
# 6/8 failures move NEITHER error rate nor latency --
# invisible to standard application monitoring.
#
#
# B. DETECTION DELAY vs FALSE ALARMS
# monitoring a daily mean score, sigma=1.0, 500 queries/day
# false alarms/month drop 0.02 drop 0.05 drop 0.10 drop 0.20
# --------------------------------------------------------------------
# 0.1 85.2d 18.1d 3.2d 1.0d
# 1.0 12.1d 4.2d 1.5d 1.0d
# 5.0 3.3d 1.8d 1.1d 1.0d
# 15.0 1.5d 1.2d 1.0d 1.0d
#
# a tighter alarm budget costs detection time on SMALL regressions
# and costs almost nothing on large ones -- so tune the budget for
# the smallest regression you actually intend to act on.
#
# volume buys detection speed (0.05 drop, 1 false alarm/month):
# queries/day days to detect
# --------------------------------
# 50 14.4
# 200 7.7
# 500 4.2
# 2,000 1.5
# 10,000 1.0
#
#
# C. WHAT RUN ARTIFACTS COST
# 5,000 queries/day, 45 KB per full run bundle
# policy GB/month GB/year runs kept
# --------------------------------------------------------------------
# keep everything 6.8G 81.0G 150,000
# sample 10% 0.7G 8.1G 15,000
# sample 1% 0.1G 0.8G 1,500
# all failures + 1% of successes 0.3G 4.1G 7,500
#
# sampling successes is safe; sampling FAILURES is not -- the whole
# point of the bundle is diagnosing the run that went wrong.
#
# asserts passed
The FAILURES table is the part to argue with. It encodes my judgement about which signals move for which failure, and yours will differ — a system with strict output validation might catch prompt regressions on the error rate, for instance. Write your own version for your own failure history, then count the columns. The count is the useful output: it tells you which signals to add and, more importantly, which failures nothing currently catches.
6. Real-world example
A team ran a document-search assistant for about eight months with clean dashboards. Then a quarterly review found that answers about anything recent were wrong or evasive.
Ingestion had stopped 51 days earlier. A credential used by the nightly sync had expired; the job caught the exception, logged it at WARNING, and exited zero. Nothing downstream noticed, because a stale index behaves exactly like a healthy one — same latency, same error rate, same score distribution, same well-formed answers. It only looks broken if you ask about something that should have been added and was not.
Nobody did, for 51 days, because users ask about recent material gradually rather than all at once, and the early ones assumed they were searching wrong.
The fix took an afternoon and was not clever: emit document count and newest-document age after every sync, and alert when the newest document is older than 48 hours. That single check would have fired on day two.
The broader lesson is what the audit found next. They enumerated their failure modes, marked which signal would catch each, and discovered that most of their monitoring was aimed at failures they had never actually had — infrastructure problems the platform team already handled — while the failures they had experienced were mostly uninstrumented. Freshness monitoring was the cheapest possible check and nobody had thought to add it, because it never occurs to anyone to monitor a thing that stopped happening.
7. Interview questions companies actually ask
Q1 [easy] "Why isn't error rate enough for a RAG system?"
A Because RAG's characteristic failure is semantic: a confident, fluent, wrongly-
grounded answer returned with a 200. Of eight realistic failure modes, six move
neither error rate nor latency, and HTTP 5xx catches 1 of 8. Semantic failures
don't raise exceptions.
Q2 [easy] "If you add one metric, what should it be?"
A The retrieval score distribution -- mean and percentiles of the top-k scores.
It caught 5 of 8 failures in the coverage table, because nearly anything that
breaks retrieval changes how well the top results match. Second most valuable
is corpus size and freshness, at 4 of 8.
Q3 [medium] "A nightly ingestion job silently stops. Which metric catches it?"
A Only corpus size or document freshness. A stale index is indistinguishable from
a healthy one on latency, error rate, score distribution, and answer format --
it looks fine until someone asks about material that should have been added.
Alert on newest-document age; it's the cheapest check you'll ever write.
Q4 [medium] "How do you pick an alert threshold?"
A From the smallest regression you'd actually act on. Big drops are caught in ~1
day at any budget, so the threshold decision is entirely about small ones: at a
0.02 drop, tightening from 5 to 0.1 false alarms per month moves detection from
3.3 days to 85.2. If a 0.02 drop wouldn't change your behaviour, don't pay 85
days trying to see it.
Q5 [medium] "Your system does 50 queries a day. How fast can you detect a 5%
regression?"
A About 14 days at one false alarm per month, versus 1 day at 10,000 queries/day
-- detection time scales with delta/SE and SE shrinks as sqrt(n). At low volume
the data simply doesn't exist yet, so rely on scheduled evaluation against a
fixed set rather than production alerting.
Q6 [medium] "Why track distributions rather than the mean?"
A A mean hides localised failures: half better and half worse leaves it unmoved,
and one broken topic or tenant barely shifts it. Track percentiles, and segment
by tenant, query type, and source -- a per-tenant index problem is obvious in
that tenant's p50 and invisible globally.
Q7 [hard] "What do you store per run, and how much can you sample?"
A Manifest (query, config version, model ids, index, flags, timings, call counts),
the retrieved candidates with scores, and the final answer with citations. Sample
successes freely -- 1% is 0.8 GB/year at 5,000 queries/day -- but keep 100% of
failures, empties, low-score runs and thumbs-downs. Sampling failures makes 99%
of your incidents undiagnosable, which defeats the purpose.
Q8 [hard] "Leading indicators moved but user metrics didn't. What does that mean?"
A Probably a harmless change -- something shifted that users don't feel. The more
interesting case is the inverse: user metrics degrade while every leading
indicator holds. That means your instrumentation has a blind spot, and finding
which failure class nothing covers is more valuable than the incident itself.
8. When to use / tradeoffs
INSTRUMENT, IN THIS ORDER:
1. run artifacts with an id surfaced in the UI (day one, any scale)
2. corpus size + newest-document age (catches silent stalls)
3. retrieval score distribution, percentiles (broadest single signal)
4. empty-result rate and citation resolve rate
5. per-stage call counts (path changes)
6. alert thresholds (once volume supports it)
SEGMENT EVERYTHING BY: tenant, query type, source, config version
| Situation | Why it breaks | Use instead |
|---|---|---|
| Relying on error rate and latency | 6 of 8 failures move neither | Score distribution + corpus freshness |
| Tracking the mean only | Hides localised and offsetting failures | Percentiles, segmented |
| One metric for everything | Best single signal covers 5 of 8 | 3–4 signals covering different classes |
| Very tight alarm budget | 85 days to see a 0.02 drop | Budget set by smallest actionable drop |
| Alerting at low volume | 50 queries/day can't see 5% quickly | Scheduled eval on a fixed set |
| Sampling failures | 99% of incidents undiagnosable | All failures + a sample of successes |
| No config version in the run | "Why was this wrong" is unanswerable | Version every run |
| Only lagging indicators | Too slow to act on | Pair leading with lagging |
| Storing runs without a retention plan | Bundles hold user queries and content | Redact and expire by policy |
Honest limits. The coverage matrix in part A is my judgement, not data — it is an explicit, arguable model of which signals move for which failures, and it is included precisely so you can disagree and write your own; the count is what matters, not my cell values. Part B assumes daily scores are independent and normally distributed with a known, stable σ, which real traffic violates in every direction: weekday/weekend seasonality, query-mix shifts, and heavy tails all inflate the false-alarm rate above the nominal budget, so treat these day counts as optimistic. A real deployment should use a control chart or a change-point method with a moving baseline rather than a fixed threshold, which is a larger topic than this article covers. The 45 KB per run and 5,000 queries/day in part C are illustrative; the arithmetic is linear, so substitute yours. Finally, this article says nothing about what to do when an alert fires — triage, rollback, and incident process are general operational concerns, not RAG-specific ones.
9. Summary + related articles
- RAG fails by returning a confident wrong answer with a 200. Six of eight failure modes move neither error rate nor latency; HTTP 5xx caught 1 of 8.
- Score distribution is the highest-value single signal (5 of 8). Corpus size and freshness is second (4 of 8) and catches the silent-stall class nothing else sees.
- No single signal suffices. Pick 3–4 covering different failure classes; correlated signals add nothing.
- Track percentiles, segmented by tenant, query type, and source. Means hide the localised failures worth acting on.
- Set the alarm budget from the smallest regression you'd act on. Large drops are caught in ~1 day regardless; at a 0.02 drop the budget swings detection from 3.3 to 85.2 days.
- Volume buys detection speed: a 0.05 drop takes 14.4 days at 50 queries/day, 1.0 day at 10,000. Low-traffic systems should use scheduled evaluation instead.
- Keep run artifacts from day one, with the run id in the UI and the config version recorded. Sample successes (1% = 0.8 GB/year); never sample failures.
- Boundary: monitoring tells you that something changed and roughly when. It cannot tell you whether the answers are good — that needs evaluation against labelled data.
Related:
- RAG Evaluation: Attributing Failure and Sizing the Eval Set — the controlled counterpart: monitoring watches production, evaluation gives you the attributable comparison, and low-traffic systems need the latter
- Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For — §3.5's silent embedding/index mismatch, the canonical green-dashboard failure
- Document Processing: What You Index Sets the Ceiling — ingestion failures that report success; the source of the stale-index story in §6
- RAG Cost Optimization: Find the Step That Runs Forty Times — per-stage call counts serve both cost attribution and path-change detection
- Multi-Index RAG: Merging Several Retrievers Into One Answer — §3.1 on independence, which is why correlated monitoring signals buy nothing
- Generation Pipeline: Assembling Context and Enforcing a Citation Contract — citation resolve rate, the one signal that catches generation-side regressions
Resources
- Sculley, Holt, Golovin et al., "Hidden Technical Debt in Machine Learning Systems", NeurIPS 2015 — the case for monitoring data and behaviour rather than only code health.
- Breck, Cai, Nielsen et al., "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction", IEEE Big Data 2017 — a concrete checklist; the monitoring section maps closely onto §3.1.
- Beyer, Jones, Petoff & Murphy, Site Reliability Engineering, O'Reilly 2016 — chapters 6 and 10 on choosing signals and on alerting that respects a false-alarm budget.
- Montgomery, Introduction to Statistical Quality Control, 8th ed., Wiley 2019 — control charts and CUSUM, the better alternative to the fixed threshold modelled in §4.1.
- Rabanser, Günnemann & Lipton, "Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift", NeurIPS 2019 — detecting distribution drift, the formal version of §3.2.