TL;DR — Accuracy is the wrong headline metric whenever one class is rare, and it fails in a specific, measurable way: in the run below, a model that predicts "not fraud" every single time scores 99.0% accuracy — higher than a real model that catches 55% of fraud and scores 96.8%. Accuracy rewards being right about the 99% of cases nobody cares about. A confusion matrix shows what it hides — 0 caught, 20 missed — and precision (of what I flagged, how much was real) and recall (of what was real, how much did I catch) separate the two error types so you can price them. They trade against each other: moving from cautious to aggressive took recall 15% → 90% and precision 42.9% → 8.8%. There is no setting that avoids both errors, so the metric you optimise has to come from which mistake costs more. It stops being about metrics at all when your test set doesn't resemble production — then every number is precise and meaningless.
1. Simple explanation
Ask "how good is this model?" and the instinctive answer is what percentage did it get right? For a balanced problem — is this email spam, roughly half and half — that's reasonable.
Now make one outcome rare. One transaction in a hundred is fraudulent. A model that ignores its input entirely and says "not fraud" every time is right 99% of the time. It is also completely useless, and it will beat your real model on the scoreboard.
The problem is that accuracy adds up two very different mistakes and treats them as equal. Missing a fraud and wrongly blocking a customer are both "errors", and they cost wildly different amounts. Any single number that averages them has thrown away the information you needed.
Analogy — a smoke alarm. An alarm that never goes off is correct on almost every day of your life; there is no fire on almost every day. Its accuracy is superb and it is worthless, because the only day that matters is the one it sleeps through. The two questions you actually care about are when there was a fire, did it sound? (recall) and when it sounded, was there a fire? (precision). Tuning it more sensitive improves the first and ruins the second — burnt toast at 2am — and there is no setting that gives you both.
2. Diagram
THE ACCURACY TRAP — 2000 transactions, 20 of them fraud (1%)
model accuracy precision recall F1
always say 'not fraud' 99.0% 0.0% 0.0% 0.000 ← WINS on accuracy
an actual model 96.8% 16.4% 55.0% 0.253 ← useful
▲
the useless model scores HIGHER
WHAT THE CONFUSION MATRIX SHOWS THAT ACCURACY HIDES
always say 'not fraud' an actual model
pred+ pred- pred+ pred-
actual+ 0 20 actual+ 11 9
actual- 0 1980 actual- 56 1924
▲ ▲
caught 0 of 20 frauds caught 11 of 20
flagged 0 customers flagged 56 customers
Same accuracy scale. Completely different systems.
THE TWO QUESTIONS, AND THE TRADE
PRECISION of everything I flagged, how much was real? → false alarms
RECALL of everything real, how much did I catch? → misses
cautious precision 42.9% recall 15% fraud slips through
balanced precision 40.6% recall 65% best available trade
aggressive precision 8.8% recall 90% good customers blocked
▼ ▲
falls as rises as you
you get bolder get bolder
no setting avoids both. choose from what each error COSTS.
3. How it works
3.1 The confusion matrix is the primitive
Every other metric is derived from four counts:
predicted positive predicted negative
actually positive TP (caught) FN (missed)
actually negative FP (false alarm) TN (correctly ignored)
Report this first, always. It is the only view that shows which errors you are making rather than how many, and the two off-diagonal cells usually have very different costs.
3.2 The metrics, and what each is blind to
accuracy = (TP + TN) / everything blind to class imbalance
precision = TP / (TP + FP) ignores what you missed
recall = TP / (TP + FN) ignores false alarms
F1 = harmonic mean of the two hides which one is weak
Each answers a different question, and each is blind to something:
- Accuracy is dominated by the majority class. At 1% positives it is essentially a report on the negatives.
- Precision says nothing about misses — flag one transaction, get it right, score 100%.
- Recall says nothing about false alarms — flag everything, score 100%.
- F1 combines them, and because it's the harmonic mean it punishes imbalance between them. Useful as a single number, and it still hides which side is failing, so don't report it alone.
3.3 Why the trade is unavoidable
Almost every classifier produces a score, and you choose a threshold. Lower it and you catch more real positives and more false alarms. There is no threshold that only catches the ones you want, because the score distributions of the two classes overlap.
The measured version: cautious flagging gave 42.9% precision at 15% recall; aggressive gave 8.8% precision at 90% recall. Both are the same model — only the threshold moved.
So the question is never "which threshold is best" in the abstract. It is which error costs more. A missed fraud costs the transaction value; a false alarm costs a blocked customer and a support call. Put numbers on both, however rough, and the threshold follows. Refuse to, and it gets chosen by default — badly.
3.4 The base rate makes precision hard, permanently
At a 1% base rate, even a good detector produces mostly false alarms. In §4 the real model caught 11 frauds and flagged 56 good customers — 16.4% precision, from a model with a 3% false-positive rate.
That is not a bad model. It's arithmetic: 3% of 1,980 legitimate transactions is 59, which swamps 20 total frauds no matter how well you detect them. When positives are rare, low precision is the default and no amount of tuning escapes it — you need a better signal, a second-stage filter, or an acceptance that human review is part of the system.
3.5 The failure that outranks all of this: a bad test set
Metrics assume your test set resembles production. When it doesn't, every number is precise and meaningless. Three ways it goes wrong, in rough order of frequency:
Leakage — information in training that won't exist at prediction time. The classic is a feature computed after the outcome; the model scores brilliantly offline and collapses live.
Temporal leakage — a random train/test split on time-ordered data, so the model trains on the future and is tested on the past. For anything with a time dimension, split by time. See Backtesting, Baselines & Sensitivity Analysis.
Distribution drift — the test set was assembled last year, or from one segment, or from the easy cases someone had labelled.
A single held-out split is also noisy on small data; cross-validation averages over several splits and gives you a spread, which tells you whether a difference between two models is real or luck.
3.6 Where these metrics stop applying
They are for classification with a known ground truth. They say nothing useful about ranking quality (use ranking metrics), regression (use error magnitudes), or generative output, where there is usually no single correct answer to compare against — for that see LLM-as-a-Judge: Using a Model to Grade Model Output and Agent Evaluation. And they all assume your labels are correct, which on a hard problem is often the weakest assumption in the whole pipeline.
4. The math
4.1 Definitions
accuracy = (TP + TN) / (TP + TN + FP + FN)
precision = TP / (TP + FP)
recall = TP / (TP + FN)
F1 = 2 * precision * recall / (precision + recall)
4.2 Why accuracy collapses under imbalance
with a positive rate p, the always-negative model scores
accuracy = 1 - p
p = 0.01 -> 99% accuracy, 0% recall, zero value
The metric and the goal have come apart entirely: you can maximise one by ignoring the other completely.
4.3 Worked example
2000 transactions, 20 genuinely fraudulent (1%). Two "models": one that always says "not fraud", and one that catches 70% of fraud with a 3% false-positive rate.
model accuracy precision recall F1
always say 'not fraud' 99.0% 0.0% 0.0% 0.000
an actual model 96.8% 16.4% 55.0% 0.253
The do-nothing model wins on accuracy, 99.0% against 96.8%. If accuracy is your headline metric, you have just shipped the wrong model — and every dashboard will agree with you.
The confusion matrices make the difference impossible to miss:
always say 'not fraud'
predicted fraud predicted ok
actual fraud 0 20
actual ok 0 1980
-> caught 0 of 20 frauds, missed 20, flagged 0 good customers
an actual model
predicted fraud predicted ok
actual fraud 11 9
actual ok 56 1924
-> caught 11 of 20 frauds, missed 9, flagged 56 good customers
Note the second model's precision: 11 real out of 67 flagged, 16.4%. That is §3.4 — the base rate, not the model, is doing that.
4.4 The trade, measured
Same model, three thresholds:
threshold behaviour precision recall who suffers
very cautious flagging 42.9% 15.0% fraud slips through
balanced 40.6% 65.0% best available trade
flag anything suspicious 8.8% 90.0% good customers blocked
Recall climbs 15% → 65% → 90%; precision holds and then falls off a cliff to 8.8%. The middle row is where the curve is kindest, and "kindest" is not a decision — only the relative cost of a missed fraud versus a blocked customer decides which row you want.
5. Real code
"""Why accuracy lies on imbalanced data, and what a confusion matrix shows instead."""
import random
random.seed(11)
# 2000 transactions, 1.5% of them fraudulent. This imbalance is typical and it is
# exactly the regime where accuracy stops carrying information.
N, RATE = 2000, 0.015
truth = [1 if random.random() < RATE else 0 for _ in range(N)]
POS = sum(truth)
def always_negative(_x):
return 0
def decent_model(y, rng):
"""Catches 70% of fraud; 3% false-positive rate on legitimate transactions."""
if y == 1:
return 1 if rng.random() < 0.70 else 0
return 1 if rng.random() < 0.03 else 0
def confusion(truth, pred):
tp = sum(1 for t, p in zip(truth, pred) if t == 1 and p == 1)
fp = sum(1 for t, p in zip(truth, pred) if t == 0 and p == 1)
fn = sum(1 for t, p in zip(truth, pred) if t == 1 and p == 0)
tn = sum(1 for t, p in zip(truth, pred) if t == 0 and p == 0)
return tp, fp, fn, tn
def metrics(truth, pred):
tp, fp, fn, tn = confusion(truth, pred)
acc = (tp + tn) / len(truth)
prec = tp / (tp + fp) if tp + fp else 0.0
rec = tp / (tp + fn) if tp + fn else 0.0
f1 = 2 * prec * rec / (prec + rec) if prec + rec else 0.0
return acc, prec, rec, f1
rng = random.Random(3)
preds = {
"always say 'not fraud'": [always_negative(t) for t in truth],
"an actual model": [decent_model(t, rng) for t in truth],
}
print(f"{N} transactions, {POS} actually fraudulent ({POS/N:.1%})\n")
print(f"{'model':<26} {'accuracy':>9} {'precision':>10} {'recall':>8} {'F1':>7}")
res = {}
for name, pred in preds.items():
a, p, r, f = metrics(truth, pred)
res[name] = (a, p, r, f)
print(f"{name:<26} {a:>8.1%} {p:>10.1%} {r:>8.1%} {f:>7.3f}")
lazy_acc = res["always say 'not fraud'"][0]
real_acc = res["an actual model"][0]
print(f"\nThe do-nothing model scores {lazy_acc:.1%} accuracy -- HIGHER than the")
print(f"model that actually catches fraud ({real_acc:.1%}). Accuracy rewards it for")
print(f"being right about the {1-POS/N:.1%} of cases nobody cares about.")
print("\nTHE CONFUSION MATRIX SHOWS WHAT ACCURACY HIDES")
for name, pred in preds.items():
tp, fp, fn, tn = confusion(truth, pred)
print(f"\n {name}")
print(f" predicted fraud predicted ok")
print(f" actual fraud {tp:>13} {fn:>12}")
print(f" actual ok {fp:>13} {tn:>12}")
print(f" -> caught {tp} of {POS} frauds, missed {fn}, "
f"flagged {fp} good customers")
print("\nPRECISION AND RECALL TRADE OFF -- pick from the COST of each error")
print(f" {'threshold behaviour':<28} {'precision':>10} {'recall':>8} who suffers")
for i, (label, catch, fpr) in enumerate([("very cautious flagging", 0.30, 0.002),
("balanced", 0.70, 0.010),
("flag anything suspicious", 0.95, 0.100)]):
r2 = random.Random(100 + i) # fresh rng per row, or the draws interleave
pred = [1 if ((t == 1 and r2.random() < catch) or
(t == 0 and r2.random() < fpr)) else 0 for t in truth]
_a, p, r, _f = metrics(truth, pred)
who = ("fraud slips through" if r < 0.5
else "good customers blocked" if p < 0.10
else "best available trade")
print(f" {label:<28} {p:>10.1%} {r:>8.1%} {who}")
# The do-nothing model wins on accuracy. That is the whole point.
assert lazy_acc > real_acc, (lazy_acc, real_acc)
assert res["always say 'not fraud'"][2] == 0.0 # recall is zero
assert res["always say 'not fraud'"][3] == 0.0 # so is F1
# The real model is worse on accuracy and vastly better on the metric that matters.
assert res["an actual model"][2] > 0.5
assert res["an actual model"][3] > res["always say 'not fraud'"][3]
print(f"\nAccuracy ranked them WRONG. Recall and F1 ranked them right.")
print("all assertions passed")
# Output:
# 2000 transactions, 20 actually fraudulent (1.0%)
#
# model accuracy precision recall F1
# always say 'not fraud' 99.0% 0.0% 0.0% 0.000
# an actual model 96.8% 16.4% 55.0% 0.253
#
# The do-nothing model scores 99.0% accuracy -- HIGHER than the
# model that actually catches fraud (96.8%). Accuracy rewards it for
# being right about the 99.0% of cases nobody cares about.
#
# THE CONFUSION MATRIX SHOWS WHAT ACCURACY HIDES
#
# always say 'not fraud'
# predicted fraud predicted ok
# actual fraud 0 20
# actual ok 0 1980
# -> caught 0 of 20 frauds, missed 20, flagged 0 good customers
#
# an actual model
# predicted fraud predicted ok
# actual fraud 11 9
# actual ok 56 1924
# -> caught 11 of 20 frauds, missed 9, flagged 56 good customers
#
# PRECISION AND RECALL TRADE OFF -- pick from the COST of each error
# threshold behaviour precision recall who suffers
# very cautious flagging 42.9% 15.0% fraud slips through
# balanced 40.6% 65.0% best available trade
# flag anything suspicious 8.8% 90.0% good customers blocked
#
# Accuracy ranked them WRONG. Recall and F1 ranked them right.
# all assertions passed
Note the seeded sample gave 20 positives from a 1.5% rate — 1.0% actual, which is small-sample variance and itself a reminder that with 20 positives, every metric here has wide error bars.
6. Real-world example
A team shipped a classifier to flag support tickets needing urgent escalation. Roughly 2% of tickets qualified. The model reported 97% accuracy in review and was approved.
For six weeks nobody escalated anything unusual, and then a serious issue reached a customer having sat in the normal queue for four days.
The model had learned to predict "not urgent" almost always. At a 2% base rate that scores 98% accuracy, so the model's 97% was worse than doing nothing — a fact nobody spotted because nobody had computed the do-nothing baseline. The review had compared 97% against a vague sense that 97% is good.
Two failures, and the second is the more instructive.
The obvious one: accuracy on imbalanced data, exactly §3. A confusion matrix in the review would have shown a nearly empty "predicted urgent" column.
The subtler one: there was no baseline. A metric with nothing to compare against is a number, not evidence. The cheapest and most valuable baseline is the majority-class predictor, because it tells you what the metric scores when the model has learned nothing — and any model that doesn't clearly beat it has not been shown to work.
The fix was to report recall as the headline, with a confusion matrix beside it and the do-nothing baseline in the same table. The model was retrained with class weighting, landed around 70% recall at 25% precision, and was accepted on the explicit basis that a missed escalation cost far more than a wasted review.
7. Interview questions companies actually ask
Q1. Why is accuracy a poor metric for imbalanced data? Because it's dominated by the majority class. At a 1% positive rate, predicting "negative" always scores 99% while catching nothing — in the worked example that useless model outscored a real one, 99.0% to 96.8%. Accuracy averages two errors with very different costs, and any single number that does that has discarded what you needed.
Q2. Explain precision and recall. Precision: of everything I flagged, how much was real — it measures false alarms. Recall: of everything real, how much did I catch — it measures misses. Precision ignores what you missed, recall ignores false alarms, so you report both. F1 is their harmonic mean, useful as one number and still hiding which side is weak.
Q3. How do you choose between them? From the cost of each error. A missed fraud costs the transaction; a false alarm costs a blocked customer and a support call. Put rough numbers on both and the threshold follows. If you won't, the threshold gets chosen by default and usually badly. There is no threshold that avoids both — measured, recall 15%→90% took precision 42.9%→8.8%.
Q4. Your model has 95% precision and 20% recall. Is it good? It depends entirely on what the misses cost. For a system that surfaces suggestions to a human, high precision and low recall is often right — everything shown is trustworthy. For cancer screening or fraud, 20% recall means missing four in five, which is usually unacceptable regardless of how clean the flags are.
Q5. Why does a good detector still produce mostly false alarms on rare events? Base rates. With 1% positives and a 3% false-positive rate, 3% of the huge negative class swamps the tiny positive class — in the example, 56 false alarms against 11 real catches, giving 16% precision from a genuinely decent model. That's arithmetic, not a tuning failure, and it means human review is part of the system rather than a sign something is broken.
Q6. What's the first baseline you compute? The majority-class predictor — always predict the common label. It tells you what your metric scores when the model has learned nothing, and any model that doesn't clearly beat it has not been shown to work. It's free, and in the §6 story it would have caught a six-week failure in the review meeting.
Q7. What's more likely to invalidate your evaluation than choosing the wrong metric? A test set that doesn't resemble production. Leakage — a feature computed after the outcome — makes offline scores brilliant and live performance collapse. Temporal leakage from randomly splitting time-ordered data trains on the future. And drift, where the test set came from one segment or last year. When the test set is wrong, every metric is precise and meaningless.
8. When to use / tradeoffs
Report accuracy when:
- Classes are roughly balanced
- Both error types genuinely cost the same
- It sits alongside a confusion matrix, never alone
Report precision/recall when:
- One class is rare — which is most real problems
- The two errors have different costs
- A human acts on the output
| Situation | Why it breaks | Report instead |
|---|---|---|
| Rare positive class | Majority-class predictor wins | Recall + precision + confusion matrix |
| Single metric, no baseline | A number with nothing to compare to | Always include the do-nothing baseline |
| F1 reported alone | Hides which side is failing | F1 and its two components |
| Random split on time-ordered data | Trains on the future | Split by time |
| Feature computed after the outcome | Leakage; collapses in production | Audit feature timing |
| Small test set | Wide error bars on every metric | Cross-validation, report the spread |
| Generative output | No single correct answer | Judge models, human review, task metrics |
| Threshold picked by feel | Implicitly prices the errors, badly | Cost per error type, then derive it |
Honest limits. The "models" in §5 are simulated with fixed catch and false-positive rates rather than trained on features, which isolates the metric behaviour cleanly and means none of the numbers say anything about achievable performance on a real problem. With only 20 positives, every metric here has error bars wide enough to swamp the differences between the threshold rows — a real evaluation needs far more positives or explicit confidence intervals, and this example would not survive its own §3.5 advice. The seeded 1.5% rate produced 1.0% in the sample, which is exactly that variance showing. The article also treats the cost of an error as a single number, where in practice it varies per case: a missed £5 fraud and a missed £50,000 fraud are not the same error, and value-weighted metrics exist for that reason.
9. Summary + related articles
- The confusion matrix is the primitive. Report it first; every other metric is derived from its four counts.
- Accuracy fails under imbalance. Measured: a do-nothing model scored 99.0% and beat a real model at 96.8%.
- Precision = of what I flagged, how much was real. Recall = of what was real, how much did I catch. Each is blind to the other's error.
- F1 is their harmonic mean — useful, and it still hides which side is weak. Never report it alone.
- The two trade against each other: recall 15%→90% cost precision 42.9%→8.8%. No setting avoids both.
- So the metric follows from which error costs more. Refuse to price them and the threshold gets chosen badly by default.
- At low base rates, low precision is arithmetic, not a bug — 3% of a huge negative class swamps a tiny positive one.
- Always compute the majority-class baseline. It is free and it catches the failure in §6 during review.
- A bad test set outranks all of this: leakage, temporal leakage, and drift make every number precise and meaningless.
Related:
- Backtesting, Baselines & Sensitivity Analysis — baselines, and why time-ordered data needs a time-ordered split
- Logistic Regression — where the score you threshold comes from
- Classification — the task these metrics evaluate
- LLM-as-a-Judge: Using a Model to Grade Model Output — evaluating output with no single correct answer
- Probability & Statistics Foundations — base rates, and why they dominate precision
- Fraud Detection — this exact trade in a full system design
- Agent Evaluation — scoring systems whose output is text
- RAG Evaluation: Attributing Failure and Sizing the Eval Set — precision and recall applied to retrieval
Resources
- Provost & Fawcett — Data Science for Business, Ch. 7-8 — the clearest treatment of expected value and cost-sensitive evaluation; the §3.3 argument in full.
- Fawcett, T. (2006) — An Introduction to ROC Analysis, Pattern Recognition Letters 27(8) — thresholds, ROC curves, and why AUC behaves differently from precision under imbalance.
- Saito & Rehmsmeier (2015) — The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets, PLoS ONE 10(3) — directly on the §3.4 base-rate problem: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0118432
- Kaufman et al. (2012) — Leakage in Data Mining: Formulation, Detection, and Avoidance, ACM TKDD 6(4) — the failure mode of §3.5, catalogued.
- scikit-learn — model evaluation user guide; the reference implementations of every metric here: https://scikit-learn.org/stable/modules/model_evaluation.html
- Hastie, Tibshirani & Friedman — The Elements of Statistical Learning, Ch. 7 — cross-validation and model assessment, free online: https://hastie.su.domains/ElemStatLearn/
