TL;DR — Three quantities explain what a language model's loss number actually means. Entropy is how spread out a prediction is — the model's own uncertainty. Surprisal is how wrong it turned out to be for the outcome that happened. Perplexity is
2^surprisal, read as "effectively choosing between N equally likely options". The measurement below is the one worth keeping: two models with almost identical entropy (0.60 vs 0.62 — equally confident) had surprisals of 0.15 vs 5.64 bits, a perplexity of 1.11 against 50.00. Confidence and correctness are different axes, and being confidently wrong is punished ~37× harder. Cross-entropy averaged over a sequence is the training loss, which is why every language model is trained to avoid exactly that failure. It stops being comparable the moment you change tokenizer or vocabulary — perplexity is only meaningful between models measured on the same data the same way.
1. Simple explanation
Information theory asks one question: how surprised should you be?
If a friend tells you the sun rose this morning, you learn nothing — you already knew. If they tell you it snowed in Chennai, that is a lot of information, because it was unlikely. Information content is inversely related to probability, and measuring it in bits makes that precise.
Language models live on this. A model doesn't output words, it outputs a probability for every possible next token. So "how good was that prediction?" becomes "how surprised was the model by what actually came next?" — a number you can compute, average, and minimise.
That average, over a lot of text, is the training loss. Everything else in this article is a different way of reading the same quantity.
Analogy — twenty questions. With no information you might need twenty yes/no questions to identify something, and twenty questions distinguishes about a million things — that is twenty bits. A good hint doesn't tell you the answer; it cuts the number of questions you still need. Entropy is how many questions remain on average, and surprisal is how many you'd have needed for the answer that actually turned up. The analogy carries the key asymmetry too: a confident hint that points the wrong way doesn't just fail to help, it costs you far more than having no hint at all.
2. Diagram
SURPRISAL: how many bits did this outcome cost?
p(outcome) surprisal = -log2(p) reading
1.00 0.00 bits certain; you learned nothing
0.90 0.15 bits expected
0.50 1.00 bit a coin flip
0.10 3.32 bits unexpected
0.02 5.64 bits shocked
THE MEASUREMENT THAT MATTERS — actual next word was 'mat'
model entropy surprisal perplexity
confident and RIGHT 0.60 0.15 1.11
unsure 1.97 1.74 3.33
confident and WRONG 0.62 5.64 50.00
^^^^ ^^^^
nearly IDENTICAL wildly DIFFERENT
certainty correctness
Entropy cannot tell those two models apart.
Surprisal separates them by 37x.
THE THREE QUANTITIES, AND WHAT EACH ONE ASKS
ENTROPY "how spread out is this PREDICTION?" needs only p
H(p) = -sum p_i * log2(p_i) model's own uncertainty
SURPRISAL "how wrong was it for THIS outcome?" needs p and the truth
-log2(p[actual])
CROSS-ENTROPY "on average, over a sequence?" = the training loss
mean of the surprisals
PERPLEXITY "effectively choosing between how many?" = 2^cross-entropy
a human-readable rescaling, nothing more
3. How it works
3.1 Surprisal: bits as a unit of "I didn't see that coming"
surprisal(x) = -log2( p(x) )
Three properties make this the right definition. Something certain (p = 1) carries zero bits — no information. Rarer events carry more. And bits add: two independent events costing 3 bits each cost 6 together, which matches the intuition that information accumulates.
Base 2 gives bits; base e gives nats, which is what most ML libraries actually report. Same quantity, different ruler — see Logarithms, Exponents & the Log Scale.
3.2 Entropy is about the prediction, not the outcome
H(p) = -sum over i of p_i * log2(p_i)
Entropy is the expected surprisal — how uncertain the model is before it finds out. It needs only the distribution, never the truth.
Maximum entropy is a uniform distribution: over 4 options, log2(4) = 2 bits, the state of knowing nothing. Minimum is zero, when one option has probability 1.
This is why entropy alone cannot grade a model. A confidently wrong model has low entropy — it is certain, just certain of the wrong thing. §4 shows both confident models at ~0.6 bits, indistinguishable on this measure.
3.3 Cross-entropy is the training loss
Take the surprisal of the token that actually appeared, at every position, and average:
cross-entropy = mean over positions of -log2( p_model[actual token] )
That's it. Training a language model is minimising this number over a corpus. Which means the objective is literally "be less surprised by real text" — and the asymmetry in §3.4 is baked into what the model learns to avoid.
Cross-entropy is also the standard loss for classification, so the same quantity connects language modelling to Model Evaluation.
3.4 Why confidently wrong is punished so hard
-log2(p) goes to infinity as p goes to zero. That shape is the whole story:
| p assigned to the truth | surprisal |
|---|---|
| 0.90 | 0.15 bits |
| 0.50 | 1.00 bit |
| 0.10 | 3.32 bits |
| 0.02 | 5.64 bits |
| 0.001 | 9.97 bits |
Being unsure costs a little. Being confidently wrong costs enormously — in §4, 37× more than being confidently right. A model trained on this loss learns to hedge rather than commit when it doesn't know, which is a genuinely desirable behaviour and one reason well-trained models spread probability rather than spiking.
It's also why a single near-zero probability on a token that appears can dominate the loss for a whole batch.
3.5 Perplexity, and why the number is so often misused
perplexity = 2 ^ cross-entropy (or e^loss if the loss is in nats)
Perplexity is cross-entropy made readable: "the model was effectively choosing between N equally likely options at each step." A perplexity of 1 is certainty; a perplexity equal to the vocabulary size means the model knows nothing.
The misuse is comparing perplexity across setups. It is not comparable when:
- The tokenizer differs. Per-token perplexity over different token units measures different things. Tokenization explains why one model's token is another's three.
- The vocabulary size differs. A bigger vocabulary raises the ceiling.
- The test data differs. Perplexity on Wikipedia and on code are unrelated numbers.
Perplexity compares models on the same data with the same tokenizer. Outside that, it's a number with units nobody wrote down.
3.6 Where this framework stops
Low perplexity means the model predicts text well. It does not mean the model is truthful, helpful, or safe — a model can be superbly unsurprised by fluent nonsense. That gap is why the field moved to task benchmarks and human preference, and why perplexity is a training diagnostic rather than a product metric.
It also assumes a well-defined probability distribution over discrete outcomes, so it says little about ranking quality, regression, or generation judged by an unbounded set of acceptable answers.
4. The math
4.1 The four definitions
surprisal(x) = -log2( p(x) )
entropy H(p) = -sum_i p_i log2(p_i) = expected surprisal
cross-entropy H(p, q) = -sum_i p_i log2(q_i) truth p, model q
perplexity PPL = 2^H same number, readable
KL divergence D(p || q) = H(p, q) - H(p) >= 0
"the extra bits you pay for using the wrong distribution"
KL is the natural way to say how far apart two distributions are, and it is never negative — using the wrong model never costs fewer bits than using the right one.
4.2 Worked example
The sentence is "the cat sat on the ___" and the actual next word is mat. Three models predict it.
model entropy surprisal perplexity reading
confident and RIGHT 0.60 0.15 1.11 barely surprised
unsure 1.97 1.74 3.33 mildly surprised
confident and WRONG 0.62 5.64 50.00 shocked
The middle model spread its probability (0.30 / 0.28 / 0.24 / 0.18) so its entropy is high — 1.97 bits, near the 2-bit maximum for four options. It was mildly surprised, and its perplexity of 3.33 says "effectively choosing between about three words."
Now the pair that matters:
both have LOW entropy: 0.60 vs 0.62 (similarly certain)
but surprisal: 0.15 vs 5.64 bits
and perplexity: 1.11 vs 50.00
Entropy cannot tell them apart. Both are confident. Only bringing in the actual outcome separates them, and it separates them by a factor of 37 in surprisal. That asymmetry is exactly what the training objective encodes.
4.3 Over a sequence
per-token surprisal: [0.15, 1.84, 2.06]
mean = cross-entropy: 1.349 bits/token
PERPLEXITY = 2^1.349 = 2.55
Across those three positions the model was effectively choosing between about 2.5 equally likely words each time. With a 4-word vocabulary, a model that knew nothing would score 4 — so 2.55 is real but unremarkable skill.
That comparison only works because both numbers use the same vocabulary and the same data, which is precisely the condition §3.5 warns about.
5. Real code
"""Entropy, cross-entropy and perplexity: what a language model's loss means."""
import math
# Three models predicting the next word after "the cat sat on the ___".
# TRUTH is what actually came next.
TRUTH = "mat"
MODELS = {
"confident and RIGHT": {"mat": 0.90, "rug": 0.06, "floor": 0.03, "moon": 0.01},
"unsure": {"mat": 0.30, "rug": 0.28, "floor": 0.24, "moon": 0.18},
"confident and WRONG": {"mat": 0.02, "rug": 0.90, "floor": 0.05, "moon": 0.03},
}
def entropy(p: dict[str, float]) -> float:
"""Average surprise of the distribution itself, in bits. High = spread out."""
return -sum(q * math.log2(q) for q in p.values() if q > 0)
def surprisal(p: dict[str, float], outcome: str) -> float:
"""How surprised the model was by what actually happened, in bits."""
return -math.log2(p[outcome])
def perplexity_from_bits(bits: float) -> float:
"""2^bits -- 'how many equally likely options was it effectively choosing from'."""
return 2 ** bits
print(f"actual next word: {TRUTH!r}\n")
print(f"{'model':<22} {'entropy':>8} {'surprisal':>10} {'perplexity':>11} reading")
res = {}
for name, p in MODELS.items():
h, s = entropy(p), surprisal(p, TRUTH)
ppl = perplexity_from_bits(s)
res[name] = (h, s, ppl)
reading = ("barely surprised" if s < 0.5
else "mildly surprised" if s < 2
else "shocked")
print(f"{name:<22} {h:>8.2f} {s:>10.2f} {ppl:>11.2f} {reading}")
print("\n entropy = how spread out the PREDICTION was (model's own uncertainty)")
print(" surprisal = how wrong it turned out to be, for THIS outcome")
print(" perplexity = 2^surprisal, read as 'effectively guessing between N options'")
print("\nCONFIDENCE IS NOT CORRECTNESS -- the two confident models differ hugely")
a = res["confident and RIGHT"]
b = res["confident and WRONG"]
print(f" both have LOW entropy: {a[0]:.2f} vs {b[0]:.2f} (similarly certain)")
print(f" but surprisal: {a[1]:.2f} vs {b[1]:.2f} bits")
print(f" and perplexity: {a[2]:.2f} vs {b[2]:.2f}")
print(" -> being confidently wrong is punished far harder than being unsure.")
print(" That asymmetry is exactly what training a language model optimises.")
print("\nCROSS-ENTROPY OVER A SEQUENCE = the training loss")
SEQ = [("mat", MODELS["confident and RIGHT"]),
("rug", MODELS["unsure"]),
("floor", MODELS["unsure"])]
bits = [surprisal(p, w) for w, p in SEQ]
mean_bits = sum(bits) / len(bits)
print(f" per-token surprisal: {[round(b,2) for b in bits]}")
print(f" mean = cross-entropy: {mean_bits:.3f} bits/token")
print(f" PERPLEXITY = 2^{mean_bits:.3f} = {2**mean_bits:.2f}")
print(" i.e. across the sequence the model was effectively choosing")
print(f" between about {2**mean_bits:.1f} equally likely words at each step.")
print("\nWHY PERPLEXITY NUMBERS ARE NOT COMPARABLE ACROSS SETUPS")
V = 4
print(f" uniform over {V} options -> perplexity {V} (knows nothing)")
print(f" our sequence -> perplexity {2**mean_bits:.2f}")
print(" A model with a BIGGER vocabulary has a higher ceiling, and one using a")
print(" different tokenizer is measuring per-token over different units entirely.")
print(" Perplexity compares models on the SAME data with the SAME tokenizer. Only.")
# Entropy is maximal when the distribution is uniform.
uni = {k: 1/V for k in MODELS["unsure"]}
assert abs(entropy(uni) - math.log2(V)) < 1e-12
assert entropy(uni) > entropy(MODELS["unsure"]) > entropy(MODELS["confident and RIGHT"])
# Confident-and-wrong is punished far harder than merely unsure.
assert res["confident and WRONG"][1] > res["unsure"][1] > res["confident and RIGHT"][1]
# Perplexity is just the exponentiated surprisal.
assert abs(res["unsure"][2] - 2 ** res["unsure"][1]) < 1e-9
# A model that put 90% on the truth is 'effectively choosing between ~1.1 options'.
assert res["confident and RIGHT"][2] < 1.2
print("\nall assertions passed")
# Output:
# actual next word: 'mat'
#
# model entropy surprisal perplexity reading
# confident and RIGHT 0.60 0.15 1.11 barely surprised
# unsure 1.97 1.74 3.33 mildly surprised
# confident and WRONG 0.62 5.64 50.00 shocked
#
# entropy = how spread out the PREDICTION was (model's own uncertainty)
# surprisal = how wrong it turned out to be, for THIS outcome
# perplexity = 2^surprisal, read as 'effectively guessing between N options'
#
# CONFIDENCE IS NOT CORRECTNESS -- the two confident models differ hugely
# both have LOW entropy: 0.60 vs 0.62 (similarly certain)
# but surprisal: 0.15 vs 5.64 bits
# and perplexity: 1.11 vs 50.00
# -> being confidently wrong is punished far harder than being unsure.
# That asymmetry is exactly what training a language model optimises.
#
# CROSS-ENTROPY OVER A SEQUENCE = the training loss
# per-token surprisal: [0.15, 1.84, 2.06]
# mean = cross-entropy: 1.349 bits/token
# PERPLEXITY = 2^1.349 = 2.55
# i.e. across the sequence the model was effectively choosing
# between about 2.5 equally likely words at each step.
#
# WHY PERPLEXITY NUMBERS ARE NOT COMPARABLE ACROSS SETUPS
# uniform over 4 options -> perplexity 4 (knows nothing)
# our sequence -> perplexity 2.55
# A model with a BIGGER vocabulary has a higher ceiling, and one using a
# different tokenizer is measuring per-token over different units entirely.
# Perplexity compares models on the SAME data with the SAME tokenizer. Only.
#
# all assertions passed
Four-word vocabularies make the arithmetic checkable by hand; real models work over tens of thousands of tokens and report loss in nats rather than bits. The formulas are unchanged — only the ruler and the scale differ.
6. Real-world example
A team fine-tuned a model and reported success: validation perplexity fell from 12.4 to 8.1. That is a large improvement by any normal reading, and it was approved on that basis.
The deployed model was worse. Users reported answers that were fluent, plausible and more often wrong than before.
Two things had gone unexamined, and both are §3.5 and §3.6.
They had changed the tokenizer during fine-tuning. The new one produced more tokens per sentence, so per-token perplexity fell simply because each token was individually easier to predict — a shorter unit is a smaller guess. Normalised per character rather than per token, the improvement mostly vanished. The number went down because the units changed.
And the improvement that remained was real but measured the wrong thing. Lower perplexity means the model predicts text in the fine-tuning distribution better. Their fine-tuning corpus was internally written and confidently phrased, so the model learned to be more fluent and more assertive — which lowers perplexity and raises the rate of confident errors. Perplexity has no term for truthfulness.
The fix was to fix the tokenizer across comparisons, report bits-per-character alongside perplexity, and gate deployment on a task-level scored set rather than loss. The general point: perplexity is a training diagnostic, not a product metric, and comparing it across setups is comparing numbers with different units.
7. Interview questions companies actually ask
Q1. What is entropy, in one sentence? The expected surprisal of a distribution — how uncertain a prediction is before you learn the outcome. It needs only the probabilities, never the truth, which is exactly why it cannot grade a model: a confidently wrong prediction has low entropy.
Q2. What's the difference between entropy and cross-entropy? Entropy measures one distribution's own spread. Cross-entropy measures how many bits you pay when you use the model's distribution to encode outcomes drawn from reality. Cross-entropy minus entropy is the KL divergence — the extra bits your model costs you, which is never negative.
Q3. What is perplexity and what does the number mean? 2^cross-entropy (or e^loss in nats). Read it as "the model was effectively choosing between N equally likely options at each step." Perplexity 1 is certainty; perplexity equal to the vocabulary size is knowing nothing. It's cross-entropy made readable, nothing more.
Q4. Why can't you compare perplexity across two models? Because it depends on the tokenizer, the vocabulary size and the test data. Per-token perplexity over different token units measures different things — a model with finer tokens gets a lower number for free. It is only comparable on the same data with the same tokenizer, and reporting bits-per-character sidesteps the tokenizer problem.
Q5. Why is a confidently wrong prediction punished so much? Because surprisal is -log(p), which goes to infinity as the probability assigned to the truth goes to zero. Assigning 0.90 to the truth costs 0.15 bits; assigning 0.02 costs 5.64 — around 37× more. That shape is what teaches a model to hedge when uncertain rather than commit, and it's why one near-zero probability on a token that appears can dominate a batch's loss.
Q6. Your model's loss went down but users say it got worse. What happened? Two common causes. The units changed — a different tokenizer makes per-token loss incomparable, and the improvement can be entirely artefact. Or the loss improved on the fine-tuning distribution while the thing you care about didn't: lower perplexity means better prediction of text like the training text, and it contains no term for truthfulness. Gate on a task-level scored set.
Q7. Where does information theory show up outside training loss? Decision trees split on information gain, which is an entropy reduction. KL divergence appears in variational methods and in RLHF objectives as a penalty for drifting from a reference model. Compression and tokenization are information theory directly — BPE is a compression algorithm. And drift detection often compares distributions with KL or a related divergence.
8. When to use / tradeoffs
Use perplexity / cross-entropy when:
- Monitoring training and validation loss
- Comparing checkpoints of the same model on the same data
- Detecting distribution shift between two corpora
- You need a cheap continuous signal, computed without human labels
Use something else when:
- Deciding whether to ship → task-level scored evaluation
- Comparing across tokenizers → bits-per-character
- Measuring truthfulness or helpfulness → these are not in the loss
- Judging generation with many acceptable answers → judge models or humans
| Situation | Why it breaks | Use instead |
|---|---|---|
| Perplexity across tokenizers | Different units per token | Bits-per-character |
| Perplexity across vocab sizes | Bigger vocab, higher ceiling | Same-vocab comparison only |
| Loss as a ship/no-ship gate | No term for truthfulness | Scored task evaluation |
| Entropy used to grade a model | Confidently wrong is low-entropy | Surprisal, which uses the truth |
| Comparing loss in bits and nats | Different log base | Convert: nats × 1.4427 = bits |
| One catastrophic token in a batch | -log(p) explodes near zero | Inspect per-token loss, not just the mean |
Honest limits. The example uses four-word vocabularies so every number is hand-checkable, and real perplexities live in a very different range — the ratios transfer, the magnitudes do not. It also reports bits while nearly every ML library reports nats, so a loss you read off a training curve is × 1.4427 from these figures. The three models are hand-constructed to make the entropy/surprisal split maximally visible; real distributions over 50,000 tokens are far flatter and the separation is less dramatic per token, though it accumulates. And the whole framework assumes the model outputs a genuine probability distribution that is calibrated — a model whose 0.9 does not mean 90% has a perplexity that is arithmetically correct and semantically misleading.
9. Summary + related articles
- Surprisal =
-log2(p): how many bits the actual outcome cost. Certain → 0 bits; near-impossible → very many. - Entropy = expected surprisal: how spread out the prediction is. Needs only
p, so it cannot grade a model. - Cross-entropy = mean surprisal over a sequence = the training loss. The objective is literally "be less surprised by real text".
- Perplexity =
2^cross-entropy, read as "effectively choosing between N options". A readable rescaling, nothing more. - Measured: two models with entropy 0.60 vs 0.62 — equally confident — had surprisal 0.15 vs 5.64 bits. Confidence ≠ correctness.
- Confidently wrong costs ~37× confidently right, because
-log(p)explodes near zero. That asymmetry teaches models to hedge. - KL divergence = cross-entropy − entropy = the extra bits your model costs. Never negative.
- Perplexity is not comparable across tokenizers, vocabularies, or datasets. Report bits-per-character to cross tokenizers.
- Low perplexity means fluent prediction, not truthfulness. It is a training diagnostic, not a product metric.
Related:
- Logarithms, Exponents & the Log Scale — why the loss is written in logs, and bits versus nats
- Probability & Statistics Foundations — the distributions all of this operates on
- How LLMs Work — the next-token distribution being scored here
- Tokenization — why per-token perplexity is not portable
- Sampling and Temperature — reshaping the same distribution at inference
- Model Evaluation — cross-entropy as a classification loss, and why loss isn't the metric
- Transformers Deep Dive — softmax, which produces the distribution
- When to Fine-Tune — the §6 failure, in its decision context
Resources
- Shannon, C. E. (1948) — A Mathematical Theory of Communication, Bell System Technical Journal 27 — entropy, surprisal and the whole framework, in the original and still remarkably readable.
- Cover & Thomas — Elements of Information Theory, Ch. 2 — the standard reference for entropy, cross-entropy and KL divergence.
- MacKay, D. — Information Theory, Inference, and Learning Algorithms — free online, and unusually good at connecting the theory to inference: https://www.inference.org.uk/mackay/itila/
- Jurafsky & Martin — Speech and Language Processing (3rd ed. draft), the n-gram chapter's section on perplexity — including the tokenizer caveat: https://web.stanford.edu/~jurafsky/slp3/
- Chip Huyen — Evaluation Metrics for Language Modeling — a practical treatment of exactly the §3.5 comparability problem: https://huyenchip.com/2019/11/07/perplexity-vs-entropy-vs-cross-entropy.html