Sampling and Temperature

SamplingDecodingBeginner21 min

By: Anacodic Team

TL;DR — A model hands you a probability for every possible next token; sampling is the separate step that picks one, and it is not part of the model. temperature=0 takes the most likely token every time, which is deterministic — in the measured run below it produces exactly 1 distinct outcome from 400 draws. Raising temperature reshapes the distribution and reaches further into the tail: at T=1.5, nonsense tokens were drawn 29 times in 400. The important asymmetry is that temperature only makes the tail less likely while top-p deletes it outright — the same run with top_p=0.85 hit nonsense zero times while keeping three plausible options. So the standard shape is a moderate temperature plus truncation, not temperature alone. It stops being the right lever when the problem is factual accuracy: turning temperature to 0 makes a wrong answer repeatable, not correct.

1. Simple explanation

A model doesn't emit text. It emits a score for every token in its vocabulary — "mat: 45%, rug: 25%, floor: 15%…". Something then has to choose one. That chooser is the sampler, and it lives outside the model, in your API call.

The simplest rule is "always take the highest". That's greedy decoding, and it's what temperature=0 means. Same prompt, same output, forever — which is what you want for extracting a field from a document and terrible for writing three different subject lines.

The alternative is to roll a weighted die. If mat is at 45% you take it about 45% of the time. Now the output varies, and the question becomes how much — which is what temperature controls.

Analogy — a chef with a recipe and a spice rack. The model produces the recipe: this much of each ingredient, ranked. The sampler decides how faithfully to follow it. Follow exactly and you get the same dish every time — reliable, never surprising. Improvise a little and it varies pleasantly. Improvise a lot and you eventually reach for something that was technically in the rack and shouldn't be in the dish. Turning up improvisation doesn't remove the bad ingredient — it just makes reaching for it more likely. Taking it off the rack entirely is a different action, and that's the difference between temperature and top-p.

2. Diagram

THE MODEL'S OUTPUT — one probability per candidate token

  mat        0.45  █████████████████████
  rug        0.25  ████████████
  floor      0.15  ███████
  carpet     0.08  ████
  table      0.04  ██
  moon       0.02  █          ← implausible, non-zero
  xylophone  0.01  ▌          ← nonsense, still reachable


TEMPERATURE reshapes it. Same tokens, different odds.

  T=0.0   mat 1.000                                  deterministic
          █████████████████████████████████████████

  T=0.5   mat 0.684  rug 0.211  floor 0.076          sharpened
          ████████████████████████████  ████  ██

  T=1.0   mat 0.450  rug 0.250  floor 0.150          as produced
          ██████████████  ████████  █████

  T=1.5   mat 0.348  rug 0.235  floor 0.167          flattened
          ███████████  ███████  █████                (tail gets easier)


TRUNCATION deletes the tail instead of demoting it

  top_k=3     keep the 3 most likely, renormalise    3 choices
  top_p=0.85  keep the smallest set summing to 0.85  3 choices
                                    ▲
                       adapts: confident model -> fewer kept
                               unsure model    -> more kept


WHAT YOU ACTUALLY GET  (400 draws each)

  setting               distinct   nonsense hits
  greedy (T=0)                 1               0    ← same answer every time
  T=0.5                        6               1
  T=1.0                        7              14
  T=1.5                        7              29    ← tail reached often
  T=1.0 + top_p=0.85           3               0    ← tail GONE, variety kept
                                                ▲
                     temperature makes nonsense unlikely.
                     top-p makes it impossible.

3. How it works

3.1 Temperature: reshaping before choosing

Temperature divides the model's raw scores before they're turned back into probabilities:

T < 1 exaggerates differences. The leader gets more likely, everything else less. At the limit, T=0 means "always the top token".
T = 1 leaves the distribution as the model produced it.
T > 1 compresses differences toward uniform, so unlikely tokens become meaningfully reachable.

Two things people get wrong. First, temperature never removes an option — at any T > 0, every non-zero token remains possible. Second, T=0 isn't a temperature at all; it's a special case meaning argmax, since dividing by zero is undefined. Providers implement it as "pick the maximum".

3.2 Top-k: keep a fixed number

Sort, keep the k most likely, discard the rest, renormalise. Genuinely removes the tail.

Its weakness is the fixed count. When the model is certain, k=40 drags in 39 tokens it had almost ruled out. When the model is genuinely torn between fifty options, k=40 cuts ten reasonable ones. The right number depends on how confident the model is at that step, and k can't know that.

3.3 Top-p (nucleus): keep a fixed amount of probability

Sort, then accumulate until the total reaches p, and keep only those. The set size adapts to the model's confidence, which is exactly what top-k can't do:

  confident step:  mat 0.95, ...          -> top_p=0.9 keeps 1 token
  uncertain step:  0.2, 0.18, 0.15, ...   -> top_p=0.9 keeps 6 or 7

That adaptivity is why top-p is the more common default. In §4 both top_k=3 and top_p=0.85 happen to keep three tokens — a coincidence of this particular distribution, not a general equivalence.

3.4 The interaction that actually matters

Temperature and truncation do different jobs and compose well:

	changes odds	removes options
temperature	✅	❌
top-k / top-p	❌ (only renormalises)	✅

This is why "just lower the temperature" is the wrong fix for occasional nonsense. Lowering T makes the bad token rarer, so the failure moves from every hundredth call to every thousandth — still there, now harder to reproduce, which is worse for debugging. Truncation removes it. The measured version: T=1.5 drew nonsense 29 times in 400; adding top_p=0.85 drew it 0 times while still offering three genuine choices.

The usual production shape is a modest temperature for naturalness plus top-p to cut the tail. Order matters — truncate first, then sample from what remains.

3.5 Determinism, and the limits of `T=0`

T=0 makes the sampler deterministic. It does not guarantee identical API responses, because batching, floating-point non-determinism on GPUs, and model updates behind a version alias all vary independently of your settings. If you need reproducibility, pin the model version, set T=0, use a seed if offered — and still verify rather than assume.

More importantly, T=0 is not an accuracy setting. If the top token is wrong, greedy decoding returns that wrong answer every single time. You've bought consistency, not correctness — and consistent wrongness is easier to miss than intermittent wrongness, because it looks stable. Accuracy comes from what's in the context; see What RAG Is and When to Use It.

3.6 Where these levers stop being the answer

Sampling settings shape which plausible continuation you get. They cannot make the model know something. They are the wrong tool for hallucination (that's grounding), for format compliance (use structured output constraints or a schema), or for length (use a max-token cap and ask in the prompt). And on reasoning tasks the relationship is not monotonic — some benchmarks improve slightly with a little randomness rather than none, so T=0 is not automatically the best choice even for correctness-sensitive work. Measure it.

4. The math

4.1 Temperature

  given probabilities p_i, temperature T > 0:

      scaled_i = exp( log(p_i) / T )
      p'_i     = scaled_i / sum(scaled)

  T -> 0+   the largest p dominates entirely      (argmax)
  T  = 1    p' == p                               (unchanged)
  T -> inf  every p' -> 1/n                       (uniform)

Note it's monotonic: temperature never changes the ranking, only the gaps. The most likely token stays the most likely at every T.

4.2 Truncation

  top_k(p, k):   keep the k largest, renormalise so they sum to 1

  top_p(p, P):   sort descending, accumulate until the running total >= P,
                 keep that set, renormalise
                 -> the SIZE of the kept set varies with the distribution

4.3 Worked example

One distribution: mat 0.45, rug 0.25, floor 0.15, carpet 0.08, table 0.04, moon 0.02, xylophone 0.01. The last two are the tail we don't want.

Temperature reshapes, keeping all seven options:

  T=0.0                  1 choices   mat:1.000
  T=0.5                  7 choices   mat:0.684  rug:0.211  floor:0.076  carpet:0.022
  T=1.0                  7 choices   mat:0.450  rug:0.250  floor:0.150  carpet:0.080
  T=1.5                  7 choices   mat:0.348  rug:0.235  floor:0.167  carpet:0.110

At T=0.5 the leader climbs from 0.45 to 0.68 and carpet falls from 0.08 to 0.02 — but it is still there. Only T=0 collapses to a single choice.

Truncation removes options:

  top_k=3                3 choices   mat:0.529  rug:0.294  floor:0.176
  top_p=0.85             3 choices   mat:0.529  rug:0.294  floor:0.176

Seven becomes three. The kept probabilities are renormalised, so mat rises from 0.45 to 0.53 without any temperature change.

And what you actually observe over 400 draws:

  setting                distinct   nonsense hits   most common
  greedy (T=0)                  1               0   'mat'
  T=0.5                         6               1   'mat'
  T=1.0                         7              14   'mat'
  T=1.5                         7              29   'mat'
  T=1.0 + top_p=0.85            3               0   'mat'

Three readings. Greedy gives 1 distinct outcome in 400 — fully deterministic. Nonsense rises with temperature: 1 → 14 → 29 hits as T goes 0.5 → 1.0 → 1.5, because flattening makes the tail reachable. And the last row is the point: top_p=0.85 produced zero nonsense while still varying across three tokens. Temperature couldn't achieve that at any setting, because it demotes the tail rather than deleting it.

5. Real code

"""Greedy, temperature, top-k and top-p, applied to one fixed distribution."""
import math
import random
from collections import Counter

# One model output: the probability of each candidate next token.
DIST = {
    "mat":      0.45,
    "rug":      0.25,
    "floor":    0.15,
    "carpet":   0.08,
    "table":    0.04,
    "moon":     0.02,   # implausible but non-zero -- the long tail
    "xylophone": 0.01,  # nonsense, and still reachable
}


def rescale(dist: dict[str, float], temperature: float) -> dict[str, float]:
    """Temperature reshapes the distribution BEFORE sampling.
    T<1 sharpens toward the top token, T>1 flattens toward uniform."""
    if temperature <= 0:                          # T=0 means 'always the argmax'
        best = max(dist, key=dist.get)
        return {best: 1.0}
    logits = {t: math.log(p) / temperature for t, p in dist.items()}
    m = max(logits.values())
    exp = {t: math.exp(v - m) for t, v in logits.items()}
    z = sum(exp.values())
    return {t: v / z for t, v in exp.items()}


def top_k(dist: dict[str, float], k: int) -> dict[str, float]:
    """Keep the k most likely, drop the rest, renormalise."""
    kept = dict(sorted(dist.items(), key=lambda kv: -kv[1])[:k])
    z = sum(kept.values())
    return {t: p / z for t, p in kept.items()}


def top_p(dist: dict[str, float], p: float) -> dict[str, float]:
    """Keep the smallest set whose probabilities sum to >= p (nucleus sampling).
    Unlike top-k, the set SIZE adapts to how confident the model is."""
    kept, run = {}, 0.0
    for t, prob in sorted(dist.items(), key=lambda kv: -kv[1]):
        kept[t] = prob
        run += prob
        if run >= p:
            break
    z = sum(kept.values())
    return {t: q / z for t, q in kept.items()}


def sample(dist: dict[str, float], rng: random.Random) -> str:
    r, acc = rng.random(), 0.0
    for t, p in dist.items():
        acc += p
        if r <= acc:
            return t
    return next(reversed(dist))


def show(label: str, dist: dict[str, float]) -> None:
    top = sorted(dist.items(), key=lambda kv: -kv[1])[:4]
    body = "  ".join(f"{t}:{p:.3f}" for t, p in top)
    print(f"  {label:<22} {len(dist)} choices   {body}")


print("THE MODEL'S RAW OUTPUT")
show("as produced", DIST)

print("\nTEMPERATURE reshapes it (same tokens, different odds)")
for T in (0.0, 0.5, 1.0, 1.5):
    show(f"T={T}", rescale(DIST, T))

print("\nTRUNCATION removes the tail entirely")
show("top_k=3", top_k(DIST, 3))
show("top_p=0.85", top_p(DIST, 0.85))

print("\nWHAT YOU ACTUALLY GET -- 400 samples per setting")
print(f"  {'setting':<22} {'distinct':>8}  {'nonsense hits':>14}   most common")
NONSENSE = {"moon", "xylophone"}
for label, d in [
    ("greedy (T=0)",      rescale(DIST, 0.0)),
    ("T=0.5",             rescale(DIST, 0.5)),
    ("T=1.0",             DIST),
    ("T=1.5",             rescale(DIST, 1.5)),
    ("T=1.0 + top_p=0.85", top_p(DIST, 0.85)),
]:
    rng = random.Random(0)
    got = Counter(sample(d, rng) for _ in range(400))
    junk = sum(got[t] for t in NONSENSE)
    print(f"  {label:<22} {len(got):>8}  {junk:>14}   {got.most_common(1)[0][0]!r}")


def draw(dist: dict[str, float], n: int = 400, seed: int = 0) -> Counter:
    """One rng for the whole run -- constructing it inside the loop would reseed
    it on every draw and return the same token n times."""
    rng = random.Random(seed)
    return Counter(sample(dist, rng) for _ in range(n))


greedy_out = draw(rescale(DIST, 0.0))
hot = draw(rescale(DIST, 1.5))
nucleus = draw(top_p(DIST, 0.85))

# Greedy is deterministic: one outcome, every time.
assert len(greedy_out) == 1 and greedy_out.most_common(1)[0][0] == "mat"
# Raising temperature widens the spread and reaches further into the tail.
assert len(hot) >= len(greedy_out)
assert sum(hot[t] for t in NONSENSE) > 0
# top-p removes the nonsense outright, rather than merely making it unlikely.
assert all(t not in nucleus for t in NONSENSE)
assert "moon" not in top_p(DIST, 0.85)
print("\ngreedy: 1 possible output. T=1.5: reaches nonsense. top_p: tail deleted.")
print("all assertions passed")

# Output:
#   THE MODEL'S RAW OUTPUT
#     as produced            7 choices   mat:0.450  rug:0.250  floor:0.150  carpet:0.080
#
#   TEMPERATURE reshapes it (same tokens, different odds)
#     T=0.0                  1 choices   mat:1.000
#     T=0.5                  7 choices   mat:0.684  rug:0.211  floor:0.076  carpet:0.022
#     T=1.0                  7 choices   mat:0.450  rug:0.250  floor:0.150  carpet:0.080
#     T=1.5                  7 choices   mat:0.348  rug:0.235  floor:0.167  carpet:0.110
#
#   TRUNCATION removes the tail entirely
#     top_k=3                3 choices   mat:0.529  rug:0.294  floor:0.176
#     top_p=0.85             3 choices   mat:0.529  rug:0.294  floor:0.176
#
#   WHAT YOU ACTUALLY GET -- 400 samples per setting
#     setting                distinct   nonsense hits   most common
#     greedy (T=0)                  1               0   'mat'
#     T=0.5                         6               1   'mat'
#     T=1.0                         7              14   'mat'
#     T=1.5                         7              29   'mat'
#     T=1.0 + top_p=0.85            3               0   'mat'
#
#   greedy: 1 possible output. T=1.5: reaches nonsense. top_p: tail deleted.
#   all assertions passed

The draw helper exists because of a bug worth avoiding: building random.Random(0) inside the comprehension reseeds it on every draw, so all 400 samples come back identical. One generator per run.

6. Real-world example

A team ran a classifier that read a support message and returned one of nine category labels. It was correct about 94% of the time, which was fine, and roughly one call in two hundred returned something that wasn't a label at all — a short sentence, or a label with an explanation attached. Downstream parsing threw, the message went to a dead-letter queue, and someone triaged it by hand.

The fix attempted first was lowering temperature: 1.0 → 0.7 → 0.3. The failure rate fell each time and never reached zero. At 0.3 it was rare enough to look solved and frequent enough to keep filling the queue — and now much harder to reproduce, which made it worse to work on.

Temperature was the wrong lever. Every non-label continuation still had non-zero probability at any T above 0; lowering T only pushed it further down the tail. The tokens were still on the rack.

Two changes fixed it properly. They set top_p low enough to cut the tail outright, which made the malformed continuations unreachable rather than unlikely. And they added a stop sequence plus a tight max_tokens, so even a wandering generation was truncated to something the parser could handle. Failures went to zero and stayed there.

The instructive part is that they'd also considered temperature=0, which would have removed the variability — and would have been the wrong instinct for a subtler reason. Their 6% misclassification rate was unaffected by any of this. Sampling settings decide which plausible continuation you get; they have nothing to say about whether the model understood the message. Conflating "unstable output" with "wrong output" sent them chasing the wrong metric for a week.

7. Interview questions companies actually ask

Q1. What does temperature actually do? It rescales the model's scores before a token is chosen, sharpening the distribution below 1 and flattening it above 1. Crucially it only changes the odds — every non-zero token remains reachable at any T > 0, and the ranking never changes. T=0 is a special case meaning "take the argmax", since dividing by zero is undefined.

Q2. Difference between temperature and top-p? Temperature changes probabilities but removes nothing; top-p removes options and then renormalises. That's why temperature is the wrong fix for occasional nonsense — it makes the bad token rarer and keeps it possible. In the measured run, T=1.5 drew nonsense 29 times in 400 while top_p=0.85 drew it zero times and still varied across three tokens.

Q3. Top-k or top-p? Top-p usually, because the size of the kept set adapts to the model's confidence: one token when it's certain, several when it's torn. Top-k keeps a fixed count, so it drags in near-ruled-out tokens on confident steps and cuts reasonable ones on uncertain steps. k can't know how confident the model is at that step; p responds to it.

Q4. Does temperature=0 give you reproducible output? It makes the sampler deterministic, which isn't the same thing. Batching, GPU floating-point non-determinism, and silent model updates behind a version alias all vary independently. Pin the model version, set T=0, use a seed if the provider offers one — then verify empirically rather than assuming.

Q5. Should you set temperature=0 for accuracy-critical work? Not automatically, and the reasoning matters. T=0 buys consistency, not correctness — if the top token is wrong you now get that wrong answer every time, which is easier to miss than intermittent wrongness because it looks stable. Some reasoning benchmarks are also slightly better with a little randomness than none. Accuracy comes from what's in the context, not from the sampler.

Q6. Your model occasionally emits malformed output. Which knob? Not temperature — that only makes it rarer. Truncate the tail with top-p so the malformed continuations become unreachable, add stop sequences, and cap max tokens. If the output must satisfy a schema, use structured-output constraints rather than sampling settings, since those make invalid output impossible instead of improbable.

Q7. When would you want a high temperature? When variety is the product: brainstorming, generating multiple distinct options, creative drafting, or synthetic data where you need diversity rather than the single most likely phrasing. Pair it with top-p, so you get variety among plausible continuations instead of variety that includes the tail.

8. When to use / tradeoffs

Use temperature=0 (greedy) when:

Output feeds a parser — classification, extraction, structured fields
You need the same input to give the same output
You're debugging and need reproducibility
Any variation is noise rather than value

Use moderate temperature + top-p when:

A human reads the output and repetition feels robotic
You want several genuinely different options
Naturalness matters and small variation is harmless

Use high temperature when:

Variety is the deliverable — brainstorming, synthetic data
You'll filter or rank the results afterwards

Situation	Why the naive fix fails	Do this instead
Occasional nonsense token	Temperature demotes, never deletes	Lower top-p; add stop sequences
Output must match a schema	Sampling can't enforce structure	Structured outputs / JSON mode
Output too long	Not a sampling problem	`max_tokens` + ask in the prompt
Wrong answers	Sampling chooses among plausible ones	Retrieval, grounding, better prompt
Needs reproducibility	`T=0` alone isn't sufficient	Pin version, seed, verify
Repetitive, robotic text	`T=0` on a flat distribution	Raise `T` a little, add top-p
Same wrong answer every time	`T=0` made it stable, not correct	Fix the context, not the sampler

Honest limits. §5 samples one hand-made 7-token distribution, where a real step is over tens of thousands of tokens whose tail is far longer and flatter — so the direction of every result here transfers and the magnitudes do not. The "nonsense hits" counts come from a single seed and 400 draws; they'd move on a different seed, and the ranking of settings is the robust part rather than the numbers. The temperature implementation applies log(p)/T to probabilities, which is equivalent to the usual logits/T up to a constant and is what lets the example run without a model — a real implementation works on raw logits before any normalisation. Providers also differ in whether T and top_p are meant to be tuned together, and some documentation advises changing only one; that's a defensible convention rather than a mathematical constraint, but follow your provider's guidance over this article's. Finally, none of these settings touch factual accuracy, and the §6 team's 6% misclassification rate was untouched by all of them.

The model outputs a probability per token. Sampling is a separate step that picks one, and it lives in your API call, not the model.
temperature=0 is deterministic — 1 distinct outcome in 400 draws. Below 1 sharpens, above 1 flattens; the ranking never changes.
Temperature never removes an option. Every non-zero token stays reachable at any T > 0.
Top-p / top-k do remove options. Measured: T=1.5 hit nonsense 29/400; top_p=0.85 hit it 0/400 while keeping three choices.
So "lower the temperature" is the wrong fix for nonsense — it makes the failure rarer and harder to reproduce. Truncation makes it impossible.
Top-p adapts the kept-set size to the model's confidence; top-k can't. That's why top-p is the usual default.
Standard shape: moderate temperature for naturalness plus top-p to cut the tail. Truncate first, then sample.
T=0 gives consistency, not correctness — a wrong top token becomes reliably wrong, which is easier to miss.
These settings choose among plausible continuations. They do nothing for accuracy, schema compliance, or length.

How LLMs Work — where the distribution comes from, and why there's always a top token
Tokenization — what the candidates being sampled actually are
Reasoning Budgets: When Thinking Tokens Are Waste — capping output length, the lever sampling doesn't give you
API Best Practices — where these parameters sit in a real request
Context Fundamentals — what the model conditions on before sampling
What RAG Is and When to Use It — the actual fix for wrong answers
Hallucination Detection & Grounding — why temperature is not a hallucination control
Guardrails & Output Validation — enforcing output shape properly
Agent Evaluation — measuring whether a settings change helped

Resources

Holtzman et al. (2019) — The Curious Case of Neural Text Degeneration, arXiv:1904.09751 — the paper that introduced nucleus (top-p) sampling and showed why pure likelihood maximisation produces degenerate text: https://arxiv.org/abs/1904.09751
Fan, Lewis & Dauphin (2018) — Hierarchical Neural Story Generation, arXiv:1805.04833 — where top-k sampling was introduced: https://arxiv.org/abs/1805.04833
Ackley, Hinton & Sejnowski (1985) — A Learning Algorithm for Boltzmann Machines, Cognitive Science 9(1) — the origin of the temperature parameter in this form.
Wang et al. (2022) — Self-Consistency Improves Chain of Thought Reasoning in Language Models, arXiv:2203.11171 — evidence that sampling several times and voting can beat greedy decoding on reasoning: https://arxiv.org/abs/2203.11171
Jurafsky & Martin — Speech and Language Processing (3rd ed. draft), the chapter on text generation and decoding strategies: https://web.stanford.edu/~jurafsky/slp3/
Companion notebook — sweep temperature and top-p over your own distribution and watch the tail-reach and diversity curves cross.

Runnable notebook

Run it end to end — the mock model needs no API key; add your own key for the real Claude section.

← Previous: How LLMs Work