Structured Outputs

TL;DR — There are two different things people call "getting JSON out of a model", and they are not on the same scale. Asking — a prompt, examples, a schema pasted in — makes invalid output unlikely. Constraining — masking any next token that would break the grammar — makes it impossible. The measured difference below: asking fails 14.65% of the time at T=1.0 and still 0.10% at T=0.3, while constrained decoding fails 0.00%, because the invalid branches are unreachable rather than merely improbable. A rare failure is also worse to live with than a common one, since it's harder to reproduce. The limit worth stating up front: constraining guarantees the shape, never the value — in the same run, 72% of the perfectly valid outputs said true, and nothing here has an opinion about whether that was correct.

---

1. Simple explanation

You want a model's output to go straight into code — a JSON object, a specific set of fields, the right types. Free text won't do.

The obvious approach is to ask: describe the format, show examples, maybe paste in the schema. This works most of the time, which is the problem. "Most of the time" means your parser throws occasionally, at a rate low enough to look like noise and high enough to fill an error queue.

The other approach changes the mechanism. Since the model produces a probability for every possible next token, you can look at the partially-generated output, work out which tokens would make it invalid, and set those probabilities to zero before one is chosen. The model cannot emit a malformed field name because that path no longer exists.

Analogy — asking someone to fill in a form versus giving them a form. You can hand over a blank sheet with careful written instructions about which fields to include and in what order. Most people comply; some add a covering note, some reorder things, one writes a paragraph. Or you hand them a form with boxes: name here, date here. The second isn't "clearer instructions" — it's a different kind of thing, because non-compliance is no longer available. The analogy carries the limit too: a correctly filled form can still contain the wrong date.

---

2. Diagram

TWO MECHANISMS, NOT TWO EFFORT LEVELS

  ASKING                              CONSTRAINING
  ───────────────────────────         ────────────────────────────
  prompt: "reply with JSON"           at each step, the grammar says
  + examples                          which tokens are legal NEXT
  + schema in the text                          │
          │                                     ▼
          ▼                           mask the illegal ones to zero
  model emits whatever                          │
  it finds most likely                          ▼
          │                           sample only from what remains
          ▼                                     │
  hope it parses                                ▼
                                      cannot produce invalid output


MEASURED — same distribution, 4000 draws

  setting                 % invalid
  T=1.0 (as produced)        14.65%   ████████████████
  T=0.7                       5.12%   █████
  T=0.3                       0.10%   ▏          ← rare, NOT zero
  T=0.0 (greedy)              0.00%              ← but no variation at all
  constrained                 0.00%   ← and full variation retained


WHAT THE MASK DOES, ONE TOKEN AT A TIME

  produced so far:  {"ok":
  legal next:       true | false                 ← grammar says boolean
  model wanted:     true(0.62) True(0.02) yes(0.01) ...
  after masking:    true(0.97) false(0.03)
                          ▲
                'True' was never an option, so the
                Python-literal failure cannot occur


AND THE LIMIT

  constrained output:  {"ok": true}   ← always parses  ✅
                              ▲
                        is that the RIGHT answer?
                        Nothing here can tell you.
                        72% of valid outputs said true.

---

3. How it works

3.1 Why asking plateaus

Every malformed output you've seen — a code fence, a "Sure!" preamble, True instead of true, an unquoted key — is a token sequence with non-zero probability. Prompting pushes those probabilities down. It cannot push them to zero.

So the failure rate falls as you improve the prompt or lower the temperature, and it asymptotes above zero. §4 shows the curve: 14.65% → 5.12% → 0.10%. Each step feels like progress and none of them arrives.

Worse, a rare failure is harder to work with than a common one. At 15% you reproduce it immediately; at 0.1% it appears in production, resists reproduction, and gets closed as "transient".

3.2 What constraining actually does

The model still produces a distribution over every token. Constrained decoding inserts a step between that and the choice:

1. Track what has been generated so far.
2. Ask a grammar or state machine which tokens are legal next.
3. Set every illegal token's probability to zero and renormalise.
4. Sample from what remains.

After {"ok": a JSON-boolean grammar permits true and false and nothing else. True isn't unlikely — it is absent. That's the entire difference.

This runs per token, so a valid document is built incrementally with no possibility of drift. It's usually exposed as a "JSON mode", a "response format" parameter, or a schema you supply; under the hood it's token masking.

3.3 Function calling is the same mechanism

Tool use — describing functions and having the model choose one with arguments — is structured output wearing a different name. The tool definition is a schema, the model's call is a constrained generation against it, and you get a validated object rather than prose to parse.

Which is why the two topics share failure modes: an under-specified schema produces plausible-but-useless arguments in exactly the way an under-specified output schema produces plausible-but-useless fields. See Tool Use and MCP Fundamentals.

3.4 What it costs you

Constraining is not free:

That third row deserves attention. If your schema has no way to express uncertainty, you have removed the model's ability to signal it. Give the schema a null, a confidence, or an explicit unknown — otherwise you've converted "I don't know" into a fabricated value, which is a worse outcome than a parse error.

3.5 Shape is not correctness

The measured run makes this concrete: constrained output parses 100% of the time, and 72% of those valid outputs said true. Whether true was the right answer is a completely separate question that no grammar can address.

So structured outputs remove one class of bug — malformed responses — and leave the more important one untouched. You still need the value checked: against a retrieved source (Grounding and Abstention by Design: Refusing Before You Generate the Wrong Answer), against a range or an enum, or against a scored evaluation set.

Valid JSON is a floor, not a guarantee.

3.6 Where to reach for something else

If your provider doesn't support constraining, the fallback ladder is: ask well (schema in the prompt plus consistent examples), validate on receipt, and retry once with the validation error fed back. That converges in practice and costs up to double on the retry path — acceptable, but note it's recovery, not prevention.

And if the output isn't really structured — prose, a summary, an explanation — forcing a schema onto it fights the model for no benefit. Structured outputs are for data.

---

4. The math

4.1 Asking versus constraining

  asking:
      P(invalid) = Σ p(t) over all invalid token sequences t
                 > 0   for any temperature above 0

  constraining:
      p'(t) = p(t) / Σ p(valid)   if t is legal
            = 0                    otherwise
      P(invalid) = 0   exactly, by construction

4.2 The retry ladder, when constraining is unavailable

  with per-call failure rate f and one retry:
      P(fail after retry) = f²
      expected calls      = 1 + f

  f = 0.05  ->  0.25% residual failure, 1.05 calls average

Better, still not zero — and you pay for it on the retry path.

4.3 Worked example

A distribution over seven candidate completions for a yes/no field. Two are valid JSON matching the schema; five are the failures people actually see — a code fence, a preamble, a Python literal, an unquoted key, plain prose.

Asking, at four temperatures:

ASKING for JSON -- failure rate by temperature
  setting                 % invalid
  T=1.0 (as produced)        14.65%
  T=0.7                       5.12%
  T=0.3                       0.10%
  T=0.0 (greedy)              0.00%

Lowering temperature works, right up until you notice what it costs. T=0.3 is at 0.10% — one call in a thousand, which will still fill a queue at volume and will not reproduce on demand. T=0.0 reaches zero here only because the top candidate happens to be valid; it also eliminates all variation, which is fine for extraction and useless if you wanted variety.

Constraining:

CONSTRAINING the output -- the invalid branches are unreachable
  candidates before 7  ->  after 2
    0.721  {"ok": true}
    0.279  {"ok": false}
  constrained                 0.00%

Seven candidates become two. Both parse. The failure rate is 0.00% not because it's small but because there is no path to an invalid output.

The three approaches side by side:

  approach                  best case   guarantee?
  ask nicely (prompt)           0.10%           no
  ask + retry on error             ~0  no (costs 2x)
  constrained decoding          0.00%          YES

The middle row is the honest fallback: it gets close, and it is recovery rather than prevention, at up to double the calls.

4.4 And the limit

  ...but 72% of valid outputs say ok=true.

Every constrained output is well-formed. 72% of them assert true. Whether that is correct is outside what any of this can determine.

---

5. Real code

"""Asking for JSON vs constraining it: unlikely-invalid against impossible-invalid."""
import json
import random

# What a model might produce for "return {"ok": true} or {"ok": false}".
# Only the first two are valid; the rest are the failures people actually see.
# FENCE is built rather than written literally so this file can itself sit inside
# a markdown code block -- a literal triple backtick would close the fence early.
FENCE = "`" * 3
CANDIDATES = {
    '{"ok": true}':                       0.62,
    '{"ok": false}':                      0.24,
    f'{FENCE}json\n{{"ok": true}}\n{FENCE}': 0.06,   # fenced
    'Sure! {"ok": true}':                 0.04,   # preamble
    '{"ok": True}':                       0.02,   # Python literal, not JSON
    '{ok: true}':                         0.01,   # unquoted key
    'The answer is yes.':                 0.01,   # prose
}
VALID_KEYS = {"ok"}


def parses(text: str) -> bool:
    try:
        obj = json.loads(text)
    except Exception:
        return False
    return isinstance(obj, dict) and set(obj) == VALID_KEYS and isinstance(obj["ok"], bool)


def sample(dist, rng):
    r, acc = rng.random(), 0.0
    for t, p in dist.items():
        acc += p
        if r <= acc:
            return t
    return next(reversed(dist))


def rescale(dist, T):
    if T <= 0:
        return {max(dist, key=dist.get): 1.0}
    import math
    lg = {t: math.log(p) / T for t, p in dist.items()}
    m = max(lg.values())
    e = {t: math.exp(v - m) for t, v in lg.items()}
    z = sum(e.values())
    return {t: v / z for t, v in e.items()}


def constrain(dist):
    """Mask every candidate the grammar forbids, then renormalise.
    This is what constrained decoding does, one token at a time."""
    kept = {t: p for t, p in dist.items() if parses(t)}
    z = sum(kept.values())
    return {t: p / z for t, p in kept.items()}


def trial(dist, n=4000, seed=0):
    rng = random.Random(seed)
    outs = [sample(dist, rng) for _ in range(n)]
    return sum(1 for o in outs if not parses(o)) / n * 100


print("ASKING for JSON -- failure rate by temperature")
print(f"  {'setting':<22} {'% invalid':>10}")
for label, d in [("T=1.0 (as produced)", CANDIDATES),
                 ("T=0.7", rescale(CANDIDATES, 0.7)),
                 ("T=0.3", rescale(CANDIDATES, 0.3)),
                 ("T=0.0 (greedy)", rescale(CANDIDATES, 0.0))]:
    print(f"  {label:<22} {trial(d):>9.2f}%")

print("\n  Lowering temperature makes the failure RARER and never zero --")
print("  except at T=0, which also removes all variation. And a rare")
print("  failure is harder to reproduce, so it is worse to debug.")

print("\nCONSTRAINING the output -- the invalid branches are unreachable")
c = constrain(CANDIDATES)
print(f"  candidates before {len(CANDIDATES)}  ->  after {len(c)}")
for t, p in c.items():
    print(f"    {p:.3f}  {t}")
print(f"  {'constrained':<22} {trial(c):>9.2f}%")

print("\nWHY IT IS DIFFERENT (not just 'better')")
print(f"  {'approach':<24} {'best case':>10}  {'guarantee?':>11}")
print(f"  {'ask nicely (prompt)':<24} {trial(rescale(CANDIDATES, 0.3)):>9.2f}%  {'no':>11}")
print(f"  {'ask + retry on error':<24} {'~0':>10}  {'no (costs 2x)':>11}")
print(f"  {'constrained decoding':<24} {trial(c):>9.2f}%  {'YES':>11}")

# A schema is not the same as a correct answer.
truthy = sum(p for t, p in c.items() if json.loads(t)["ok"] is True)
print(f"\n  ...but {truthy:.0%} of valid outputs say ok=true. Constraining makes the")
print("  SHAPE certain. It says nothing about whether the VALUE is right.")

assert trial(CANDIDATES) > 10
assert trial(rescale(CANDIDATES, 0.3)) > 0        # still fails, just less often
assert trial(rescale(CANDIDATES, 0.3)) < trial(CANDIDATES)
assert trial(c) == 0.0                            # impossible, not improbable
assert len(c) == 2 and all(parses(t) for t in c)
assert abs(sum(c.values()) - 1.0) < 1e-9
print("\nall assertions passed")

# Output:
#   ASKING for JSON -- failure rate by temperature
#     setting                 % invalid
#     T=1.0 (as produced)        14.65%
#     T=0.7                       5.12%
#     T=0.3                       0.10%
#     T=0.0 (greedy)              0.00%
#
#     Lowering temperature makes the failure RARER and never zero --
#     except at T=0, which also removes all variation. And a rare
#     failure is harder to reproduce, so it is worse to debug.
#
#   CONSTRAINING the output -- the invalid branches are unreachable
#     candidates before 7  ->  after 2
#       0.721  {"ok": true}
#       0.279  {"ok": false}
#     constrained                 0.00%
#
#   WHY IT IS DIFFERENT (not just 'better')
#     approach                  best case   guarantee?
#     ask nicely (prompt)           0.10%           no
#     ask + retry on error             ~0  no (costs 2x)
#     constrained decoding          0.00%          YES
#
#     ...but 72% of valid outputs say ok=true. Constraining makes the
#     SHAPE certain. It says nothing about whether the VALUE is right.
#
#   all assertions passed

constrain masks whole candidate strings so the idea fits on a page; a real implementation masks one token at a time against a grammar or a compiled state machine. The effect is identical — illegal continuations get probability zero — and doing it per token is what lets it build arbitrarily long valid documents.

---

6. Real-world example

A team extracted structured fields from documents with a prompt containing the schema and three examples. Parse failures ran at about 2%, handled by a retry, and everyone considered it solved.

Then they added a field for an optional review date. The schema said the key was required and the value could be a date string or null.

Extraction accuracy on that field was poor in a specific way: when no date was present, the model returned plausible-looking dates rather than null. Roughly one in seven absent dates came back as a confident fabrication, and downstream systems scheduled reviews that had never been agreed.

Two causes, and both are §3.4. None of the three examples showed the null case — every example had a date — so the model had never seen the absent case demonstrated. And when they later moved to a strict JSON mode, they made the field "type": "string" rather than allowing null, which removed the model's only way to say "not present". Constraining then guaranteed it produced a string, and the only strings available were invented ones.

The fix had two parts: allow null in the schema, and add an example showing it. Fabrications went to near zero.

The lesson generalises past JSON. A schema that cannot express uncertainty converts "I don't know" into a fabrication, and constrained decoding enforces that conversion rigorously. Always give the shape somewhere to put "absent", "unknown", or "not confident" — otherwise you have used a correctness tool to guarantee a wrong answer.

---

7. Interview questions companies actually ask

Q1. What's the difference between asking for JSON and constrained decoding? Asking lowers the probability of invalid output; constraining removes the possibility. Every malformed response — a code fence, a preamble, True instead of true — is a token sequence with non-zero probability, and prompting cannot drive that to zero. Constrained decoding masks illegal tokens before sampling, so the invalid branches don't exist. Measured: 0.10% versus 0.00%.

Q2. Why not just set temperature to 0? It removes variation, not invalidity. It reached zero in the example only because the top candidate happened to be valid — with a malformed top candidate you get that malformed output every single time, reliably. And you lose all variety, which matters if the task wanted any.

Q3. How does constrained decoding actually work? At each step the model produces a distribution over all tokens. A grammar or state machine, tracking what's been generated, determines which tokens are legal next; the rest have their probability set to zero and the remainder is renormalised. After {"ok": a JSON grammar allows true and false and nothing else, so True isn't unlikely — it's absent.

Q4. What does constraining not give you? Correctness. It guarantees the shape and says nothing about the values. In the worked example every constrained output parsed and 72% said true, and no grammar can tell you whether that was right. You still need the value checked against a source, a range, or a scored evaluation set.

Q5. When would you avoid it? When the provider doesn't support it, when the schema is recursive or highly dynamic, when the output genuinely isn't structured data, and when forcing a shape degrades the content — a model that would say "I'm not sure" being forced into a confident field. Fallback: schema in the prompt, consistent examples, validate on receipt, retry once with the error fed back.

Q6. Your model returns valid JSON with fabricated values. What went wrong? Almost certainly the schema has no way to express absence, and possibly no example shows it either. If a field is a required string with no null allowed, you have removed the model's only way to signal "not present" — and constraining then enforces that a string is produced, so the only strings available are invented. Allow null, and demonstrate it in an example.

Q7. How is function calling related? It's the same mechanism. A tool definition is a schema, the model's call is a constrained generation against it, and you get a validated object instead of prose to parse. Which is why they share failure modes: an under-specified tool schema produces plausible-but-useless arguments exactly as an under-specified output schema produces plausible-but-useless fields.

---

8. When to use / tradeoffs

Use constrained decoding when:

• Output feeds code directly — extraction, classification, tool arguments
• A parse failure has real cost
• The schema is fixed and reasonably simple
• Your provider supports it

Use ask-and-validate when:

• Constraining is unavailable
• The schema is recursive or changes per request
• You can absorb an occasional retry

Use neither when:

• The output is prose, a summary, or an explanation

Honest limits. The example masks whole candidate strings, where a real implementation masks per token against a compiled grammar — the effect is the same but the engineering is considerably more involved, and support for it varies widely across providers and schema features. The seven candidates and their probabilities are illustrative: your real failure mix and rate will differ, and the specific 14.65% is a property of this made-up distribution rather than a benchmark. The claim that constraining is exactly 0% assumes the grammar is correct and the implementation is sound; a buggy grammar produces confidently valid nonsense, which is not obviously better than a parse error. And §3.4's warning about quality degradation is real but hard to quantify — there's evidence in both directions on whether constraining hurts content, so measure it on your task rather than assuming either way.

---

9. Summary + related articles

• Asking makes invalid output unlikely. Constraining makes it impossible. Different mechanisms, not different effort levels.
• Measured: asking fails 14.65% at T=1.0, 0.10% at T=0.3; constrained fails 0.00% — because there is no path to an invalid output.
• A rare failure is worse to live with than a common one: same queue, no reproduction.
• T=0 reaches zero only when the top candidate happens to be valid, and it removes all variation.
• Constraining works by masking illegal next tokens against a grammar, per token, then renormalising.
• Function calling is the same mechanism with a different name, and shares the failure modes.
• Shape is not correctness. Every constrained output parsed; 72% said true; nothing here knows if that's right.
• A schema with no way to express absence converts "I don't know" into a fabrication — and constraining enforces it rigorously.
• Fallback when unavailable: schema in the prompt, consistent examples, validate, retry once. Recovery, not prevention, at up to 2× calls.

Related:

• Prompt Fundamentals — the examples and delimiters that make asking work as well as it can
• Sampling and Temperature — why lowering temperature plateaus above zero
• How LLMs Work — the per-token distribution that masking operates on
• Structured Extraction from Vision-Language Models — the same problem with images as input
• Tool Use — function calling as constrained generation
• MCP Fundamentals — tool schemas in a protocol
• Guardrails & Output Validation — validating values, not just shapes
• Grounding and Abstention by Design: Refusing Before You Generate the Wrong Answer — giving the model a way to say "I don't know"
• API Best Practices — retries and validation around the call

Resources

• Willard & Louf (2023) — Efficient Guided Generation for Large Language Models, arXiv:2307.09702 — the finite-state-machine approach behind several constrained-decoding libraries: https://arxiv.org/abs/2307.09702
• Geng et al. (2023) — Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning, arXiv:2305.13971 — grammar-based constraining and its effect on task quality: https://arxiv.org/abs/2305.13971
• Tam et al. (2024) — Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models, arXiv:2408.02442 — evidence that forcing a format can degrade content, the §3.4 caveat: https://arxiv.org/abs/2408.02442
• JSON Schema specification — what your schema can and cannot express, including nullability: https://json-schema.org/
• OpenAI — structured outputs guide, including which schema features are supported: https://platform.openai.com/docs/guides/structured-outputs
• Anthropic — tool use documentation; the same mechanism applied to function arguments: https://docs.claude.com/en/docs/agents-and-tools/tool-use/overview