TL;DR — Reasoning models produce hidden "thinking" tokens before their visible answer. Those tokens are billed at the output rate and are decoded one at a time, exactly like visible output — so they cost money and wall-clock time while the user sees nothing. On a task with a short, format-constrained answer, thinking can be the majority of both: in the worked example below it is 53% of the cost and 75% of the latency for a forty-token reply. Most APIs let you set a thinking budget, and some let you set it to zero — but the top reasoning tiers often enforce a minimum, so "just turn it off" silently does nothing and the real fix is choosing a different tier. It stops being waste the moment the task genuinely requires multi-step work, which is why the budget is a per-task decision that has to be validated against a scored test set rather than set globally by taste.
1. Simple explanation
A reasoning model works through a problem before answering. The working-out is generated as tokens, just like the answer, but it is not shown to the user — the API returns the answer and a count of how many hidden tokens it took to get there.
Two facts about those hidden tokens determine everything else. They are billed at the same rate as visible output, which is typically several times the input rate. And they are produced serially, one token after another, which is the slow part of any model call — so a thousand hidden tokens takes about as long as writing a thousand words of answer.
That is a superb trade when the problem needs it. Asking a model to check a proof, reconcile conflicting sources, or plan a multi-step action gets meaningfully better with room to think. It is a terrible trade when the answer was always going to be one sentence chosen from a short list of known facts, because you are paying full price for deliberation that changes nothing.
Analogy — a consultant who bills for thinking time. For a genuinely hard restructuring question, the two hours of thinking is the value you are buying. For "what are your opening hours," those same two hours are billed, add two hours of delay, and produce the same sentence a receptionist would have given instantly. The consultant is not defective — you briefed them wrongly. The analogy carries the mechanism precisely: the hidden work is metered at the premium rate and sits on the critical path, so the only question is whether the task's difficulty justifies it.
2. Diagram
WHAT THE USER SEES vs WHAT YOU PAY FOR
user sees: "We close at six on weekdays."
└──── 40 tokens ────┘
you are billed for: [·············· 300 hidden tokens ··············][40]
└──────── invisible, output rate, serial ──────┘
53% of cost 75% of the wall clock
THE SAME PROMPT ON TWO TIERS
reasoning tier, budget=300
prefill ▏ ░░░░░░░░░░░░░░░ thinking ░░░░░░░░░░░░░░░ ▏ answer ▏
0.6s 3.0s 0.4s = 4.00 s
$0.00565
small tier, budget=0
prefill ▏ answer ▏
0.6s 0.16s = 0.76 s
$0.00020
identical 40-token answer. 29x cheaper. 3.24 s faster.
THE TRAP
budget = 0 on a reasoning tier
│
└──▶ clamped to the tier minimum (e.g. 128)
no error, no warning, bill unchanged
you did not turn anything off
3. How it works
3.1 Where thinking tokens sit in the bill
A model call has three token counts, and most usage dashboards show them separately:
| bucket | billed at | generated | visible |
|---|---|---|---|
| input / prompt | input rate (low) | in parallel (prefill) | you wrote it |
| thinking | output rate (high) | serially | no |
| output / completion | output rate (high) | serially | yes |
The asymmetry that matters: input is cheap and fast because the whole prompt is processed in parallel, while both thinking and output are expensive and slow because they are produced token by token. So a 2000-token prompt is a smaller problem than a 500-token thinking trace, in both dimensions, which is the opposite of most people's intuition about "big prompts are the expensive thing."
3.2 The budget parameter, and the minimum that ignores you
Current APIs expose a thinking or reasoning budget — a token cap, or a coarse effort level. The pattern to know is that support for zero is not uniform across a provider's own line-up. Smaller and mid-tier models generally accept a budget of zero and genuinely stop deliberating. The flagship reasoning tiers frequently enforce a floor, so a request for zero is clamped upward.
The failure mode is quiet. No exception, no warning field, and the response looks normal — you simply keep being billed for thinking you thought you had disabled. Anyone reasoning from the code alone will conclude the setting works. The only way to know is to read the returned usage counts, which is the first argument for metering every call.
Practically: if a task does not need reasoning, the lever is the model tier, not the budget parameter. The budget parameter is for tuning how much on tasks that need some.
3.3 Caps you should set alongside the budget
A call made with no generation controls at all is the common starting state, and it leaves three things unbounded:
- no thinking budget — deliberation runs as long as the model likes
- no maximum output length — a one-sentence task can return five paragraphs
- no stop sequences — nothing halts a model that starts writing a list
For a task with a known answer shape, all three should be set. The output cap is the one with the most direct latency effect, because output tokens are serial: halving expected output roughly halves generation time. It is also a safety net rather than a style control — the model should be asked for one sentence and capped at two, so a drifting response gets truncated instead of read aloud.
3.4 Examples beat instructions for shaping length
If the goal is short, consistent answers, a few worked examples do it more reliably than prose rules, and usually for the same number of input tokens. Instructions like "answer in one short sentence, no lists, no markdown" are wordy, easy to drift from, and cost tokens on every call. Four to six examples that are all one short sentence demonstrate the target instead of describing it, and they cap output length by imitation.
They also compose well with caching: examples are static, so they belong in the cached prefix and cost a fraction after the first call. See RAG Cost Optimization: Find the Step That Runs Forty Times §3.2 for the per-role framing this fits into.
3.5 The interaction with caching, and the mistake that breaks it
Static prompt content — instructions, approved facts, few-shot examples — can be cached and re-billed at a large discount. Two rules make it work:
- Put the static content first. Caching matches on a prefix, so anything variable at the front invalidates everything behind it.
- Keep the prefix genuinely static. The classic bug is interpolating something that looks constant but is not — a timestamp, a greeting that changes with the hour, a request ID. The prefix then differs on most calls and never gets a cache hit, and because the cost difference is a discount rather than an error, nothing surfaces the problem.
3.6 When thinking is worth every token, and where this stops applying
Reasoning budgets pay for themselves on tasks with a verifiable multi-step structure: mathematical and logical work, code that must compile, planning with constraints, reconciling sources that disagree, and any judging task where you want the rationale as an artefact. The signal to look for is whether the task has intermediate steps that can be wrong independently of the final answer.
This whole framing assumes you can characterise your task's difficulty up front. For genuinely mixed traffic — some trivial requests, some hard ones, arriving on the same endpoint — a single global budget is wrong for most of it, and the answer is routing rather than tuning. It also assumes hidden token counts are reported; if a provider does not return them you cannot do any of this arithmetic, and you should treat that as a reason to be cautious rather than an excuse to guess.
4. The math
4.1 Cost of a call
C = (in_fresh * r_in + in_cached * r_in * discount) / 1e6
+ (thinking + out) * r_out / 1e6
where r_in, r_out are per-million rates and discount is typically ~0.25
The term to stare at is thinking * r_out. Thinking is multiplied by the output rate, which is commonly 8–10x the input rate, so a few hundred hidden tokens can outweigh a couple of thousand prompt tokens.
4.2 Latency of a call
T = in_total / prefill_rate + (thinking + out) / decode_rate
prefill_rate >> decode_rate (parallel vs serial)
Since prefill_rate is often an order of magnitude larger than decode_rate, the second term dominates whenever thinking is non-trivial. This is why prompt size is a weak latency lever and generated-token count is a strong one.
4.3 Worked example
A short factual reply: 1800 input tokens (instructions plus retrieved facts plus a little history), 40 output tokens. The answer length never changes in what follows. Rates are illustrative — substitute your provider's current published figures.
reasoning tier: $1.25 /M in, $10.00 /M out, ~100 tok/s decode, min thinking 128
small tier: $0.10 /M in, $0.40 /M out, ~250 tok/s decode, min thinking 0
prefill: ~3000 tok/s for both
Sweeping the budget on the reasoning tier:
thinking cost of which time of which
tokens /call thinking /call thinking
128 $ 0.00393 33% 2.28s 56%
128 $ 0.00393 33% 2.28s 56%
300 $ 0.00565 53% 4.00s 75%
800 $ 0.01065 75% 9.00s 89%
2048 $ 0.02313 89% 21.48s 95%
Two things to read off it. The first two rows are identical because the first was a request for zero, clamped to the tier's minimum of 128 — the parameter was accepted and ignored. And by a budget of 300, three quarters of the elapsed time is invisible deliberation; by 2048, it is 95%, and a forty-token answer takes twenty-one seconds.
Now the same prompt and the same forty-token answer on the small tier with thinking genuinely off:
reasoning, thinking=300 $0.00565 4.00s
small tier, thinking=0 $0.00020 0.76s
---> 29x cheaper, 3.24s faster
Adding prefix caching for 1200 of the 1800 static input tokens at a 25% rate takes the small-tier call to $0.00011 — another 46% off. The compounding matters: tier choice, budget, output cap, and caching are multiplicative, not alternatives.
5. Real code
"""What thinking tokens cost in money and in wall-clock time.
Thinking tokens are billed at the OUTPUT rate and are decoded serially, exactly
like visible output -- but the caller never sees them. Rates below are
illustrative; substitute your provider's current published numbers.
"""
from dataclasses import dataclass
PREFILL_TOK_PER_S = 3000.0 # input is processed in parallel -- cheap in time
@dataclass
class Tier:
name: str
usd_in_per_m: float
usd_out_per_m: float
decode_tok_per_s: float
min_thinking: int # 0 means thinking can be switched off entirely
REASONING = Tier("reasoning tier", 1.25, 10.00, 100.0, min_thinking=128)
CHEAP = Tier("small tier", 0.10, 0.40, 250.0, min_thinking=0)
IN_TOK = 1800 # system prompt + retrieved facts + short history
OUT_TOK = 40 # two spoken sentences. This never changes.
def call(tier: Tier, thinking: int, cached_in: int = 0) -> dict:
thinking = max(thinking, tier.min_thinking)
fresh_in = IN_TOK - cached_in
usd_in = (fresh_in * tier.usd_in_per_m + cached_in * tier.usd_in_per_m * 0.25) / 1e6
usd_out = (thinking + OUT_TOK) * tier.usd_out_per_m / 1e6
secs = IN_TOK / PREFILL_TOK_PER_S + (thinking + OUT_TOK) / tier.decode_tok_per_s
return {
"thinking": thinking,
"usd": usd_in + usd_out,
"usd_thinking": thinking * tier.usd_out_per_m / 1e6,
"secs": secs,
"secs_thinking": thinking / tier.decode_tok_per_s,
}
print(f"Answer length is fixed at {OUT_TOK} output tokens in every row below.\n")
print("REASONING TIER — sweeping the thinking budget")
print(f" {'thinking':>9} {'cost':>9} {'of which':>10} {'time':>8} {'of which':>10}")
print(f" {'tokens':>9} {'/call':>9} {'thinking':>10} {'/call':>8} {'thinking':>10}")
sweep = {}
for t in (0, 128, 300, 800, 2048):
r = call(REASONING, t)
sweep[t] = r
print(f" {r['thinking']:>9} ${r['usd']:>8.5f} "
f"{r['usd_thinking'] / r['usd'] * 100:>9.0f}% {r['secs']:>7.2f}s "
f"{r['secs_thinking'] / r['secs'] * 100:>9.0f}%")
print("\nNote the first two rows are identical: asking for 0 is clamped to 128,")
print("because this tier cannot switch thinking off.\n")
obs = call(REASONING, 300)
fix = call(CHEAP, 0)
print("SIDE BY SIDE — same prompt, same 40-token answer")
for label, tier, r in (("reasoning, thinking=300", REASONING, obs),
("small tier, thinking=0 ", CHEAP, fix)):
print(f" {label} ${r['usd']:.5f} {r['secs']:.2f}s")
print(f" ---> {obs['usd'] / fix['usd']:.0f}x cheaper, "
f"{obs['secs'] - fix['secs']:.2f}s faster")
cached = call(CHEAP, 0, cached_in=1200)
print(f"\nplus caching 1200 static input tokens at 25%: "
f"${cached['usd']:.5f} ({(1 - cached['usd'] / fix['usd']) * 100:.0f}% off input)")
# Asking a reasoning tier for zero thinking does nothing.
assert sweep[0]["thinking"] == 128 and sweep[0]["usd"] == sweep[128]["usd"]
# At the observed budget, thinking is the majority of BOTH cost and time.
assert obs["usd_thinking"] / obs["usd"] > 0.5
assert obs["secs_thinking"] / obs["secs"] > 0.5
# The visible answer is identical; only the invisible part differs.
assert round(obs["usd"] / fix["usd"]) == 29
assert round(obs["secs"], 2) == 4.00 and round(fix["secs"], 2) == 0.76
print("\nall assertions passed")
# Output:
# Answer length is fixed at 40 output tokens in every row below.
#
# REASONING TIER — sweeping the thinking budget
# thinking cost of which time of which
# tokens /call thinking /call thinking
# 128 $ 0.00393 33% 2.28s 56%
# 128 $ 0.00393 33% 2.28s 56%
# 300 $ 0.00565 53% 4.00s 75%
# 800 $ 0.01065 75% 9.00s 89%
# 2048 $ 0.02313 89% 21.48s 95%
#
# Note the first two rows are identical: asking for 0 is clamped to 128,
# because this tier cannot switch thinking off.
#
# SIDE BY SIDE — same prompt, same 40-token answer
# reasoning, thinking=300 $0.00565 4.00s
# small tier, thinking=0 $0.00020 0.76s
# ---> 29x cheaper, 3.24s faster
#
# plus caching 1200 static input tokens at 25%: $0.00011 (46% off input)
#
# all assertions passed
The decode rates are the least portable constants here — measure your own by timing a call with a known output length, and the rest of the model follows.
6. Real-world example
A team built a phone assistant for a clinic. The job was narrow: answer from a short list of approved facts, collect an appointment date, and otherwise hand off to a human. The system prompt explicitly forbade inventing anything.
They chose the flagship reasoning tier, on the reasonable-sounding grounds that it was the most capable model and the client cared about accuracy. The call was made with no generation config at all — no thinking budget, no output cap, no temperature. Answers were correct. Two problems surfaced instead.
Callers talked over the assistant, because the first word arrived three to four seconds after they stopped speaking, and a phone line goes hostile past about a second. The team's response was to add short filler phrases — "Sure." — to cover the gap, then a regular expression to stop the assistant saying "Sure" twice when its real answer also began with an acknowledgement, then de-duplication logic when that produced repeats. Three subsystems, all of them scaffolding around a latency problem nobody had measured.
The second problem only appeared on the invoice. The model line item was roughly half the total cost of running the service, above transcription, synthesis, and telephony combined — for a workload whose hardest question was which insurers the practice accepted.
When they finally logged the returned usage counts, the shape was obvious: the majority of both the cost and the elapsed time was hidden deliberation on questions with one-sentence answers. Someone had already tried setting the budget to zero, seen no error, and assumed it worked; the tier had a minimum and had silently clamped it. Moving to a smaller tier with thinking genuinely disabled, capping output, and streaming the reply cut first-word latency by roughly three quarters and the model bill by more than an order of magnitude — and the three filler subsystems were deleted, because there was no longer a gap to hide.
The lesson is the ordering. They had spent weeks on prompts and bandages before reading the token counts that would have pointed at the cause in an afternoon.
7. Interview questions companies actually ask
Q1. Why are thinking tokens more expensive than prompt tokens for the same count? Because they are billed at the output rate, which is commonly 8–10x the input rate, and because they are decoded serially while input is prefilled in parallel. So the same number of tokens costs several times as much money and vastly more wall-clock time on the output side. This inverts the usual intuition: a 2000-token prompt is a smaller problem than a 500-token thinking trace in both dimensions.
Q2. You set the thinking budget to zero and the bill did not change. What happened? Almost certainly the tier enforces a minimum and clamped your request upward without raising an error. Flagship reasoning models frequently cannot disable deliberation at all, while smaller models in the same family can. The diagnostic is the returned usage counts rather than the request — and the fix is a different model tier, because the budget parameter is for tuning tasks that need some reasoning, not for switching it off.
Q3. How would you decide the right thinking budget for a task? Empirically, against a scored test set, sweeping the budget and plotting accuracy against cost and latency. You are looking for the knee — the point where more thinking stops buying correctness. Do it per task type rather than globally, because a single budget across mixed traffic is wrong for most of it. If you cannot score the task, you cannot tune the budget, and building that harness is the prerequisite.
Q4. What are the three generation controls people forget to set, and which matters most for latency? Thinking budget, maximum output length, and stop sequences. Output length matters most for latency because output tokens are serial — halving expected output roughly halves generation time. Treat it as a safety net rather than a style control: ask for the length you want in the prompt, and cap slightly above it so drift gets truncated rather than delivered.
Q5. When is a reasoning tier clearly the right call? When the task has intermediate steps that can be independently wrong: mathematics, code that must compile, planning under constraints, reconciling contradictory sources, or grading where you want the rationale as an artefact. The test is whether there is verifiable structure between the question and the answer. If the answer is a lookup or a rephrasing, there is nothing for deliberation to improve.
Q6. Your prompt is mostly static but caching never hits. Why? Something variable is sitting inside what you believe is the static prefix — a timestamp, a generated identifier, a greeting that changes with the time of day. Prefix caching matches from the beginning of the prompt, so any early variation invalidates everything after it. The bug is quiet because a cache miss is a higher bill rather than an error, which is why cached-token counts belong in your per-call metrics.
Q7. How do few-shot examples interact with cost, given that they add input tokens? Favourably in most cases. They add cheap, parallel-prefilled, cacheable input, and they reduce expensive serial output by demonstrating the target length instead of describing it. They frequently replace an equivalent number of instruction tokens, making the swap roughly token-neutral, and they let a smaller model hit a quality bar that previously seemed to need a larger one — which is the much bigger saving.
8. When to use / tradeoffs
Reach for a reasoning tier / generous budget when:
- The task has verifiable intermediate steps — maths, code, constrained planning
- Sources conflict and must be reconciled rather than summarised
- You want the rationale itself as an auditable artefact
- Correctness dominates cost and latency in the product's value
Reach for a small tier with thinking off when:
- The answer is a lookup, a rephrasing, or a short classification
- Output shape is fixed and known in advance
- Latency is a product requirement, especially in a real-time or spoken interface
- Per-call cost is multiplied by high volume
| Situation | Why it breaks | Use instead |
|---|---|---|
| Real-time voice, one-sentence answers | Serial hidden tokens dominate first-word latency | Small tier, budget 0, capped output, streamed |
| Mixed easy/hard traffic on one endpoint | Any single global budget is wrong for most requests | Route by task type, budget per route |
| Set budget 0 on a flagship tier | Clamped to a minimum, silently | Change tier; verify with returned usage counts |
| Multi-step maths or code with thinking off | Removes the mechanism the task actually needs | Reasoning tier, tuned against a scored set |
| Provider does not report hidden token counts | None of this arithmetic is possible | Treat as a procurement concern; be conservative |
| No evaluation harness | Tier changes are unfalsifiable guesses | Build the scored set first |
Honest limits. Every number in §4 and §5 is a model, not a measurement of your workload. The rates are illustrative and change often; the decode rates are the least portable constants and vary with load, region, and context length. Real thinking-token counts are not controllable to a single value — a budget is a ceiling, actual usage varies per request, and short-answer tasks can occasionally spend far more than their average. The framing also assumes thinking is either useful or not for a whole task class, when the reality is a distribution: a small fraction of "easy" requests genuinely benefit, and disabling deliberation loses those. That is a real quality cost, not a free win, and the only defence is a scored test set with a slice for the hard tail. Finally, the multiplicative story about tier, budget, cap, and caching assumes they do not interact badly — in practice a smaller model with a tight output cap sometimes truncates mid-sentence in a way the larger one did not, so measure the composition rather than each lever alone.
9. Summary + related articles
- Reasoning models emit hidden thinking tokens, billed at the output rate and decoded serially — so they cost both money and latency while the user sees nothing.
- Input is cheap and parallel; generated tokens are expensive and serial. A 500-token thinking trace outweighs a 2000-token prompt in both dimensions.
- Measured on a 40-token answer: at a 300-token budget, thinking was 53% of cost and 75% of elapsed time. At 2048 it was 95%, taking 21 seconds.
- Setting the budget to zero on a flagship tier often does nothing — it clamps to a minimum with no error. The lever for "no reasoning needed" is the model tier.
- Set all three controls: thinking budget, maximum output length, stop sequences. The output cap is the strongest latency lever.
- Few-shot examples beat prose instructions for length control — cheap parallel input replacing expensive serial output, and cacheable.
- Keep the cached prefix genuinely static; an interpolated timestamp or hourly greeting silently destroys every cache hit.
- Thinking stops being waste when the task has verifiable intermediate steps. Deciding which is which requires a scored test set, not taste.
Related:
- API Best Practices — retries, timeouts, and the request-level hygiene these settings sit inside
- RAG Cost Optimization: Find the Step That Runs Forty Times — §3.2 per-role model routing and §3.5 on not making the call at all
- Voice Agent Architectures: Cascaded vs Speech-to-Speech — where the model term sits in a real-time latency budget
- Turn-Taking in Voice Agents: Endpointing, VAD and Barge-In — why cheap models make overlapping the endpointing wait affordable
- Production Agents — cost per task with routing, and caching break-even
- Streaming Modes in LangGraph — emitting output as it is generated instead of waiting for the last token
- Agent Evaluation — the scored harness §8 says you need before changing tiers
Resources
- Wei et al. (2022) — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, arXiv:2201.11903 — the paper that established explicit intermediate steps as a capability lever: https://arxiv.org/abs/2201.11903
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models, arXiv:2001.08361 — background on why compute spent at inference trades against model size: https://arxiv.org/abs/2001.08361
- Snell et al. (2024) — Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters, arXiv:2408.03314 — the test-time-compute framing that reasoning budgets are an instance of: https://arxiv.org/abs/2408.03314
- Anthropic — extended thinking documentation, including budget parameters and how thinking tokens are billed: https://docs.claude.com/en/docs/build-with-claude/extended-thinking
- OpenAI — reasoning models guide, including effort levels and reasoning-token accounting: https://platform.openai.com/docs/guides/reasoning
- Google — Gemini thinking documentation, including which tiers accept a zero budget: https://ai.google.dev/gemini-api/docs/thinking
- Provider pricing pages are the only authority on the rates used in §4; they change frequently, so verify before repeating any figure from this article in a decision.
