Context Fundamentals

TL;DR — The context window is a token budget covering everything in one call — system prompt, examples, retrieved documents, conversation history, and the reply you haven't generated yet. When a conversation outgrows it, something must be discarded, and which thing you discard is a design decision you either make or make by accident. In the measured run below, three eviction policies inside one identical budget each lose something different: dropping from the front silently deletes the safety rules while keeping the facts; pinning the system prompt keeps the rules and loses the fact the question depends on; only summarising the dropped prefix keeps both. None of the three errors — they all produce a fluent answer. It stops being a capacity problem the moment the window is large enough, at which point the real limits become cost, latency, and the fact that a model uses the middle of a long prompt less reliably than the ends.

---

1. Simple explanation

Everything the model knows about your request has to arrive in one message. There's no memory between calls — a chatbot that "remembers" your name is just re-sending the earlier turns every time.

That message has a size limit, counted in tokens, and the limit covers everything at once: the instructions, any examples, retrieved documents, the entire conversation so far, and space for the reply. It is one shared budget, not several.

So a long conversation eventually doesn't fit. At that point something gets thrown away. The question — the only question that matters here — is what.

Analogy — a whiteboard in a meeting room. Everything the group can reason about has to be on the board. It's the right size for a while, then it's full, and to write anything new you must rub something out. If you rub out whatever's nearest the left edge, you'll eventually erase the agenda and the constraint someone stated at the start, and the meeting will carry on confidently in the wrong direction — because nobody remembers there was a constraint. Nothing about a full whiteboard tells you which part mattered. Deciding that in advance is the whole job.

---

2. Diagram

ONE BUDGET, SHARED BY EVERYTHING

  ┌─────────────────────────────────────────────────────┬──────────┐
  │ system │ examples │ retrieved docs │ history         │ reply    │
  │ prompt │          │                │                 │ (reserve)│
  └─────────────────────────────────────────────────────┴──────────┘
  ◀──────────────────── context window (tokens) ────────────────────▶
                                                          ▲
                          forget to reserve this and the model
                          runs out of room mid-sentence


AS THE CONVERSATION GROWS

  turn 1   ██░░░░░░░░░░░░░░░░░░  fits easily
  turn 5   ████████░░░░░░░░░░░░  fits
  turn 9   ████████████████████  FULL -> something must go
                                 ▲
                       and now you are choosing, whether or not
                       you wrote any code to choose


THREE POLICIES, ONE BUDGET, THREE DIFFERENT LOSSES

  the last question needs BOTH:  the amount (turn 2)
                                 the "never promise a refund" rule (system)

  drop oldest, incl. system   amount ✅   rules ❌   -> answers, breaks policy
  pin system, drop oldest     amount ❌   rules ✅   -> on policy, can't answer
  pin system + summarise      amount ✅   rules ✅   -> the only one that works

     all three fit the SAME budget.  none of them errors.
     all three produce fluent, confident output.


WHY SUMMARISING WINS

  9 turns of history        ████████████████  ~90 tokens
  one summary line          ███               ~14 tokens
                            ▲
             compression, not deletion: the fact survives
             at a fraction of the token cost

---

3. How it works

3.1 The window is a budget, and the reply is part of it

Every provider states a maximum tokens per request. Everything counts against it, including the output — which is the part people forget. If you fill the window with input, there is no room left to answer, and you get a truncated reply or an error.

So the usable input budget is:

  input_budget = context_window - max_output_tokens - safety_margin

Always reserve output space explicitly. And measure the budget in tokens, not characters or words — the ratio varies enormously by content, and Tokenization explains why a word-count estimate will be wrong in the direction that hurts.

3.2 There is no memory — history is re-sent every turn

Models are stateless. A "conversation" is the whole transcript re-sent on every call, which has two consequences worth stating plainly.

Cost grows with conversation length. Turn 20 pays for turns 1–19 again. Naively, total cost over a conversation is quadratic in the number of turns, which is why long chats get expensive faster than people expect.

The static prefix should come first. System prompt and examples are identical every turn, so putting them at the front lets prefix caching bill them at a fraction — see Reasoning Budgets: When Thinking Tokens Are Waste §3.5. Interpolating anything variable early (a timestamp, a greeting that changes with the hour) silently destroys that.

3.3 The four eviction policies

When history won't fit, you pick one. Roughly in order of sophistication:

The last is history-as-RAG: index past turns and retrieve the ones the current question needs, rather than assuming recency equals relevance. It's the most robust and the most machinery — Memory Management covers it properly.

3.4 The failure: recency is not importance

Every naive policy assumes the newest turns matter most. They often don't.

Facts get stated once, early — an order number, a constraint, a preference — and then referred to obliquely for the rest of the conversation. "So how much am I getting back?" depends on an amount mentioned seven turns ago. A sliding window drops exactly that.

And nothing signals the loss. The model doesn't know a fact was evicted; it sees a shorter conversation that reads as complete. So it answers from what's left, fluently. §4 measures all three variants of this, and the worst one isn't the one you'd expect.

3.5 A bigger window doesn't remove the problem

Long-context models change the arithmetic but not the shape.

Cost and latency still scale with what you send. Filling a million-token window because you can is expensive on every call.

The middle is used less reliably. The documented lost in the middle effect means content buried in a long prompt has less influence than content at either end. A fact at position 400,000 is technically present and practically unreliable — so "it fits" is not the same as "the model will use it."

Attention isn't free. More context means more compute per token, so a full window is slower as well as dearer.

The practical upshot: even with a huge window, deliberately selecting what to include beats dumping everything in. Bigger windows raise the ceiling; they don't remove the need to choose.

3.6 Where this stops being the right frame

If your conversations comfortably fit, none of this is your problem — go and optimise cost or latency instead. This article also treats context as a flat token budget, which breaks down for multimodal input, where an image consumes a fixed patch count unrelated to any text arithmetic. And it says nothing about persistent memory across sessions — facts a system should recall next week — which is a storage-and-retrieval design question rather than a windowing one.

---

4. The math

4.1 The budget

  input_budget = window - max_output - margin

  used = tokens(system) + tokens(examples) + tokens(docs) + tokens(history)
  must satisfy:  used <= input_budget

4.2 Why conversations get expensive

  turn i re-sends the whole transcript, so
      total_tokens ≈ sum over i of (prefix + i * avg_turn)
                   = n * prefix + avg_turn * n(n+1)/2

  -> QUADRATIC in the number of turns

Summarising cuts the growing term, which is why it's a cost strategy as much as a capacity one.

4.3 Worked example

A deliberately tiny window makes the mechanics visible: 100 tokens total, 20 reserved for the reply, so 80 for the prompt. A system prompt ("Never promise a refund. Always confirm the order id.") plus nine conversation turns.

window 100 tokens, 20 reserved for the reply -> 80 for the prompt
the full conversation needs 113 tokens, so something must go.

The last user turn is "so how much am I getting back", which needs the amount stated back at turn 2. Three policies, same 80-token budget:

policy                            used  turns kept   system?   the fact?
drop_oldest_including_system        72           7      LOST         YES
pin_system_drop_oldest              80           6       YES        LOST
pin_system_and_summarise            79           5       YES         YES

Read the middle two columns against each other, because this is the result.

drop_oldest_including_system keeps the amount and deletes the safety rules. It can answer the question — and it has lost the instruction never to promise a refund. That's the worst outcome available: confident, fluent, and against policy, with the facts intact so nothing looks wrong.

pin_system_drop_oldest is the obvious improvement, and it's necessary but not sufficient. Rules kept, amount gone. It stays on policy and cannot answer correctly — and since nothing marks the gap, it may invent a figure rather than ask again.

pin_system_and_summarise keeps both inside the same 80 tokens, by replacing four dropped turns with one summary line. It keeps fewer raw turns (5 vs 6) and more information, which is the whole point: compression beats deletion because a summary costs a fraction of what it replaces.

None of the three raises an error. All three fit. All three produce fluent output. The difference is only visible if you know what the answer needed.

4.4 How the window size changes it

  window  100 -> 6 of 9 turns kept
  window  150 -> 9 of 9 turns kept
  window  200 -> 9 of 9 turns kept
  window  400 -> 9 of 9 turns kept

Above 150 tokens this conversation fits entirely and the policy stops mattering — which is exactly §3.5's point. A bigger window removes this instance of the problem and not the class: make the conversation longer, or add retrieved documents, and you are choosing again.

---

5. Real code

"""A fixed token budget, a growing conversation, and what each eviction policy loses."""

WINDOW = 100          # total token budget, deliberately tiny
RESERVE_OUT = 20      # tokens held back for the reply
BUDGET = WINDOW - RESERVE_OUT

SYSTEM = ("You are a support agent. Never promise a refund. "
          "Always confirm the order id before acting.")

# (speaker, text). Turn 2 carries the fact the final question depends on.
TURNS = [
    ("user", "hi my order is late"),
    ("agent", "sorry to hear that, can you give me the order id"),
    ("user", "it is A-1002 and I paid forty five pounds fifty"),   # <- THE FACT
    ("agent", "thanks, I can see it left the warehouse on Tuesday"),
    ("user", "ok but it still has not arrived"),
    ("agent", "I understand, let me check the courier status for you"),
    ("user", "any update"),
    ("agent", "the courier says it is delayed by one day"),
    ("user", "so how much am I getting back"),                     # needs the fact
]
FACT_TURN = 2
KEY = "forty five"


def tokens(text: str) -> int:
    return round(len(text.split()) * 1.3)


def assemble(policy: str) -> tuple[list[str], int]:
    """Build the prompt under `policy`, never exceeding BUDGET."""
    sys_cost = tokens(SYSTEM)

    if policy == "drop_oldest_including_system":
        parts = [SYSTEM] + [t for _s, t in TURNS]
        while sum(tokens(p) for p in parts) > BUDGET:
            parts.pop(0)                      # blindly drops the SYSTEM prompt first
        return parts, sum(tokens(p) for p in parts)

    if policy == "pin_system_drop_oldest":
        kept: list[str] = []
        for _s, t in reversed(TURNS):         # newest first
            if sys_cost + sum(tokens(k) for k in kept) + tokens(t) > BUDGET:
                break
            kept.insert(0, t)
        return [SYSTEM] + kept, sys_cost + sum(tokens(k) for k in kept)

    if policy == "pin_system_and_summarise":
        # Compress the dropped prefix instead of deleting it.
        kept: list[str] = []
        for _s, t in reversed(TURNS):
            if sys_cost + sum(tokens(k) for k in kept) + tokens(t) > BUDGET - 12:
                break
            kept.insert(0, t)
        dropped = [t for _s, t in TURNS][:len(TURNS) - len(kept)]
        summary = ""
        if dropped:
            facts = [d for d in dropped if KEY in d or "A-1002" in d]
            summary = ("earlier: order A-1002, customer paid forty five pounds fifty"
                       if facts else "earlier: customer reported a late delivery")
        parts = [SYSTEM] + ([summary] if summary else []) + kept
        return parts, sum(tokens(p) for p in parts)

    raise ValueError(policy)


full = tokens(SYSTEM) + sum(tokens(t) for _s, t in TURNS)
print(f"window {WINDOW} tokens, {RESERVE_OUT} reserved for the reply "
      f"-> {BUDGET} for the prompt")
print(f"the full conversation needs {full} tokens, so something must go.\n")

print(f"{'policy':<32} {'used':>5} {'turns kept':>11}  {'system?':>8}  {'the fact?':>10}")
results = {}
for policy in ("drop_oldest_including_system",
               "pin_system_drop_oldest",
               "pin_system_and_summarise"):
    parts, used = assemble(policy)
    has_sys = SYSTEM in parts
    has_fact = any(KEY in p for p in parts)
    n_turns = sum(1 for p in parts if p != SYSTEM and not p.startswith("earlier:"))
    results[policy] = (has_sys, has_fact, used)
    print(f"{policy:<32} {used:>5} {n_turns:>11}  "
          f"{'YES' if has_sys else 'LOST':>8}  {'YES' if has_fact else 'LOST':>10}")

print("\nThe last question needs BOTH: the amount (turn 2) and the no-refund rule.")
print("Each policy loses a DIFFERENT one:\n")
print("  drop_oldest_including_system  amount kept, SAFETY RULES lost")
print("                                -> can answer, and may promise the refund it")
print("                                   was told never to promise. Worst outcome:")
print("                                   confident, fluent, and against policy.")
print("  pin_system_drop_oldest        rules kept, AMOUNT lost")
print("                                -> stays on policy but cannot answer; and with")
print("                                   nothing signalling the gap, it may invent")
print("                                   a figure rather than ask again.")
print("  pin_system_and_summarise      both kept, in the SAME budget")
print("                                -> the only policy that can answer correctly.")
print("                                   Compression beats deletion because a summary")
print("                                   costs far fewer tokens than the turns it replaces.")

print("\nHow many turns fit, as the window grows:")
for w in (100, 150, 200, 400):
    globals()["BUDGET"] = w - RESERVE_OUT
    parts, used = assemble("pin_system_drop_oldest")
    n = sum(1 for p in parts if p != SYSTEM)
    print(f"  window {w:>4} -> {n} of {len(TURNS)} turns kept")
globals()["BUDGET"] = WINDOW - RESERVE_OUT

# Dropping from the front silently discards the instructions you rely on.
assert results["drop_oldest_including_system"][0] is False
# Pinning the system prompt is necessary but not sufficient.
assert results["pin_system_drop_oldest"][0] is True
assert results["pin_system_drop_oldest"][1] is False
# Only compression keeps BOTH the rules and the fact inside the same budget.
assert results["pin_system_and_summarise"][0] is True
assert results["pin_system_and_summarise"][1] is True
for _p, (_s, _f, used) in results.items():
    assert used <= BUDGET, used
print("\nall assertions passed")

# Output:
#   window 100 tokens, 20 reserved for the reply -> 80 for the prompt
#   the full conversation needs 113 tokens, so something must go.
#
#   policy                            used  turns kept   system?   the fact?
#   drop_oldest_including_system        72           7      LOST         YES
#   pin_system_drop_oldest              80           6       YES        LOST
#   pin_system_and_summarise            79           5       YES         YES
#
#   The last question needs BOTH: the amount (turn 2) and the no-refund rule.
#   Each policy loses a DIFFERENT one:
#
#     drop_oldest_including_system  amount kept, SAFETY RULES lost
#                                   -> can answer, and may promise the refund it
#                                      was told never to promise. Worst outcome:
#                                      confident, fluent, and against policy.
#     pin_system_drop_oldest        rules kept, AMOUNT lost
#                                   -> stays on policy but cannot answer; and with
#                                      nothing signalling the gap, it may invent
#                                      a figure rather than ask again.
#     pin_system_and_summarise      both kept, in the SAME budget
#                                   -> the only policy that can answer correctly.
#                                      Compression beats deletion because a summary
#                                      costs far fewer tokens than the turns it replaces.
#
#   How many turns fit, as the window grows:
#     window  100 -> 6 of 9 turns kept
#     window  150 -> 9 of 9 turns kept
#     window  200 -> 9 of 9 turns kept
#     window  400 -> 9 of 9 turns kept
#
#   all assertions passed

The summariser is hard-coded to demonstrate the mechanism; a real one is another model call, which costs tokens itself — so summarise on a schedule (every N turns) rather than on every request.

---

6. Real-world example

A team ran a customer-service assistant with a sliding window: system prompt pinned, most recent turns kept, oldest dropped. Sensible, and better than the naive version.

Complaints arrived about the assistant "forgetting" — asking again for an order number the customer had given, or contradicting something agreed earlier. Always in longer conversations, which the team read as "long conversations are harder" rather than as a specific mechanism.

It was exactly the §3.4 failure. Customers state identifying facts once, at the start, then talk around them. By turn twelve the order number had been evicted, and the assistant had no way to know it ever existed — so it asked again, or worse, proceeded with a plausible substitute.

Two properties made it hard to see. Short conversations were unaffected, so it never reproduced in testing. And the assistant's behaviour was correct given its input — it wasn't malfunctioning, it was reasoning from a transcript that had been silently edited.

The fix had two halves. A running summary, regenerated every few turns, carried the durable facts forward — order number, agreed actions, stated constraints — at a fraction of the token cost of the turns it replaced. And a structured slot held the identifiers explicitly, outside the conversational text, so they could never be evicted by a windowing decision at all.

The general lesson: decide what must never be dropped, and store it somewhere a windowing policy can't reach. Leaving that to "keep the most recent N" is choosing by accident.

---

7. Interview questions companies actually ask

Q1. What's in the context window? Everything in the request: system prompt, examples, retrieved documents, the full conversation so far, and the space for the reply. One shared budget in tokens, and the output counts against it — which is why you reserve output space explicitly rather than filling the window with input and hoping.

Q2. How does a model remember earlier turns? It doesn't. Models are stateless; the whole transcript is re-sent on every call. Two consequences: cost grows roughly quadratically over a long conversation, since turn 20 pays for turns 1–19 again; and the static prefix should come first so prefix caching can bill it at a fraction.

Q3. What happens when the conversation exceeds the window? Something must be discarded, and you either choose or choose by accident. Options in order of sophistication: truncate oldest, pin the system prompt and slide, summarise the dropped prefix, or index history and retrieve only the relevant turns. The critical thing is that eviction is silent — nothing errors, and the model can't tell a fact was removed.

Q4. Why isn't "keep the most recent N turns" good enough? Because recency isn't importance. Facts are stated once, early — an order number, a constraint — then referred to obliquely. A sliding window drops precisely those. In the worked example, pinning the system prompt kept the rules and lost the amount the final question depended on, and the model answered anyway.

Q5. Does a million-token window solve this? It raises the ceiling and doesn't change the shape. Cost and latency still scale with what you send; attention isn't free, so a full window is slower; and the documented "lost in the middle" effect means content buried in a long prompt influences the output less than content at either end. "It fits" is not "the model will use it."

Q6. How would you keep a fact from ever being evicted? Take it out of the conversational text. Hold identifiers and agreed constraints in structured fields injected into the prompt separately, so no windowing policy can reach them, and carry the narrative gist in a periodically regenerated summary. Deciding what must never be dropped is the design work; "keep the last N" is a default, not a decision.

Q7. Why summarise rather than truncate? Because compression preserves the information at a fraction of the tokens. In the measured run, summarising kept fewer raw turns than the sliding window (5 vs 6) and more actual information — both the safety rules and the fact — inside an identical budget. It also cuts the quadratic growth term, so it's a cost strategy too. The cost is that summarising is itself a model call, so do it every N turns rather than every request.

---

8. When to use / tradeoffs

You need a context strategy when:

• Conversations run long enough to approach the window
• Retrieved documents compete with history for space
• Facts stated early matter later
• Cost per conversation is growing faster than conversation length

You can defer it when:

• Every interaction is single-turn or comfortably short
• The window dwarfs your realistic maximum input

Honest limits. The 100-token window in §4 is a demonstration scale, and the specific policy that wins is a property of this conversation — put the deciding fact in the last turn instead of turn 2 and the sliding window is perfectly adequate. The token function is a word-count proxy, so the arithmetic is internally consistent and not transferable. The summariser is hard-coded to retain the right fact, which is the part real systems find hard: a real summariser is a model call that can drop the very detail that later matters, and it introduces a second place for information to be silently lost. The quadratic cost model assumes no prefix caching, which changes the constant substantially in practice. And "lost in the middle" is a documented effect whose strength varies by model and by how the content is formatted — treat it as a reason to be deliberate, not a precise law.

---

9. Summary + related articles

• The context window is a token budget covering system prompt, examples, documents, history, and the reply. Reserve output space explicitly.
• Models are stateless — history is re-sent every turn, so conversation cost grows roughly quadratically.
• When it doesn't fit, something is discarded. Eviction is silent: nothing errors, and the model can't tell a fact was removed.
• Measured, in one identical budget: dropping from the front lost the safety rules while keeping the facts; pinning the system prompt lost the fact; only summarising kept both.
• The worst outcome isn't the obvious one — keeping facts and losing the rules produced an answer that was confident, fluent, and against policy.
• Summarising kept fewer turns and more information. Compression beats deletion.
• Recency is not importance. Facts are stated once, early, then referred to obliquely.
• A bigger window raises the ceiling, not the shape: cost, latency, and lost in the middle still apply. "It fits" ≠ "it will be used."
• Decide what must never be dropped and store it outside the transcript, in structured slots.

Related:

• Tokenization — why the budget must be counted in tokens, not characters
• Prompt Fundamentals — what goes in the prompt, and in what order
• Context Strategies — the eviction policies in depth
• Context Window Optimization — fitting more useful content into the same budget
• Memory Management — history-as-retrieval, and memory across sessions
• Reasoning Budgets: When Thinking Tokens Are Waste — prefix caching, and reserving room for hidden tokens
• How LLMs Work — why there's no state between calls
• RAG Cost Optimization: Find the Step That Runs Forty Times — controlling the quadratic growth
• Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems — when documents compete with history for space
• Consistency Memory: Making Repeated AI Judgments Agree With Each Other — keeping facts stable across a long interaction

Resources

• Liu et al. (2023) — Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172 — the evidence behind §3.5; content position changes whether it's used: https://arxiv.org/abs/2307.03172
• Vaswani et al. (2017) — Attention Is All You Need, arXiv:1706.03762 — why attention cost grows with sequence length: https://arxiv.org/abs/1706.03762
• Dao et al. (2022) — FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, arXiv:2205.14135 — part of why long windows became practical: https://arxiv.org/abs/2205.14135
• Press, Smith & Lewis (2021) — Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation, arXiv:2108.12409 — how models handle inputs longer than they trained on: https://arxiv.org/abs/2108.12409
• Packer et al. (2023) — MemGPT: Towards LLMs as Operating Systems, arXiv:2310.08560 — treating the window as managed memory, the generalisation of §3.3: https://arxiv.org/abs/2310.08560
• Provider documentation is the authority on your actual window size and on whether output counts against it; both change between model versions.