TL;DR — Chunking decides what retrieval is able to return, so it caps the quality of everything downstream. The metric that matters is not chunk size but whether a rule survives intact — claim and exception in the same chunk. Measured below: fixed 60-character chunks keep 0 of 3 rules whole, fixed 120 keeps 1 of 3, adding 40 characters of overlap recovers all 3 of 3 at 1.43× storage, and structure-aware recursive splitting gets 3/3 with no duplication at all. Bigger is not simply better — one chunk containing the whole document keeps every rule and destroys retrieval's ability to discriminate between them. The failure this prevents is the nastiest one in RAG: a chunk that is true, relevant, highly scored, and leads to the opposite answer. It stops being a strategy question when your documents have no structure to exploit, at which point overlap is the only lever left and you pay for it in duplication.
1. Simple explanation
Documents are too long to hand a model whole, so you cut them into pieces and retrieve the pieces. Chunking is where you decide where to cut.
It sounds like a formatting detail. It is the single highest-leverage decision in a RAG pipeline, because retrieval can only ever return a chunk you created. If the sentence that answers a question got split down the middle, no amount of clever retrieval, reranking or prompting recovers it.
The mistake is thinking about chunk size. What matters is whether each chunk is a complete unit of meaning. Real documents state a rule and then qualify it — "you may do X, except when Y" — and a chunk containing only the first half is perfectly readable, entirely true, and actively misleading.
Analogy — photocopying a contract at the wrong page break. Copy the clause "the tenant may sublet the property" onto one page and "unless written consent is withheld" onto the next, then file them separately. Anyone retrieving the first page reads a true sentence and reaches a false conclusion. Nothing is corrupted, nothing is missing from the filing cabinet, and the answer is still wrong. That is exactly what a bad chunk boundary does, and why "how big should chunks be" is the wrong first question.
2. Diagram
THE METRIC THAT MATTERS: does the rule survive?
source: "Items may be returned within 30 days, except sale items which are final."
└──────── claim ────────┘ └──── exception ────┘
both must land in the SAME chunk
fixed 60 |Items may be returned within 30 days, except sale i|tems which...|
▲
split mid-word, rule broken
fixed 120 |Items may be returned within 30 days, except sale items which are|
rule survives here -- but the NEXT rule breaks instead
+ overlap |...30 days, except sale items which are final.|
|except sale items which are final. Shipping...|
▲ duplicated, so it appears whole SOMEWHERE
recursive |Returns. Items may be returned within 30 days, except sale items
which are final.| ← split on the sentence boundary
MEASURED — 3 rules, each a claim plus an exception
strategy chunks avg chars total chars rules intact
fixed 60 5 56 279 0/3
fixed 120 3 93 279 1/3
fixed 120 + 40 over 4 100 398 3/3 ← 1.43x storage
sentence 6 46 274 3/3
recursive <=160 3 92 277 3/3 ← 3/3, no duplication
BIGGER IS NOT SIMPLY BETTER
size chunks rules intact
60 5 0/3
120 3 1/3
200 2 3/3
400 1 3/3 ← whole doc in one chunk:
every rule intact, and retrieval
can no longer tell them apart
3. How it works
3.1 The four strategies
Fixed-size — cut every N characters or tokens. Trivial to implement, predictable cost, and it splits mid-word and mid-rule with no awareness of anything. It is the default in most tutorials and the source of most chunking bugs.
Sentence — split on sentence boundaries. Chunks are always readable, and it works only if a rule fits in one sentence. It scores 3/3 in §4 because every rule in that document is one sentence; in Anatomy of a RAG Pipeline the same strategy fails, because there the exception is its own sentence. The strategy didn't change — the documents did.
Recursive — try to split on the largest natural boundary that fits: sections, then paragraphs, then sentences, then words. This respects the document's own structure and is the best default for prose. In §4 it achieves 3/3 while storing fewer characters than the original document had.
Overlap — extend each chunk backwards into the previous one, so material near a boundary appears in two chunks. It is not a strategy on its own; it is a modifier that buys safety with duplication.
3.2 Overlap: what it actually buys
Overlap works by making boundary material appear twice, so a rule split by one boundary is whole in the chunk that straddles it.
The price is exact and worth stating: with chunk size S and overlap V, you store roughly S/(S−V) times the original text. At 120 with 40 overlap that is 1.5× in theory and 1.43× measured. You pay it three times — storage, embedding cost at index time, and a slightly noisier index because near-duplicate chunks compete with each other.
The rule of thumb is 10–20% overlap. Below that it rarely spans a real rule; above it you are mostly paying to store the same sentences repeatedly.
3.3 Why "just use bigger chunks" fails
If small chunks split rules, the obvious fix is bigger chunks. §4 shows why that dead-ends: at 400 characters the whole document is one chunk, every rule is intact, and retrieval has nothing to choose between.
Three costs to large chunks:
| cost | why |
|---|---|
| Retrieval precision | a chunk covering five topics is somewhat similar to queries about all five, and precisely relevant to none |
| Token spend | every retrieved chunk is input tokens on every call |
| Dilution | irrelevant surrounding text competes for the model's attention |
So there is a genuine optimum in the middle, and it is a property of your documents — how long a self-contained unit of meaning tends to be — not a number you can copy from a blog post.
3.4 Metadata is part of the chunk
A chunk lifted out of its document loses its context: which document, which section, what date, which product version. Retrieval returns a paragraph that begins "This does not apply to enterprise accounts" with no indication of what "this" was.
Prepend the structural path — document title, section heading — to each chunk's text before embedding. It costs a few tokens, improves retrieval (the heading carries topic signal), and makes citation possible. Store the date and version as filterable metadata so you can exclude superseded documents, which similarity ranking will otherwise happily return.
3.5 Measure the thing you care about
"Rules intact" is a stand-in for the real question: can retrieval return everything needed to answer correctly? Measure it directly. Take questions with known answers, and for each one check whether the chunk containing the answer exists and is complete.
That number is computable without generating anything, which makes it the cheapest evaluation in the pipeline and the right one to optimise chunking against. Optimising chunk size against final answer quality works too and is far slower, because every change requires a full generation pass. See RAG Evaluation: Attributing Failure and Sizing the Eval Set.
3.6 Where chunking strategy stops mattering
If your documents are already short and self-contained — FAQ entries, product records, support tickets — one document is one chunk and none of this applies. Don't split what is already atomic.
At the other extreme, some content resists chunking entirely: tables where meaning spans rows, code where a function's behaviour depends on definitions elsewhere, transcripts where a pronoun refers to something ten turns back. For those the answer is structural — keep the table whole, chunk code by function, resolve references at ingest — rather than a better splitter. And if a rule is genuinely spread across separate documents, no chunking strategy can help; that needs multi-document retrieval.
4. The math
4.1 Overlap cost
chunks ≈ len(text) / (S - V) S = chunk size, V = overlap
stored ≈ len(text) * S / (S - V)
S=120, V=40 -> 1.5x in theory, 1.43x measured (boundary effects)
S=120, V=12 -> 1.11x
4.2 What to optimise
retrievable(q) = 1 if some chunk contains everything needed to answer q
maximise Σ retrievable(q) over your evaluation questions
subject to chunk size small enough that retrieval can discriminate
Note both halves. Maximising the first alone gives you one giant chunk.
4.3 Worked example
A 279-character policy document with three rules. Each rule is a claim plus an exception, and a rule is only usable if both land in the same chunk.
strategy chunks avg chars total chars rules intact
fixed 60 5 56 279 0/3
fixed 120 3 93 279 1/3
fixed 120 + 40 over 4 100 398 3/3
sentence 6 46 274 3/3
recursive <=160 3 92 277 3/3
Fixed 60 keeps zero rules whole. Its chunks split mid-word:
'Returns. Items may be returned within 30 days, except sale i'
'tems which are final. Shipping. Delivery takes 3 to 5 days, '
'unless the address is rural, which adds 2 days. Refunds. Ref'
Every one of those is a plausible-looking chunk that will embed, index, and retrieve normally — and none contains a complete rule.
Overlap fixes it, and the price is visible:
fixed 120 3 chunks, 279 chars stored, 1/3 intact
fixed 120 + 40 over 4 chunks, 398 chars stored, 3/3 intact
-> 1.43x storage and embedding cost, +2 rules recovered
Recursive gets the same 3/3 for 277 characters — less than the original document, and 30% less than the overlap approach. Respecting the document's own boundaries is strictly better than duplicating blindly, when the document has boundaries to respect.
4.4 And the ceiling
size chunks rules intact note
60 5 0/3
120 3 1/3
200 2 3/3
400 1 3/3 whole doc in one chunk -- retrieval can no
longer discriminate
Rules-intact rises monotonically with size and hits its maximum exactly when retrieval becomes useless. That is why chunking cannot be optimised on one metric.
5. Real code
"""Four chunking strategies, scored on whether a rule survives intact."""
DOC = (
"Returns. Items may be returned within 30 days, except sale items which are final. "
"Shipping. Delivery takes 3 to 5 days, unless the address is rural, which adds 2 days. "
"Refunds. Refunds are issued to the original payment method, except gift purchases "
"which are refunded as credit."
)
# Each rule is only USABLE if its claim and its exception land in the same chunk.
RULES = [
("returned within 30 days", "except sale items"),
("3 to 5 days", "unless the address is rural"),
("original payment method", "except gift purchases"),
]
def fixed(text, size):
return [text[i:i + size] for i in range(0, len(text), size)]
def fixed_overlap(text, size, overlap):
step = size - overlap
return [text[i:i + size] for i in range(0, len(text), step)]
def sentences(text):
return [s.strip() + "." for s in text.split(".") if s.strip()]
def recursive(text, size):
"""Split on the largest natural boundary that fits: sentences, then words."""
out, buf = [], ""
for s in sentences(text):
if len(buf) + len(s) + 1 <= size:
buf = (buf + " " + s).strip()
else:
if buf:
out.append(buf)
buf = s if len(s) <= size else ""
if len(s) > size: # a single sentence too big: fall back
out.extend(s[i:i + size] for i in range(0, len(s), size))
if buf:
out.append(buf)
return out
def intact(chunks):
"""How many rules appear COMPLETE (claim and exception together) in some chunk?"""
n = 0
for claim, exc in RULES:
if any(claim in c and exc in c for c in chunks):
n += 1
return n
STRATEGIES = {
"fixed 60": lambda: fixed(DOC, 60),
"fixed 120": lambda: fixed(DOC, 120),
"fixed 120 + 40 over": lambda: fixed_overlap(DOC, 120, 40),
"sentence": lambda: sentences(DOC),
"recursive <=160": lambda: recursive(DOC, 160),
}
print(f"document {len(DOC)} chars, {len(RULES)} rules, each a claim + an exception\n")
print(f"{'strategy':<22} {'chunks':>7} {'avg chars':>10} {'total chars':>12} "
f"{'rules intact':>13}")
res = {}
for name, fn in STRATEGIES.items():
ch = fn()
tot = sum(len(c) for c in ch)
ok = intact(ch)
res[name] = (len(ch), tot, ok)
print(f"{name:<22} {len(ch):>7} {tot/len(ch):>10.0f} {tot:>12} "
f"{f'{ok}/{len(RULES)}':>13}")
print("\nWHAT 'RULES INTACT' MEANS")
print(" A chunk can be perfectly good TEXT and an incomplete RULE. If the claim")
print(" lands in one chunk and its exception in the next, retrieval can return a")
print(" chunk that is true, relevant, and leads to the OPPOSITE answer.\n")
worst = min(res, key=lambda k: res[k][2])
print(f" {worst!r} keeps only {res[worst][2]}/{len(RULES)} rules whole. Its chunks:")
for c in STRATEGIES[worst]()[:3]:
print(f" {c!r}")
print("\nOVERLAP BUYS SAFETY WITH DUPLICATION")
base = res["fixed 120"]
over = res["fixed 120 + 40 over"]
print(f" fixed 120 {base[0]} chunks, {base[1]} chars stored, "
f"{base[2]}/{len(RULES)} intact")
print(f" fixed 120 + 40 over {over[0]} chunks, {over[1]} chars stored, "
f"{over[2]}/{len(RULES)} intact")
print(f" -> {over[1]/base[1]:.2f}x storage and embedding cost, "
f"{over[2]-base[2]:+d} rules recovered")
print("\nBIGGER IS NOT SIMPLY BETTER")
print(f" {'size':>6} {'chunks':>7} {'rules intact':>13} note")
for size in (60, 120, 200, 400, len(DOC)):
ch = fixed(DOC, size)
note = "whole doc in one chunk -- retrieval can no longer discriminate" \
if len(ch) == 1 else ""
print(f" {size:>6} {len(ch):>7} {f'{intact(ch)}/{len(RULES)}':>13} {note}")
# Small fixed chunks strand the exceptions.
assert res["fixed 60"][2] == 0
# Overlap recovers rules that plain fixed-size chunking splits ...
assert over[2] > base[2]
# ... and it costs real duplication.
assert over[1] > base[1]
# Structure-aware splitting does best per stored character.
assert res["recursive <=160"][2] == len(RULES)
assert res["recursive <=160"][1] <= over[1]
# One giant chunk keeps every rule and destroys retrieval's ability to select.
assert intact(fixed(DOC, len(DOC))) == len(RULES) and len(fixed(DOC, len(DOC))) == 1
print("\nall assertions passed")
# Output:
# document 279 chars, 3 rules, each a claim + an exception
#
# strategy chunks avg chars total chars rules intact
# fixed 60 5 56 279 0/3
# fixed 120 3 93 279 1/3
# fixed 120 + 40 over 4 100 398 3/3
# sentence 6 46 274 3/3
# recursive <=160 3 92 277 3/3
#
# WHAT 'RULES INTACT' MEANS
# A chunk can be perfectly good TEXT and an incomplete RULE. If the claim
# lands in one chunk and its exception in the next, retrieval can return a
# chunk that is true, relevant, and leads to the OPPOSITE answer.
#
# 'fixed 60' keeps only 0/3 rules whole. Its chunks:
# 'Returns. Items may be returned within 30 days, except sale i'
# 'tems which are final. Shipping. Delivery takes 3 to 5 days, '
# 'unless the address is rural, which adds 2 days. Refunds. Ref'
#
# OVERLAP BUYS SAFETY WITH DUPLICATION
# fixed 120 3 chunks, 279 chars stored, 1/3 intact
# fixed 120 + 40 over 4 chunks, 398 chars stored, 3/3 intact
# -> 1.43x storage and embedding cost, +2 rules recovered
#
# BIGGER IS NOT SIMPLY BETTER
# size chunks rules intact note
# 60 5 0/3
# 120 3 1/3
# 200 2 3/3
# 400 1 3/3 whole doc in one chunk -- retrieval can no longer discriminate
# 279 1 3/3 whole doc in one chunk -- retrieval can no longer discriminate
#
# all assertions passed
intact uses substring matching against known rule fragments, which is only possible because this document is small enough to enumerate. On a real corpus the equivalent is a set of questions with known answers, and the check is whether the answering passage exists whole in some chunk.
6. Real-world example
A team chunked a policy handbook at 500 characters with no overlap and shipped it. Retrieval looked healthy — high similarity scores, on-topic passages — and answers were right most of the time.
The wrong answers had a pattern nobody spotted for weeks: they were all permissive. The assistant said yes to things the handbook said no to.
The handbook's style was the cause. Every policy was written as a general statement followed by a separate paragraph of exceptions — a house style that reads well and chunks terribly. At 500 characters those reliably landed in different chunks, and because the general statements were phrased like the questions users asked, they scored higher than the exception paragraphs. Retrieval returned the permissive half and ranked it first.
Two things made this hard to catch. The retrieved chunks always looked correct — on topic, well scored, genuinely from the handbook. And there was no missing-information signal anywhere: nothing scored low, nothing was empty, no threshold tripped. See Anatomy of a RAG Pipeline §3.4 for why a score threshold cannot catch this.
They fixed it by chunking on the handbook's own heading structure so a policy and its exceptions stayed together, and adding modest overlap as a backstop. Accuracy on conditional questions went from roughly half to nearly all.
The lesson worth keeping: the right chunking strategy is a function of how your documents are written. The team's mistake wasn't picking 500 characters; it was picking a number without reading how their own content was structured.
7. Interview questions companies actually ask
Q1. Why does chunking matter so much? Because retrieval can only return a chunk you created, so chunking caps the quality of everything downstream. If the passage that answers a question was split across two chunks, no reranker, threshold or prompt recovers it — and the resulting failure is the worst kind, because the retrieved chunk is true, relevant, and leads to the wrong conclusion.
Q2. What's the right chunk size? There isn't a universal one, and size is the wrong first question. The right question is whether each chunk is a complete unit of meaning for your documents — how long a self-contained rule tends to be in your corpus. Measure "does some chunk contain everything needed to answer this question" over a set of known questions, and optimise that subject to chunks staying small enough for retrieval to discriminate.
Q3. What does overlap buy and what does it cost? It makes boundary material appear in two chunks, so a rule split by one boundary is whole in the chunk that straddles it. It costs roughly S/(S−V) times the storage and embedding — 1.43× measured at 120/40 — plus a slightly noisier index from near-duplicate chunks. Typical is 10–20%.
Q4. Why not just make chunks very large? Because rules-intact hits its maximum exactly when retrieval becomes useless. One chunk containing the whole document keeps every rule and gives retrieval nothing to choose between. Large chunks also cost more input tokens per call and dilute the prompt with irrelevant surrounding text.
Q5. When does sentence chunking fail? Whenever a rule spans more than one sentence. In one of our examples it scores 3/3 because every rule is a single sentence; in another it fails, because the exception is its own sentence. The strategy didn't change — the documents did, which is the general point about chunking.
Q6. What should go in a chunk besides the text? The structural path — document title and section heading — prepended before embedding, because it costs a few tokens, carries topic signal that improves retrieval, and makes citation possible. Plus date and version as filterable metadata, so superseded documents can be excluded; similarity ranking has no notion of currency and will return stale content happily.
Q7. How do you evaluate a chunking change? Directly, and without generating anything: for a set of questions with known answers, check whether a chunk containing the complete answer exists. That's the cheapest measurement in the pipeline and it isolates chunking from the model entirely. Optimising against final answer quality also works and is much slower, since every change needs a full generation pass.
8. When to use / tradeoffs
Use recursive/structure-aware splitting when:
- Documents have headings, sections, or consistent paragraph structure
- Rules are stated with qualifications
- You want the best result per stored character
Use fixed-size with overlap when:
- Documents have no exploitable structure
- Content is uniform and you want predictable chunk counts
- You can afford ~1.2–1.5× storage
Don't chunk at all when:
- Documents are already short and atomic — FAQ entries, product records
| Situation | Why it breaks | Do this instead |
|---|---|---|
| Fixed size, no overlap | Splits rules mid-clause; 0/3 in the example | Add overlap, or split on structure |
| "Make chunks bigger" | Retrieval loses the ability to discriminate | Find the middle; measure both halves |
| Size copied from a tutorial | The right size depends on your documents | Measure rules-intact on your corpus |
| Sentence chunking on multi-sentence rules | Exception lands in its own chunk | Recursive, or retrieve neighbours |
| Chunk text with no heading | "This does not apply to..." — apply to what? | Prepend the structural path |
| No date/version metadata | Superseded content retrieves happily | Filterable metadata |
| Tables split by a character count | Meaning spans rows | Keep the table whole |
| Rule spread across documents | No chunking can fix it | Multi-document retrieval |
Honest limits. The document in §5 is 279 characters and the rules are enumerated by hand, which makes "rules intact" exactly computable and is not available to you on a real corpus — there the equivalent is a labelled question set, which is more work and the actual prerequisite. The recursive splitter is a two-level toy (sentences, then characters); production implementations walk several levels of structure and handle markdown, headings and lists. All measurements are in characters rather than tokens, which is convenient and wrong in the way Tokenization describes — real chunk limits are token limits, and the ratio varies by content. And the comparison assumes retrieval quality is determined by chunk completeness alone, when chunk length also affects embedding quality: very short chunks embed noisily, very long ones blur, and that interaction is not modelled here.
9. Summary + related articles
- Chunking caps everything downstream: retrieval can only return a chunk you created.
- The metric is not size, it's whether a rule survives intact — claim and exception in the same chunk.
- Measured: fixed 60 → 0/3 rules whole; fixed 120 → 1/3; +40 overlap → 3/3 at 1.43× storage; recursive → 3/3 with no duplication.
- A broken chunk fails in the worst way: true, relevant, highly scored, and leads to the opposite answer.
- Bigger is not simply better. Rules-intact maxes out exactly when the whole document is one chunk and retrieval can't discriminate.
- Overlap costs
S/(S−V)in storage, embedding, and index noise. 10–20% is typical. - Structure-aware splitting beat overlap on both axes here — 3/3 for fewer stored characters.
- Sentence chunking scores 3/3 on one document and fails on another. The strategy didn't change; the documents did.
- Prepend the heading path; store date and version as filterable metadata.
- Measure chunking directly — "does a complete answering chunk exist" — without generating anything.
Related:
- Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems — choosing the score floor that sits downstream of this
- Retrieval Techniques — what happens to the chunks you just made
- Document Processing: What You Index Sets the Ceiling — getting to clean text before you split it
- Anatomy of a RAG Pipeline — where chunking sits, and the threshold that can't rescue it
- Vector Search for Retrieval — how chunks get ranked once created
- Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For — storing the chunks and their metadata
- RAG Evaluation: Attributing Failure and Sizing the Eval Set — measuring retrieval separately from the answer
- Reranking: The Second Pass That Decides What the Model Sees — recovering precision when you retrieve generously
- Tokenization — why chunk limits are token limits, not character limits
- Context Fundamentals — the budget the retrieved chunks compete for
Resources
- Manning, Raghavan & Schütze — Introduction to Information Retrieval, Ch. 2 on the document unit — the pre-LLM treatment of exactly this decision, and still the clearest: https://nlp.stanford.edu/IR-book/
- LangChain — text splitter documentation, including the recursive character splitter §3.1 describes: https://python.langchain.com/docs/concepts/text_splitters/
- LlamaIndex — node parsers and the sentence-window pattern, which retrieves a small chunk and expands to its neighbours: https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/
- Günther et al. (2024) — Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models, arXiv:2409.04701 — embedding the whole document first and chunking afterwards, which sidesteps part of the §3.4 metadata problem: https://arxiv.org/abs/2409.04701
- Es et al. (2023) — RAGAS: Automated Evaluation of Retrieval Augmented Generation, arXiv:2309.15217 — context-recall metrics for §3.5: https://arxiv.org/abs/2309.15217