TL;DR — A chunk budget expressed in words or characters is a promise about the wrong unit. The model bills and truncates in tokens, and the tokens-per-word ratio varies with the content: in the run below, three passages of near-identical word count came out at 1.10, 2.03 and 5.14 tokens per word, so a single "600 word" setting means anywhere from 660 to 3,086 real tokens — a 4.7× spread from one number. Sizing by tokens fixes the budget; splitting on structure rather than a word offset fixes the other half, because a fixed offset cuts tables mid-row and a half-row retrieves as noise. Both fixes are worthless later unless the build settings — tokenizer, token budget, overlap, splitter — are written into the index itself, because a store whose build settings are unknown cannot be reproduced or compared against another store. This stops helping when your corpus is homogeneous single-script prose, where words and tokens stay proportional and the whole problem is theoretical.
1. Simple explanation
Every retrieval system cuts documents into chunks before embedding them. The cut has to happen somewhere, and two decisions govern it: how big each chunk is allowed to be, and where the boundary is permitted to fall. Most implementations answer the first in words or characters and the second with "wherever the counter runs out." Both answers are wrong in ways that stay invisible on English prose and become expensive on anything else.
Size first. The number that actually constrains you is the token count, because that is what the embedding model's context limit is denominated in and what the bill is computed from. Words are a proxy for tokens, and a poor one — a word of Devanagari or of mathematical notation can be several tokens, while a common English word is one. So the same "600" means different things in different documents, and two indexes built with the same setting are not comparable.
Position second. A word counter has no idea what it is cutting through. Give it a table and it will happily place a boundary between a row's identifier and the instruction that identifier refers to, producing two chunks where the first is a truncated fragment and the second is a floating instruction with nothing to attach it to. Neither retrieves.
Analogy — cutting a printed price list with a guillotine set to "every 30 centimetres." The guillotine is consistent and fast and knows nothing about the page. On a column of flowing prose you get slightly ragged paragraphs, which is survivable. On the tariff table it comes down between the product code and the price, and now you own two useless pieces of paper: a list of codes with no prices, and a list of prices with nothing to identify them. The fix is not a sharper blade or a more careful measurement — it is to cut on the horizontal rules that the table already prints, and to measure in rows rather than centimetres. Token-aware, structure-aware chunking is that change: measure in the unit the consumer actually uses, and cut where the document already offers a seam.
2. Diagram
THE UNIT PROBLEM — one setting, three real sizes
(measured, §5)
chunk_size_words = 600
|
+--> English prose 1.10 tok/word -> 660 tokens [ ok ]
+--> Maths notation 2.03 tok/word -> 1,219 tokens [ 2x ]
+--> Devanagari 5.14 tok/word -> 3,086 tokens [ 5x, may overflow ]
one number, 4.7x spread => cannot be maximised safely,
and two runs are not comparable
THE POSITION PROBLEM — same table, two splitters
SOURCE WORD OFFSET (12) TOKEN + ROW BOUNDARY (30)
+------+--------------+ chunk0: chunk0:
| Code | Fault | | Code | Fault | Action | | Code | Fault | Action |
| E01 | Sensor open | | E01 | Sensor open <-cut | E01 | Sensor | Replace|
| E02 | Over-temp | | E02 | Over- | Check |
| E03 | Low pressure | chunk1: chunk1:
+------+--------------+ | Replace probe | E02 | | Code | Fault | Action | <- header repeated
... | E03 | Low pr| Refill |
rows intact: 2 / 3 rows intact: 3 / 3
chunk0 ends mid-row; "Replace probe" is
stranded from the code it belongs to
The right-hand splitter is not smarter about language. It is only obeying two rules the left-hand one does not have: never end a chunk inside a row, and count the budget in tokens.
3. How it works
3.1 Tokens are the unit the system actually enforces
The context limit of an embedding model, the truncation rule it applies when you exceed that limit, and the price you pay are all denominated in tokens. Nothing downstream is denominated in words. When you set a chunk budget in words you are setting it in a unit that gets converted, at an exchange rate you do not control, by the tokenizer. Tokenization covers why the rate varies — byte-pair encoding merges frequent sequences into single tokens and fragments unfamiliar ones down toward individual characters or bytes — and this article takes that result as given.
The consequence for chunking specifically is that a word budget has to be set pessimistically. If you know some of your corpus runs at five tokens per word, you must choose a word count that is safe at five, which means every English document in the same corpus is chunked at a fifth of the size it could have used. You pay for the worst case everywhere. Setting the budget in tokens directly removes that coupling: each document fills the budget it is actually allowed.
3.2 A word offset is not a boundary, it is an interruption
Splitting on a word count treats the document as a flat sequence. Prose tolerates this because a paragraph cut slightly early is still mostly a paragraph, and overlap patches the rest — the mechanism Chunking Strategies analyses in detail. Structured content does not tolerate it, because in a table the unit of meaning is the row, and a row is only usable when its key and its value are in the same chunk. Cut between them and you have not produced a smaller chunk; you have produced two non-chunks.
This is why the fix is qualitatively different from tuning. There is no chunk size at which a structure-blind splitter reliably keeps rows whole — the boundary lands where the counter runs out, and the counter does not know where rows are. You either teach the splitter about rows or you keep losing them at a rate set by how the row lengths happen to line up with your chunk size.
3.3 Structure-aware splitting, and why the header gets repeated
A structure-aware splitter walks the document's own hierarchy — sections, then paragraphs, then rows — and only places a boundary at a seam the document already has. Applied to a table, it packs whole rows until the next row would exceed the token budget, then starts a new chunk.
There is a second, easily missed step: the new chunk must repeat the header row. A chunk containing only | E03 | Low pressure | Refill circuit | has lost the column names, so neither a reader nor a model can tell which column is the fault and which is the action. Repeating the header costs a handful of tokens per chunk and makes each chunk independently interpretable. This is the same principle as prepending the section heading to a prose chunk, which the sibling article treats as metadata; for tables it is not decoration but a correctness requirement.
3.4 What must not change: the metadata contract
Retrieval depends on the fields attached to each chunk, not just the text. If a chunk carries a document identifier, a source filename, a subsystem grouping, a page and a section heading today, an upgrade to the splitter must keep emitting exactly those fields with the same meanings. Changing the sizing logic and the metadata schema in one step means that when retrieval quality moves, you cannot attribute the movement. Upgrade the splitter; leave the contract alone; measure; then change the contract separately if you still want to.
3.5 Write the build settings into the index
An index is a derived artefact. The chunks in it are a function of the tokenizer, the token budget, the overlap, and the splitter that produced them — and none of those are recoverable from the chunks themselves. Two stores with different recall might differ because one chunker is better, or because one simply had a larger budget. Without the settings recorded, that question is unanswerable and every later retrieval comparison is uninterpretable.
Environment variables are not the record. They describe the machine now, not the run that built the store weeks ago, and they change without touching the index. The record has to travel with the artefact: a small header block alongside the chunks naming the tokenizer, the max tokens, the overlap and the splitter version.
The analogous discipline for the embedding half is already standard practice, and Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For documents the failure it prevents — an index queried with a different model than it was built with returns plausible nonsense until someone notices. Chunk settings deserve the same treatment for the same reason, with one difference in severity: an embedding mismatch produces garbage immediately and loudly, whereas an unrecorded chunk config produces results that look fine and are merely un-comparable, so nobody discovers the problem until they try to run an experiment.
3.6 Where this stops applying
If the corpus is uniformly single-script prose with no tables, the tokens-per-word ratio is roughly constant, a word budget is a fixed multiple of a token budget, and structure-aware splitting reduces to paragraph splitting. The whole apparatus buys nothing measurable. The cost of adopting it is real — a heavier dependency, a slower ingest, and a re-index — so on a homogeneous English prose corpus the honest answer is to keep the simple splitter and spend the effort on retrieval instead.
4. The math
4.1 The budget conversion
For a passage with W words and a tokenizer producing T tokens, define the observed ratio:
r = T / W tokens per word, content-dependent
tokens(budget_words) = budget_words * r
A word budget is safe for every document in the corpus only if it is chosen against the maximum ratio:
budget_words_safe = budget_tokens / r_max
and the resulting waste on the cheapest content is:
waste_fraction = 1 - r_min / r_max
4.2 Worked example
Using the three measured ratios from §5 (r = 1.10, 2.03, 5.14) and a model chunk limit of 3,200 tokens:
r_max = 5.14 -> budget_words_safe = 3200 / 5.14 = 622 words
At 622 words, the actual token usage per passage type:
Devanagari 622 * 5.14 = 3197 tokens (99.9% of the limit — as intended)
Maths notation 622 * 2.03 = 1263 tokens (39.5% of the limit)
English prose 622 * 1.10 = 684 tokens (21.4% of the limit)
waste_fraction = 1 - (1.10 / 5.14) = 1 - 0.214 = 0.786
So a single safe word budget leaves the English documents using 21.4% of the space they were entitled to — 78.6% wasted, and that waste is structural, not a tuning error. Setting the budget in tokens instead lets every document fill the same 3,200, and the ratio never enters the calculation.
The 4.7× figure quoted in the TL;DR is the same phenomenon seen from the other end: r_max / r_min = 5.14 / 1.10 = 4.67, the spread in real size produced by holding the word number fixed.
5. Real code
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
# Three passages of the SAME word count, from one multilingual service-manual corpus.
PASSAGES = {
"English prose": (
"Disconnect the mains supply before removing the access panel. "
"Wait two minutes for the capacitor to discharge fully. "
"Check the seal for cracks and replace it if any are visible."
),
"Devanagari": (
"पैनल हटाने से पहले मुख्य आपूर्ति बंद करें। "
"संधारित्र को पूरी तरह निर्वहन होने के लिए दो मिनट प्रतीक्षा करें। "
"सील में दरारें जांचें और दिखने पर बदलें।"
),
"Maths notation": (
"Let P_max = V^2 / (4*R_load) where R_load >= 0.5 ohm. "
"If dT/dt > 3.2 K/s then derate by sqrt(P_max / P_rated). "
"Tolerance is +/- 0.05 across 12 <= V <= 24."
),
}
print(f"{'passage':<16}{'words':>7}{'tokens':>8}{'tok/word':>10}")
print("-" * 41)
ratios = {}
for name, text in PASSAGES.items():
w = len(text.split())
t = len(enc.encode(text))
ratios[name] = t / w
print(f"{name:<16}{w:>7}{t:>8}{t/w:>10.2f}")
# A word budget is a promise about words, not about the model's real input.
BUDGET_WORDS = 600
print(f"\nA '{BUDGET_WORDS} word' chunk budget, priced in real tokens:")
for name, r in ratios.items():
print(f" {name:<16} -> {BUDGET_WORDS * r:>7.0f} tokens")
spread = max(ratios.values()) / min(ratios.values())
print(f"\nSame setting, {spread:.1f}x spread in actual size.")
# ---- what a boundary does to a table -------------------------------------
TABLE = """| Code | Fault | Action |
| E01 | Sensor open | Replace probe |
| E02 | Over-temp | Check airflow |
| E03 | Low pressure | Refill circuit |"""
def chunk_by_words(text, size):
"""Word-count splitting: structure-blind, the common default."""
words = text.split()
return [" ".join(words[i:i + size]) for i in range(0, len(words), size)]
def chunk_by_rows(text, max_tokens, enc):
"""Structure-aware: never break a row; pack rows up to a TOKEN budget."""
header, *rows = text.strip().split("\n")
out, cur = [], [header]
for row in rows:
trial = "\n".join(cur + [row])
if len(enc.encode(trial)) > max_tokens and len(cur) > 1:
out.append("\n".join(cur))
cur = [header, row] # repeat the header so the chunk stands alone
else:
cur.append(row)
out.append("\n".join(cur))
return out
word_chunks = chunk_by_words(TABLE, 12)
row_chunks = chunk_by_rows(TABLE, max_tokens=30, enc=enc)
def rows_intact(chunks):
"""A row is usable only if its code and its action land in the same chunk."""
pairs = [("E01", "Replace probe"), ("E02", "Check airflow"), ("E03", "Refill circuit")]
return sum(any(c in ch and a in ch for ch in chunks) for c, a in pairs)
print(f"\nword-split (12 words) : {len(word_chunks)} chunks, "
f"{rows_intact(word_chunks)}/3 rows intact")
print(f"row-split (30 tokens) : {len(row_chunks)} chunks, "
f"{rows_intact(row_chunks)}/3 rows intact")
print("\nfirst word-split chunk, verbatim:")
print(f" {word_chunks[0]!r}")
assert rows_intact(word_chunks) < 3, "word split should strand at least one row"
assert rows_intact(row_chunks) == 3, "row split must keep every row whole"
assert spread > 2.0, "token/word ratio spread should exceed 2x across scripts"
print("\nasserts passed")
# Output:
# passage words tokens tok/word
# -----------------------------------------
# English prose 30 33 1.10
# Devanagari 28 144 5.14
# Maths notation 32 65 2.03
#
# A '600 word' chunk budget, priced in real tokens:
# English prose -> 660 tokens
# Devanagari -> 3086 tokens
# Maths notation -> 1219 tokens
#
# Same setting, 4.7x spread in actual size.
#
# word-split (12 words) : 3 chunks, 2/3 rows intact
# row-split (30 tokens) : 2 chunks, 3/3 rows intact
#
# first word-split chunk, verbatim:
# '| Code | Fault | Action | | E01 | Sensor open'
#
# asserts passed
The stranded row is worth reading literally. chunk0 ends at '| E01 | Sensor open' — the fault code and its symptom are present, the instruction Replace probe is in the next chunk, and a question like "what do I do about E01?" now has to retrieve two chunks and join them to be answerable. It will usually retrieve one.
Production splitters do this hierarchically rather than for tables alone. Docling's HybridChunker, for example, takes a tokenizer and a max-token budget and walks a parsed document's structure — sections, lists, tables — emitting token-bounded chunks that respect those boundaries. The mechanism above is the same idea reduced to one document type so it fits in a runnable block.
6. Real-world example
A team indexed a multilingual equipment-maintenance corpus: English procedures, translated procedures in Devanagari, and fault-code tables shared across both. Chunking was word-based at roughly 600 words with an overlap of a little over ten percent, and English retrieval was good enough that nobody questioned the settings for months.
Two failures surfaced together. Technicians searching in the translated documents reported that answers were being cut off mid-procedure — the 600-word chunks were running past the embedding model's context limit and being silently truncated, because 600 words of Devanagari is roughly 3,000 tokens, not 660. And fault-code lookups, in both languages, returned a code with no remedy about a third of the time, because the word counter had been cutting tables mid-row all along. Both had been happening since the first ingest.
The instructive part was the attempted fix. The team dropped the chunk size to 200 words, which stopped the truncation and made the table problem worse — smaller chunks meant more boundaries, so more rows got cut. The two symptoms had one cause each and needed one fix each: a token budget for the truncation, a structure-aware splitter for the rows. Tuning the single knob they had could only ever trade one failure for the other.
The third finding came later and cost the most time. When they rebuilt with the new chunker, recall improved — and nobody could say by how much, because the original index had no record of the settings that produced it. The old numbers had been measured against a store whose budget, overlap and tokenizer were no longer known. The comparison had to be thrown away and both configurations re-ingested and re-measured from scratch, purely to recover a baseline that a four-line header block would have preserved.
7. Interview questions companies actually ask
Q1. Why size chunks in tokens rather than words or characters? Because tokens are the unit every downstream constraint is expressed in — the embedding model's context limit, the truncation it applies when you exceed that limit, and the cost per call. Words convert to tokens at a rate that depends on the content: common English runs near 1:1, while non-Latin scripts and dense notation can run 2–5×. A word budget therefore has to be set safe for the worst content in the corpus, which wastes most of the budget on everything else, and two indexes built with the same word setting are not the same size in any meaningful sense.
Q2. A table keeps getting split across chunks and retrieval degrades. What is happening and how do you fix it? A structure-blind splitter places boundaries wherever its counter runs out, which for a table means between a row's key and its value. Each half is unusable: the key half has no answer attached, the value half has nothing identifying it. It is not fixable by tuning chunk size — smaller chunks create more boundaries and make it worse. The fix is a splitter that treats a row as atomic and packs whole rows up to a token budget, repeating the header row in each chunk so every chunk remains independently interpretable.
Q3. Why repeat the table header in every chunk? Because a chunk is retrieved and read alone. A chunk holding only data rows has lost the column names, so neither a model nor a person can tell which field is the identifier and which is the instruction. The cost is a few tokens per chunk; the alternative is chunks whose contents are ambiguous. It is the same reasoning that leads to prepending section headings to prose chunks, but for tables it affects correctness rather than just retrieval quality.
Q4. What should be stored alongside a vector index, beyond the vectors? The build provenance: the embedding model and its version, and the chunking settings — tokenizer, maximum tokens, overlap, and splitter identity. The embedding half is the better-known requirement because querying with a mismatched model silently returns nonsense. The chunking half matters for a quieter reason: without it, a store cannot be reproduced or compared with another store, so no retrieval experiment run against it is interpretable.
Q5. You improved the chunker and recall went up. What would make that result untrustworthy? If anything else changed at the same time — the embedding model, the metadata fields attached to each chunk, the query set, the retrieval depth — the improvement cannot be attributed to the chunker. The specific trap with chunking upgrades is changing the metadata contract in the same commit, because retrieval depends on those fields, so a movement in recall has two candidate causes. Change the sizing logic, hold the contract fixed, measure, and only then revisit the schema.
Q6. When is token-aware, structure-aware chunking not worth adopting? When the corpus is homogeneous single-script prose with no tabular or code content. There the tokens-per-word ratio is roughly constant, so a word budget is just a token budget in different units, and structure-aware splitting collapses into paragraph splitting. The dependency, the slower ingest and the mandatory re-index are real costs paid for no measurable gain.
8. When to use / tradeoffs
Reach for token-aware, structure-aware chunking when:
- The corpus spans more than one script or mixes prose with notation, so the tokens-per-word ratio is not constant.
- Documents contain tables, code blocks, or config where a row, function or block is the unit of meaning.
- You intend to run retrieval experiments and compare index builds, which requires the builds to be described.
- You are near the embedding model's context limit and want to use the budget rather than leave 70% of it unused for safety.
| Situation | Why it breaks | Use instead |
|---|---|---|
| Homogeneous English prose, no tables | Ratio is near-constant, so a word budget is already a token budget in disguise; structure-awareness reduces to paragraph splitting | Keep the simple splitter; spend the effort on retrieval or reranking |
| Ingest is CPU-bound and already the bottleneck | Tokenizing to measure every candidate boundary costs real time on a large corpus | Size by tokens but split on cheap structural seams; do not tokenize speculatively at every offset |
| The answer routinely spans several rows or sections | Keeping each row whole does not help when the question needs three of them joined | Retrieve more chunks, or restructure the source; this is a retrieval-strategy problem, not a chunking one |
| No labelled question set exists | You can measure "rows intact" but not whether retrieval actually improved | Build the question set first — it is the prerequisite for every claim in this article |
Honest limits. The rows_intact metric in §5 is computable because the table has three rows enumerated by hand; on a real corpus the equivalent is a labelled question set, which is substantially more work and is the actual prerequisite for any of these claims. The row-packing splitter shown is a single-level toy that understands one document type — production implementations walk several levels of a parsed hierarchy and have to handle merged cells, tables split across pages, and documents whose structure the parser recovers imperfectly, which is a harder problem than the chunking logic sitting on top of it. The measured ratios come from one tokenizer (cl100k_base) and three short passages; both the magnitude and the ordering will differ for another tokenizer, and a different Devanagari passage would not give exactly 5.14. Recording the build settings makes two indexes comparable, which is not the same as making them correct — it removes one confound from an experiment that still needs a real question set and a defensible metric to mean anything. And the whole argument assumes chunk completeness drives retrieval quality, when chunk length independently affects embedding quality: very short chunks embed noisily and very long ones blur, an interaction not modelled here.
9. Summary + related articles
- Tokens are the unit every downstream limit is denominated in; words convert at a content-dependent rate that measured 1.10 to 5.14 tokens per word across three passages, a 4.7× spread from one setting.
- A safe word budget must be chosen against the worst ratio in the corpus, which wasted 78.6% of the available budget on English content in §4's worked example. A token budget removes the coupling.
- A word offset is not a boundary. It cuts tables mid-row, and shrinking chunk size makes that worse, not better — the two failures need two different fixes.
- Structure-aware splitting packs whole rows to a token budget and repeats the header, so every chunk is independently interpretable.
- Upgrade the sizing logic while holding the metadata contract fixed, or a change in retrieval quality cannot be attributed.
- Write tokenizer, max tokens, overlap and splitter into the index. Environment variables describe the machine now, not the run that built the store.
- Boundary condition: on homogeneous single-script prose with no structured content, the ratio is near-constant and structure-awareness collapses to paragraph splitting — the apparatus costs a dependency, a slower ingest and a re-index for no measurable gain.
Related:
- Chunking Strategies — the four splitting strategies and what overlap actually buys; this article is the token-and-structure upgrade to the sizing it describes in characters
- Chunk Size, Overlap, and Retrieval Thresholds in RAG Systems — why a similarity threshold is not portable across scoring functions
- Tokenization — why the tokens-per-word ratio varies by script and domain in the first place
- Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For — the embedding-model half of index provenance, and the silent failure it prevents
- Document Processing: What You Index Sets the Ceiling — parsing and extraction upstream of the splitter
Resources
- Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft — Chapter 2 (regular expressions, text normalization, and subword tokenization) for why subword vocabularies fragment unfamiliar text.
- Sennrich, Haddow & Birch, "Neural Machine Translation of Rare Words with Subword Units," ACL 2016 — the byte-pair-encoding paper underlying the tokenizers in use today.
- tiktoken — the tokenizer used for every measurement in §5;
get_encoding("cl100k_base")reproduces the numbers. - Docling — document parser whose
HybridChunkerperforms tokenizer-aware, structure-respecting chunking of the kind described in §5. - The Unicode Standard, Annex #29: Text Segmentation — why "word" is itself a script-dependent notion before tokenization enters the picture.