Tokenization

TL;DR — A model never sees your text. It sees tokens: chunks of characters produced by a compression algorithm learned from training data, where frequent sequences become one token and rare ones fragment. That single fact explains most surprises about LLM cost and behaviour. In the worked example below, a common word is 1 token and an unfamiliar word of the same length is 9 — nine times the price for one word, on the same model. Nothing is ever "out of vocabulary": worst case, text degrades to individual characters, silently and expensively. So token counts are not proportional to word counts, they vary by language and domain, and any cost estimate built on word counts will be wrong in the direction that hurts. It stops being predictable the moment your text stops resembling the tokenizer's training data — code, names, non-Latin scripts, and long numbers all fragment far worse than prose.

---

1. Simple explanation

You type words. The model consumes numbers. Something has to convert between them, and that something is the tokenizer.

The obvious approach — one number per word — fails badly. You'd need a dictionary of every word in every language, and you'd still be stuck the first time someone types a name or a typo. The opposite extreme — one number per letter — never runs out of coverage but throws away all structure, so the model must relearn "the" from three characters every time.

Real tokenizers sit in between. They learn, from a large pile of text, which character sequences appear often enough to deserve their own token. the gets one. ing gets one. A rare surname gets chopped into pieces. It's a compression scheme fitted to whatever text the tokenizer was trained on.

Analogy — shorthand for a specific job. A court stenographer has single strokes for "objection" and "the witness", because those recur constantly. Hand the same stenographer a chemistry lecture and they're spelling out "polytetrafluoroethylene" letter by letter — slower, more strokes, same skill. Nothing is broken; the shorthand was fitted to courtrooms. That is exactly why your token counts jump when your text leaves the tokenizer's comfort zone, and why you are billed by strokes, not by words.

---

2. Diagram

WHAT ACTUALLY REACHES THE MODEL

  "the reader"  ──tokenizer──▶  ['the</w>', 'reader</w>']  ──▶  [1832, 9041]
                                 ▲                              ▲
                          learned chunks                   the model's real input


HOW THE CHUNKS ARE LEARNED  (byte-pair encoding)

  start: every word is a list of characters
     r e a d e r </w>

  repeatedly merge the most frequent adjacent pair:
     step 1   'e'+'a'   -> 'ea'        r ea d e r </w>
     step 2   'ea'+'d'  -> 'ead'       r ead e r </w>
     step 4   'r'+'ead' -> 'read'      read e r </w>
     step 6   'er'+'</w>'-> 'er</w>'   read er</w>
     step 10  'read'+'er</w>' -> 'reader</w>'      ONE token

  frequent sequence  -> single token
  rare sequence      -> stays in pieces


THE COST CONSEQUENCE — same model, same one word

  'reader'      1 token   ████                          1.0x
  'readership'  7 tokens  ████████████████████████████  7.0x
  'xylophone'   9 tokens  ████████████████████████████████████  9.0x
                          ▲
                  never seen in training -> falls back to characters
                  no error. no warning. just 9x the bill.

---

3. How it works

3.1 Byte-pair encoding, the algorithm you can read in ten lines

Most modern tokenizers are variants of byte-pair encoding (BPE). Training it is genuinely simple:

1. Split every word in the corpus into characters, with a marker for word-end.
2. Count every adjacent pair of symbols across the whole corpus.
3. Merge the most frequent pair into a single new symbol.
4. Repeat until you have as many tokens as you want.

The merge list is the tokenizer. To tokenize new text you replay those merges, in the order they were learned, and whatever symbols survive are your tokens. §5 implements this from scratch and it's about forty lines.

The consequence worth internalising: the vocabulary is a function of the training corpus. A tokenizer trained mostly on English web text will have single tokens for common English words and will fragment everything else. That isn't a bug; it's the compression working as designed on data it wasn't fitted to.

3.2 Nothing is ever out-of-vocabulary — and that's the expensive part

Older tokenizers had an <UNK> token for unknown words, which lost information outright. BPE cannot fail: if no merge applies, the text decomposes to individual characters, and characters are always in the vocabulary.

This is a real improvement and it hides a cost. There is no error, no warning, and no signal in your logs when text tokenizes badly. A word that should cost 1 token quietly costs 9. The failure is silent and financial rather than loud and functional, which is why it's usually discovered on an invoice.

3.3 Where token counts explode

Roughly, English prose runs about 0.75 words per token — so ~1.3 tokens per word. Things that break that ratio, worst first:

Two practical warnings. Whitespace usually belongs to the token — " the" and "the" are frequently different tokens, which is why a stray leading space can change behaviour. And numbers are worse than people expect: 1234567 may be several tokens with no relationship to its arithmetic value, which is part of why models are unreliable at long-number arithmetic.

3.4 Why this determines your bill and your limits

Every commercial concept in LLM engineering is denominated in tokens:

• Pricing is per million input and output tokens.
• The context window is a token limit, not a character or word limit.
• Latency tracks generated token count, because output is produced serially.
• Rate limits are usually tokens per minute.

So a word-based estimate is not a rough approximation — it's the wrong unit. And it errs optimistically for exactly the content most likely to be in your system: identifiers, code, and non-English text. See Reasoning Budgets: When Thinking Tokens Are Waste for what happens when invisible tokens join the bill, and RAG Cost Optimization: Find the Step That Runs Forty Times for controlling the total.

3.5 Practical rules

Count, don't estimate. Every provider ships a tokenizer library or a counting endpoint. Use it on your real text before you promise anyone a cost.

Count with the right tokenizer. Different model families tokenize differently, sometimes by 20% on the same string. A count from one vendor's tool tells you little about another's.

Watch the ratio, not just the total. Tokens-per-character is a useful health metric: when it drifts up, something in your input has changed shape — a new document type, a new language, a JSON blob that used to be prose.

3.6 Where this model of tokenization stops applying

Some newer systems are byte-level or tokenizer-free, operating on raw bytes so there is no vocabulary at all — the fragmentation story changes shape (everything is uniform, and sequences get longer). Multimodal models tokenize images and audio by entirely different schemes, where an image is a fixed patch count unrelated to anything here. And this article says nothing about how tokens become meaning; that's the next stage, covered in How LLMs Work.

---

4. The math

4.1 Counting and cost

  tokens(text) = length of the symbol sequence after replaying all merges

  cost = (tokens_in * rate_in + tokens_out * rate_out) / 1e6

  tokens ≈ words * R      where R ≈ 1.3 for English prose
                          and R can be 3-10 for identifiers, code, other scripts

Because R is a property of the text, not the model, a fixed conversion factor is only ever valid for the content class you measured it on.

4.2 Worked example

Train BPE for 12 merges on a small corpus about readers and leaders. The merges learned, in order:

   1. 'e' + 'a'  ->  'ea'
   2. 'ea' + 'd'  ->  'ead'
   3. 'e' + '</w>'  ->  'e</w>'
   4. 'r' + 'ead'  ->  'read'
   5. 'e' + 'r'  ->  'er'
   6. 'er' + '</w>'  ->  'er</w>'
   7. 'l' + 'ead'  ->  'lead'
   8. 't' + 'h'  ->  'th'
   9. 'th' + 'e</w>'  ->  'the</w>'
  10. 'read' + 'er</w>'  ->  'reader</w>'
  11. 's' + '</w>'  ->  's</w>'
  12. 'lead' + 'er</w>'  ->  'leader</w>'

Watch the order. ea had to exist before ead, which had to exist before read, which had to exist before reader. Tokens are built up from smaller merges — the vocabulary is a hierarchy, not a flat list.

Now tokenize five words:

  reader       1 tokens  ['reader</w>']   (in corpus)
  leader       1 tokens  ['leader</w>']   (in corpus)
  readable     5 tokens  ['read', 'a', 'b', 'l', 'e</w>']   (in corpus)
  readership   7 tokens  ['read', 'er', 's', 'h', 'i', 'p', '</w>']   (NEVER SEEN)
  xylophone    9 tokens  ['x', 'y', 'l', 'o', 'p', 'h', 'o', 'n', 'e</w>']   (NEVER SEEN)

Three things to read off this.

reader and leader are 1 token each because they appeared often enough to earn a merge all the way to the full word. readable appears in the corpus but only three times, so it never got merged past read — appearing in training is not enough; it has to be frequent.

readership was never seen, yet it still tokenizes sensibly into read + er + the rest. That's the subword payoff: an unseen word made of familiar parts is still partly compressed.

xylophone shares nothing with the corpus and decomposes to one token per character. No error, no <UNK>, just nine tokens.

4.3 The cost, stated plainly

  'reader'        1 tokens  ->  1.0x the cost of 'reader'
  'readership'    7 tokens  ->  7.0x the cost of 'reader'
  'xylophone'     9 tokens  ->  9.0x the cost of 'reader'

One word, same model, 9× the price depending only on whether the tokenizer's training data happened to contain it. Scale that to a corpus of product codes or a non-English language and it is the difference between a viable product and an unviable one.

---

5. Real code

"""Byte-pair encoding from scratch: learn merges, then split words with them."""
from collections import Counter

CORPUS = ("the reader reads the readable reader " * 3 +
          "a leader leads the leading leader " * 3)


def word_freqs(text: str) -> dict[tuple[str, ...], int]:
    """Each word starts as a sequence of characters, with a marker for word-end."""
    freqs: Counter = Counter()
    for w in text.split():
        freqs[tuple(w) + ("</w>",)] += 1
    return dict(freqs)


def most_common_pair(freqs: dict[tuple[str, ...], int]) -> tuple[str, str] | None:
    pairs: Counter = Counter()
    for symbols, n in freqs.items():
        for a, b in zip(symbols, symbols[1:]):
            pairs[(a, b)] += n
    return pairs.most_common(1)[0][0] if pairs else None


def apply_merge(freqs: dict, pair: tuple[str, str]) -> dict:
    a, b = pair
    out = {}
    for symbols, n in freqs.items():
        merged, i = [], 0
        while i < len(symbols):
            if i < len(symbols) - 1 and symbols[i] == a and symbols[i + 1] == b:
                merged.append(a + b)
                i += 2
            else:
                merged.append(symbols[i])
                i += 1
        out[tuple(merged)] = n
    return out


def train_bpe(text: str, n_merges: int) -> list[tuple[str, str]]:
    freqs = word_freqs(text)
    merges = []
    for _ in range(n_merges):
        pair = most_common_pair(freqs)
        if pair is None:
            break
        merges.append(pair)
        freqs = apply_merge(freqs, pair)
    return merges


def tokenize(word: str, merges: list[tuple[str, str]]) -> list[str]:
    """Apply the learned merges in the order they were learned."""
    symbols = tuple(word) + ("</w>",)
    for a, b in merges:
        symbols = tuple(apply_merge({symbols: 1}, (a, b)).keys())[0]
    return list(symbols)


merges = train_bpe(CORPUS, n_merges=12)
print("MERGES LEARNED, in order (each becomes one token)")
for i, (a, b) in enumerate(merges, 1):
    print(f"  {i:>2}. {a!r} + {b!r}  ->  {a + b!r}")

print("\nSPLITTING WORDS with those merges")
for w in ["reader", "leader", "readable", "readership", "xylophone"]:
    toks = tokenize(w, merges)
    seen = "in corpus" if w in CORPUS.split() else "NEVER SEEN"
    print(f"  {w:<12} {len(toks)} tokens  {toks}   ({seen})")

print("\nTOKENS != WORDS -- this is why token counts surprise people")
SAMPLES = {
    "the reader reads":        "common words, seen often",
    "readership":             "rarer word -> more pieces",
    "xylophone":              "unseen -> falls back to characters",
}
for text, why in SAMPLES.items():
    n_words = len(text.split())
    n_toks = sum(len(tokenize(w, merges)) for w in text.split())
    print(f"  {n_words} word(s) -> {n_toks:>2} tokens   {text!r:<22} {why}")

rare = sum(len(tokenize(w, merges)) for w in "readership".split())
common = sum(len(tokenize(w, merges)) for w in "reader".split())
unseen = sum(len(tokenize(w, merges)) for w in "xylophone".split())

print(f"\nCOST CONSEQUENCE (billing is per TOKEN, not per word)")
for label, n in (("'reader'", common), ("'readership'", rare), ("'xylophone'", unseen)):
    print(f"  {label:<14} {n:>2} tokens  ->  {n / common:.1f}x the cost of 'reader'")

# Frequent sequences become single tokens; rare and unseen text fragments.
assert len(tokenize("reader", merges)) < len(tokenize("xylophone", merges))
assert len(tokenize("readership", merges)) > len(tokenize("reader", merges))
# Nothing is ever out-of-vocabulary: worst case, it decomposes to characters.
assert all(len(t) >= 1 for t in tokenize("xylophone", merges))
# 9 characters -> 9 tokens: one per character, except the final 'e' which merged
# with the word-end marker into 'e</w>'. Unseen text degrades to characters, never
# to an out-of-vocabulary error.
assert unseen == 9, unseen
print("\nall assertions passed")

# Output:
#   MERGES LEARNED, in order (each becomes one token)
#      1. 'e' + 'a'  ->  'ea'
#      2. 'ea' + 'd'  ->  'ead'
#      3. 'e' + '</w>'  ->  'e</w>'
#      4. 'r' + 'ead'  ->  'read'
#      5. 'e' + 'r'  ->  'er'
#      6. 'er' + '</w>'  ->  'er</w>'
#      7. 'l' + 'ead'  ->  'lead'
#      8. 't' + 'h'  ->  'th'
#      9. 'th' + 'e</w>'  ->  'the</w>'
#     10. 'read' + 'er</w>'  ->  'reader</w>'
#     11. 's' + '</w>'  ->  's</w>'
#     12. 'lead' + 'er</w>'  ->  'leader</w>'
#
#   SPLITTING WORDS with those merges
#     reader       1 tokens  ['reader</w>']   (in corpus)
#     leader       1 tokens  ['leader</w>']   (in corpus)
#     readable     5 tokens  ['read', 'a', 'b', 'l', 'e</w>']   (in corpus)
#     readership   7 tokens  ['read', 'er', 's', 'h', 'i', 'p', '</w>']   (NEVER SEEN)
#     xylophone    9 tokens  ['x', 'y', 'l', 'o', 'p', 'h', 'o', 'n', 'e</w>']   (NEVER SEEN)
#
#   TOKENS != WORDS -- this is why token counts surprise people
#     3 word(s) ->  4 tokens   'the reader reads'     common words, seen often
#     1 word(s) ->  7 tokens   'readership'           rarer word -> more pieces
#     1 word(s) ->  9 tokens   'xylophone'            unseen -> falls back to characters
#
#   COST CONSEQUENCE (billing is per TOKEN, not per word)
#     'reader'        1 tokens  ->  1.0x the cost of 'reader'
#     'readership'    7 tokens  ->  7.0x the cost of 'reader'
#     'xylophone'     9 tokens  ->  9.0x the cost of 'reader'
#
#   all assertions passed

Twelve merges on a toy corpus; real tokenizers learn 30,000–200,000 on terabytes. The algorithm is the same one — this is not a simplified analogue of BPE, it is BPE.

---

6. Real-world example

A team launched a support assistant in one market and priced it from a pilot: average conversation, average tokens, a comfortable margin. The pilot was in English.

They expanded to a second market where customers wrote in a non-Latin script, often mixing two languages in a sentence. Cost per conversation roughly tripled. Nothing had changed in the code, the model, or the prompt.

The tokenizer was the whole story. For that script it was producing close to one token per character, and code-mixed sentences fragmented worse still because neither language's merges applied cleanly. Conversations of identical length — same number of words, same information — cost three times as much.

Two things made it hurt. The margin had been calculated from an English word-count ratio, so the model was wrong for the new market from the first day. And nothing surfaced it: no errors, no latency change, no quality complaints. It appeared as a monthly invoice, six weeks in.

The fixes were unglamorous: log tokens per conversation broken down by detected language, price per market rather than globally, and — the one that actually moved the number — trim the system prompt, which was being re-sent in full on every turn and was itself fragmenting badly in the new locale.

The general lesson: tokenization is where text properties become financial properties. Any per-unit assumption you carry from one content type to another is a guess, and it's usually optimistic.

---

7. Interview questions companies actually ask

Q1. What is a token and why not just use words? A token is a chunk of characters the tokenizer learned to treat as one unit. Words don't work because you'd need every word in every language and would still fail on names and typos; characters work but throw away structure the model would have to relearn constantly. Subword tokenization sits between: frequent sequences get one token, rare ones fragment, so coverage is total and common text stays compact.

Q2. How does byte-pair encoding work? Start with every word as characters, count all adjacent symbol pairs across the corpus, merge the most frequent pair into a new symbol, repeat until you hit your target vocabulary size. The ordered merge list is the tokenizer, and you tokenize new text by replaying those merges. Note the merges are hierarchical — ea before ead before read before reader.

Q3. Why do token counts differ so much from word counts? Because the ratio depends on how well your text matches the tokenizer's training data. English prose is roughly 1.3 tokens per word. Long numbers, hashes, code, rare names and non-Latin scripts can be 3–10× that, since no useful merges were learned for them. That's why estimating cost from word counts is wrong in the direction that hurts.

Q4. What happens when a model meets a word that wasn't in training? With BPE, nothing dramatic — it decomposes into smaller learned pieces, and worst case into individual characters. There's no out-of-vocabulary error. The catch is that this is silent: a word that should cost one token quietly costs nine, with no error, no warning, and no log line. It's a financial failure rather than a functional one.

Q5. Why are LLMs bad at arithmetic on long numbers? Partly tokenization. A number like 1234567 may be split into several tokens whose boundaries have nothing to do with place value, so the model isn't seeing digits in a positional structure — it's seeing arbitrary chunks. That's not the whole story, but it's a real contributor and it's why digit-level tasks improve when numbers are spaced or split deliberately.

Q6. Does a leading space matter? Often yes. Most tokenizers attach whitespace to the following token, so " the" and "the" can be different tokens with different learned behaviour. It matters when you concatenate strings programmatically, and it's a classic source of "the prompt looks identical but behaves differently."

Q7. How would you reduce token spend without changing the model? Measure first with the real tokenizer on real traffic, broken down by content type — you'll usually find one class dominating. Then: trim anything re-sent every turn (system prompts especially), cache the static prefix, cap output length since output bills higher, and reduce retrieved context. Only then consider a different model.

---

8. When to use / tradeoffs

You need to think about tokenization when:

• Estimating or forecasting cost for anything
• Working in a language other than the tokenizer's dominant one
• Your text is identifiers, code, JSON, or long numbers
• You're near a context-window limit
• Cost per request changed and nothing in the code did

You can mostly ignore it when:

• Prototyping with small volumes of English prose
• The workload is small enough that cost doesn't matter yet

Honest limits. The implementation in §5 is real BPE but a toy configuration: 12 merges on a two-sentence corpus, so the specific numbers illustrate the mechanism and predict nothing about a production tokenizer with 100,000 merges. Real tokenizers also add machinery this omits — byte-level fallbacks, pre-tokenization rules, special tokens, normalisation — and some models use WordPiece or Unigram rather than BPE, which differ in how they select merges though not in the consequences described here. The "1.3 tokens per word" figure is a rule of thumb for English prose from a handful of tokenizers and should be measured rather than trusted. Finally, tokenization is one contributor to poor arithmetic, not the whole explanation; treat §7 Q5 as a partial answer.

---

9. Summary + related articles

• The model never sees text. It sees tokens — learned chunks where frequent sequences become one token and rare ones fragment.
• BPE: merge the most frequent adjacent pair, repeat. The ordered merge list is the tokenizer, and merges are hierarchical.
• Nothing is out-of-vocabulary. Unseen text degrades to characters — silently, with no error, at up to 9× the token count.
• Measured: reader = 1 token, xylophone = 9 tokens. One word, same model, 9× the price.
• Appearing in training isn't enough — readable was in the corpus and still fragmented, because it wasn't frequent.
• Everything commercial is denominated in tokens: pricing, context window, latency, rate limits. Word counts are the wrong unit.
• Token counts explode on non-Latin scripts, long numbers, code, hashes, and rare names — and this is where text properties become financial properties.
• Count with the real tokenizer on your text. Watch tokens-per-character as a health metric.
• Byte-level and multimodal models don't follow this model at all.

Related:

• How LLMs Work — what happens to tokens once the model has them
• Sampling and Temperature — how the next token gets chosen
• Reasoning Budgets: When Thinking Tokens Are Waste — hidden tokens you pay for and never see
• Context Fundamentals — the window is a token limit, not a text limit
• RAG Cost Optimization: Find the Step That Runs Forty Times — controlling total token spend
• LLM Observability — logging token counts per call, per route
• Embeddings and Cosine Similarity — the other way text becomes numbers, for a different purpose
• Cost-Latency Tradeoffs — where token cost sits among the levers

Resources

• Sennrich, Haddow & Birch (2016) — Neural Machine Translation of Rare Words with Subword Units, arXiv:1508.07909 — the paper that brought BPE to NLP; §5 implements this: https://arxiv.org/abs/1508.07909
• Kudo & Richardson (2018) — SentencePiece: A simple and language independent subword tokenizer, arXiv:1808.06226 — the language-agnostic approach behind many current tokenizers: https://arxiv.org/abs/1808.06226
• Kudo (2018) — Subword Regularization: Improving NMT Models with Multiple Subword Candidates, arXiv:1804.10959 — the Unigram alternative to BPE: https://arxiv.org/abs/1804.10959
• Hugging Face — Tokenizers library and its tokenization course chapter; the practical tool for counting and comparing: https://huggingface.co/learn/nlp-course/chapter6
• Jurafsky & Martin — Speech and Language Processing (3rd ed. draft), the chapter on words and tokenization: https://web.stanford.edu/~jurafsky/slp3/
• Companion notebook — train BPE on your own text, then compare token counts across content types and languages.