Transcription Error Rates: CER, WER, and What They Hide

TL;DR — "95% accurate" is not a measurement until you say accurate at what unit. The same transcription of a line of council minutes scores 5.26% CER and 20.00% WER — a 3.8x gap from the same two errors, because one wrong character ruins a whole word. Both are edit distance over reference length; only the symbol changes. Two decisions move the number more than most model changes do: normalization (case and punctuation folding takes one pair from 16.13% to 6.45% to 0.00%) and aggregation (a naive mean over pages reports 9.16% where the corpus figure is 3.94%, because a nine-character page carries the same vote as a sixty-character one). The metric stops working when errors are not equally costly — a wrong century and a stray comma weigh the same, and a system that invents fluent text can score better than one that honestly marks a gap.

---

1. Simple explanation

You have a scanned page and a machine's attempt at reading it. You want one number for how wrong it is. Count the smallest number of single-character fixes — insert, delete, or swap — that turn the machine's text into the correct text, then divide by the length of the correct text. That is Character Error Rate. Do the same counting whole words instead of characters and you have Word Error Rate.

The catch is that the two numbers disagree, often by a factor of four, on exactly the same transcription.

Analogy — marking a spelling test two ways. A pupil writes aproved for approved. A lenient marker counts letters: seven of eight are right, so the pupil scored 88% on that word. A strict marker counts words: the word is wrong, so the pupil scored 0%. Neither marker is cheating. They are answering different questions — how close was this to correct? versus would a reader accept it? — and if you do not say which marker you used, your "accuracy" figure means nothing. CER is the lenient marker; WER is the strict one.

---

2. Diagram

  THE SAME TWO ERRORS, COUNTED TWO WAYS

  reference : the council approved the drainage works  on the north road
  hypothesis: the council aproved  the drainage vvorks on the north road
                          ^^^^^^^                ^^^^^^

  CHARACTER VIEW                        WORD VIEW
  ──────────────                        ─────────
  a p p r o v e d                       [ approved ]
  a _ p r o v e d                       [ aproved  ]
    ^ 1 deletion                            ^ whole token wrong

  3 edits / 57 chars = 5.26%            2 edits / 10 words = 20.00%
  └──────── partial credit ────┘        └──── all or nothing ────┘

                        ratio = 3.8x

The gap is structural, not an artefact of this sentence: a word is wrong if any character in it is wrong. For English-like text averaging five to six characters per word, WER typically lands three to six times CER. A reported WER below CER on the same data means the two were computed on differently normalized text.

---

3. How it works

3.1 One primitive, two symbol sets

Edit distance — Levenshtein distance — is the smallest number of insertions, deletions and substitutions that turn one sequence into another. CER runs it over characters, WER over whitespace-delimited tokens. Nothing else differs between them.

Two properties matter in practice:

3.2 Which one to report

The choice follows from what consumes the text.

That last row is underused. Two engines can post identical CER with very different character: CER 4% / WER 8% means errors are concentrated in a few mangled words, usually a layout or segmentation failure. CER 4% / WER 22% means nearly every word carries one wrong character, usually a systematic confusion of one letter pair. The second is normally the easier fix, and reporting CER alone hides the difference.

3.3 Normalization is a policy, not a preprocessing step

Take a reference and hypothesis differing only in case and punctuation — Approved: the North Road works. against approved the north road works. Measured three ways, the same pair is 16.13% wrong, 6.45% wrong, or perfect, depending only on what was folded away first. No model changed.

A defensible pipeline states its steps in order: Unicode normalization (NFC or NFKC) first, since ligatures and combining accents otherwise count as substitutions; then case folding; then punctuation, the most contested, because folding it hides real errors in numeric and legal text; then whitespace collapse, almost always safe since line-wrapping is a layout artefact.

The rule that is not negotiable: apply exactly the same normalization to both sides, before measuring. Normalizing one side produces numbers that look plausible and are meaningless.

3.4 Where the method stops applying

All of the above assumes a single correct answer exists. Where more than one transcription is legitimately valid — an abbreviation expanded or left as printed — a single-reference error rate penalises a correct output, and you need multi-reference variants or a normalization policy that folds the variation away.

---

4. The math

4.1 The formula

                 edit_distance(reference, hypothesis)
    error_rate = ────────────────────────────────────
                        len(reference)

    CER : symbols are characters
    WER : symbols are whitespace-delimited tokens

Aggregating across pages is a second formula, and using the wrong one is the most common mistake in published transcription numbers:

                      sum_p edit_distance(p)
    corpus_rate = ─────────────────────────────      <- correct
                      sum_p len(reference_p)

    naive_mean  = ( sum_p rate_p ) / n_pages          <- weights a 9-char page
                                                         like a 61-char page

4.2 Worked example

Three pages of a council register:

  page  reference                                            len   edits   CER
  ----  ---------------------------------------------------  ---   -----   ------
  0     the council approved the drainage works ... road       57       3   0.0526
  1     adjourned                            -> adjoumed        9       2   0.2222
  2     no further business ... closed at nine (correct)       61       0   0.0000
                                                             ---   -----
                                                             127       5

  naive mean   = (0.0526 + 0.2222 + 0.0000) / 3 = 0.0916   ->  9.16%
  corpus rate  = 5 / 127                        = 0.0394   ->  3.94%

Page 1 is the single word adjourned read as adjoumed — one rn/m confusion, nine characters long. In the naive mean it carries exactly the weight of the sixty-one-character page and more than doubles the reported figure. The difference between the two rows is 5.22 points on three pages, and it grows as you add more short pages.

Scanned collections are full of short pages: title pages, plates with a caption, blank leaves carrying only a folio number. Any corpus with that tail reports an inflated error rate under a naive mean.

---

5. Real code

def levenshtein(a, b):
    if len(a) < len(b):
        a, b = b, a
    prev = list(range(len(b) + 1))
    for i, ca in enumerate(a, 1):
        cur = [i]
        for j, cb in enumerate(b, 1):
            cur.append(min(prev[j] + 1,                    # deletion
                           cur[j - 1] + 1,                 # insertion
                           prev[j - 1] + (ca != cb)))      # substitution
        prev = cur
    return prev[-1]

def cer(ref, hyp):
    return levenshtein(list(ref), list(hyp)) / max(1, len(ref))

def wer(ref, hyp):
    return levenshtein(ref.split(), hyp.split()) / max(1, len(ref.split()))

def corpus_cer(pairs):
    """Sum the numerators and denominators separately — never average the rates."""
    num = sum(levenshtein(list(r), list(h)) for r, h in pairs)
    den = sum(len(r) for r, _ in pairs)
    return num / max(1, den)

REF = "the council approved the drainage works on the north road"
HYP = "the council aproved the drainage vvorks on the north road"

print(f"CER  {cer(REF, HYP):.4f}")
print(f"WER  {wer(REF, HYP):.4f}")
print(f"ratio {wer(REF, HYP) / cer(REF, HYP):.1f}x")

PAGES = [
    (REF, HYP),
    ("adjourned", "adjoumed"),
    ("no further business was raised and the meeting closed at nine",
     "no further business was raised and the meeting closed at nine"),
]
naive = sum(cer(r, h) for r, h in PAGES) / len(PAGES)
print(f"naive mean       {naive:.4f}")
print(f"corpus-aggregated {corpus_cer(PAGES):.4f}")

assert round(cer(REF, HYP), 4) == 0.0526
assert round(wer(REF, HYP), 4) == 0.2000
assert round(naive, 4) == 0.0916
assert round(corpus_cer(PAGES), 4) == 0.0394
print("asserts passed")

# Output:
#   CER  0.0526
#   WER  0.2000
#   ratio 3.8x
#   naive mean       0.0916
#   corpus-aggregated 0.0394
#   asserts passed

---

6. Real-world example

A digitisation project reported 4% CER on a run of scanned registers and moved on. Six months later a name index built from the same text was missing roughly one entry in eight, and nobody could explain how a 96%-accurate transcription produced an index that unreliable.

Three things had gone wrong, and none was visible in the headline number.

The 4% was a naive mean. The collection had a long tail of short leaves — plates, blanks, title pages — each contributing a page-level rate to an unweighted average. Re-aggregated over characters, the true corpus figure was worse on the substantive pages and better on the tail, and the single number had been describing neither.

The errors were not uniformly distributed. Grouping the alignment operations by content showed two confusions — rn read as m, and w read as vv — accounting for the large majority of edits. Both are shape confusions driven by the resolution of the scan, and both fall disproportionately on capitalised words, which is where surnames live. A 4% character rate concentrated on proper nouns is close to useless for a name index while being perfectly adequate for topic search.

Nobody had measured coverage. The engine silently produced fluent text for passages that were physically unreadable. Those pages scored well, because invented text aligns better against a reference than a long run of gap markers would.

The lesson is not that CER is a bad metric. It is that a single aggregate, without the error distribution and without a coverage figure beside it, cannot tell you whether the text is fit for the purpose you have in mind.

---

7. Interview questions companies actually ask

Q1. Why can a character error rate exceed 100%? Because the denominator is the reference length, not the hypothesis length, and insertions are unbounded. A system that emits three times as much text as the page contains accumulates edits without increasing the divisor, so the rate rises past 1.0. A figure like 240% is diagnostic rather than nonsensical: it says insertions dominate, which usually means the recogniser is hallucinating or a segmentation step duplicated a region.

Q2. You have per-page error rates and need one number for the collection. What do you do? Sum the edit distances and sum the reference lengths, then divide — never average the per-page rates. An unweighted mean gives a nine-character page the same vote as a six-hundred-character one, and scanned collections have a long tail of short pages, so the naive mean is biased upward. If you genuinely want every page weighted equally, say so explicitly and report a median and interquartile range rather than a mean, because the distribution is skewed.

Q3. Your WER is lower than your CER. What happened? Almost certainly the two were computed on differently normalized text — for instance the word-level pass collapsed punctuation and case while the character-level pass did not. Structurally WER should exceed CER for word-like text, because any wrong character condemns the entire token, so an inversion is a pipeline bug rather than a property of the data.

Q4. Two engines both score 4% CER. How do you choose between them? Report WER alongside it and compare the ratio. A CER of 4% with WER 8% means errors are concentrated — a few badly mangled regions, often a layout failure — while 4% with WER 22% means nearly every word carries a single wrong character, which points to a systematic confusion. Then extract the confusion pairs from the alignment; if a handful of substitutions dominates, the cheaper fix is a post-processing rule, not a different engine.

Q5. Why might a system that refuses to transcribe unreadable text score worse than one that guesses? Because edit distance rewards fluent invention. A plausible guess aligns partially against the reference and accrues a few substitutions; an honest gap marker aligns against nothing and accrues a long run of insertions or deletions. An evaluation that reports only error rate therefore selects for confident fabrication, which is the opposite of what an archive wants. The fix is to report coverage — the fraction of the page the system attempted — beside the error rate, and to never quote one without the other.

Q6. How does normalization change the number, and what should you publish? Substantially: folding case and punctuation can take the same pair of strings from 16% error to zero. Publish the ordered list of normalization steps with the figure, apply exactly the same policy to reference and hypothesis, and apply it before measuring. Two error rates computed under different policies are not comparable, and most published transcription numbers omit the policy entirely.

Q7. What does edit distance fail to capture that matters in a document pipeline? Cost asymmetry and reading order. Transcribing 1957 as 1857 is one substitution, identical in cost to a stray comma, though one is catastrophic in a table of dates. And a two-column page read straight across produces text in which every character is correct and the sequence is meaningless — word-level WER registers heavy error while character-level CER on the concatenated page can look mild, and neither diagnoses the cause.

---

8. When to use / tradeoffs

Reach for CER/WER when:

• you have, or can produce, a trustworthy reference transcription;
• you are comparing engines or configurations on the same corpus under one normalization policy;
• you need a single tracked number that regressions can be measured against;
• the errors you care about are roughly uniform in cost.

Do NOT use when:

Honest limits. The implementations here are the textbook definitions and are deliberately unoptimized — O(n·m) time, fine for lines and slow for whole corpora, where you want a C-backed library. Every figure in this article rests on the assumption that a reference exists, and producing one is the genuinely expensive part: it usually means double-keying by hand, at a volume most projects underbudget, and no choice of metric substitutes for it. The examples are short constructed lines chosen so each mechanism is visible; they demonstrate behaviour, they are not a benchmark. Above all, these metrics say how much a transcription differs from the truth and never whether it is fit for purpose — 4% CER falling entirely on proper nouns is excellent for topic search and useless for a name index, and only you know which you are building.

---

9. Summary + related articles

• CER and WER are one formula — edit distance over reference length — applied to different symbols. The same two errors give 5.26% and 20.00% because a word is wrong if any character in it is wrong.
• Report both when comparing engines: the ratio distinguishes concentrated failures from systematic ones, and CER alone hides the difference.
• Aggregate over the corpus, not over pages. Three pages gave 9.16% as a naive mean and 3.94% aggregated correctly; a tail of short pages widens that gap.
• Normalization is a policy you publish, not a step you assume. The same pair of strings scored 16.13%, 6.45% and 0.00% under three policies.
• Extract the confusion pairs before retraining anything — the error distribution is almost always more skewed than the rate suggests, and the fix is often a rule.
• The boundary: the metric assumes all errors cost the same and that a reference exists. Where costs differ, or where a system may decline to answer, an error rate on its own will mislead you — pair it with field-level accuracy and a coverage figure.

Related:

• Aligning and Voting Across Transcription Engines — what to do once you can measure: combining several imperfect transcriptions into one better than any of them.
• Text Preprocessing — the tokenization and normalization operations that §3.3 turns into a measurement policy.
• Document Processing: What You Index Sets the Ceiling — why extraction quality caps everything downstream of ingestion.

Resources

• Levenshtein, V. I. (1966). "Binary codes capable of correcting deletions, insertions, and reversals." Soviet Physics Doklady, 10(8), 707–710. The original edit-distance paper.
• Wagner, R. A. & Fischer, M. J. (1974). "The string-to-string correction problem." Journal of the ACM, 21(1), 168–173. The dynamic-programming formulation used in §5.
• Unicode Consortium. Unicode Standard Annex #15: Unicode Normalization Forms. The reference for choosing between NFC, NFD, NFKC and NFKD before any comparison.
• jiwer and torchmetrics.text both implement CER and WER with normalization and aggregation hooks; reading their default normalization is the fastest way to see how much two published numbers can differ.