Inter-Rater Agreement: Why 90% Agreement Can Mean Nothing

TL;DR — Raw agreement between two annotators is almost always a flattering number, because two people labelling a skewed dataset agree constantly by accident. Cohen's kappa subtracts the agreement you would expect from chance alone, and the correction is large: three datasets in this article all show exactly 90% raw agreement while their kappa values are +0.85, +0.49 and +0.00. The third is a rater who applies one label to everything. The boundary condition is prevalence — as one class approaches the whole dataset, expected agreement approaches 1 and kappa collapses toward 0 even for genuinely careful raters, which is why a low kappa on a heavily skewed set is a signal to redefine the labels rather than to blame the annotators.

---

1. Simple explanation

You ask two people to sort a hundred photographs into three piles: clear, ambiguous, unusable. They agree on ninety of them. Ninety percent — the labels must be reliable, and the dataset is ready to train on.

Now suppose ninety-four of those photographs were obviously clear. Both people put them in the clear pile without thinking, because anyone would. Their ninety percent agreement was mostly the dataset being easy, not the two of them applying a shared standard. Hand them a genuinely hard photograph and there is no evidence at all that they would agree.

Chance-corrected agreement asks a different question: how much did they agree beyond what two people labelling randomly, with those same label habits, would have managed anyway? On the skewed set, two random labellers would have agreed on about eighty percent of items just by both reaching for clear. Ninety against a baseline of eighty is a much smaller achievement than ninety against a baseline of thirty-three.

Analogy — marking a multiple-choice exam with four options. A candidate scores 40%. Impressive or not? Guessing alone scores 25%, so they are 15 points above chance, and the honest statement of their knowledge is that 15, not the 40. If the exam were true/false, chance is 50% and a 40% score is below guessing. The raw score has no meaning until you know what the floor is. Raw agreement is a raw score; kappa is the part above the floor.

---

2. Diagram

  THREE DATASETS. IDENTICAL RAW AGREEMENT. VERY DIFFERENT RELIABILITY.

  CASE 1 — balanced classes                 CASE 2 — one class dominates
  34 clear / 33 ambig / 33 unusable         94 clear / 3 ambig / 3 unusable

  agree     ##################  90          agree     ##################  90
  by chance ######              33          by chance ################    80
            ^^^^^^^^^^^^^^^^                          ^^
            a big gap                                 a small gap
            kappa = +0.85  strong                     kappa = +0.49  moderate


  CASE 3 — rater B always says "clear"
  90 clear / 10 unusable, B labels everything clear

  agree     ##################  90
  by chance ##################  90
            (no gap at all)
            kappa =  0.00   the labels carry NO information


  WHAT THE TWO NUMBERS MEASURE

     p_o  (observed)   how often they matched
     p_e  (expected)   how often they WOULD match with those same
                       label habits and no shared standard at all

                   p_o - p_e          "how far above the floor"
     kappa  =  ----------------
                   1   - p_e          "out of how far it was possible to get"

---

3. How it works

3.1 Raw agreement has no floor

Raw agreement, p_o, is the fraction of items where two raters chose the same label. Its problem is that its floor moves with the dataset. On a balanced three-class set, two indifferent raters land on the same label about a third of the time. On a set where 94% of items belong to one class, two indifferent raters agree about 80% of the time. The same p_o = 0.90 therefore means two completely different things, and nothing in the number tells you which.

This matters most exactly when you are least likely to check. Skewed datasets are the normal case — most content is fine, most transactions are legitimate, most pages are clean — so the flattering version of the number is the one you will usually be looking at.

3.2 Chance correction

Cohen's kappa estimates the floor from the raters' own behaviour rather than assuming labels are used equally. If rater A used clear on 94% of items and rater B on 92%, then the chance of both independently choosing clear on a given item is 0.94 × 0.92. Summing that product across all labels gives p_e, the expected agreement.

Kappa then reports the achieved fraction of the available headroom:

  kappa = (p_o - p_e) / (1 - p_e)

Two properties follow directly and are worth internalising:

• kappa = 0 means the raters did no better than their own label habits predict. Not "they disagreed" — they agreed exactly as much as two independent coin-weightings would.
• kappa = 1 requires perfect agreement. Anything less than perfect cannot reach 1, however large p_o is.

Kappa can also go negative, which means systematically worse than chance — usually a sign that one rater has inverted a category, or that the two are working from different versions of the instructions.

3.3 The prevalence paradox, and what it is telling you

The uncomfortable property of kappa is that a fixed number of disagreements produces a lower kappa on a skewed dataset. §4.3 shows the same ten disagreements giving +0.85 on balanced classes and +0.49 on skewed ones. Nothing about the raters changed.

Treat this as diagnostic rather than as a defect in the statistic. A low kappa on a heavily skewed set is telling you that the labelling task carries little information as currently defined. If 94% of items are clear, then agreeing about clear is close to free, and the only items where a shared standard could be demonstrated are the six that are not. A six-item basis for a reliability claim is not a reliability claim.

The responses are about the task, not the statistic:

3.4 Which coefficient for which labels

Kappa is for two raters and unordered categories. Reaching for it outside that is a common and quiet error:

Where this stops applying. Every coefficient here assumes the raters worked independently and from the same instructions. If one rater saw the other's labels, adjudicated a subset together, or the protocol changed partway through the batch, the arithmetic still produces a number and that number means nothing. It also assumes items are exchangeable — if the two raters split a corpus by source rather than labelling the same items, there is no agreement to compute at all, only two separate label distributions.

---

4. The math

4.1 Cohen's kappa

For two raters over n items with label set L:

  p_o = (1/n) * |{ i : a_i = b_i }|                    observed agreement

  p_e = sum_L ( count_A(l)/n ) * ( count_B(l)/n )      expected by chance

  kappa = (p_o - p_e) / (1 - p_e)

Conventional reading, and it is a convention rather than a law:

  kappa < 0.00   worse than chance
  0.00 - 0.40    poor
  0.40 - 0.60    moderate
  0.60 - 0.80    substantial
  0.80 - 1.00    strong

4.2 Weighted kappa, for ordered labels

When labels are ordered, a disagreement of one step is not as bad as a disagreement of two. Introduce a weight w_jk for the cost of rater A saying j while B says k:

  p_o(w) = 1 - sum_jk  w_jk * O_jk / n
  p_e(w) = 1 - sum_jk  w_jk * E_jk / n
  kappa_w = (p_o(w) - p_e(w)) / (1 - p_e(w))

  linear weights:    w_jk = |j - k| / (L - 1)
  quadratic weights: w_jk = (j - k)^2 / (L - 1)^2

Unweighted kappa is the special case w_jk = 0 when j = k and 1 otherwise.

4.3 Worked example

100 items, three labels, and exactly ten disagreements in every case below — the measured output in §5.

Case 1, balanced classes. Rater A uses 34 clear, 33 ambiguous, 33 unusable; B's counts are nearly the same after ten flips.

  p_o = 90 / 100                                       = 0.90
  p_e ~ (0.34)(0.34) + (0.33)(0.33) + (0.33)(0.33)     = 0.33
  kappa = (0.90 - 0.33) / (1 - 0.33) = 0.57 / 0.67     = +0.85

Case 2, skewed classes. 94 clear, 3 ambiguous, 3 unusable. Same ten flips.

  p_o = 90 / 100                                       = 0.90
  p_e ~ (0.94)(0.91) + small terms                     = 0.80
  kappa = (0.90 - 0.80) / (1 - 0.80) = 0.10 / 0.20     = +0.49

Case 3, one rater always says clear. A labels 90 clear and 10 unusable; B labels all 100 clear.

  p_o = 90 / 100                                       = 0.90
  p_e = (0.90)(1.00) + (0.10)(0.00)                    = 0.90
  kappa = (0.90 - 0.90) / (1 - 0.90) = 0.00 / 0.10     =  0.00

The third case is the one to remember: a rater who never uses a label scores 90% agreement and exactly zero reliability. Raw agreement cannot distinguish that rater from a careful one. Kappa reports zero, which is the correct answer.

---

5. Real code

"""Two annotators, the same items. Raw agreement vs Cohen's kappa."""
from collections import Counter

def kappa(a, b):
    """Cohen's kappa for two raters over the same ordered items."""
    assert len(a) == len(b) and a, "raters must label the same non-empty items"
    n = len(a)
    p_o = sum(x == y for x, y in zip(a, b)) / n          # observed agreement
    ca, cb = Counter(a), Counter(b)
    labels = set(ca) | set(cb)
    p_e = sum((ca[l] / n) * (cb[l] / n) for l in labels)  # agreement by chance
    return p_o, p_e, (p_o - p_e) / (1 - p_e) if p_e != 1 else float("nan")

def show(name, a, b):
    p_o, p_e, k = kappa(a, b)
    grade = ("poor" if k < 0.40 else "moderate" if k < 0.60
             else "substantial" if k < 0.80 else "strong")
    print(f"{name:26} n={len(a):3}  p_o={p_o:.2f}  p_e={p_e:.2f}  "
          f"kappa={k:+.2f}  ({grade})")

# 100 images, three labels: clear / ambiguous / unusable
# CASE 1 — balanced classes, the raters disagree on a tenth of items
bal_a = (["clear"] * 34 + ["ambiguous"] * 33 + ["unusable"] * 33)
bal_b = bal_a.copy()
for i in (5, 15, 25, 40, 50, 60, 70, 80, 90, 95):        # 10 flips
    bal_b[i] = {"clear": "ambiguous", "ambiguous": "unusable",
                "unusable": "clear"}[bal_b[i]]

# CASE 2 — same 10 disagreements, but one class dominates
skew_a = ["clear"] * 94 + ["ambiguous"] * 3 + ["unusable"] * 3
skew_b = skew_a.copy()
for i in (5, 15, 25, 40, 50, 60, 70, 80, 90, 95):
    skew_b[i] = {"clear": "ambiguous", "ambiguous": "unusable",
                 "unusable": "clear"}[skew_b[i]]

show("balanced classes", bal_a, bal_b)
show("skewed (94% one class)", skew_a, skew_b)

# CASE 3 — a rater who never uses a label still "agrees" a lot
lazy_a = ["clear"] * 90 + ["unusable"] * 10
lazy_b = ["clear"] * 100                                  # always says clear
show("one rater always 'clear'", lazy_a, lazy_b)

p_o_bal, _, k_bal = kappa(bal_a, bal_b)
p_o_skew, _, k_skew = kappa(skew_a, skew_b)
assert abs(p_o_bal - p_o_skew) < 1e-9, "same raw agreement in both cases"
assert k_bal > k_skew, "kappa must fall when one class dominates"
print(f"\nsame raw agreement ({p_o_bal:.2f}) in both cases, "
      f"kappa {k_bal:+.2f} vs {k_skew:+.2f}")

# Output:
#   balanced classes           n=100  p_o=0.90  p_e=0.33  kappa=+0.85  (strong)
#   skewed (94% one class)     n=100  p_o=0.90  p_e=0.80  kappa=+0.49  (moderate)
#   one rater always 'clear'   n=100  p_o=0.90  p_e=0.90  kappa=+0.00  (poor)
#
#   same raw agreement (0.90) in both cases, kappa +0.85 vs +0.49

The two asserts encode the article's claim: raw agreement is held constant at 0.90 across cases, and kappa still falls. If a future edit breaks that, the block fails rather than quietly printing something else.

---

6. Real-world example

A team building a content-moderation classifier needed labelled training data, so two reviewers independently labelled 2,000 posts as allow, review, or remove. The reviewers agreed on 93% of items. The team recorded "93% inter-annotator agreement" in the model card and moved on to training.

The classifier performed well on aggregate accuracy and badly on the only thing anyone cared about: the remove class. Recall on remove sat around a third, and no amount of reweighting or threshold tuning improved it much.

Computing kappa on the original labels afterwards gave +0.31 — poor. The cause was visible the moment anyone looked at the confusion between raters rather than the headline: 96% of the 2,000 posts were allow, both reviewers said allow almost reflexively, and nearly every disagreement was concentrated in the review versus remove boundary. There were roughly eighty items where the distinction the model needed to learn was actually at stake, and the two reviewers agreed on fewer than half of those.

The labels were not wrong so much as uninformative about the decision the model had to make. What fixed it was not more labelling at the same ratio; it was stratifying the next batch so that borderline candidates were deliberately over-sampled, writing an explicit boundary rule for review versus remove, and reporting kappa computed on the non-allow subset alongside the overall figure. The overall kappa stayed unimpressive and the subset kappa became the number the team actually steered on.

Two things are worth taking from this. The 93% was never a lie — it was a true statement about an easy dataset. And the failure showed up two stages downstream, in a model metric, with nothing in the labelling report to point at.

---

7. Interview questions companies actually ask

Q1. Two annotators agree on 95% of your labels. Is the dataset reliable? Not knowable from that number alone, because raw agreement has a floor that moves with class balance. If one class covers 95% of items, two indifferent annotators would also agree about 90% of the time, so 95% is a small achievement above chance and kappa would come out low. Ask for the class distribution and the chance-corrected figure. The number to be suspicious of is high agreement on a heavily skewed set, which is the most common situation in practice.

Q2. What does a kappa of exactly 0 mean? That the raters agreed precisely as often as their own individual label habits would predict if they were labelling independently with no shared standard. It does not mean they disagreed — the clean example is a rater who applies one label to everything, who can score 90% raw agreement and exactly zero kappa. Zero means the labels carry no information about a shared standard, which is a different and more serious finding than "they disagreed sometimes".

Q3. Your kappa is 0.45 and your annotators insist they followed the guidelines. Who is right? Both can be. A fixed number of disagreements yields a lower kappa on a skewed dataset, so careful annotators on a 95/5 split can produce a mediocre kappa. Look at where the disagreements sit: if they cluster on one category boundary, the guideline for that boundary is underspecified and the fix is a written boundary rule plus re-labelling, not a lecture about carefulness. If disagreements are spread evenly, the task itself may not be learnable as defined.

Q4. When would you use weighted kappa instead of plain kappa? When the categories are ordered and the size of a disagreement matters. If labels run reject / weak / moderate / strong, one rater saying weak while the other says moderate is a much smaller problem than weak against strong, and unweighted kappa charges both the same. Linear weights make the penalty proportional to the number of steps; quadratic weights punish large gaps harder. Using unweighted kappa on ordinal labels systematically understates agreement.

Q5. Three people labelled your data, not two. What changes? Cohen's kappa is defined for two raters, so it does not apply directly. Fleiss' kappa generalises to any fixed number of raters per item, and Krippendorff's alpha handles the messier real situation where different items were seen by different numbers of raters. A common shortcut — averaging pairwise Cohen's kappas — is defensible for reporting but is not the same statistic, so say which you computed.

Q6. How many items need double-labelling? Enough that the rare categories appear in the double-labelled subset often enough to be informative, which is a statement about the skew, not a fixed percentage. A frequently used starting point is 20% of the corpus, but on a 95/5 split, 20% of items gives you very few of the minority class and the resulting kappa will be dominated by the easy majority. The better move on skewed data is to stratify the double-labelled subset so hard cases are over-represented, and report that you did.

Q7. Can you use a model as the second rater? You can compute the number, and it answers a different question than you probably want. Kappa between a human and a model measures how well the model reproduces that human, which is a useful evaluation of the model and is not evidence that the labelling task is well defined. Reliability claims about a dataset need two independent humans; the model-versus-human figure belongs in the model's evaluation section, not in the dataset's reliability section.

---

8. When to use / tradeoffs

Reach for chance-corrected agreement when:

• you are about to publish or rely on a claim that labels are reliable
• two people labelled the same items independently into unordered categories
• the class distribution is skewed, which is when raw agreement is most misleading
• a downstream model or decision depends on the rare class

Honest limits. The 0.00–0.40–0.60–0.80 bands are a widely repeated convention, not a result; they have no theoretical basis and differ between fields, so quoting "substantial agreement" without the raw number and the class distribution hides more than it conveys. Kappa's dependence on prevalence is genuine and much criticised — two datasets with identical rater behaviour can produce very different kappas, which is why prevalence and bias indices are sometimes reported alongside it. The statistic is also silent about whether the labels are correct: two raters who share the same misunderstanding will produce a strong kappa and a systematically wrong dataset, because agreement is not accuracy. And on small double-labelled subsets kappa is unstable — with 30 items and a rare class, a single flipped label can move it by more than a band, so a point estimate without an interval invites over-reading.

---

9. Summary + related articles

• Raw agreement has no fixed floor; the floor rises with class skew, so the same 90% can be strong or worthless.
• Kappa reports the fraction of available headroom achieved: (p_o - p_e) / (1 - p_e).
• Three datasets in §5 share an identical p_o = 0.90 and score +0.85, +0.49 and 0.00. The last is a rater applying one label to everything.
• kappa = 0 means "no better than these raters' own label habits predict" — not "they disagreed".
• The prevalence effect is diagnostic: low kappa on a skewed set says the task as defined carries little information, so change the task — redefine categories, stratify the hard cases — rather than the statistic.
• Match the coefficient to the labels: weighted kappa for ordered categories, Fleiss' for 3+ raters, Krippendorff's alpha with gaps, ICC for continuous scores.
• The boundary: all of it assumes independent raters, one protocol, and matched items. Agreement is never evidence of correctness — two raters can share a misunderstanding and score highly.

Related:

• Model Evaluation — why accuracy misleads on skewed classes, the same prevalence problem one stage later
• Backtesting, Baselines & Sensitivity Analysis — establishing the chance floor before reading any score
• LLM-as-a-Judge: Using a Model to Grade Model Output — what changes when the second rater is a model, and why that is a model metric rather than a dataset one
• Classification — the downstream task whose rare-class performance these labels determine
• Study Screening & Selection: Dual Reviewers, PRISMA, and Kappa — dual independent screening as a working protocol
• Probability & Statistics Foundations — the expected-value reasoning behind p_e
• RAG Evaluation: Attributing Failure and Sizing the Eval Set — building a labelled evaluation set that needs exactly this check

Resources

• Cohen, Jacob. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1), 1960. The original definition of kappa.
• Cohen, Jacob. Weighted Kappa: Nominal Scale Agreement with Provision for Scaled Disagreement or Partial Credit. Psychological Bulletin, 70(4), 1968. The ordered-category extension in §4.2.
• Fleiss, Joseph L. Measuring Nominal Scale Agreement Among Many Raters. Psychological Bulletin, 76(5), 1971. The generalisation beyond two raters.
• Landis, J. Richard & Koch, Gary G. The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 1977. The source of the conventional 0.40/0.60/0.80 interpretation bands — worth reading precisely because it presents them as arbitrary.
• Byrt, Ted & Bishop, Janet & Carlin, John B. Bias, Prevalence and Kappa. Journal of Clinical Epidemiology, 46(5), 1993. The standard treatment of the prevalence problem in §3.3.
• Krippendorff, Klaus. Content Analysis: An Introduction to Its Methodology. Sage. Chapter 11 covers alpha and the missing-data case.
• scikit-learn — sklearn.metrics.cohen_kappa_score, including the weights="linear" and weights="quadratic" options: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html