Consistency Memory: Making Repeated AI Judgments Agree With Each Other

TL;DR — An LLM asked to make the same judgment call twice, independently, will not reliably give the same answer — sampling variance means two runs over the identical error produce two different scores. Consistency memory fixes this by storing the first judgment for a given category of decision and reusing it for every later occurrence, rather than re-deriving a fresh, independently noisy answer each time. In the harness below, ten independent judgments of the same error vary from 2.2 to 4.0 (stdev 0.461); with a memory store, the first judgment is derived once and every later occurrence of the same error gets the identical value (stdev 0.0). This stops being the right fix the moment two occurrences of "the same" category actually deserve different judgments — memory that's too eager to match will paper over a real distinction instead of preserving one.

---

1. Simple explanation

Ask an LLM to score the same mistake in isolation, in two separate calls, and you'll typically get two different numbers — not because the model is broken, but because generation is a sampling process, and a judgment expressed as "somewhere around here" varies within that range every time it's re-derived from scratch. This is invisible for a single judgment, but it becomes a real problem the moment a system is expected to be consistent: two people who made the identical mistake should get the identical consequence, and "the AI happened to sample a slightly different number both times" is not a defensible reason for it to disagree with itself.

Analogy — a judge who forgets every previous ruling. Imagine a judge who, for every case, decides the appropriate sentence entirely from first principles, with no memory of how similar cases were sentenced last week or an hour ago. Two defendants who committed the identical offense under identical circumstances could receive noticeably different sentences purely because the judge reasoned it out independently each time, arriving at a plausible number rather than the same number as last time. A judge who keeps notes — "this exact category of offense gets this sentence, because I already decided that" — doesn't re-litigate a settled question; they look up the precedent and apply it. Consistency memory is the notes.

---

2. Diagram

WITHOUT MEMORY                              WITH CONSISTENCY MEMORY
(every judgment re-derived independently)   (first judgment is precedent)

  submission 1 --judge-->  2.9                submission 1 --judge-->  2.9  --store-->
  submission 2 --judge-->  3.1                                                  |
  submission 3 --judge-->  4.0                submission 2 --lookup---> 2.9 <--+
  submission 4 --judge-->  2.9                submission 3 --lookup---> 2.9 <--+
  submission 5 --judge-->  3.0                submission 4 --lookup---> 2.9 <--+
  submission 6 --judge-->  3.2                        ...
  submission 7 --judge-->  2.2                submission 10 -lookup--> 2.9 <--+
  submission 8 --judge-->  3.0
  submission 9 --judge-->  3.3               MEASURED (harness in §5, same
  submission 10-judge-->   3.7               error type, 10 occurrences):

  spread: 2.2 to 4.0                            no-memory   stdev = 0.461
  (same mistake, different penalty                with-memory stdev = 0.000
   depending only on which run you ask)          (identical after occurrence 1)

---

3. How it works

3.1 The judgment is only as consistent as the process that produced it

A single LLM call scoring a single case is a sample from a distribution, not a lookup of a fixed fact. Two independent calls over the identical input are two independent samples, and unless the distribution has essentially zero variance — rare for anything resembling a judgment call rather than a deterministic calculation — the two samples will differ. This is normal, expected model behavior, not a bug in any individual call. The bug, if there is one, is in a system that needs consistency across many calls but treats every call as an isolated, independent event with no relationship to any other.

3.2 Store the first judgment, don't re-derive every one

Consistency memory keeps a store keyed by the category of decision — an error type, a fault classification, any label that identifies "this situation is the same kind of situation as that one." The first time a category is seen, a judgment is derived the normal way (an LLM call, a rule, whatever the underlying decision process is) and the result is written to the store alongside the category key. Every subsequent occurrence of the same category looks the result up rather than re-deriving it, so the second, tenth, and hundredth occurrence of the identical error type all inherit the exact value the first occurrence produced.

This only works to the extent the category key genuinely captures "same situation." A key that's too coarse (grouping meaningfully different situations under one label) forces different things to get the same answer; a key that's too fine (treating trivially different phrasings of the same underlying issue as different categories) misses the consistency the memory was supposed to provide in the first place. Choosing the right granularity for the category key is most of the actual design work.

3.3 What to do when a category has been seen before but the new occurrence isn't identical

Real occurrences of "the same" error are rarely byte-identical — the surrounding context differs even when the core mistake is the same. A practical middle ground, shown in §5, is to blend a new occurrence's independently-derived judgment into the stored value (a running average, weighted by how many times the category has been seen) rather than either always trusting the stored value blindly or always re-deriving from scratch. This lets a category's stored judgment settle toward a stable value over its first few occurrences and then stay essentially fixed, while still allowing genuine drift if the true right judgment for that category turns out to be different from the very first guess.

Where this stops working: consistency memory assumes that matching category keys really do deserve the same judgment. If a category is genuinely heterogeneous — the "same" labeled error actually covers cases that deserve different treatment depending on context the category key doesn't capture — memory forces an inconsistency in the other direction: it makes clearly different cases receive an identical judgment because they were mislabeled as the same category. That failure mode is worse than ordinary sampling noise, because it's silent and systematic rather than random and visible on inspection.

---

4. The math

4.1 Why independent samples don't cancel out into consistency

If each independent judgment for a category is drawn from a distribution with some spread (call its standard deviation sigma), then across n independent occurrences of the identical category, the individual judgments still each carry the full spread sigma — averaging only reduces the spread of an aggregate statistic like the mean, not the disagreement between any two specific individual judgments. Two people who committed the identical offense are still each getting one draw from that distribution; the fact that the distribution's mean is stable across many draws doesn't make any two particular draws agree with each other.

4.2 Worked comparison, from the actual run in §5

no-memory   deductions: min=2.2  max=4.0  stdev=0.461   (10 independent samples)
with-memory deductions: min=2.9  max=2.9  stdev=0.000   (1 sample, reused 9 times)

The with-memory number isn't a smaller amount of variance — it's zero variance by construction, because occurrences 2 through 10 are not independent samples at all; they are the same stored value read back. The no-memory spread (2.2 to 4.0, a range of 1.8 on judgments centered around 3.0) represents roughly a 60% swing relative to the mean, entirely attributable to re-deriving the same judgment independently ten times rather than to any real difference between the ten submissions.

---

5. Real code

import random
import statistics


class ConsistencyMemory:
    """Stores {error_type: (deduction, reasoning, count)}. A new occurrence
    of an already-seen error type reuses the stored deduction instead of
    re-deriving one from scratch."""

    def __init__(self):
        self._store = {}

    def lookup(self, error_type):
        return self._store.get(error_type)

    def record(self, error_type, deduction, reasoning):
        if error_type in self._store:
            prev_deduction, _, count = self._store[error_type]
            # Reinforce the existing value rather than overwrite it outright --
            # this is what keeps a single noisy re-derivation from moving the
            # anchor after it's established.
            blended = round((prev_deduction * count + deduction) / (count + 1), 1)
            self._store[error_type] = (blended, reasoning, count + 1)
        else:
            self._store[error_type] = (deduction, reasoning, 1)


def grade_without_memory(error_type, rng):
    """Simulates an independent LLM call re-deriving a deduction each time,
    with the natural run-to-run variance a real grader shows for the same
    underlying error."""
    base = {"off_by_one_index": 3.0, "missing_null_check": 2.0}[error_type]
    return round(base + rng.uniform(-1.2, 1.2), 1)


def grade_with_memory(error_type, rng, memory):
    prior = memory.lookup(error_type)
    if prior is not None:
        deduction = prior[0]
    else:
        deduction = grade_without_memory(error_type, rng)
    memory.record(error_type, deduction, f"deduction for {error_type}")
    return deduction


rng_a = random.Random(11)
rng_b = random.Random(11)  # identical seed sequence for a fair comparison
memory = ConsistencyMemory()

# Ten different student submissions all happen to make the SAME off-by-one
# indexing error -- realistic, since the same assignment tends to produce
# the same handful of common mistakes across many submissions.
submissions = ["off_by_one_index"] * 10

without_memory = [grade_without_memory(e, rng_a) for e in submissions]
with_memory = [grade_with_memory(e, rng_b, memory) for e in submissions]

print(f"{'submission':12}{'no memory':12}{'with memory':12}")
for i, (a, b) in enumerate(zip(without_memory, with_memory)):
    print(f"{i:<12}{a:<12}{b:<12}")

print(f"\nno-memory   deductions: min={min(without_memory)} max={max(without_memory)} "
      f"stdev={statistics.pstdev(without_memory):.3f}")
print(f"with-memory deductions: min={min(with_memory)} max={max(with_memory)} "
      f"stdev={statistics.pstdev(with_memory):.3f}")

assert with_memory[0] == with_memory[1] == with_memory[-1]
assert statistics.pstdev(with_memory) == 0.0
assert statistics.pstdev(without_memory) > 0.0
print(f"\nasserts passed: with-memory deductions are identical after the "
      f"first occurrence; no-memory deductions vary every time")

second_error = grade_with_memory("missing_null_check", rng_b, memory)
print(f"\na different error type ('missing_null_check') still gets its "
      f"own independent deduction: {second_error}")
assert memory.lookup("off_by_one_index")[0] == with_memory[-1]
print("assert passed: unrelated error types don't share or overwrite each other's memory")

# Output:
# submission  no memory   with memory
# 0           2.9         2.9
# 1           3.1         2.9
# 2           4.0         2.9
# 3           2.9         2.9
# 4           3.0         2.9
# 5           3.2         2.9
# 6           2.2         2.9
# 7           3.0         2.9
# 8           3.3         2.9
# 9           3.7         2.9
#
# no-memory   deductions: min=2.2 max=4.0 stdev=0.461
# with-memory deductions: min=2.9 max=2.9 stdev=0.000
#
# asserts passed: with-memory deductions are identical after the first occurrence; no-memory deductions vary every time
#
# a different error type ('missing_null_check') still gets its own independent deduction: 2.1
# assert passed: unrelated error types don't share or overwrite each other's memory

All four asserts passed on the run that produced this output: with-memory judgments are provably identical from the second occurrence onward, no-memory judgments provably vary, and a second, unrelated category is confirmed to get its own independent judgment rather than inheriting the first category's stored value.

---

6. Real-world example

A team building an automated review system for a high volume of similar submissions noticed that two reviewers — well, one human and one AI reviewer, run twice on the same case as a sanity check — sometimes disagreed with themselves: the same AI reviewer, given the identical case a second time in a re-run, occasionally produced a different verdict than it had the first time. This wasn't caught until a submitter appealed a decision by pointing out that another submission with what looked like the identical issue had received a different, more lenient outcome a day earlier.

Investigating, the team found there was no shared record of past judgments at all — every review was generated fresh, independently, with no reference to how the same category of issue had been judged before. Two submissions with the identical underlying problem could land on either side of a threshold purely because of ordinary sampling variance in two separate LLM calls, and nothing in the system would ever have flagged this as an error, because both individual judgments were, in isolation, entirely plausible.

The fix was consistency memory: the first time a given issue category was judged, that judgment became the record for the category, and every subsequent occurrence looked it up rather than re-deriving it. This didn't just fix future disagreements — it also let the team run a batch consistency check on past decisions, grouping historical judgments by category and confirming that outcomes within a category actually clustered together after the change, which they demonstrably had not before it.

---

7. Interview questions companies actually ask

Q1. Why would the same LLM produce two different scores for what looks like the same decision, run twice? Generation from an LLM is a sampling process, not a deterministic lookup, so a judgment expressed as "roughly this severity" is drawn from a distribution rather than computed as a single fixed value — two independent calls are two independent draws, and unless that distribution has essentially no spread, the two draws can differ meaningfully even though neither one is "wrong" in isolation.

Q2. How does consistency memory actually fix that, given the underlying model still has the same sampling variance? It doesn't reduce the model's variance — it avoids re-sampling at all for repeat occurrences. The first occurrence of a category is judged normally, with all the usual variance; every later occurrence of the same category retrieves that stored judgment instead of asking the model again, so there is no second sample to disagree with the first.

Q3. What's the risk of choosing a category key that's too coarse — grouping meaningfully different situations under one label? Cases that actually deserve different treatment get forced into the same judgment, because the memory can't distinguish them once they share a key — this failure is worse than ordinary sampling noise because it's systematic and silent rather than random and visible; nothing about the system looks broken, it's just quietly wrong for every case that got mis-grouped.

Q4. And what's the risk of a category key that's too fine-grained? Trivially different phrasings of the same underlying issue get treated as different categories, so the memory never actually fires — every occurrence looks "new" to the lookup even though a human would recognize it as the same case, and the system pays the cost of maintaining a memory store without getting the consistency benefit it exists to provide.

Q5. Why blend a new occurrence into the stored value with a running average instead of either always trusting the store or always re-deriving fresh? Always trusting the store locks in whatever the very first judgment happened to be, even if it was an unusually noisy sample; always re-deriving fresh defeats the purpose of having memory at all. Blending lets the stored value settle toward a stable answer across the first few occurrences of a category while still being responsive if the true right judgment turns out to differ from the initial guess, rather than freezing on a potentially bad first draw forever.

Q6. How would you detect that a category key is too coarse in a live system, rather than assuming your key design is correct? Look for a category whose occurrences, before memory was introduced, showed unusually high variance or a bimodal spread rather than a single cluster — that's a signal the category may actually contain two or more situations that deserve different treatment. A category that's correctly scoped should show occurrences clustering tightly even before consistency memory forces them to; one that doesn't cluster is a candidate for splitting into finer categories.

Q7. This pattern trades consistency for responsiveness to new information. When is that the wrong tradeoff? When the correct judgment for a category can legitimately change over time — new guidance, a policy update, corrected understanding of what a given issue actually warrants — and the memory is aggressively anchored to old occurrences, it will keep reproducing an outdated judgment long after it stopped being correct. Systems where the "right answer" is expected to evolve need an explicit mechanism to invalidate or reset stale memory, not just accumulate it indefinitely.

---

8. When to use / tradeoffs

Reach for consistency memory when:

• the same category of decision recurs often enough that inconsistency between occurrences would be noticed and would matter
• the underlying judgment process (an LLM call, a scoring rule) has real sampling variance between independent invocations
• you can define a category key that reliably captures "this is genuinely the same situation as that one"

Honest limits. Consistency memory guarantees that repeat occurrences of the same category agree with each other; it says nothing about whether the very first judgment for that category was actually correct — a bad first draw becomes a consistently repeated bad draw rather than an inconsistently varying one, which is only an improvement if you separately have reason to trust the first-occurrence judgment. It also depends entirely on category-key quality, which is a modeling decision made outside the memory mechanism itself and is easy to get subtly wrong in either direction. And the measured numbers in §5 describe a small, deliberately clean synthetic case with an obvious, unambiguous category; real categories are rarely that clean, and the harder engineering work is almost always in defining the key, not in building the store.

---

9. Summary + related articles

• Independent LLM judgments over the same category vary because generation is sampling, not deterministic lookup — this is normal model behavior, not a defect in any one call.
• Consistency memory fixes cross-occurrence disagreement by deriving a judgment once per category and reusing it, rather than re-deriving independently every time.
• Measured: ten independent judgments of one error type varied with stdev 0.461 (range 2.2–4.0); the same ten judgments through a consistency memory were identical after the first occurrence, stdev 0.0.
• Boundary: this only helps to the extent the category key genuinely captures "same situation" — a category that's too coarse forces different things to agree, and one that's too fine never lets memory fire at all.

Related:

• Memory Management — episodic memory (an audit-log style record of what happened) versus this article's semantic/consistency memory (a record used to constrain future judgments) are different memory shapes serving different purposes
• API Best Practices — another case where a value derived once (a truncation budget, here a judgment) should be reused deliberately rather than silently re-derived under different conditions