LLM-as-a-Judge: Using a Model to Grade Model Output

EvaluationLLMBeginner14 min

By: Anacodic Team

TL;DR — When there's no single correct answer to check against (open-ended text, a generated rule, an agent's reasoning), you can use a strong LLM as an automatic judge. Two modes: pointwise (score one output against a rubric) and pairwise (say which of two is better). It scales evaluation and, on benchmarks like MT-Bench, agrees with humans about as often as humans agree with each other (~80%). But a judge is biased, not neutral: it favors the answer shown first (position bias), longer answers (verbosity bias), and text that looks like its own (self-enhancement bias). The fixes are mechanical — swap positions and average, force a rubric, calibrate against human labels — and a judge should never grade a safety decision (that needs a deterministic gate).

1. Simple explanation

Some outputs are easy to grade automatically: a classifier's label is right or wrong, a number is close or far. But much of modern AI produces things with no key to check against — a summary, a chatbot reply, an agent's multi-step reasoning, a rule inferred from examples. Humans can judge these, but human grading is slow and expensive, and you need it on every code change.

LLM-as-a-judge uses a capable model to do that grading. You show the judge the task, the output(s), and either a rubric ("score 1–5 for factual accuracy") or a choice ("which answer is more helpful, A or B?"), and it returns a verdict. Done well, it turns a week of human annotation into minutes of API calls, and it correlates with human preference well enough to rank systems.

The catch: the judge is another LLM, so it inherits LLM quirks. It is suggestible — nudge the order, the length, or the phrasing, and its verdict moves for reasons that have nothing to do with quality. Treating the judge as an objective oracle is the mistake; treating it as a useful but biased annotator you must debias and validate is the craft.

Analogy — a competition judge who hasn't read the rules carefully. They can tell a great performance from a poor one, but they also unconsciously score the first act higher, reward whoever talked longest, and favor a style like their own. You don't fire them — you rotate the running order, hand them a scoring sheet, and check their scores against a trusted judge on a few acts. Same moves here.

2. Diagram

  POINTWISE (score one)                 PAIRWISE (compare two)
  ┌──────────────────────┐              ┌────────────────────────────┐
  │ task + output + rubric│             │ task + output A + output B  │
  │        ▼             │              │            ▼                │
  │   JUDGE (LLM)         │             │      JUDGE (LLM)             │
  │        ▼             │              │            ▼                │
  │   score 1..5 + reason │             │  "A" / "B" / "tie" + reason │
  └──────────────────────┘              └────────────────────────────┘

  POSITION BIAS + THE FIX
     ask(A,B) → "A"          judge leans toward the FIRST slot
     ask(B,A) → "B"          swap the order …
     consistent? → trust     A both times = real win
     inconsistent? → tie     flipped with order = bias, not quality

3. How it works

3.1 Pointwise, pairwise, listwise

Three shapes. Pointwise scores a single output against a rubric (e.g. 1–5 for "coherence"); good for absolute tracking over time, but scores drift and are hard to calibrate across runs. Pairwise shows two outputs and asks which is better; more reliable because relative judgments are easier than absolute ones, and it maps cleanly to "did my change beat the baseline?" Listwise ranks several at once; efficient but the hardest for the judge to do consistently. Pairwise is the workhorse for comparing a candidate against a baseline.

3.2 Give the judge a rubric and let it reason

A bare "is this good?" invites vibes. Two things sharply improve agreement with humans: (1) a rubric that names the criteria and the scale, and (2) asking the judge to reason before it scores (chain-of-thought), or to generate the evaluation steps first and then fill them in — the idea behind G-Eval. Force the verdict into a structured format ({"winner": "...", "reason": "..."}) so it is parseable and the reason is auditable.

3.3 The biases (and why they exist)

LLM judges are not neutral. The well-documented ones (from the MT-Bench study and follow-ups):

Position bias — prefers whichever answer is shown first (sometimes last).
Verbosity bias — prefers longer, more detailed answers even when no better.
Self-enhancement bias — prefers text in a style similar to its own outputs.
Plus sycophancy (agreeing with a stated opinion) and format effects. These are systematic, so they don't wash out by running more samples — you must counteract them directly.

3.4 Debiasing and validation

Swap and average for position bias: judge (A,B) and (B,A); only count a win if the verdict is consistent, else call it a tie. This is the single most important fix and is shown in the code below.
Control length to blunt verbosity bias (cap or normalize length; tell the judge length is not quality).
Use a different model as judge than the one being judged, to reduce self-enhancement.
Calibrate: on a small human-labeled set, measure the judge's agreement with humans (and its consistency under swaps). Report that number — a judge you haven't validated is a guess.

3.5 Panels and juries of judges

For high-stakes evaluation, one judge is a single point of bias. Two cheap upgrades: run an ensemble of different judge models and take the majority/average verdict, which diversifies away any one model's idiosyncrasies; and, for close or important calls, add a tie-break rubric or escalate to a human. A panel costs more per item, but its verdicts are steadier and its biases partially cancel — the same diversity logic that makes multi-agent debate work. Reserve panels for the evals that gate real decisions; a single validated judge is fine for routine regression tracking.

Boundary condition. A judge is an approximate, biased evaluator. It is fine for ranking and regression-testing quality, but it is not a source of ground truth, and it must never decide safety (an allergen, a policy violation) — those are hard constraints for a deterministic gate, not a probabilistic opinion. Section 8 lists where it breaks.

4. The math

4.1 Agreement and the swap test

Let a judge produce a verdict J(x, y) on an ordered pair. Position-invariance would require J(a,b) and J(b,a) to name the same winner. Define the consistent (debiased) verdict:

  v1 = J(a, b)
  v2 = J(b, a)
  debiased(a,b) =  a      if v1 == "a wins" and v2 == "a wins"
                   b      if v1 == "b wins" and v2 == "b wins"
                   tie    otherwise      # flipped with order -> bias, not quality

Quality of the judge itself is measured by agreement with humans: agreement = (# items where judge verdict == human verdict) / (# items). On MT-Bench, a strong judge reaches ~80% agreement with humans — about the human–human agreement rate, which is the practical ceiling.

4.2 Worked example

Two answers with hidden true quality A = 0.55, B = 0.60 (so B is better). Model the judge as adding a +0.10 bonus to whichever answer is shown first.

Naive (one order, A first): the judge scores A = 0.55 + 0.10 = 0.65 vs B = 0.60, and picks A — the worse answer, purely from position. Now swap: (A,B) → A; (B,A) → B. The verdict flipped with the order, so the debiased rule returns tie — it refuses to crown a winner that only wins by position. The bias didn't vanish, but the swap exposed it, turning a confident wrong answer into an honest "too close to call."

5. Real code

# A pairwise LLM judge is stochastic and POSITION-BIASED: it tends to prefer
# whichever answer is shown FIRST. We simulate that bias and show the standard
# fix — judge BOTH orders and only trust a consistent verdict.
TRUE = {"A": 0.55, "B": 0.60}   # hidden quality: B is truly slightly better
POS_BIAS = 0.10                  # the judge adds this to whichever is shown first

def judge(first, second):        # returns the answer the judge prefers
    score = {first: TRUE[first] + POS_BIAS, second: TRUE[second]}
    return max(score, key=score.get)

naive = judge("A", "B")                       # one order only
v1, v2 = judge("A", "B"), judge("B", "A")     # both orders
debiased = v1 if v1 == v2 else "tie"          # consistent -> trust; else tie

print("truly better    :", max(TRUE, key=TRUE.get))
print("naive (A first) :", naive)
print("A-first:", v1, "| B-first:", v2, "| debiased:", debiased)
assert naive == "A"        # position bias made the judge pick the WORSE answer
assert debiased == "tie"   # swapping exposes the flip -> no false winner
print("OK: order-swapping caught the position bias")

# Output:
#   truly better    : B
#   naive (A first) : A
#   A-first: A | B-first: B | debiased: tie
#   OK: order-swapping caught the position bias

The naive verdict crowned the worse answer because it appeared first. The swap-and-check turned that into an honest "tie" — a false win averted with one extra call.

6. Real-world example

A team automated their model-quality gate with a single pairwise judge call (candidate vs. current production, candidate always shown first). For weeks it "showed improvement," and they shipped changes on its say-so. A skeptic re-ran the eval with the two answers swapped and roughly half the "wins" evaporated — the judge had been rewarding the candidate largely for being in slot A. Worse, a change that made answers longer scored well across the board (verbosity bias), even on a slice where a human panel rated the longer answers as padded.

The fix was three cheap changes, not a fancier judge: randomize/duplicate order and require a consistent verdict, normalize for length (and tell the judge length ≠ quality), and calibrate once against 100 human-labeled pairs to report the judge's true agreement (which came out at 74%, not the implied 100%). The recurring lesson: an unvalidated LLM judge measures the prompt layout as much as the quality — until you debias and calibrate it, its numbers are partly artifact.

7. Interview questions companies actually ask

Q1. When would you use an LLM as a judge instead of an exact metric? When the output has no single correct answer to match — open-ended generation, summaries, agent trajectories, inferred rules — so string/label metrics don't apply, and human grading doesn't scale to every change. The judge gives a cheap, repeatable proxy for human preference.

Q2. Pointwise vs pairwise — which and why? Pairwise is usually more reliable because relative judgments ("which is better") are easier and more stable than absolute scores, and it maps directly to "did my change beat the baseline?" Pointwise is better when you need an absolute number tracked over time, but its scores drift and need careful rubric calibration.

Q3. Name the main biases of an LLM judge. Position bias (favors first/last), verbosity bias (favors longer answers), and self-enhancement bias (favors its own style), plus sycophancy and format effects. They're systematic, so more samples don't remove them — you debias directly.

Q4. How do you handle position bias specifically? Judge both orders — (A,B) and (B,A) — and only count a win when the verdict is consistent; otherwise call it a tie. Averaging over both orders cancels the first-slot advantage; the inconsistent cases are exactly the ones the bias was deciding.

Q5. How do you know your judge is any good? Calibrate it against a small human-labeled set and report agreement with humans (and consistency under swaps). A strong judge lands near the human–human agreement rate (~80% on MT-Bench); a judge you haven't measured is not evidence.

Q6. Should an LLM judge ever gate safety? No. Safety constraints (allergens, policy, legal limits) are hard rules that need a deterministic, testable gate — a probabilistic opinion can be wrong, and "usually safe" is not safe. Use the judge for quality ranking, a gate for safety.

Q7. The same judge grades a model from its own family — what's the risk? Self-enhancement bias: it tends to prefer outputs that look like its own, inflating that family's scores. Mitigate by using a different judge model, an ensemble of judges, or by calibrating the specific judge–candidate pairing against humans.

8. When to use / tradeoffs

Reach for an LLM judge when:

Outputs are open-ended and have no exact key to check against.
You need repeatable, scalable evaluation on every change (regression testing).
Relative ranking (candidate vs baseline) is what you actually need.

Do NOT use it when:

Situation	Why it breaks	Use instead
A safety/policy hard constraint	a probabilistic opinion can be wrong	a deterministic, tested gate
You need ground truth, not a proxy	the judge is itself a fallible model	human labels / verifiable checks
An exact metric exists	a judge adds cost, noise, and bias	the exact metric (accuracy, BLEU-where-valid, unit tests)
Judge shares the model family being judged	self-enhancement inflates scores	a different judge model / ensemble
Verdicts must be legally defensible	bias + non-determinism	human adjudication

Honest limits. A judge's numbers are a calibrated proxy for human preference, not truth — always report its measured agreement, not an implied 100%. Its biases are systematic and survive averaging, so you must debias structurally (swap, length-control, cross-model). It is non-deterministic: temperature and prompt wording move verdicts, so pin them and version the judge prompt like code. And it can be gamed — optimizing a system against a judge teaches the system the judge's biases (longer, first-slot, on-style answers), so a judge used as a training signal degrades over time unless refreshed and human-anchored.

Use a strong LLM to grade open-ended output when there's no exact key; it scales human-like preference to every change.
Pairwise (which is better) is more reliable than pointwise (absolute score); give a rubric and let the judge reason.
Judges are biased — position, verbosity, self-enhancement — and the biases are systematic, not noise.
Debias mechanically: swap order and require consistency, control length, use a different judge model; then calibrate against humans and report agreement.
Boundary: a judge ranks quality, it is not ground truth and must never gate safety — that needs a deterministic gate.

Multi-Agent Debate: When Letting Models Argue Helps (and When It Hurts) — a judge reads the debate transcript and picks the best-argued answer instead of the most popular one.
Shapley Values: Fairly Attributing Credit to Features and Agents — when the judge's score is your value function, Shapley attributes that score across features or agents.
Backtesting, Baselines & Sensitivity Analysis — how to compare systems honestly once you have a judge.
Agent Evaluation — judging multi-step agent trajectories, not just final answers.
Hallucination Detection & Grounding — verifying claims, a complement to preference judging.

Resources

Zheng, L. et al. (2023). "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS Datasets & Benchmarks — https://arxiv.org/abs/2306.05685 (verified arXiv id) — introduces the paradigm, the ~80% human-agreement result, and the bias taxonomy.
Liu, Y. et al. (2023). "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment." EMNLP — chain-of-thought + form-filling pointwise scoring. (Venue confirmed; arXiv id not verified here.)
Evidently AI, "LLM-as-a-judge: a complete guide to using LLMs for evaluations" — a practitioner guide to rubrics, pairwise setups, and calibration. (Vendor doc; treat as practical, not peer-reviewed.)

← Previous: Backtesting, Baselines & Sensitivity Analysis Next: Model Evaluation →