TL;DR — A single end-to-end score cannot tell you what to fix. Retrieval and generation compose, so two systems with the same end-to-end accuracy can need opposite repairs: measured below, retrieval 0.70 / generation 0.95 scores 66.0% and retrieval 0.95 / generation 0.70 scores 66.5% — indistinguishable, yet the first fails on retrieval for 31% of queries and the second on generation for 28%. The fix is to hold one stage fixed while measuring the other: run the generator against gold context to isolate it, and score retrieval against gold documents. The second half of the problem is sample size, where practice is far worse than most teams believe. Detecting a genuine 70%→75% improvement needs about 200 queries for 80% power; the 20-query eval set everyone actually uses picks the better system 57% of the time — a coin flip with extra steps. Putting the same queries to both systems roughly halves the requirement to ~100 by cancelling shared query difficulty. And effect size dominates everything: a 2-point gain needs ~500 queries, a 20-point gain needs ~10.
1. Simple explanation
You change the chunk size, run your eval, and the score goes from 71% to 74%. Ship it?
Two separate questions hide in that decision, and most teams answer neither. Is 3 points real, or noise? And which part of the system did it improve — did retrieval start finding better documents, or did the generator start using them better?
The second question matters because RAG is a chain. The retriever finds documents; the generator writes an answer from them. If the retriever missed the answer, the generator cannot recover it — the best writer in the world cannot cite a document it was never given. So a bad end-to-end score is compatible with an excellent generator, and also with an excellent retriever, and the aggregate number cannot distinguish those cases. They call for opposite work.
Analogy — a restaurant getting bad reviews. Diners say the food is bad. That is your end-to-end metric, and acting on it directly is guesswork: is the kitchen cooking badly, or is the supplier delivering poor ingredients? The way to find out is to hold one fixed. Give the kitchen known-excellent ingredients and taste the result — if the dish is still bad, it is the kitchen. Then inspect the deliveries against a standard. That is exactly what evaluating generation against gold context does, and it is the single most useful thing you can add to a RAG eval.
2. Diagram
query
│
▼
┌─────────────┐ did it return the right documents?
│ RETRIEVAL │ -> recall@k, MRR, nDCG@k (needs gold DOCS)
└──────┬──────┘
│ context
▼
┌─────────────┐ did it use them faithfully and answer the question?
│ GENERATION │ -> faithfulness, answer relevance (needs gold ANSWERS)
└──────┬──────┘
│
▼
end-to-end <- ONE number that conflates both. Necessary, but
useless for deciding what to fix.
THE CONFLATION, MEASURED (§5)
──────────────────────────────
retrieval generation end-to-end gen fault ret fault
0.70 0.95 66.0% 3.5% 30.6% ← fix RETRIEVAL
0.95 0.70 66.5% 28.4% 5.1% ← fix GENERATION
▲▲▲▲
same score, opposite diagnosis
ISOLATING A STAGE
─────────────────
generation quality = run the generator on GOLD context
(retrieval held at 100% by construction)
retrieval quality = score returned docs against GOLD docs
(generator not involved at all)
HOW BIG AN EVAL SET? true 70% vs 75%
───────────────────────────────────────
n unpaired paired
20 56.5% 58.9% ← the eval set everyone has
50 67.5% 73.2%
100 75.9% 83.3% ← paired crosses 80% here
200 85.7% 92.3% ← unpaired crosses 80% here
500 95.7% 99.1%
3. How it works
3.1 Three layers, three eval sets
RETRIEVAL input: query gold: the documents that should be found
metrics: recall@k, MRR, nDCG@k
cheap to label, reusable forever, no model calls to score
GENERATION input: query + context gold: a correct answer, or a rubric
metrics: faithfulness (is every claim supported by the
context?), answer relevance (does it address the question?)
expensive to label, needs judging
END TO END input: query gold: a correct answer
metrics: task success
what the user experiences; useless for attribution
You need all three, and they are not interchangeable. Build the retrieval set first — it is the cheapest, the most reusable, and it constrains everything downstream.
3.2 Retrieval metrics
recall@k asks whether the gold documents made it into the top k. MRR asks how high the first relevant one ranked. nDCG@k handles graded relevance and discounts by position; it is derived with a worked example in Reranking: The Second Pass That Decides What the Model Sees, so it is not re-derived here.
Which to use depends on what happens downstream. If your generator reads all k chunks equally, recall@k is the metric that matters and ordering within k is close to irrelevant. If you truncate to fit a context budget, or the model attends more to earlier chunks, order matters and you want nDCG. Choosing the metric that does not match your consumption pattern is a common way to optimise something the system does not care about.
3.3 Generation metrics, and the gold-context trick
The important move in this whole article: evaluate the generator on gold context.
Feed the generator the documents that should have been retrieved rather than the ones that were. Retrieval is now perfect by construction, so anything that goes wrong is the generator's fault. Compare that to its score on real retrieved context, and the gap is precisely what retrieval is costing you.
generation on GOLD context = 0.94 <- the generator is fine
generation on RETRIEVED context = 0.66 <- retrieval is the bottleneck
------------------------------------------
gap attributable to retrieval = 0.28
Two metrics matter on the generation side. Faithfulness asks whether every claim in the answer is supported by the supplied context — this is hallucination detection pointed at your own output, and the judging machinery (NLI, LLM-as-judge, citation checking) is covered in Hallucination Detection & Grounding. Answer relevance asks whether the answer addresses the question, which is a genuinely separate axis: an answer can be perfectly faithful to the context and still not answer what was asked.
Both are usually scored by an LLM judge, which brings its own problem — the judge is a model with its own error rate, and a judge that agrees with human labels 85% of the time puts a ceiling on what you can measure. Validate the judge against human labels on a sample before trusting it on thousands.
3.4 Sample size, where most eval sets fail
Here is the uncomfortable arithmetic. A typical hand-built eval set has 20–50 queries. Measured in §5, comparing a genuinely better system (75%) against a worse one (70%) on 20 queries identifies the better one 57% of the time. On 50 queries, 68%. You need about 200 for the conventional 80% power.
This means most reported RAG improvements are unmeasured. The number moved, the team shipped, and the direction of the change was close to a coin flip. It also means a regression of the same size is equally invisible, which is worse.
Three consequences worth internalising:
- Effect size dominates sample size. A 20-point improvement is visible on ~10 queries. A 2-point improvement needs ~500. If you are hunting for 1–2 point gains on 50 queries, you are measuring noise, and no amount of care in labelling fixes that.
- Pair your comparisons. Put the same queries to both systems. Shared query difficulty then cancels out of the comparison rather than adding variance to it — worth roughly 2× here (about 100 queries instead of 200).
- Report the uncertainty. "74% ± 6%" invites a different decision from "74%". A bootstrap over your eval set is a few lines and turns a misleading point estimate into an honest interval.
3.5 Building the eval set
Sample from real traffic, not imagination. Hand-invented queries are systematically cleaner, better-spelled, and more answerable than what users actually type, and a system tuned on them fails in ways the eval cannot see. Stratify so the hard categories are represented in proportion — or deliberately over-represented if they matter more.
Include queries whose answer is not in the corpus. A system that never says "I don't know" will score well on an eval set where everything is answerable, and then confidently fabricate in production. This is the single most common gap in a RAG eval set.
Freeze the set, version it, and treat it as code. Growing it opportunistically — adding a query whenever someone reports a bug — silently biases it toward whatever failed recently, and makes scores incomparable across time.
3.6 What offline evaluation cannot tell you
Offline evaluation measures ranking and answer quality. It cannot measure latency, and it cannot measure what users do. A change that improves nDCG by 3 points and adds 4 seconds is a regression that every offline metric will call an improvement — the failure described in Reranking: The Second Pass That Decides What the Model Sees. Pair offline eval with a latency budget and, where you can, an online metric.
4. The math
4.1 Why the stages compose, and what that hides
For a query to succeed end to end, retrieval must supply the evidence and generation must use it:
P(success) = P(retrieval ok) * P(generation ok | retrieval ok)
Multiplication is what makes the aggregate uninformative. Any two factors with the same product give the same score:
0.70 * 0.95 = 0.665 measured 66.0%
0.95 * 0.70 = 0.665 measured 66.5%
Identical end-to-end, opposite diagnoses. The failure mass sits in retrieval for the first (30.6% of queries) and in generation for the second (28.4%). Only the factored view tells you which.
It also tells you where the leverage is. Improving a factor multiplies the whole product, so the smaller factor has more headroom: taking 0.70 → 0.80 gains 9.5 points, while 0.95 → 1.00 gains only 3.3. Fix the weaker stage first, always.
4.2 Sizing the eval set
Comparing two systems on n queries is comparing two proportions. For unpaired comparison, the standard error of the difference is:
SE = sqrt( p1(1-p1)/n + p2(1-p2)/n )
With p1 = 0.70, p2 = 0.75, n = 20:
SE = sqrt(0.21/20 + 0.1875/20) = sqrt(0.0199) = 0.141
The effect you are hunting is 0.05. The noise is 0.141 — nearly three times the signal. No labelling care rescues that; the experiment cannot see the effect. Measured power at n=20 is 56.5%, barely above the 50% you would get by guessing.
Pairing removes the shared-difficulty component of the variance. Only discordant pairs — queries where the two systems disagree — carry information, which is why the same queries put to both systems is so much more efficient:
n for 80% power, 0.70 vs 0.75: unpaired ~200 paired ~100
And effect size dominates everything else:
gap n for 80% power (paired)
0.02 500
0.05 100
0.10 50
0.20 10
Quadrupling n roughly halves the detectable effect, so chasing small gains gets expensive very fast. Decide the smallest difference worth acting on, then size the eval set for that — rather than building a set of convenient size and hoping.
5. Real code
Standard library only, deterministic, ~5 seconds. Part A models the composition of two stages; part B is a power simulation using a correlated latent-variable model so that both systems make their own mistakes.
import math
import random
from statistics import NormalDist
SEED = 5
N = 20000 # for stage-attribution estimates
def simulate_pipeline(p_retrieval, p_generation, n=N, seed=SEED):
"""Generation can only succeed if retrieval supplied the evidence."""
rng = random.Random(seed)
both = ret_only = gen_fail = 0
for _ in range(n):
r = rng.random() < p_retrieval
g = rng.random() < p_generation
if r and g:
both += 1
elif r and not g:
gen_fail += 1
elif not r:
ret_only += 1
return both / n, gen_fail / n, ret_only / n
def attribution():
print("A. WHERE DOES END-TO-END FAILURE COME FROM?")
print(f"{'retrieval':>10} {'generation':>11} {'end-to-end':>11} "
f"{'gen fault':>10} {'ret fault':>10}")
print("-" * 56)
for pr, pg in ((0.90, 0.90), (0.70, 0.95), (0.95, 0.70), (0.60, 0.60)):
e2e, gfail, rfail = simulate_pipeline(pr, pg)
print(f"{pr:10.2f} {pg:11.2f} {e2e:11.1%} {gfail:10.1%} {rfail:10.1%}")
print("\n Two systems can share an end-to-end score and need opposite fixes:")
a = simulate_pipeline(0.70, 0.95)
b = simulate_pipeline(0.95, 0.70)
print(f" retrieval .70 / generation .95 -> {a[0]:.1%} end to end")
print(f" retrieval .95 / generation .70 -> {b[0]:.1%} end to end")
print(f" identical score, but the first is a RETRIEVAL problem "
f"({a[2]:.0%} of queries) and the second a GENERATION one "
f"({b[1]:.0%}).")
print("\n The fix: measure generation against GOLD context. That holds")
print(" retrieval fixed at 100% and isolates the generator.")
# Latent-variable model. Each query has a shared difficulty z; each system
# also has idiosyncratic error. rho = how correlated the two systems are.
# rho=0 behaves like two unrelated systems; rho->1 like perfectly nested ones.
RHO = 0.7
_ND = NormalDist()
def power(p1, p2, n_queries, paired, rho=RHO, trials=4000, seed=SEED):
"""How often does an eval of n queries correctly rank system 2 above 1?"""
rng = random.Random(seed)
t1, t2 = _ND.inv_cdf(p1), _ND.inv_cdf(p2)
a, b = math.sqrt(rho), math.sqrt(1 - rho)
wins = 0
for _ in range(trials):
s1 = s2 = 0
for _ in range(n_queries):
if paired:
# SAME query put to both systems: shared difficulty z cancels
z = rng.gauss(0, 1)
s1 += (a * z + b * rng.gauss(0, 1)) < t1
s2 += (a * z + b * rng.gauss(0, 1)) < t2
else:
# different query samples per system: z differs too
s1 += (a * rng.gauss(0, 1) + b * rng.gauss(0, 1)) < t1
s2 += (a * rng.gauss(0, 1) + b * rng.gauss(0, 1)) < t2
wins += (s2 > s1)
return wins / trials
def sample_size():
P1, P2 = 0.70, 0.75
print(f"\n\nB. HOW MANY EVAL QUERIES? (true accuracy {P1:.0%} vs {P2:.0%})")
print(f"{'queries':>8} {'unpaired':>10} {'paired':>8}")
print("-" * 28)
res = {}
for n in (20, 50, 100, 200, 500, 1000, 2000):
u = power(P1, P2, n, paired=False)
p = power(P1, P2, n, paired=True)
res[n] = (u, p)
print(f"{n:>8} {u:10.1%} {p:8.1%}")
print(f"\n at n=20 an eval picks the better system {res[20][0]:.0%} of the"
f" time -- barely better than a coin flip")
fu = next((n for n in res if res[n][0] >= 0.80), None)
fp = next((n for n in res if res[n][1] >= 0.80), None)
print(f" 80% power needs ~{fu} queries unpaired, ~{fp} paired")
print(f" pairing the SAME queries to both systems removes shared query")
print(f" difficulty from the comparison -- worth ~{fu/fp:.0f}x here")
print("\n effect size dominates everything:")
print(f"{'gap':>6} {'n for 80% power (paired)':>26}")
print("-" * 34)
for p2 in (0.72, 0.75, 0.80, 0.90):
need = next((n for n in (10, 20, 50, 100, 200, 500, 1000, 2000, 5000)
if power(P1, p2, n, paired=True) >= 0.80), ">5000")
print(f"{p2-P1:6.2f} {str(need):>26}")
attribution()
sample_size()
# claims made in the prose
a = simulate_pipeline(0.70, 0.95)
b = simulate_pipeline(0.95, 0.70)
assert abs(a[0] - b[0]) < 0.02, "the two configs must score alike end to end"
assert a[2] > 3 * a[1], "config A must be retrieval-dominated"
assert b[1] > 3 * b[2], "config B must be generation-dominated"
assert power(0.70, 0.75, 20, paired=False) < 0.70, "n=20 must be weak"
assert power(0.70, 0.75, 2000, paired=True) > 0.95
assert (power(0.70, 0.75, 100, paired=True)
> power(0.70, 0.75, 100, paired=False)), "pairing must help"
print("\nasserts passed")
# Output:
# A. WHERE DOES END-TO-END FAILURE COME FROM?
# retrieval generation end-to-end gen fault ret fault
# --------------------------------------------------------
# 0.90 0.90 80.7% 9.0% 10.2%
# 0.70 0.95 66.0% 3.5% 30.6%
# 0.95 0.70 66.5% 28.4% 5.1%
# 0.60 0.60 35.8% 23.4% 40.8%
#
# Two systems can share an end-to-end score and need opposite fixes:
# retrieval .70 / generation .95 -> 66.0% end to end
# retrieval .95 / generation .70 -> 66.5% end to end
# identical score, but the first is a RETRIEVAL problem (31% of queries) and the second a GENERATION one (28%).
#
# The fix: measure generation against GOLD context. That holds
# retrieval fixed at 100% and isolates the generator.
#
#
# B. HOW MANY EVAL QUERIES? (true accuracy 70% vs 75%)
# queries unpaired paired
# ----------------------------
# 20 56.5% 58.9%
# 50 67.5% 73.2%
# 100 75.9% 83.3%
# 200 85.7% 92.3%
# 500 95.7% 99.1%
# 1000 99.2% 100.0%
# 2000 100.0% 100.0%
#
# at n=20 an eval picks the better system 57% of the time -- barely better than a coin flip
# 80% power needs ~200 queries unpaired, ~100 paired
# pairing the SAME queries to both systems removes shared query
# difficulty from the comparison -- worth ~2x here
#
# effect size dominates everything:
# gap n for 80% power (paired)
# ----------------------------------
# 0.02 500
# 0.05 100
# 0.10 50
# 0.20 10
#
# asserts passed
The 0.90 / 0.90 row is worth a second look: an 80.7% end-to-end score with failure split almost evenly (9.0% generation, 10.2% retrieval). That is the case where the aggregate number is least misleading — and also the case where you have no single high-leverage fix. Lopsided failure is better news than balanced failure, because it tells you where to work.
6. Real-world example
A team spent a quarter improving their generator. Better prompts, a stronger model, a citation-formatting pass, few-shot examples for tone. Their end-to-end score moved from 61% to 64% and then stopped, no matter what they tried.
They finally ran the generator against gold context — feeding it the documents that should have been retrieved. It scored 93%.
The generator had never been the problem. It had been at roughly 90% the whole quarter, and every prompt improvement was squeezing the 10% that remained while retrieval quietly failed on nearly a third of queries. The quarter's work was real, and it was aimed at the factor with almost no headroom.
Making it worse, their eval set was 40 queries. The 61%→64% "improvement" that had justified continuing down this path was, on 40 queries, indistinguishable from noise — they had been steering by a number that could not resolve the changes they were making.
Two changes fixed the process, and neither was a model change. They added the gold-context run as a standing metric, so retrieval and generation now had separate numbers on the same dashboard. And they grew the eval set to 300 queries sampled from real traffic, including 40 whose answers were deliberately absent from the corpus. The first honest measurement showed retrieval at 0.68 — and the next quarter's work, aimed there, moved the end-to-end score more than the entire previous quarter had.
7. Interview questions companies actually ask
Q1 [easy] "Why isn't one end-to-end score enough?"
A Because retrieval and generation MULTIPLY, so different factor pairs give the
same product. Measured: 0.70 retrieval x 0.95 generation = 66.0%, and
0.95 x 0.70 = 66.5%. Same score, opposite diagnosis -- the first fails on
retrieval for 31% of queries, the second on generation for 28%. The aggregate
tells you something is wrong, never what.
Q2 [easy] "How do you isolate the generator's quality?"
A Run it on GOLD context -- the documents that should have been retrieved rather
than the ones that were. Retrieval is then perfect by construction, so anything
wrong is the generator's. The gap between its gold-context score and its
real-context score is exactly what retrieval is costing you.
Q3 [medium] "Which stage should you fix first?"
A The weaker factor, because improving a factor multiplies the whole product and
the smaller one has more headroom. Going 0.70 -> 0.80 gains 9.5 points end to
end; going 0.95 -> 1.00 gains 3.3. Teams routinely optimise the stage they
understand best rather than the one with room.
Q4 [medium] "You have 20 eval queries. Is that enough?"
A No. Comparing a true 70% system against a true 75% one, n=20 picks the better
one 56.5% of the time -- essentially a coin flip. SE of the difference is 0.141
against an effect of 0.05, so noise is ~3x signal. You need ~200 for 80% power
unpaired, ~100 paired.
Q5 [medium] "How do you make a small eval set go further?"
A Pair it: put the SAME queries to both systems so shared query difficulty cancels
out instead of adding variance. Worth about 2x here. Also report a bootstrap
confidence interval rather than a point estimate -- '74% ± 6%' prompts a very
different decision from '74%'.
Q6 [medium] "Faithfulness and answer relevance -- what's the difference?"
A Faithfulness asks whether every claim is supported by the supplied context.
Answer relevance asks whether the answer addresses the QUESTION. They're
orthogonal: an answer can be perfectly faithful to retrieved context and still
not answer what was asked. You need both, or you optimise a grounded system that
misses the point.
Q7 [hard] "What's missing from most RAG eval sets?"
A Queries whose answer isn't in the corpus. If everything in your eval is
answerable, a system that never says 'I don't know' scores perfectly and then
fabricates confidently in production -- you've selected for exactly the wrong
behaviour. Also: hand-written queries are cleaner and more answerable than real
traffic, so sample from logs.
Q8 [hard] "Your offline metrics improved but users are unhappy. What did you miss?"
A Almost certainly latency, which no offline ranking metric captures -- +3 nDCG
and +4 seconds is an offline win and a product regression. Possibly also an
empty-result or refusal path your eval set doesn't contain. And check the eval
set hasn't drifted: sets grown by adding whatever failed recently are biased
toward old bugs and not comparable over time.
8. When to use / tradeoffs
BUILD THE THREE-LAYER EVAL WHEN:
+ you're making more than one change and need to know which one worked
+ you can't tell whether to invest in retrieval or generation
+ regressions would otherwise ship silently
ORDER OF CONSTRUCTION (cheapest and most reusable first):
1. retrieval set: query -> gold documents (no model calls to score)
2. gold-context generation run (isolates the generator)
3. end-to-end set: query -> gold answer (what users experience)
4. unanswerable queries: ~10-15% of the set (tests refusal)
| Situation | Why it breaks | Use instead |
|---|---|---|
| One end-to-end number | Conflates factors with identical products | Factor into per-stage metrics |
| 20–50 query eval set | 57% power at n=20; noise ≈ 3× signal | ~200 queries, or ~100 paired |
| Different queries per system | Query difficulty inflates variance | Pair — same queries both systems |
| Point estimates, no interval | "74%" hides ±6% | Bootstrap CI on the eval set |
| Only answerable queries | Selects for systems that never refuse | 10–15% unanswerable |
| Hand-invented queries | Cleaner and more answerable than real traffic | Sample from logs, stratified |
| Eval set grown ad hoc | Biased to recent bugs; scores incomparable over time | Freeze and version it |
| Chasing 1–2 point gains | Below the resolution of any practical eval set | Decide the minimum worthwhile effect first |
| Trusting the LLM judge | Judge error caps measurable quality | Validate judge vs humans on a sample |
| Offline metrics only | Blind to latency and user behaviour | Add a latency budget + an online metric |
Honest limits. Part A treats retrieval and generation success as independent Bernoulli events given the stage before. They are not: hard queries tend to be hard for both stages, and partial retrieval (two of three needed documents) is common and is modelled here as binary success. Positive correlation means real attribution is muddier than the clean split shown, and partial credit needs graded scoring rather than a coin flip. Part B's power figures assume a binary per-query outcome; graded metrics like nDCG carry more information per query and need somewhat smaller samples, so treat ~200 as a conservative ceiling for binary success and re-derive it for your metric. The rho = 0.7 correlation between systems is a guess — the more similar your two systems, the more pairing helps, so the ~2× figure moves with it. All of these numbers assume queries are independent draws from a stable distribution; if your eval set over-samples one topic, or traffic shifts, the effective sample size is smaller than the count. Finally, statistical power is about detecting a difference, not about whether the difference matters: a real, well-measured 1-point gain can still be worthless if it costs 3 seconds of latency.
9. Summary + related articles
- Retrieval and generation multiply, so one end-to-end number cannot tell you what to fix: 66.0% and 66.5% for systems needing opposite repairs.
- Run the generator on gold context. It holds retrieval at 100% by construction and isolates the generator. This is the highest-value addition to most RAG evals.
- Fix the weaker factor first — 0.70→0.80 gains 9.5 end-to-end points, 0.95→1.00 gains 3.3.
- Most eval sets are too small to see their own results. n=20 picks the better of a 70%/75% pair 57% of the time. 80% power needs ~200 queries, ~100 paired.
- Pair your comparisons — same queries to both systems cancels shared difficulty, worth ~2×.
- Effect size dominates: 2 points needs ~500 queries, 20 points needs ~10. Decide the smallest worthwhile difference, then size for it.
- Include unanswerable queries. Otherwise you select for a system that never refuses and always fabricates.
- Boundary: offline evaluation is blind to latency and to user behaviour. A metric win with a latency loss is a product regression that every number here will call an improvement.
Related:
- Reranking: The Second Pass That Decides What the Model Sees — §4.3 derives nDCG; §4.1's ceiling decomposition is the retrieval-side analogue of the stage attribution here
- Hallucination Detection & Grounding — how faithfulness is actually scored: NLI, LLM-as-judge, citation checking, and their failure modes
- Backtesting, Baselines & Sensitivity Analysis — holding everything else fixed so a measured delta is attributable
- RAG Cost Optimization: Find the Step That Runs Forty Times — the other axis every change trades against; evaluate quality and cost together
- Query Transformation: Fixing the Question Before You Retrieve — a worked example of per-query reporting revealing what the mean hid
- Probability & Statistics Foundations — proportions, standard error, and sampling variation underneath §4.2
Resources
- Es, James, Espinosa-Anke & Schockaert, "RAGAS: Automated Evaluation of Retrieval Augmented Generation", EACL 2024 (arXiv:2309.15217) — the faithfulness / answer-relevance / context-relevance decomposition.
- Saad-Falcon, Khattab, Potts & Zaharia, "ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems", NAACL 2024 (arXiv:2311.09476) — training judges and validating them against human labels.
- Chen et al., "Benchmarking Large Language Models in Retrieval-Augmented Generation", AAAI 2024 (arXiv:2309.01431) — includes negative rejection, the unanswerable-query axis from §3.5.
- Järvelin & Kekäläinen, "Cumulated Gain-Based Evaluation of IR Techniques", ACM TOIS 20(4), 2002 — the nDCG definition referenced in §3.2.
- Efron & Tibshirani, An Introduction to the Bootstrap, Chapman & Hall 1993 — chapters 6 and 13 cover the confidence intervals recommended in §3.4.
- Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed., Routledge 1988 — chapter 6 covers power for differences between proportions, the analytic form of §4.2.