Study Screening & Selection: Dual Reviewers, PRISMA, and Kappa

TL;DR — After the search returns hundreds or thousands of records, screening narrows them to the studies you'll actually appraise — and it must be reproducible, not one person's judgment. The standard is two independent reviewers screening each record (title/abstract, then full text), with disagreements reconciled (discuss, or a third reviewer). You measure how much they agree beyond chance with Cohen's kappa, and you report the whole funnel — identified → deduplicated → screened → included → excluded-with-reasons — as a PRISMA flow diagram. The guiding bias is conservative: at title/abstract, if either reviewer says include, it advances (you'd rather over-include early than lose a relevant study). It breaks when reviewers share the same blind spot, or when "screen with AI" skips the human reconciliation that makes selection trustworthy.

---

1. Simple explanation

A systematic search is deliberately broad, so it returns far more than you can use — most records are irrelevant, some are duplicates, a few are the studies you need. Screening is the funnel that gets from "everything the search found" to "the studies we'll appraise," and the whole point of doing it systematically is that someone else could repeat your process and reach the same set.

Two design choices make screening trustworthy. First, two independent reviewers screen each record without seeing each other's decisions, then reconcile disagreements — because a single screener's calls are unrepeatable and biased. Second, you screen in stages: a fast title/abstract pass on everything, then a slower full-text pass on what survives, recording why each full-text exclusion happened. You quantify reviewer agreement with Cohen's kappa (agreement beyond what you'd expect by chance) and you draw the counts as a PRISMA flow diagram so the funnel is transparent.

A key habit: at the cheap title/abstract stage, be conservative — when the two reviewers disagree, keep the record for full-text review rather than dropping it. Missing a relevant study (a false exclude) is a much worse error than reading one extra abstract (a false include), and the full-text stage will catch the over-inclusions.

Analogy — two graders and a rubric, not one. A single grader marking a stack of exams introduces their mood and bias. Two graders marking independently, then resolving the papers they scored differently, produces a grade you can defend — and their disagreement rate tells you how clear the rubric was. Screening is that, applied to studies; kappa is the "how much did the two graders actually agree" number.

---

2. Diagram

  SEARCH RESULTS ──▶ DEDUPLICATE ──▶ TITLE/ABSTRACT SCREEN ──▶ FULL-TEXT SCREEN ──▶ INCLUDED
     (identified)      (remove       (2 reviewers, fast)        (2 reviewers,        (appraise)
                        dupes)         │                          record reasons)
                                       ▼
                             reviewer A vs reviewer B
                               agree  → decision stands
                               differ → RECONCILE (discuss / 3rd reviewer)
                               (title/abstract: if EITHER includes → advance)

  PRISMA FLOW (report the counts)              AGREEMENT (report kappa)
     identified            10                     observed agreement  0.67
     after deduplication    6                     expected by chance  0.50
     screened               6                     kappa               0.33  (fair)
     → to full-text         4                     (1.0 perfect · 0 = chance)

---

3. How it works

3.1 Deduplicate first

The same study appears across databases (PubMed, Embase, Cochrane…), so the raw "identified" count is inflated. Remove duplicates before screening — otherwise two reviewers waste effort screening the same record twice and your PRISMA counts mislead. The gap between "identified" and "after deduplication" is a real number you report.

3.2 Two independent reviewers

Each record is judged by two people who can't see each other's decisions. This is the core of systematic selection: it removes any single reviewer's idiosyncrasy and makes the process auditable. Where they agree, the decision stands; where they disagree, you reconcile — a discussion, or a pre-designated third reviewer / senior adjudicator. You never silently let one reviewer overrule; disagreement is data (and it flags an unclear inclusion criterion).

3.3 Stage it: title/abstract, then full text

Screening happens in passes of increasing cost. The title/abstract pass is fast and errs toward inclusion. The full-text pass is slow, reads the whole paper against the eligibility criteria, and — crucially — records a reason for every exclusion (wrong population, wrong comparator, wrong outcome, wrong design…). Those reasons are required in the PRISMA diagram and are what make exclusions defensible.

3.4 Measure agreement: Cohen's kappa

Raw agreement (the fraction of records both reviewers decided the same way) is misleading because two reviewers agree by chance a lot when most records are "exclude." Cohen's kappa corrects for that: it's the observed agreement minus the chance agreement, scaled so 1.0 is perfect and 0 is chance-level. Low kappa is a signal your eligibility criteria are ambiguous — fix the criteria, don't just push through. (Rough reading: <0.4 fair/poor, 0.4–0.6 moderate, 0.6–0.8 substantial.)

3.5 Report the funnel: PRISMA

The PRISMA flow diagram reports the numbers at each stage — records identified, duplicates removed, screened, full-texts assessed, included, and excluded with reasons. It is what lets a reader (or a replicator) see exactly how you went from thousands of hits to a handful of included studies. A review without this funnel isn't reproducible.

Boundary condition. Independent dual screening only helps if the reviewers can actually disagree — two people (or two AI screeners) with the same blind spot will agree confidently and wrongly, and high kappa then measures shared bias, not correctness. And "let an LLM screen it" is fine as a first pass or a second reviewer, but replacing human reconciliation entirely removes the check that makes selection trustworthy (Section 8).

---

4. The math

4.1 Cohen's kappa

For two reviewers over n records, let p_o = observed agreement (fraction decided the same), and p_e = agreement expected by chance from each reviewer's inclusion rate:

  p_e   = p_A·p_B + (1 − p_A)·(1 − p_B)         # p_A, p_B = each reviewer's include rate
  kappa = (p_o − p_e) / (1 − p_e)               # 1 = perfect, 0 = chance, <0 = worse than chance

Conservative reconcile at title/abstract (advance if either includes):

  advance(record) = A_includes(record) OR B_includes(record)
  disagreements   = { records where A_includes ≠ B_includes }   # go to reconciliation

4.2 Worked example

Six records after dedup. Reviewer A includes {r1, r2, r4}; Reviewer B includes {r1, r4, r6}. They agree on r1 (both in), r3 (both out), r4 (both in), r5 (both out) — 4 of 6, so p_o = 0.667. Each includes 3 of 6, so p_A = p_B = 0.5 and p_e = 0.5·0.5 + 0.5·0.5 = 0.5; kappa = (0.667 − 0.5)/(1 − 0.5) = 0.333 (fair — the criteria need tightening). The disagreements are r2 and r6; under conservative reconcile both advance to full text (better safe than sorry), so the included set is {r1, r2, r4, r6}. As a PRISMA funnel: 10 identified → 6 after deduplication → 6 screened → 4 to full text.

---

5. Real code

# Two independent reviewers screen records at title/abstract stage.
# Disagreements resolve CONSERVATIVELY: if EITHER includes, it advances to full text
# (better to over-include now than lose a relevant study).
records = ["r1","r2","r3","r4","r5","r6"]
rev_A = {"r1":1,"r2":1,"r3":0,"r4":1,"r5":0,"r6":0}   # 1=include, 0=exclude
rev_B = {"r1":1,"r2":0,"r3":0,"r4":1,"r5":0,"r6":1}

agree = sum(rev_A[r]==rev_B[r] for r in records)/len(records)
pa = sum(rev_A.values())/len(records); pb = sum(rev_B.values())/len(records)
pe = pa*pb + (1-pa)*(1-pb)                            # agreement expected by chance
kappa = (agree-pe)/(1-pe)                             # Cohen's kappa
included = [r for r in records if rev_A[r] or rev_B[r]]
disagree = [r for r in records if rev_A[r]!=rev_B[r]]

identified, duplicates = 10, 4                        # PRISMA counts
print(f"observed agreement={agree:.3f}  chance={pe:.3f}  kappa={kappa:.3f}")
print(f"disagreements (need reconciliation): {disagree}")
print(f"PRISMA: identified={identified} -> after dedup={identified-duplicates} "
      f"-> screened={len(records)} -> to full-text={len(included)}")
assert included == ["r1","r2","r4","r6"]              # r2 & r6 advance despite one 'exclude'
assert disagree == ["r2","r6"]
print("OK: dual-reviewer screen — conservative reconcile keeps borderline records")

# Output:
#   observed agreement=0.667  chance=0.500  kappa=0.333
#   disagreements (need reconciliation): ['r2', 'r6']
#   PRISMA: identified=10 -> after dedup=6 -> screened=6 -> to full-text=4
#   OK: dual-reviewer screen — conservative reconcile keeps borderline records

Note the raw agreement (0.667) sounds fine, but kappa (0.333) reveals it's only "fair" once you remove the agreement expected by chance — the signal that the inclusion criteria need sharpening.

---

6. Real-world example

A team let a single reviewer screen a 2,000-record search to save time. The review later failed peer review: a relevant trial had been excluded at the title/abstract stage on one person's misreading, and because there was no second reviewer and no kappa, nobody had caught it or could show the process was reproducible. Redone properly — dedup first, two independent reviewers, conservative advancement of disagreements, reasons logged at full text, and a PRISMA diagram — the same search yielded a defensible included set, and the reviewers' initial kappa of ~0.3 exposed two vague eligibility criteria that were then rewritten (raising agreement on a re-screen).

The recurring lesson: screening is a measurement, not a chore. Skipping the second reviewer or the kappa doesn't just risk a missed study — it removes the evidence that the selection was systematic at all, which is the whole claim a systematic review is making.

---

7. Interview questions companies actually ask

Q1. Why two independent reviewers instead of one? Because a single screener's decisions are unrepeatable and carry their individual bias. Two independent reviewers plus reconciliation removes idiosyncrasy, makes the process auditable, and their disagreement rate itself flags unclear eligibility criteria.

Q2. Why report kappa instead of raw agreement? Because raw agreement is inflated by chance — when most records are "exclude," two reviewers agree a lot just by guessing. Cohen's kappa subtracts the chance agreement, so it reflects real concordance; a low kappa means the criteria are ambiguous, not that the reviewers are careless.

Q3. How do you handle a disagreement at title/abstract? Conservatively: if either reviewer includes, advance the record to full-text review. A false exclude (losing a relevant study) is far costlier than a false include, and the full-text stage will filter the extras. Persistent disagreements go to discussion or a third reviewer.

Q4. What is the PRISMA flow diagram and why is it required? It reports the counts through the funnel — identified, duplicates removed, screened, full-texts assessed, included, and excluded with reasons. It's what makes the selection transparent and reproducible; without it, a reader can't verify how you reached your included set.

Q5. Why screen in stages (title/abstract then full text)? Cost. The title/abstract pass is fast and cheaply removes the obviously irrelevant majority; the expensive full-text pass reads the whole paper against eligibility and records exclusion reasons. Staging spends careful effort only where it's needed.

Q6. Can you use an LLM to screen? As a first pass or a second reviewer, usefully — but not as a replacement for human reconciliation. Two AI screeners can share the same blind spot and agree confidently and wrongly, so an LLM screen needs a human check, calibration against a labeled sample, and the same conservative, auditable reconciliation.

---

8. When to use / tradeoffs

Reach for dual-reviewer screening when:

• You're conducting a systematic review or guideline and selection must be reproducible.
• A missed relevant study is a serious error (the usual case in evidence synthesis).
• You need an auditable funnel (PRISMA) and a defensible agreement metric (kappa).

Adapt or don't over-apply it when:

Honest limits. Dual screening guarantees reproducibility and inter-rater reliability, not correctness — two reviewers can agree on the wrong call if the criteria are wrong or both share a blind spot, and kappa will look great. Kappa itself is sensitive to prevalence (when almost everything is excluded, it can look low even with high raw agreement), so read it alongside the counts, not alone. Conservative advancement controls false excludes at the cost of more full-text reading. And automating screening trades human labor for the risk of correlated machine errors — usable with calibration and a human check, dangerous as a silent replacement.

---

9. Summary + related articles

• Screening turns a broad search into the studies you'll appraise, and must be reproducible, not one person's judgment.
• Use two independent reviewers, reconcile disagreements, and stage it (title/abstract → full text with recorded exclusion reasons).
• Measure agreement with Cohen's kappa (beyond chance); low kappa means fix the criteria.
• Report the PRISMA funnel (identified → dedup → screened → included/excluded-with-reasons).
• Be conservative at title/abstract (advance if either includes) — a false exclude is costlier than a false include.
• Boundary: dual screening ensures reliability, not truth; shared reviewer bias (human or AI) fools kappa, and AI screening needs human reconciliation, not replacement.

Related:

• PICO: Turning a Vague Question into an Answerable One — the eligibility criteria you screen against come straight from PICO.
• Study Designs & the Evidence Hierarchy — one of the main things you screen for.
• Multi-Agent Debate: When Letting Models Argue Helps (and When It Hurts) — the AI analogue of two independent reviewers who must reconcile (and its echo-chamber failure mode).

Resources

• Page, M. J. et al. (2021). "The PRISMA 2020 statement: an updated guideline for reporting systematic reviews." BMJ, 372:n71 — the flow diagram and reporting standard. (Venue confirmed; verify article number.)
• Cohen, J. (1960). "A Coefficient of Agreement for Nominal Scales." Educational and Psychological Measurement, 20(1), 37–46 — the origin of kappa. (Venue confirmed; verify pages.)
• Higgins, J. P. T. et al. (eds.), Cochrane Handbook for Systematic Reviews of Interventions — chapter on selecting studies and dual independent screening. https://training.cochrane.org/handbook