Shapley Values: Fairly Attributing Credit to Features and Agents

Linear algebraProbabilityBeginner15 min

By: Anacodic Team

TL;DR — When several parts cooperate to produce one result — features in a model, agents on a team, steps in a pipeline — the Shapley value answers "how much did each part actually contribute?" It comes from cooperative game theory: average each part's marginal contribution across every order in which the parts could be added. It is the unique attribution satisfying efficiency (contributions sum to the whole), symmetry, dummy (zero for a part that never helps), and additivity — which is why SHAP built model explanations on it. The cost is exponential in the number of parts (2^n coalitions), so for more than a handful you approximate (sampling / SHAP). It attributes credit, not causation, and can mislead when parts are strongly correlated.

1. Simple explanation

Say a team of three ships a result together and you want to split the credit fairly. You can't just ask "what did each person do alone?" — the value often comes from combinations (two people together achieve more than the sum of their solos). The Shapley value solves exactly this: it's a principled way to divide a joint payoff among contributors so that the split is fair by clear rules.

The same question shows up all over AI: which feature drove this prediction? which agent/specialist carried a multi-agent answer? which pipeline step mattered? A single ablation ("remove part X, see the drop") is misleading, because a part's importance depends on what else is present — X may be redundant when Y is there but essential when Y is absent. Shapley fixes this by averaging X's marginal contribution over all possible teammates-already-present situations.

Analogy — splitting a group project grade. You don't grade each student in isolation (that ignores how they combined) and you don't just eyeball it. You ask, for every order the team could have "joined" the project, how much the grade went up when each student joined — then average. A student who only helps when paired with a specific teammate gets partial credit for exactly those situations. That average is their Shapley value.

2. Diagram

  Three parts {S, C, Z}.  v(coalition) = value that subset can reach.
  A part's marginal contribution depends on WHO IS ALREADY IN.

  order of joining      marginal credit to S at the moment S joins
  ────────────────      ─────────────────────────────────────────
  S C Z                 v(S)      − v(∅)
  S Z C                 v(S)      − v(∅)
  C S Z                 v(C,S)    − v(C)
  C Z S                 v(C,Z,S)  − v(C,Z)
  Z S C                 v(Z,S)    − v(Z)
  Z C S                 v(Z,C,S)  − v(Z,C)
                        ───────────────────
  Shapley(S) = AVERAGE of those 6 marginals

  Guarantee:  Shapley(S) + Shapley(C) + Shapley(Z) = v(whole team)   (efficiency)

3. How it works

3.1 The value function

Everything starts from a value function v(S): the payoff a subset (coalition) S of parts can achieve on its own. For features, v(S) might be the model's prediction (or accuracy) using only features in S; for agents, the accuracy the subset of agents reaches; for pipeline steps, the score with only those steps on. v(∅) is the baseline with nothing. Defining v well is the real work — Shapley is only as meaningful as the game you set up.

3.2 Marginal contribution depends on context

Part x's marginal contribution to a coalition S is v(S ∪ {x}) − v(S) — how much adding x improves things given S is already present. The key insight is that this differs by context: x can be worth a lot when it joins early and little when it joins after a redundant partner. A single "leave-one-out" ablation only measures one context (adding x last, to everyone else) and so systematically misjudges parts that overlap.

3.3 Average over all orders

The Shapley value averages x's marginal contribution over every order in which the parts could be assembled (equivalently, over all coalitions, weighted so each coalition size counts equally). Averaging over all orders is what makes it context-fair: x gets credited for the situations where it's pivotal and not for the ones where it's redundant.

3.4 The four axioms (why Shapley, not something else)

Shapley is the unique attribution satisfying all four:

Efficiency — the parts' values sum exactly to v(whole) (no credit lost or invented).
Symmetry — two parts that contribute identically in every coalition get equal credit.
Dummy/null — a part that never changes any coalition's value gets exactly 0.
Additivity — attribution of a sum of games is the sum of the attributions. This uniqueness is why SHAP adopted Shapley values as the basis for model explanations: any attribution that violates these can be gamed or made incoherent.

3.5 Approximating Shapley at scale

Exact Shapley enumerates every ordering — O(2^n) — fine for the handful of parts here, hopeless for the hundreds of features in a real model. Two standard escapes. Monte-Carlo sampling: draw random orderings, average each part's marginal contribution over the sample, and you get an unbiased estimate whose error shrinks as you add samples — you trade compute for precision. Structure-specific exact methods: for tree ensembles, TreeSHAP computes exact Shapley values in polynomial time by exploiting the tree structure, and kernel/linear variants exist for other model classes. The takeaway: the definition is universal, but you rarely brute-force it — you sample it, or use a method that exploits your model's shape.

3.6 Local vs global: explain one prediction, or the whole model

The same machinery answers two different questions, and the only thing that changes is the value function.

Local (per-instance). v(S) = the model's output for one row using only features S. Then phi_i explains this decision — which features drove this prediction. This is what SHAP produces per row.
Global (whole dataset). v(S) = accuracy (or −loss) over the entire dataset when only features S are available. Now phi_i is a single number per feature: its average contribution to overall performance. This is SAGE (Covert et al., 2020) — Shapley for global feature importance.

Same formula, swap "output for this row" for "accuracy over all rows." Use local to debug a single case; use global to rank features for the model as a whole. A common mistake is to average many local attributions and call it importance — related, but SAGE's dataset-level value function is the principled version.

Validate that importance transfers. Feature importance is itself an estimate that can overfit the set you measured it on. The standard check is a train/test split: compute Shapley importance on a training split, then confirm the ranking still holds (and that keeping only the top features preserves accuracy) on a held-out split. Importance that doesn't transfer was specific to those rows, not a property of the problem.

Boundary condition. Exact Shapley needs all 2^n coalitions — fine for ~3–10 parts, infeasible beyond, so you approximate (Monte-Carlo over random orders, or model-specific tricks like TreeSHAP). And it attributes credit under your value function, not real-world causation — with strongly correlated features it can split credit in ways that surprise you (Section 8).

4. The math

4.1 The formula

For part i among parts N (with n = |N|) and value function v:

  phi_i = (1/n!) * sum over all orderings O of [ v(pred(O,i) ∪ {i}) − v(pred(O,i)) ]

  where pred(O,i) = the parts that come before i in ordering O.

  Equivalent coalition form:
  phi_i = sum over S ⊆ N\{i}  of   |S|! (n−|S|−1)! / n!  · [ v(S ∪ {i}) − v(S) ]

In plain words: average i's marginal contribution over every order the parts could join (the two forms are the same; the coalition form just groups equal orders together). Efficiency then guarantees sum_i phi_i = v(N) − v(∅).

4.2 Worked example

Three specialists shape (S), count (C), size (Z); v(coalition) = accuracy that subset reaches:

  v(∅)=0.00
  v(S)=0.50   v(C)=0.40   v(Z)=0.20
  v(S,C)=0.75 v(S,Z)=0.55 v(C,Z)=0.50
  v(S,C,Z)=0.80

Averaging each part's marginal over all 3! = 6 orders gives Shapley values shape = 0.383, count = 0.308, size = 0.108. They sum to 0.800, exactly v(whole) — efficiency. Read off the story: shape is the biggest contributor, size adds little on its own and little on the margin (it's largely redundant once shape/count are present). A naive "size alone scores 0.20" would overstate it; Shapley shows its marginal worth to the team is only 0.108.

5. Real code

from itertools import permutations
players = ["shape", "count", "size"]
# v(S) = accuracy a coalition S can reach (a lookup, for the demo)
V = {frozenset(): 0.00,
     frozenset(["shape"]): 0.50, frozenset(["count"]): 0.40, frozenset(["size"]): 0.20,
     frozenset(["shape","count"]): 0.75, frozenset(["shape","size"]): 0.55,
     frozenset(["count","size"]): 0.50, frozenset(["shape","count","size"]): 0.80}

def shapley(players, V):
    phi = {p: 0.0 for p in players}
    perms = list(permutations(players))
    for order in perms:
        S = set()
        for p in order:
            before = V[frozenset(S)]; S.add(p); after = V[frozenset(S)]
            phi[p] += after - before          # marginal contribution in this order
    return {p: phi[p] / len(perms) for p in players}

phi = shapley(players, V)
for p in players: print(f"  {p:<6} Shapley = {phi[p]:.3f}")
print("  sum        =", round(sum(phi.values()), 3), "(= full-team accuracy)")
assert abs(sum(phi.values()) - V[frozenset(players)]) < 1e-9    # efficiency axiom
print("OK: contributions sum to the full-coalition value (efficiency)")

# Output:
#   shape  Shapley = 0.383
#   count  Shapley = 0.308
#   size   Shapley = 0.108
#   sum        = 0.8 (= full-team accuracy)
#   OK: contributions sum to the full-coalition value (efficiency)

This is the exact Shapley value (all 3! = 6 orders enumerated). For many parts you'd sample orders instead — same formula, estimated.

6. Real-world example

A team ran a multi-specialist system and reported importance with leave-one-out ablations: drop each specialist, measure the accuracy loss. One specialist showed almost zero loss when removed, so they nearly cut it — until a Shapley analysis over coalitions told a different story: that specialist was redundant with a second one (either alone recovered most of the signal), so removing just one barely hurt, but it was genuinely pivotal whenever the other was uncertain. Leave-one-out had measured only "remove it last, with everyone else present" and missed its contribution in the coalitions where it mattered.

Switching to Shapley (sampled over orders, since exact was too costly) redistributed credit sensibly, the "useless" specialist kept a modest but real value, and the team instead merged the two actually redundant components. The lesson recurs wherever parts overlap: single ablations answer one narrow question; Shapley averages over all contexts, which is why it — and SHAP built on it — became the standard for attribution.

7. Interview questions companies actually ask

Q1. What problem does the Shapley value solve? Fairly dividing a joint payoff among cooperating parts when value comes from combinations, not solos. It answers "how much did each feature/agent/step actually contribute?" in a way that accounts for interactions.

Q2. Why is a single leave-one-out ablation not enough? Because a part's importance depends on what else is present. Leave-one-out measures only the "remove it last, everyone else present" context, so it understates parts that are redundant with a present partner but pivotal when that partner is absent. Shapley averages over all such contexts.

Q3. State the Shapley value in words. A part's average marginal contribution across every order in which the parts could be added — equivalently, a size-weighted average of v(S∪{i}) − v(S) over all coalitions S not containing i.

Q4. Which axioms make it unique, and why care? Efficiency (contributions sum to the whole), symmetry (equal contributors get equal credit), dummy (a non-contributor gets zero), and additivity. Uniqueness matters because any attribution violating them can lose/invent credit or rank parts incoherently — SHAP relies on these guarantees.

Q5. What's the computational catch and how do you handle it? Exact Shapley needs all 2^n coalitions, exponential in the number of parts. For more than a handful you approximate — Monte-Carlo sampling over random orders, or structure-specific methods like TreeSHAP for tree models.

Q6. Where does Shapley mislead? With strongly correlated features it can split or shift credit in counterintuitive ways (e.g. spreading one real driver's credit across its correlated proxies), and it attributes credit under your chosen value function — that's association/importance, not causation. Design v and interpret accordingly.

Q7. How does this connect to SHAP? SHAP (Lundberg & Lee, 2017) computes Shapley values with features as the "players" and the model's prediction as the payoff, giving per-feature attributions that satisfy the axioms — a unified, theoretically grounded explanation method.

Q8. A feature has a high Shapley value — does that mean it causes the outcome? No. Shapley attributes how much the model used that feature under your value function — it's association within the model, not a causal claim about the world. A feature can score high purely by proxying a true cause or by correlation with one. Causal questions need interventions or experiments; attribution tells you what the model leaned on, not what actually drives the target.

Q9. What's the difference between local and global Shapley attribution? Local (SHAP) explains one prediction — value function = the model's output for a single row — so you get per-feature credit for that decision. Global (SAGE) explains the model overall — value function = accuracy/loss over the whole dataset — so you get one importance number per feature. Same Shapley formula; the value function is what differs. Validate a global ranking with a train/test split so it isn't overfit.

8. When to use / tradeoffs

Reach for Shapley when:

Several parts cooperate and you need a fair, interaction-aware split of credit.
Single ablations disagree or feel misleading (redundant/complementary parts).
You want attributions with guarantees (they sum to the whole; non-contributors get zero).

Do NOT use it (as-is) when:

Situation	Why it breaks	Use instead
Many parts (n large)	`2^n` coalitions is infeasible	sampling / SHAP / TreeSHAP
Features strongly correlated	credit splits counterintuitively	grouped/coalitional Shapley; report correlations
You need causation	Shapley gives association under `v`	causal analysis / interventions
`v(S)` is ill-defined for subsets	garbage in, garbage out	design a principled value function first
A cheap single signal suffices	Shapley is expensive overkill	one ablation / permutation importance

Honest limits. Shapley's fairness is relative to the value function you choose — pick v badly and the "fair" split is meaningless. It is expensive (exponential exact; approximations add variance). It measures contribution/ importance, not causal effect, and it is known to behave unintuitively under strong feature correlation, so pair it with a look at the correlation structure. And a large Shapley value explains that a part mattered to the model, not whether the model is right — attribution is not validation.

The Shapley value fairly splits a joint payoff by averaging each part's marginal contribution over all orders it could join.
It's the unique attribution with efficiency, symmetry, dummy, and additivity — which is why SHAP uses it for model explanations.
It beats single leave-one-out ablations because importance depends on what else is present.
Cost: exact is 2^n; approximate (sampling/SHAP) beyond a handful of parts.
Boundary: it's credit under your value function, not causation, and it can mislead under strong correlation — design v carefully and interpret with the correlation structure in mind.

Multi-Agent Debate: When Letting Models Argue Helps (and When It Hurts) — attributing which agent carried a debate answer is a Shapley game over agents.
LLM-as-a-Judge: Using a Model to Grade Model Output — when the value function is a judge's score, Shapley attributes credit to whatever the judge graded.
Fair Aggregation: Balancing Utility and Fairness — another cooperative-game idea (fairly combining many parties' satisfaction).
Weighted Averages & Aggregation — the averaging machinery underneath.
Backtesting, Baselines & Sensitivity Analysis — ablations and attribution done honestly in evaluation.

Resources

Shapley, L. S. (1953). "A Value for n-Person Games." In Contributions to the Theory of Games II, Princeton University Press — the original definition and axioms. (Volume confirmed; pages not verified here.)
Lundberg, S., Lee, S.-I. (2017). "A Unified Approach to Interpreting Model Predictions" (SHAP). NeurIPS — Shapley values for feature attribution. (Venue confirmed; verify arXiv id 1705.07874 before citing.)
Ghorbani, A., Zou, J. (2019). "Data Shapley: Equitable Valuation of Data for Machine Learning." ICML — Shapley applied to valuing training data. (Venue confirmed; arXiv id not verified here.)
Covert, I., Lundberg, S., Lee, S.-I. (2020). "Understanding Global Feature Contributions With Additive Importance Measures" (SAGE). NeurIPS — Shapley for global (dataset-level) feature importance. (Venue confirmed; verify arXiv id 2004.00668 before citing.)
Molnar, C., Interpretable Machine Learning — chapters "Shapley Values" and "SHAP" — an accessible, worked treatment. https://christophm.github.io/interpretable-ml-book/shapley.html

← Previous: Measuring Fairness: Jain, Gini, Min, and Envy Next: Information Theory →