TL;DR — Before you dare run a live A/B test, you replay your policy over historical data and compare it to a baseline (a simple, honest control). Report the % improvement over the baseline, not a lonely absolute number. Then stress-test the conclusion with a sensitivity sweep (vary one assumption; does the win survive?). Most fake wins die from four things: look-ahead bias / data leakage, survivorship bias, regime change, and a cherry-picked window. A win is only "real" when it beats a fair baseline, holds across sensitivity, and clears a noise/significance check.
1. Simple explanation
You have a new policy — a rule or model that makes decisions. A trading strategy. A recommender. A scheduler that picks when to run a job. You believe it is better than what you do today. But you cannot flip it on for real users yet: that is risky, slow, and sometimes impossible.
So you do the next best thing. You take history you already recorded, and you replay the policy over it: "if I had used this policy back then, what would have happened?" You do the same replay for a baseline (a dumb, safe default). Then you compare.
Analogy — practising in a driving simulator before the real road. The simulator replays real road recordings. You (the new policy) drive; a learner-driver baseline drives the same roads. If you only crash half as often as the baseline across many different roads — city, rain, night — you have real evidence. If you only tested one sunny straight road you liked, you learned nothing. Backtesting is that simulator. The baseline is the learner-driver you must beat. Sensitivity analysis is deliberately trying rain and night to see if your win holds.
Two traps the analogy makes obvious:
- If the simulator secretly lets you see the future of the recording, you will "drive" perfectly and learn nothing real. That is look-ahead bias / leakage.
- If you only kept recordings of roads where cars survived, your test is rigged. That is survivorship bias.
2. Diagram
HISTORY (recorded past: prices / clicks / grid signal)
│
┌──────────────────────┴──────────────────────┐
▼ ▼
┌──────────────────┐ ┌──────────────────────┐
│ BASELINE policy │ replay same data │ CANDIDATE policy │
│ (naive control) │ ───── in lockstep ──► │ (the new idea) │
└────────┬─────────┘ └──────────┬───────────┘
│ metric_base │ metric_policy
▼ ▼
┌─────────────────────────────────────────────────────────────────┐
│ COMPARE: improvement% = (metric_policy − metric_base)/base×100 │
│ (sign flips if LOWER is better — cost, latency) │
└───────────────────────────────┬─────────────────────────────────┘
▼
┌───────────────────────────────────────────────────┐
│ SENSITIVITY SWEEP: vary ONE assumption / param │
│ fee=5bps→10bps? window=2023→2020-24? EF=avg→marg? │
│ → does the SIGN of the win survive every setting? │
└───────────────────────────────┬───────────────────┘
▼
THREATS-TO-VALIDITY GATE (must pass ALL before you trust it)
[ leakage? ] [ survivorship? ] [ regime change? ] [ cherry-picked window? ]
▼
only now → propose a LIVE A/B test
3. How it works
3.1 The pieces of a backtest
| Piece | What it is | Example |
|---|---|---|
| History | Recorded, timestamped past data | Hourly prices; logged user clicks; hourly grid carbon |
| Policy | The decision rule under test | Momentum strategy; new ranker; greedy carbon scheduler |
| Baseline | A naive, fair control | Buy-and-hold; popularity ranker; FIFO scheduler |
| Metric | The number you optimise | Return; NDCG@10; total gCO2 |
| Replay loop | Step through history in time order, let each policy decide | see §5 |
| Improvement | Policy vs baseline, as a percentage | Savings%, uplift% |
The golden rule: the policy and the baseline see exactly the same history, in the same order, and are scored by the same metric. Any asymmetry (policy gets more data, sees later timestamps, uses a different metric) invalidates the comparison.
3.2 Why you need a baseline at all
An absolute number is meaningless without a reference. "My strategy made 8%." Good? The index made 20% that year — you lost 12% of opportunity. "My scheduler emitted 42 kg CO2." Compared to what?
A baseline turns a number into a claim about improvement. A good baseline is:
| Property | Why it matters |
|---|---|
| Naive / cheap | It is the "do nothing clever" option you must beat to justify complexity |
| What you actually do today | Otherwise you are beating a strawman |
| Reproducible & fixed | It cannot be tuned to lose on purpose |
| Same inputs as the policy | Fair race — no information asymmetry |
Common baselines by domain: buy-and-hold (trading), most-popular / current production model (recommenders), FIFO / earliest-start (scheduling), predict-the-mean or predict-last-value (forecasting). If you cannot beat "predict last value," you do not have a model — you have a random number generator with extra steps.
3.3 Report % improvement, not the absolute
Absolute numbers do not transfer across time, scale, or units. Percentages do. Always report:
improvement% = (metric_policy − metric_baseline) / metric_baseline × 100 (higher-is-better)
savings% = (metric_baseline − metric_policy) / metric_baseline × 100 (lower-is-better: cost)
Report the direction explicitly (is lower better?), the baseline value, and the window. "+36% savings vs FIFO over ISO-NE Jan–Mar 2024" is a claim. "3300 grams" is trivia.
3.4 Sensitivity analysis — does the win survive?
A backtest result rests on assumptions: a fee, a slippage estimate, a chosen date window, an emission factor, a forecast you trusted. Sensitivity analysis varies one assumption at a time and re-checks the conclusion.
You are not looking for the exact number to stay fixed — it will move. You are looking for the sign of the win to survive. If your strategy is profitable at 5 bps fees but loses at 10 bps, your "edge" was really a fee assumption. If the carbon savings vanish when you switch from average to marginal emission factors (see §3.5), your headline number depended on an accounting choice.
PARAMETER baseline setting stress setting win holds?
trading fee 5 bps 10 bps ✓ / ✗
backtest window 2023 2019–2024 ✓ / ✗
emission factor average CI marginal (gas) ✓ / ✗
forecast error perfect signal +10% noise ✓ / ✗
A robust result is one that is positive across the whole sweep. A fragile one flips sign somewhere — and you must report exactly where.
3.5 Measurement-basis honesty (the average-vs-marginal lesson)
This is the deepest habit in the whole article: your metric may rest on an assumption that biases the result. The carbon scheduler is the clean teaching case.
We compute grid carbon intensity as an average:
CI(t) = ( Σ_f gen_f(t)·EF_f ) / ( Σ_f gen_f(t) ) gCO2/kWh, over fuels f
But when your job runs, it does not consume "average" electricity. It causes one more generator to ramp up — the marginal generator, which on many grids is gas (≈490), not the average mix (which nuclear/wind/hydro pull down). So a decision that "moves load to a low-average-CI hour" may displace a different marginal plant than the average implies. The honest report:
| Basis | What it answers | Source | Caveat |
|---|---|---|---|
| Average CI | "What was the grid's mean intensity?" | EIA fuel mix (transparent, public) | Not what your marginal decision displaces |
| Marginal CI | "What does my extra load actually cause?" | Electricity Maps / WattTime | Modelled, less transparent, provider-specific |
| Combustion-only EF | Emissions at the smokestack | AR5 combustion factors | Ignores upstream (mining, transport, build) |
| Lifecycle EF | Cradle-to-grave per kWh | AR5 lifecycle factors | Higher for all fuels; different ranking |
Report on the transparent average basis (reproducible from public data) and state the marginal caveat. The general lesson transfers to every domain: a trading backtest that ignores market impact assumes your trades do not move the price (an average-like assumption); a recommender replay that ignores your ranker changing user behaviour assumes clicks are independent of what you showed. Name the assumption your metric rests on.
4. The math
Metric per policy. Score both policies with the same function M over the same history H:
metric_baseline = M(baseline, H)
metric_policy = M(policy, H)
Improvement (lower-is-better example — cost/emissions):
savings% = (metric_baseline − metric_policy) / metric_baseline × 100
Worked numeric example — the carbon scheduler backtested vs FIFO.
One deferrable job: power P = 10 kW, duration D = 2 h, release = 0, deadline = 4.
Slack = deadline − release − D = 4 − 0 − 2 = 2, so feasible starts are s ∈ {0, 1, 2}.
Grid carbon signal (gCO2/kWh) for hours 0–4:
hour t : 0 1 2 3 4
CI(t) : 300 220 150 180 260
Cost of starting at s (energy P·D = 20 kWh is identical for every s):
cost(s) = Σ_{h=0}^{D-1} P · CI(s+h)
cost(0) = 10·(CI0 + CI1) = 10·(300 + 220) = 5200 gCO2
cost(1) = 10·(CI1 + CI2) = 10·(220 + 150) = 3700 gCO2
cost(2) = 10·(CI2 + CI3) = 10·(150 + 180) = 3300 gCO2 ← minimum
- FIFO baseline = earliest feasible start =
s = 0→ 5200 gCO2. - Greedy policy = feasible start minimising cost =
s = 2→ 3300 gCO2.
savings% = (5200 − 3300) / 5200 × 100 = 1900 / 5200 × 100 ≈ 36.5%
Where the CI numbers come from (the average formula, one hour). Say hour 2's fuel mix (MW) is WIND 200, GAS 300, NUCLEAR 500 with AR5 combustion factors WIND 11, GAS 490, NUCLEAR 12:
CI(2) = (200·11 + 300·490 + 500·12) / (200 + 300 + 500)
= (2200 + 147000 + 6000) / 1000
= 155200 / 1000 ≈ 155 gCO2/kWh (we rounded to 150 in the signal above)
Sensitivity check on the same example. Suppose your CI forecast was wrong and hour 2 was actually 210, not 150. Recompute: cost(2) = 10·(210+180) = 3900, now worse than cost(1)=3700. Greedy (using the bad forecast) still picks s=2 and pays 3900 → realised savings drop to (5200−3900)/5200 ≈ 25%. The win survived (still positive) but shrank — exactly what a sensitivity sweep is meant to expose.
5. Real code
Runnable, dependency-free. The backtest loop is domain-agnostic: pass any two policies and a metric.
"""Generic backtesting: replay a policy and a baseline over history, report % improvement,
then sweep one parameter to test robustness. Lower-is-better and higher-is-better both handled."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Callable, Iterable, Any
# ---------- 1. The generic backtest loop ----------
def backtest(policy: Callable, baseline: Callable, history: Any,
metric: Callable, lower_is_better: bool = True) -> dict:
"""Score `policy` and `baseline` on the SAME history with the SAME metric.
Returns both metrics and the % improvement of policy over baseline."""
metric_baseline = metric(baseline, history) # control, same inputs
metric_policy = metric(policy, history) # candidate, same inputs
if lower_is_better: # cost / emissions / latency
improvement = (metric_baseline - metric_policy) / metric_baseline * 100
else: # return / NDCG / CTR
improvement = (metric_policy - metric_baseline) / metric_baseline * 100
return {"baseline": round(metric_baseline, 4),
"policy": round(metric_policy, 4),
"improvement_pct": round(improvement, 2)}
# ---------- 2. Domain: carbon-aware scheduler vs FIFO ----------
@dataclass(frozen=True)
class Job:
P: float # power, kW
D: int # duration, hours
release: int # earliest start hour
deadline: int # must finish by this hour (exclusive end)
def feasible_starts(job: Job):
# start s is feasible if release <= s and s + D <= deadline
return range(job.release, job.deadline - job.D + 1)
def cost(job: Job, s: int, CI: list[float]) -> float:
# same energy P*D for every s; only the CI hours differ
return sum(job.P * CI[s + h] for h in range(job.D))
def greedy_start(job: Job, CI: list[float]) -> int:
return min(feasible_starts(job), key=lambda s: cost(job, s, CI)) # cheapest feasible
def fifo_start(job: Job, CI: list[float]) -> int:
return next(iter(feasible_starts(job))) # earliest feasible
def total_emissions(start_fn: Callable, history: dict) -> float:
"""metric(policy, history): sum cost over all jobs, given a start-choosing policy."""
CI, jobs = history["CI"], history["jobs"]
return sum(cost(j, start_fn(j, CI), CI) for j in jobs)
# ---------- 3. Sensitivity sweep: vary ONE parameter, re-run the backtest ----------
def sensitivity_sweep(policy, baseline, base_history, metric, param_name: str,
values: Iterable, mutate: Callable, lower_is_better=True) -> list[dict]:
"""For each value, build a stressed history via `mutate(base_history, value)`,
re-run backtest, and record whether the win survived (improvement stays positive)."""
rows = []
for v in values:
stressed = mutate(base_history, v)
r = backtest(policy, baseline, stressed, metric, lower_is_better)
r[param_name] = v
r["win_holds"] = r["improvement_pct"] > 0
rows.append(r)
return rows
if __name__ == "__main__":
history = {
"CI": [300, 220, 150, 180, 260], # gCO2/kWh, hours 0..4
"jobs": [Job(P=10, D=2, release=0, deadline=4)], # the worked example
}
# Backtest greedy (policy) vs FIFO (baseline). Lower emissions is better.
result = backtest(greedy_start, fifo_start, history, total_emissions, lower_is_better=True)
print(result) # {'baseline': 5200.0, 'policy': 3300.0, 'improvement_pct': 36.54}
# Sensitivity: what if hour 2's CI was mis-forecast? Sweep CI[2] and watch the win.
def bump_hour2(h, ci2):
new = dict(h); new["CI"] = h["CI"].copy(); new["CI"][2] = ci2; return new
sweep = sensitivity_sweep(greedy_start, fifo_start, history, total_emissions,
"CI_hour2", [150, 180, 210, 250, 300], bump_hour2)
for row in sweep:
print(row["CI_hour2"], "->", row["improvement_pct"], "% holds:", row["win_holds"])
# As CI[2] rises, greedy's chosen hour gets worse and savings shrink — the sweep exposes fragility.
Expected output:
{'baseline': 5200.0, 'policy': 3300.0, 'improvement_pct': 36.54}
150 -> 36.54 % holds: True
180 -> 30.77 % holds: True
210 -> 25.0 % holds: True
250 -> 17.31 % holds: True
300 -> 7.69 % holds: True # still positive, but thin — the edge depends on the signal being right
6. Real-world example — three domains, one method
The exact same backtest(policy, baseline, history) shape covers wildly different systems. That is the point.
(a) Trading strategy backtest.
- History: daily prices for the S&P 500, 2019–2024.
- Policy: a momentum rule (buy when 50-day > 200-day moving average).
- Baseline: buy-and-hold the index.
- Metric: total return (and Sharpe). Higher is better.
- Result shape: "momentum returned 61% vs buy-and-hold 78% → −22% relative — it lost." The baseline saved you from a bad idea. Sensitivity: add 10 bps fees per trade; momentum trades often, so its return drops further. Threats: look-ahead bias if you used the 200-day average that includes today's close to decide today's trade; survivorship if your universe only includes companies that still exist (the bankruptcies are missing).
(b) Recommender offline replay.
- History: 3 months of logged impressions and clicks.
- Policy: a new ranking model.
- Baseline: the current production ranker (and a most-popular fallback).
- Metric: NDCG@10 / recall@10 on held-out clicks; or an IPS-corrected estimate.
- Result shape: "new ranker NDCG@10 = 0.42 vs prod 0.38 → +10.5%." Sensitivity: re-run per week — does +10% hold in every week or only the week of a big promo? Threats: leakage if a feature encodes the label (a "was_clicked" feature sneaks in); replay only scores items that were shown, so you cannot fairly credit the new ranker for surfacing something the old one never logged — a known bias of offline replay.
(c) Carbon-aware scheduler vs FIFO (the §4 worked example, scaled up).
- History: EIA hourly fuel mix for ISO New England (ISO-NE), converted to CI(t) via the average formula; a batch of deferrable jobs.
- Policy: greedy (cheapest feasible start). Baseline: FIFO (earliest feasible start).
- Metric: total gCO2. Lower is better.
- Result shape: "greedy emitted 33 kg vs FIFO 52 kg over the quarter → ≈36% savings." Sensitivity: switch average → marginal EF; widen the window; add forecast noise (see §3.5). Threats: regime change if the grid added a lot of solar mid-window (the pattern greedy exploited changed); cherry-picked window if you only reported the sunniest, windiest month. Generalises to PJM, MISO, CAISO, NYISO, ERCOT, SPP.
7. Interview questions companies actually ask
Q [Two Sigma / Jane Street] "You can't run a live A/B test yet. How do you evaluate a strategy?"
A Backtest: replay the policy over recorded history and compare to a fair baseline
(buy-and-hold), scored by the same metric over the same window. Report % improvement,
not an absolute return. Then run a sensitivity sweep (fees, slippage, window) and pass a
threats-to-validity gate (look-ahead, survivorship, regime change, cherry-picking) before
trusting it. A backtest is necessary evidence, never proof — a live test still decides.
Q [Netflix / Spotify] "What makes a GOOD baseline?"
A Naive, cheap, reproducible, fixed, and fed the SAME inputs as the policy — ideally it is
what you actually run today (current production ranker) plus a trivial floor (most-popular
/ predict-last-value). If you can't beat the trivial floor you have no model. A good
baseline turns an absolute number into a defensible improvement claim.
Q [Google / Meta] "What is look-ahead bias / data leakage, and how do you prevent it?"
A Using information at decision time that wouldn't have been available then — e.g. a moving
average that includes today's close, a feature derived from the label, or normalising with
stats computed over the whole dataset (including the future). Prevent it with strict
time-ordered replay, point-in-time-correct feature joins, fitting all transforms on the
train split only, and an embargo/gap between train and test.
Q [Citadel / HRT] "Your backtest shows a great win. How do you know it's REAL and not noise?"
A Effect size vs variability: paired per-period improvements, a t-test or sign test on them,
and a block/stationary bootstrap for confidence intervals (returns are autocorrelated, so
naive iid bootstrap lies). Check it holds out-of-sample and across sub-windows, and correct
for multiple hypotheses if you tried many variants (deflated Sharpe / Bonferroni). One
lucky window with wide error bars is not a win.
Q [Stripe / any DS role] "Absolute number vs % improvement — why insist on the percentage?"
A Absolute numbers don't transfer across scale, time, or units and hide the counterfactual.
"8% return" is bad if the index made 20%. Always report improvement% = (policy − base)/base,
the direction (is lower better?), the baseline value, and the window.
Q [Electricity Maps / Google carbon team] "Average vs marginal — what assumption underlies
your carbon metric, and does it bias the result?"
A Average CI = mixed grid intensity from public fuel-mix data (transparent, reproducible).
But my *extra* load displaces the MARGINAL generator (often gas ≈490), not the average mix
that nuclear/wind pull down. So average-basis savings can overstate the real avoided
emissions. I report the transparent average basis AND the marginal caveat (Electricity Maps
/ WattTime), and note combustion-only vs lifecycle factors. The general lesson: name the
assumption your metric rests on and sweep it.
Q [Amazon / Booking] "What is survivorship bias in a backtest?"
A Testing only on entities that survived to today — delisted stocks, churned users, shut-down
products are missing, so the sample is rosier than reality. Fix by using point-in-time
universes that include everything that existed then, including the ones that died.
Q [Airbnb] "Offline replay for a recommender looks great but the online lift is tiny. Why?"
A Offline replay can only score items that were actually shown/logged, and it assumes user
behaviour is independent of what you display (it isn't). Position bias, feedback loops, and
unseen-item coverage make offline metrics optimistic. Use IPS/counterfactual estimators,
but treat offline replay as a filter, not a verdict — confirm with an online A/B test.
Q [Robinhood] "What is regime change and how does it threaten a backtest?"
A The data-generating process shifts (rates regime, a grid adding solar, COVID). A policy
tuned to the old regime can look great in-sample and fail live. Defend with rolling/walk-
forward validation across multiple regimes, and report per-regime results instead of one
blended number.
Q [OpenAI / research roles] "How would you design a sensitivity analysis?"
A List every assumption the conclusion rests on (fee, slippage, window, emission-factor basis,
forecast accuracy). Vary ONE at a time across a plausible range, re-run the backtest, and
check the SIGN of the improvement survives. Report where it flips. A one-at-a-time sweep
localises fragility; a robust result is positive across the whole grid.
8. When to use / tradeoffs
USE backtesting when:
✓ a live A/B test is impossible, slow, risky, or costly to start
✓ you have clean, time-stamped historical data with no leakage
✓ you want a cheap filter to kill bad ideas before they touch users
BE HONEST that a backtest is NOT proof:
✗ overfitting the past — try enough variants and one "wins" by luck (multiple testing)
✗ offline ≠ online — replay can't model your policy changing user/market behaviour
✗ garbage history in → confident garbage out (survivorship, leakage, bad fills)
✗ a single window / single metric can be cherry-picked; the sign can flip out-of-sample
THE NO-FREE-LUNCH REMINDER:
a backtest buys you cheap, fast, low-risk evidence — but pays in realism.
Backtest to FILTER; A/B test to DECIDE. Never ship on a backtest alone.
Practical guardrails: split train/test with a time gap, use walk-forward validation, keep the baseline frozen, report confidence intervals via block bootstrap, and always pass the four-item threats gate before proposing the live test.
9. Summary + related articles
- Backtest = replay a policy over history and compare to a baseline with the same data and metric.
- A good baseline is naive, fixed, reproducible, and what you do today — it turns a number into a claim.
- Report % improvement (with direction, baseline value, and window), never a lonely absolute.
- Sensitivity analysis varies one assumption at a time; a robust win keeps its sign across the sweep.
- Measurement-basis honesty: your metric rests on an assumption (average-vs-marginal CI, no-market-impact, independent clicks) — name it and stress it.
- Four threats to validity: look-ahead / leakage, survivorship, regime change, cherry-picked window.
- Prove a win is real, not noise, with paired tests, block bootstrap, out-of-sample and per-regime checks.
- Backtest to filter; A/B test to decide. A backtest is evidence, not proof.
Related: Scheduling Under a Time-Varying Cost Signal · Carbon-Aware Scheduling of Flexible Loads
Resources
- Marcos López de Prado, Advances in Financial Machine Learning (backtest overfitting, deflated Sharpe) — https://www.wiley.com/en-us/Advances+in+Financial+Machine+Learning-p-9781119482086
- Bailey & López de Prado, "The Deflated Sharpe Ratio" — https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2460551
- Counterfactual evaluation / IPS for recommenders (Cornell CS4780 notes) — https://www.cs.cornell.edu/courses/cs7792/2016fa/
- Politis & Romano, the stationary (block) bootstrap — https://www.tandfonline.com/doi/abs/10.1080/01621459.1994.10476870
- EIA Open Data API (hourly fuel mix by balancing authority) — https://www.eia.gov/opendata/
- Electricity Maps (marginal vs average carbon signals) — https://www.electricitymaps.com/
- WattTime (marginal emissions API) — https://www.watttime.org/