Forecast Evaluation & Backtesting

TL;DR — You evaluate a forecaster by replaying it forward through history — a walk-forward backtest (expanding or rolling window), never a random split. Score point forecasts with MAE (average size of error), RMSE (punishes big misses), and MAPE (percent error — but it explodes near zero). Always compare against a seasonal-naïve baseline: if you can't beat "same hour last week," you have nothing. Put confidence intervals on your metrics with a bootstrap, prove which feature or model actually helps with an ablation study, and run a leakage audit before you trust any number. A good evaluation is skeptical by construction: it assumes your win is fake until the backtest, the baseline, the CI, and the audit all agree.

---

1. Simple explanation

A forecast is only as trustworthy as the test that measured it. And the only honest test for a time series is to pretend you are back in the past, forecast the future you cannot see, then check what really happened — and repeat, sliding forward. That is backtesting.

Analogy — a weather forecaster's report card. You don't grade a weather forecaster by letting them see tomorrow's weather. You freeze them at "now," take their forecast, wait, and compare to reality. Then you do it again the next day, and the next, over a whole year. Their grade is the average error across hundreds of these honest, forward-looking trials. And you always compare them to a lazy rival who just says "tomorrow = today" (the naïve baseline). A forecaster who can't beat the lazy rival isn't earning their salary.

That is exactly walk-forward backtesting for electricity load:

• Freeze at an origin, forecast the next 24 hours, score the error.
• Move the origin forward a day, refit, forecast again.
• Average the errors → an honest estimate of production performance.
• Compare to seasonal-naïve ("same hour last week") the whole way.

The three questions this article answers: How do I split time correctly? Which error number do I report? How do I know the win is real and not luck?

---

2. Diagram

               WALK-FORWARD BACKTEST (expanding window)
   time ──────────────────────────────────────────────────────────►
   fold 1:  ████████████ train ████████████ │ ▓▓▓ test(H) ▓▓▓
   fold 2:  ████████████ train ██████████████████ │ ▓▓▓ test ▓▓▓
   fold 3:  ████████████ train ████████████████████████ │ ▓▓▓ test ▓▓▓
                                                 ▲
                                    train ALWAYS before test (+ optional gap)
   at each fold:  refit on train  →  forecast H steps  →  score error

              SCORING (per fold, then averaged)
   ┌──────────────────────────────────────────────────────────────┐
   │  actual:  y_{t0+1} … y_{t0+H}      forecast: ŷ_{t0+1} … ŷ_{t0+H}│
   │  MAE  = mean |ŷ − y|          (average error size, same units) │
   │  RMSE = sqrt(mean (ŷ − y)²)   (punishes big misses)            │
   │  MAPE = mean |ŷ − y| / |y|    (percent; blows up near y≈0)     │
   └──────────────────────────────────────────────────────────────┘
                              │  compare to
                              ▼
        SEASONAL-NAIVE baseline  ŷ = y_{t0+h−168}   (same hour last week)
                              │
                              ▼
   skill = 1 − MAE_model / MAE_naive     (>0 means you beat the baseline)

   TRUST GATE:  [ bootstrap CI on the metric ]  [ ablation: does feature X help? ]
                [ leakage audit: lag_1 == shift(1), scaler fit on train only ]

---

3. How it works

3.1 Walk-forward (rolling / expanding) cross-validation

Standard k-fold shuffles rows — fatal for time series, because it trains on the future to predict the past. Instead, walk forward: repeatedly train on a prefix of history and test on the immediately following block.

Add a gap (embargo) of at least the longest lag between train and test so a lag/rolling feature cannot straddle the boundary. Each fold: refit → forecast H → score. Average the per-fold scores. This mirrors production, where the origin advances and you re-forecast.

3.2 Point metrics — and their pitfalls

Pitfalls to say out loud in an interview:

• MAPE explodes when actual load is near zero (rare for grid load, common for individual feeders at night). Guard against y = 0, or use sMAPE or MASE instead.
• RMSE ≥ MAE always; a big gap between them means a few large errors dominate. Report both.
• Metrics are not comparable across series of different scale unless percentage-based — which is exactly why MAPE exists and why you still need a baseline.
• Average over the horizon can hide that error grows with h (recursive compounding). Report error per horizon step, not just one blended number.

3.3 Pick a baseline first: seasonal-naïve

Before any model, compute the seasonal-naïve forecast — copy the value one full season back. For hourly load the weekly season (lag_168) is usually the strongest naïve; the daily (lag_24) is a second reference.

seasonal-naïve:  ŷ_{t0+h} = y_{t0+h−168}     (same hour, last week)

Report skill relative to the baseline, not a lonely absolute:

skill = 1 − MAE_model / MAE_naive      (0 = tied, >0 = better, <0 = worse)

If skill ≤ 0, your model loses to copying last week — stop and rethink. This is the forecasting-specific case of the "always beat a fair baseline" rule from Backtesting, Baselines & Sensitivity Analysis.

3.4 Bootstrap confidence intervals on the metric

A single MAE number is a point estimate with noise. Put an interval on it by bootstrapping: resample the per-fold (or per-horizon) errors with replacement many times, recompute the metric each time, and take the 2.5th and 97.5th percentiles for a 95% CI. Because forecast errors are autocorrelated, resample blocks of consecutive errors, not single points (a block/stationary bootstrap), or resample whole folds. If two models' CIs overlap heavily, you cannot claim one is better.

3.5 Ablation studies

To prove a feature or model earns its place, remove it and re-measure. The change in error is its contribution.

ablation contribution of feature X = MAE(without X) − MAE(with X)

Positive means dropping X hurt (X helps). Do this for temperature, lag_168, the holiday flag, and for each model in an ensemble (drop it, re-score the combo). An ablation converts "I think temperature matters" into "removing temperature raised MAE by 0.18 MW (skill −6%)."

3.6 The leakage audit checklist

Run this before reporting any metric. A great-looking backtest with leakage is worse than useless — it is confidently wrong.

---

4. The math

Let the actuals over one horizon block be y_1 … y_H and forecasts ŷ_1 … ŷ_H.

error_h = ŷ_h − y_h

MAE  = (1/H) Σ_h |error_h|
RMSE = sqrt( (1/H) Σ_h error_h² )
MAPE = (100/H) Σ_h |error_h| / |y_h|          (requires y_h ≠ 0)

Skill vs a baseline (using MAE):

skill = 1 − MAE_model / MAE_baseline

Worked numeric example — one 4-hour horizon block.

Actual load and two forecasts (model vs seasonal-naïve), in GW:

   h        actual y   model ŷ    naive ŷ
   1          4.0        3.8        4.2
   2          4.4        4.6        4.0
   3          4.1        4.0        4.5
   4          3.8        3.9        3.5

Model errors and metrics:

errors:  3.8−4.0=−0.2   4.6−4.4=0.2   4.0−4.1=−0.1   3.9−3.8=0.1
|err|:    0.2            0.2            0.1            0.1
MAE_model  = (0.2+0.2+0.1+0.1)/4 = 0.6/4 = 0.150 GW

err²:     0.04           0.04           0.01           0.01
RMSE_model = sqrt((0.04+0.04+0.01+0.01)/4) = sqrt(0.10/4) = sqrt(0.025) = 0.158 GW

|err|/|y|: 0.2/4.0=0.050  0.2/4.4=0.0455  0.1/4.1=0.0244  0.1/3.8=0.0263
MAPE_model = 100 × (0.050+0.0455+0.0244+0.0263)/4 = 100 × 0.1462/4 = 3.65 %

Seasonal-naïve errors and MAE:

|err|:  |4.2−4.0|=0.2   |4.0−4.4|=0.4   |4.5−4.1|=0.4   |3.5−3.8|=0.3
MAE_naive = (0.2+0.4+0.4+0.3)/4 = 1.3/4 = 0.325 GW

Skill:

skill = 1 − MAE_model / MAE_naive = 1 − 0.150 / 0.325 = 1 − 0.462 = 0.538

The model cuts error by ~54% vs copying last week — a real, reportable win. Note RMSE (0.158) > MAE (0.150): the gap is small, so no single giant error dominates. If one horizon had been off by 1.0 GW, RMSE would have jumped far above MAE, flagging a big-miss problem MAE alone would hide.

---

5. Real code

A dependency-light walk-forward backtest. It slides an expanding origin over the series, forecasts H steps with both a model and a seasonal-naïve baseline, computes MAE/RMSE/MAPE per fold, reports skill, and bootstraps a 95% CI on the MAE gap.

"""Walk-forward backtest for hourly load: MAE / RMSE / MAPE vs a seasonal-naive baseline,
with a bootstrap CI on the skill. No leakage: train is always strictly before test."""
import numpy as np
import pandas as pd

# ---------- data ----------
rng = np.random.default_rng(0)
n = 24 * 120
idx = pd.date_range("2024-01-01", periods=n, freq="h")
load = (5.0
        + 2.0 * np.sin((idx.hour - 6) / 24 * 2 * np.pi)
        + np.where(idx.dayofweek >= 5, -0.8, 0.0)
        + np.linspace(0, 0.5, n)
        + rng.normal(0, 0.15, n))
series = pd.Series(load, index=idx, name="load")

# ---------- metrics ----------
def mae(y, p):  return float(np.mean(np.abs(p - y)))
def rmse(y, p): return float(np.sqrt(np.mean((p - y) ** 2)))
def mape(y, p, eps=1e-6):
    mask = np.abs(y) > eps                       # guard against divide-by-zero
    return float(np.mean(np.abs((p[mask] - y[mask]) / y[mask])) * 100)

# ---------- forecasters (same fit/predict shape) ----------
def seasonal_naive_forecast(train: pd.Series, H: int, season=168) -> np.ndarray:
    """Copy the value one weekly season back (same hour last week)."""
    return train.iloc[-season:-season + H].values if season > H else \
           np.resize(train.iloc[-season:].values, H)

def ridge_forecast(train: pd.Series, H: int) -> np.ndarray:
    """A simple lag/calendar Ridge, forecast recursively. Transforms fit on TRAIN only."""
    from sklearn.linear_model import Ridge
    from sklearn.preprocessing import StandardScaler
    def feats(s):
        d = pd.DataFrame({"load": s})
        d["lag_1"], d["lag_24"], d["lag_168"] = s.shift(1), s.shift(24), s.shift(168)
        d["hour"], d["dow"] = d.index.hour, d.index.dayofweek
        return d
    d = feats(train).dropna()
    cols = ["lag_1", "lag_24", "lag_168", "hour", "dow"]
    scaler = StandardScaler().fit(d[cols])       # FIT ON TRAIN ONLY (no leakage)
    model = Ridge(alpha=1.0).fit(scaler.transform(d[cols]), d["load"])
    hist = train.copy(); out = []
    for _ in range(H):
        f = feats(hist).iloc[[-1]][cols]
        yhat = float(model.predict(scaler.transform(f))[0])
        out.append(yhat)
        hist.loc[hist.index[-1] + pd.Timedelta(hours=1)] = yhat   # recursive
    return np.array(out)

# ---------- walk-forward backtest ----------
def backtest(series, H=24, season=168, n_folds=20, step=24, gap=0):
    """Expanding-window walk-forward. Returns per-fold errors for model and naive baseline."""
    rows = []
    start = len(series) - n_folds * step - H
    for k in range(n_folds):
        cut = start + k * step                    # forecast origin index
        train = series.iloc[:cut]                 # strictly before test
        test  = series.iloc[cut + gap: cut + gap + H].values
        if len(test) < H or len(train) < season + 24:
            continue
        p_model = ridge_forecast(train, H)
        p_naive = seasonal_naive_forecast(train, H, season)
        rows.append({
            "fold": k,
            "mae_model": mae(test, p_model), "rmse_model": rmse(test, p_model),
            "mape_model": mape(test, p_model),
            "mae_naive": mae(test, p_naive),
        })
    return pd.DataFrame(rows)

# ---------- bootstrap CI on the skill ----------
def bootstrap_skill_ci(df, B=2000, seed=0):
    """Resample WHOLE FOLDS with replacement; recompute skill each time -> 95% CI."""
    rng = np.random.default_rng(seed)
    skills = []
    m, nv = df["mae_model"].values, df["mae_naive"].values
    for _ in range(B):
        i = rng.integers(0, len(df), len(df))     # resample folds
        skills.append(1 - m[i].mean() / nv[i].mean())
    lo, hi = np.percentile(skills, [2.5, 97.5])
    return float(np.mean(skills)), float(lo), float(hi)

# ---------- leakage audit ----------
def leakage_audit(series):
    lag1 = series.shift(1)
    ok = np.allclose(lag1.values[1:], series.values[:-1])
    assert ok, "LEAK: lag_1 != load.shift(1)"
    return "leakage audit passed: lag_1 == load.shift(1)"

if __name__ == "__main__":
    print(leakage_audit(series))
    df = backtest(series, H=24, season=168, n_folds=20)
    print(f"MAE  model={df.mae_model.mean():.3f}  naive={df.mae_naive.mean():.3f} GW")
    print(f"RMSE model={df.rmse_model.mean():.3f} GW   MAPE model={df.mape_model.mean():.2f} %")
    skill = 1 - df.mae_model.mean() / df.mae_naive.mean()
    mean_s, lo, hi = bootstrap_skill_ci(df)
    print(f"skill vs seasonal-naive = {skill:.3f}  (95% CI {lo:.3f} .. {hi:.3f})")
    # If the CI includes 0, you cannot claim you beat the baseline.

Typical output (numbers vary with seed/library versions):

leakage audit passed: lag_1 == load.shift(1)
MAE  model=0.1xx  naive=0.3xx GW
RMSE model=0.1xx GW   MAPE model=3.xx %
skill vs seasonal-naive = 0.5xx  (95% CI 0.4xx .. 0.6xx)

Because the CI lies well above 0, the win survives — a defensible result, not a lucky fold.

---

6. Real-world example

Grading a day-ahead load forecaster before it goes live.

• Setup. Two years of hourly load. You want to know if the new LightGBM+weather model is genuinely better than the incumbent seasonal-naïve, and by how much, before trusting it to schedule generation.
• Backtest design. Expanding-window walk-forward, H = 24, origin steps forward one day, over the last 180 folds (~6 months of day-ahead forecasts). A 24-hour gap guards the boundary so no lag feature straddles it.
• Metrics. Report MAE (MW), RMSE (MW), and MAPE (%) — per horizon hour and averaged. You find MAE ≈ 210 MW vs naïve 480 MW → skill ≈ 0.56; a 2000-resample fold bootstrap gives a 95% CI of roughly [0.49, 0.62], comfortably above 0.
• The horizon curve. Plotting MAE by hour-ahead shows error rising from ~120 MW at h=1 to ~300 MW at h=24 — recursive compounding. The blended average hid this; the per-hour view exposes it and tells you the evening peak (h≈19) is where the model most needs help.
• Ablation. Drop temperature: MAE rises to 260 MW (skill falls to 0.46) → temperature is worth ~50 MW. Drop lag_168: MAE rises to 240 MW → the weekly lag matters too. Drop the holiday flag: barely moves on normal weeks but MAE on the 6 holidays in the window jumps 3×. So the holiday flag earns its place specifically on holidays — a per-segment ablation you'd miss in the blended average.
• Leakage caught. An earlier version fit the temperature scaler on all two years. The backtest MAE looked ~15% better. The audit's "scaler fit on train only" check flagged it; refitting per fold restored the honest (worse but real) number. Shipping the leaky version would have under-forecast the summer peak in production.
• Decision. The honest, leakage-free, CI-backed skill of ~0.56 justifies a live shadow deployment, where the model runs alongside production without controlling anything, confirming the backtest before it takes over.

---

7. Interview questions companies actually ask

Q [Amazon Forecast / Uber] "Walk me through how you'd backtest a forecasting model."
  A Walk-forward: freeze at an origin, train on all history before it, forecast H steps, score
    the error against actuals, then slide the origin forward and repeat over many folds — with a
    gap so lag features can't straddle the boundary. Average per-fold errors, compare to a
    seasonal-naive baseline via skill, and put a bootstrap CI on it. Never shuffle or k-fold;
    train must always precede test in time.

Q [Google / Meta] "MAE vs RMSE vs MAPE — when do you use which?"
  A MAE = average error in the series' units, robust and easy to read. RMSE = same units but
    squares errors, so it punishes big misses — use it when large errors are especially costly
    (peak under-forecast). MAPE = percent error, scale-free and good for comparing series, but it
    blows up when actuals are near zero and is asymmetric. Report MAE and RMSE together; a big
    gap means a few large errors dominate.

Q [Netflix / any DS] "Why must you compare against a baseline, and which one for load?"
  A An absolute error is meaningless alone. For strongly seasonal load the right baseline is
    seasonal-naive — copy the value one season back (same hour last week, lag_168). Report skill
    = 1 − MAE_model/MAE_naive. If skill ≤ 0 you lose to copying last week and have no model. The
    baseline turns a number into a defensible improvement claim.

Q [Two Sigma / quant] "Your backtest shows a win. How do you know it isn't luck?"
  A Put a confidence interval on the metric: bootstrap by resampling whole folds (or blocks of
    consecutive errors, since errors are autocorrelated) and take the 2.5/97.5 percentiles of the
    skill. If the CI includes 0, you can't claim a win. Also check it holds across sub-windows and
    per horizon, and correct for multiple comparisons if you tried many model variants.

Q [Microsoft / energy] "What is MAPE's biggest failure mode?"
  A Divide-by-zero / blow-up when the actual is near zero — one tiny-load hour can dominate the
    whole average and make MAPE meaningless. It's also asymmetric, penalizing over-forecasts and
    under-forecasts unequally. Guard the denominator, or switch to sMAPE or MASE (which scales by
    a naive forecast's error and is defined even near zero).

Q [Stripe / Uber] "What is an ablation study in forecasting and why run one?"
  A Remove one feature or one model and re-measure the error; the increase is that component's
    contribution. It converts a belief ('temperature matters') into evidence ('dropping
    temperature raised MAE by 50 MW, skill −0.10'). Run it per feature and per ensemble member,
    and per segment (e.g. holidays) since a feature can matter only on rare days.

Q [Anany infra role] "Give me your leakage audit checklist for a time-series backtest."
  A (1) lag_1 exactly equals load.shift(1); (2) rolling features are shifted before rolling; (3)
    every transform/scaler is fit on train only; (4) max(train time) < min(test time) with a gap;
    (5) no feature is a function of the target at time t; (6) lags survive daylight-saving
    transitions. A great backtest with leakage is confidently wrong — audit before you report.

Q [Robinhood] "Expanding vs rolling window — how do you choose?"
  A Expanding keeps all history and suits roughly stationary data where more data helps. Rolling
    uses a fixed recent window and suits non-stationary data where old behavior misleads (a grid
    that just added lots of solar). If recent regimes differ from old ones, rolling; otherwise
    expanding. You can test both in the backtest and see which generalizes.

Q [DoorDash] "Your average MAE looks fine but production feels worse. What might you be missing?"
  A The blended average can hide that error grows with the horizon (recursive compounding) and
    that certain segments — holidays, heat waves, the evening peak — are much worse than average.
    Report error per horizon step and per segment, not one number. Production 'feels' the tail,
    not the mean.

---

8. When to use / tradeoffs

   ALWAYS, for any time-series model:
     ✓ walk-forward backtest (never shuffle / never random k-fold)
     ✓ a seasonal-naive baseline and a SKILL number, not a lonely absolute
     ✓ MAE + RMSE together (+ MAPE with a zero-guard), reported per horizon
     ✓ a bootstrap CI on the metric and a leakage audit before you trust it
   HONEST LIMITS of backtesting:
     ✗ it's an estimate, not a guarantee — regime change can still break you live
     ✗ too many model variants on one dataset => one 'wins' by luck (multiple testing)
     ✗ metrics can be gamed by a cherry-picked window; report the full period
     ✗ point metrics ignore uncertainty entirely — a good point score with terrible
       intervals is still a bad forecast (see interval metrics, PICP/MPIW, in 14-2)
   THE SKEPTIC'S REMINDER:
     assume the win is fake until the backtest, the baseline, the CI, and the audit agree.
     Backtest to FILTER and estimate; a live shadow/A-B run DECIDES.

Point metrics answer "how close is the middle?" but not "how sure are you?" The interval metrics that answer the second question — PICP (coverage) and MPIW (interval width) — get a first mention here and a full treatment in Probabilistic Forecasting & Prediction Intervals.

---

9. Summary + related articles

• Evaluate by walk-forward backtest (expanding or rolling), never shuffle or random k-fold; train always precedes test, with a gap.
• Point metrics: MAE (average error), RMSE (punishes big misses), MAPE (percent, but blows up near zero) — report per horizon, not one blended number.
• Always beat a seasonal-naïve baseline; report skill = 1 − MAE_model/MAE_naive.
• Put a bootstrap CI on the metric (resample whole folds or error blocks); if it includes 0, no win.
• Ablation studies prove which feature/model earns its place; run them per segment too.
• Run the leakage audit (lag identity, rolling causality, scaler scope, split order, target derivation, DST) before reporting anything.
• Point metrics ignore uncertainty — interval metrics PICP/MPIW come next in 14-2.

Related: Backtesting, Baselines & Sensitivity Analysis · Probabilistic Forecasting & Prediction Intervals · Time-Series Forecasting Fundamentals

Resources

• Hyndman & Athanasopoulos, Forecasting: Principles and Practice — evaluation & cross-validation — https://otexts.com/fpp3/accuracy.html
• Rob Hyndman, "Why every statistician should know about cross-validation" (time-series CV) — https://robjhyndman.com/hyndsight/tscv/
• scikit-learn TimeSeriesSplit — https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html
• Hyndman & Koehler, "Another look at measures of forecast accuracy" (MASE, MAPE pitfalls) — https://doi.org/10.1016/j.ijforecast.2006.03.001
• Politis & Romano, the stationary (block) bootstrap — https://doi.org/10.1080/01621459.1994.10476870
• M4 / M5 forecasting competitions (benchmarks, baselines) — https://mofc.unic.ac.cy/