Diebold–Mariano: Is One Forecast Significantly Better?

ForecastingEvaluationIntermediate12 min

By: Anacodic Team

TL;DR — Model A has a lower error than Model B on your test set — but is that a real difference or just luck on this particular stretch of data? The Diebold–Mariano (DM) test answers it. Form the per-step loss differential d_t = loss_A(t) − loss_B(t), and test whether its mean is significantly different from zero. DM = mean(d) / sqrt(var(d)/n); if |DM| > 1.96, the accuracy gap is significant at 5%. It's the standard, model-agnostic way to say "A beats B" in forecasting. Two must-dos: use a variance that accounts for autocorrelation (forecast errors are serially correlated), and apply the Harvey–Leybourne–Newbold (HLN) small-sample correction for short series — skipping them makes the test over-confident.

1. Simple explanation

You compare two forecasting models and A has a lower MAPE than B. Ship A? Not yet — a single test set is one draw from a noisy world, and A might be ahead only because this period happened to suit it. If you'd tested on a different month, B might have won. You need to know whether A's advantage is systematic or within the noise.

The Diebold–Mariano test makes this rigorous without assuming anything about how the models work. The trick is to stop comparing the two error series and instead look at their difference at each time step: d_t = loss_A(t) − loss_B(t). If A is genuinely better, d_t is negative on average (A's loss is lower). So the whole question reduces to a familiar one: is the mean of the d_t series significantly below zero? That's a one-sample test on the loss differential.

The DM statistic is just that mean divided by its standard error: DM = mean(d) / standard_error(mean(d)). Compare it to the normal distribution — a |DM| above 1.96 means the gap is significant at the 5% level; near zero means "too close to call, the difference is noise." It works with any loss (squared error, absolute error, pinball loss for intervals) and any pair of models, which is why it's the default significance test in forecasting papers.

Analogy — two runners, many races. Runner A finished ahead in today's race. Is A actually faster, or was it a fluke? You look at the margin in each of many races. If A wins by a consistent margin, the average margin is clearly positive and it's real. If the margins bounce around zero, today's win was luck. DM is that "average margin, relative to how much it bounces around."

2. Diagram

  model A errors:  eA_1 eA_2 ... eA_n         model B errors:  eB_1 eB_2 ... eB_n
        │                                            │
        ▼   loss (e.g. squared)                      ▼
  loss_A_t                                     loss_B_t
        └──────────────┬──────────────────────────┘
                       ▼
         d_t = loss_A_t − loss_B_t        (the loss DIFFERENTIAL)

              mean(d)                 DM = ───────────────────   ~ N(0,1)
                                            sqrt( var(d) / n )

     mean(d) < 0  → A has lower loss (better)
     |DM| > 1.96  → SIGNIFICANT at 5%   ·   |DM| ≈ 0 → too close to call (noise)

     ⚠ var(d) must be autocorrelation-robust; small n → HLN correction

3. How it works

3.1 Reduce two models to one differential series

Instead of testing two error series against each other, compute the per-step loss differential d_t = loss_A(t) − loss_B(t). This collapses the comparison into a single series whose sign tells you who won each step and whose mean tells you who wins overall. The loss can be anything you care about — squared error, absolute error, or pinball/quantile loss if you're comparing interval forecasts — and the test is identical.

3.2 Test whether the mean differential is zero

Under the null hypothesis "the two models are equally accurate," mean(d) = 0. The DM statistic standardizes the sample mean by its standard error: DM = mean(d) / sqrt(var(d)/n). It's asymptotically standard normal, so you read it off like a z-score: |DM| > 1.96 rejects "equal accuracy" at 5%, > 2.58 at 1%. A negative mean(d) means A's loss is lower (A better); positive means B better.

3.3 Use an autocorrelation-robust variance

Here's the part people botch: forecast errors are serially correlated (an error this hour is related to the next), so the naive var(d)/n understates the true variance and the test becomes over-confident (too many false "significant" results). The proper DM variance uses the long-run variance of d — summing autocovariances up to the forecast horizon (a HAC / Newey–West-style estimator). For 1-step forecasts the correction is small; for multi-step it matters a lot.

3.4 Correct for small samples (HLN)

On short test sets the asymptotic normal approximation is too optimistic. The Harvey–Leybourne–Newbold (1997) correction rescales the statistic by a factor depending on n and the horizon, and compares it to a t-distribution instead of the normal. Always apply HLN for modest n — it's the difference between a defensible claim and one a reviewer will reject.

Boundary condition. DM tests whether two given forecasts differ in accuracy on this data-generating process; it is not valid for comparing nested models (where one is a special case of the other — use an encompassing/Clark–West test), and it says nothing about whether either model is any good in absolute terms. It also inherits your test set's representativeness — a significant result on a fluke period is still about a fluke period. Section 8 lists the misuses.

4. The math

4.1 The DM statistic

For a chosen loss L, define the differential and its mean:

  d_t   = L(e_A,t) − L(e_B,t)                 # e.g. L = squared error
  dbar  = (1/n) Σ_t d_t

  DM    = dbar / sqrt( V / n )                # V = variance of d (autocorr-robust!)
          V = γ_0 + 2 Σ_{k=1..h−1} γ_k        # long-run variance; γ_k = autocovariance
                                              # (h = forecast horizon)

Decision: |DM| > z_{1−α/2} (e.g. 1.96 at 5%) ⇒ reject equal accuracy. For small n, apply the HLN multiplier and compare to t_{n−1}.

4.2 Worked example

Eight forecast steps. Model A's errors are small; Model B's are ~2× larger. Using squared-error loss, the differential d_t = e_A²−e_B² is negative at every step (A always has lower squared error). The mean differential is dbar = −0.945 — A's loss is lower on average. The (simple) DM statistic works out to DM = −11.02, whose magnitude is far above 1.96, so A's advantage is significant at 5% (and 1%) — this isn't luck, A is genuinely more accurate on this process. (In a real report you'd use the autocorrelation-robust variance and HLN correction, which would shrink the magnitude but, given how lopsided this example is, keep it significant.)

5. Real code

import math
# Diebold-Mariano test: is model A significantly more accurate than model B?
# Work on the loss DIFFERENTIAL d_t = loss_A(t) - loss_B(t) (squared error here).
err_A = [0.5,-0.4, 0.3,-0.2, 0.6,-0.1, 0.2,-0.3]     # A's errors (smaller)
err_B = [1.2,-1.0, 0.9,-1.1, 1.3,-0.8, 1.0,-0.9]     # B's errors (larger)
d = [a*a - b*b for a,b in zip(err_A, err_B)]         # squared-error differential
n = len(d); dbar = sum(d)/n
var = sum((x-dbar)**2 for x in d)/n
dm = dbar / math.sqrt(var/n)                          # DM statistic ~ N(0,1)
print(f"mean loss differential (A-B): {dbar:.3f}")
print(f"DM statistic: {dm:.3f}")
print("significant at 5%?", abs(dm) > 1.96, "| A more accurate?", dbar < 0)
assert dbar < 0 and abs(dm) > 1.96                    # A significantly better, not luck
print("OK: A's lower loss is statistically significant")

# Output:
#   mean loss differential (A-B): -0.945
#   DM statistic: -11.020
#   significant at 5%? True | A more accurate? True
#   OK: A's lower loss is statistically significant

This uses the simple variance for clarity. Production code replaces var with an autocorrelation-robust long-run variance and applies the HLN small-sample correction (comparing to a t-distribution) — essential for multi-step forecasts and short test sets.

6. Real-world example

A forecasting team reported their new model beat the incumbent because its test-set MAPE was lower, and asked to deploy. A reviewer asked the obvious question: is the difference significant? A DM test on the two models' error series settled it — for the flagship 1-hour horizon the differential was clearly negative and |DM| well above 2, so the improvement was real; but for the 24-hour-ahead horizon |DM| was under 1, meaning the "improvement" there was within the noise and shouldn't be claimed. Reporting the DM result per horizon turned a vague "our model is better" into a precise, defensible claim (better at short horizons, tied at long ones).

They also caught the classic trap: their first DM implementation used the naive variance and flagged everything as significant. Switching to an autocorrelation- robust variance with the HLN correction removed several spurious "wins" — the errors were serially correlated, and the uncorrected test had been badly over-confident. The recurring lesson: "lower average error" is a claim, not a conclusion — DM (done with the robust variance and small-sample correction) is what turns it into evidence.

7. Interview questions companies actually ask

Q1. Model A has lower test error than B — why not just deploy A? Because one test set is a noisy sample; A might be ahead only on this period. You need to test whether the accuracy difference is statistically significant, not just numerically present — that's what the Diebold–Mariano test does.

Q2. What does the DM test actually test? Whether the mean of the loss differential d_t = loss_A(t) − loss_B(t) is significantly different from zero. Negative mean ⇒ A has lower loss; |DM| > 1.96 ⇒ the difference is significant at 5%. It reduces "compare two models" to a one-sample test on one series.

Q3. Why must the variance be autocorrelation-robust? Forecast errors are serially correlated, so the naive var/n understates the true variance of the mean differential and the test becomes over-confident (too many false significants). You use a long-run/HAC variance summing autocovariances up to the horizon.

Q4. What is the HLN correction and when do you need it? Harvey–Leybourne–Newbold (1997) rescales the DM statistic for small samples and compares it to a t-distribution instead of the normal, because the asymptotic approximation is too optimistic on short test sets. Apply it whenever n is modest or the horizon is long.

Q5. When is the DM test invalid? For nested models (one is a restricted case of the other) — DM's assumptions break; use an encompassing / Clark–West test instead. It's also silent on absolute quality (both models could be terrible) and inherits any non-representativeness of your test set.

Q6. Can DM compare interval or probabilistic forecasts? Yes — swap the loss. Use pinball/quantile loss to compare quantile forecasts or CRPS-type losses for distributions; the differential-and-test machinery is identical. The loss encodes what "better" means.

8. When to use / tradeoffs

Reach for the DM test when:

You're comparing two forecasting models and need to claim one is significantly better.
The models are non-nested (neither is a special case of the other).
You'll use the right loss (point, interval, or probabilistic) and a robust variance.

Do NOT use it (as-is) when:

Situation	Why it breaks	Use instead
Nested models	DM assumptions fail	Clark–West / encompassing test
Naive variance on correlated errors	over-confident (false significants)	HAC/long-run variance
Short test set / long horizon	asymptotic normal too optimistic	HLN correction + t-distribution
Comparing many models at once	multiple-comparisons inflation	Model Confidence Set (MCS) / corrections
You want "is A any good?"	DM is relative only	absolute metrics + baselines

Honest limits. DM is a relative test — it says A beats B on this data, not that A is good, and a significant result on an unrepresentative test period is still about that period. Its validity hinges on the variance being computed correctly (autocorrelation-robust) and, for short series, the HLN correction — the uncorrected version is a common source of over-claimed "significant" improvements. It doesn't apply to nested models, and comparing many candidates with pairwise DM tests inflates false positives, so use a multiple-comparison procedure (like the Model Confidence Set) when you're screening a whole panel.

Diebold–Mariano tests whether one forecast is significantly more accurate than another — turning "lower average error" into evidence.
Reduce the comparison to the loss differential d_t = loss_A − loss_B and test whether its mean ≠ 0: DM = mean(d)/sqrt(var(d)/n), |DM|>1.96 ⇒ significant at 5%.
Works with any loss (squared, absolute, pinball for intervals) and any non-nested model pair.
Must-dos: an autocorrelation-robust variance and the HLN small-sample correction, or the test is over-confident.
Boundary: relative not absolute, invalid for nested models, and only as representative as your test set.

Forecast Evaluation & Backtesting — producing the honest out-of-sample errors DM consumes.
Forecasting Models: From SARIMA to Foundation Models — the models you'll be comparing.
Backtesting, Baselines & Sensitivity Analysis — comparing against a baseline, the natural companion to a significance test.

Resources

Diebold, F. X., Mariano, R. S. (1995). "Comparing Predictive Accuracy." Journal of Business & Economic Statistics, 13(3), 253–263 — the original test. (Venue confirmed; verify pages.)
Harvey, D., Leybourne, S., Newbold, P. (1997). "Testing the equality of prediction mean squared errors." International Journal of Forecasting, 13(2), 281–291 — the small-sample (HLN) correction. (Venue confirmed; verify pages.)
Clark, T. E., West, K. D. (2007). "Approximately normal tests for equal predictive accuracy in nested models." Journal of Econometrics — the nested-model alternative. (Venue confirmed; verify pages.)

← Previous: Forecast Evaluation & Backtesting