Conformal Prediction: Coverage-Guaranteed Intervals

TL;DR — You built a "90% prediction interval" and it only covers 78% of real outcomes. Conformal prediction fixes that with a distribution-free guarantee: hold out a calibration set, measure how big the model's errors actually are (nonconformity scores sᵢ = |yᵢ − ŷᵢ|), take the right empirical quantile q, and output ŷ ± q. Under exchangeability, this guarantees marginal coverage ≥ 1 − α — no distribution assumption, wraps any base model (a mean model, a quantile model via CQR, or even an ensemble-spread score). The one catch for us: time series break exchangeability, so you split by time/block or go adaptive (ACI). This is the calibration step the Probabilistic Forecasting & Prediction Intervals article motivates.

---

1. Simple explanation

Your model promises a "90% interval." The honest question is: over many days, does the truth actually land inside it 90% of the time? Often the answer is no — the band is too narrow (overconfident) or too wide (useless). You need a way to make the promise true, not just claim it.

Conformal prediction is a shockingly simple recipe that does exactly that. You set aside a chunk of recent data the model never trained on — the calibration set. You look at how wrong the model was on that held-out data: the sizes of its mistakes. Then you build an interval big enough that, historically, mistakes that big-or-smaller happened 90% of the time. Add that margin to future predictions and — under a mild assumption — you get the coverage you promised, with a mathematical guarantee.

Analogy — the commuter's buffer. You need to catch a train. Google Maps says the drive is "22 minutes" (a point forecast). But you know some days it's 25, some days 34. So you look back at your last 20 commutes, find that 90% of them took 31 minutes or less, and leave 31 minutes early. You didn't model traffic physics. You just measured your past errors and padded by the 90th percentile of them. That padding is the conformal quantile q. Do it right and you catch the train 9 times out of 10 — guaranteed by your own history, not by a weather model.

The beauty: conformal doesn't care how you made the point forecast. Linear model, gradient boosting, a neural net, an ensemble — conformal wraps any of them, because it only ever looks at the errors, never the internals.

---

2. Diagram

   SPLIT CONFORMAL PREDICTION  (distribution-free coverage)

   ALL DATA (ordered in time for a time series)
   |=================== TRAIN ===================|==== CALIBRATION ====|== TEST ==|
                 fit base model f                 measure its errors    deploy

   STEP 1  fit f on TRAIN                         ->  f(x) = point forecast yhat
   STEP 2  on CALIBRATION (n points, unseen):
              score_i = | y_i - f(x_i) |          ->  how wrong f was, each point
   STEP 3  sort scores; take q = the k-th smallest,  k = ceil((n+1)(1-alpha))
   STEP 4  for any NEW x:   interval = f(x) +/- q

           scores (sorted):  s1  s2  s3 ......... s_k .......... s_n
                             |----------- 90% of mass -----------|
                                                 ^
                                                 q = this one     -> the padding

   GUARANTEE (if data is EXCHANGEABLE):
         P( y_new  in  [ yhat - q , yhat + q ] )  >=  1 - alpha

   WRAPS ANY BASE SCORE
   --------------------
      mean model      : s = |y - yhat|                 -> yhat +/- q   (symmetric)
      quantile model  : s = max(qlo - y, y - qhi)      -> [qlo-q, qhi+q]  (CQR)
      ensemble spread : s = |y - mean| / std           -> normalized band
                        (conformal can calibrate ANY raw uncertainty score)

   TIME-SERIES CATCH: rows are NOT exchangeable (autocorrelation, drift)
      fix -> split by TIME / BLOCK, or go ADAPTIVE (ACI) and update q as coverage drifts

---

3. How it works

3.1 The coverage problem

A model's claimed coverage and its achieved coverage are different things. Achieved coverage is PICP — the fraction of test actuals that fall inside the interval (see Probabilistic Forecasting & Prediction Intervals). When PICP < 1−α the band is overconfident; when PICP > 1−α it's wastefully wide. Hand-tuning the width to fix this is fragile and has no guarantee. Conformal replaces the guessing with a procedure that comes with a proof.

3.2 Split (inductive) conformal, step by step

sᵢ is called a nonconformity score: it's big when the point (xᵢ, yᵢ) "does not conform" to what the model expected. The (n+1) and the ceiling are not cosmetic — they are exactly what makes the finite-sample guarantee hold (they account for the new test point being one more exchangeable draw).

3.3 The exchangeability guarantee

If the calibration points and the future test point are exchangeable (their joint distribution is unchanged by reordering — a weaker condition than i.i.d.), then split conformal guarantees:

   1 - alpha  <=  P( y_new in interval )  <  1 - alpha + 1/(n+1)

The lower bound 1−α is the promise; the upper bound shows it's only mildly conservative (by 1/(n+1), which shrinks as calibration grows). Crucially this holds for any base model and any data distribution — hence distribution-free. You are trading a strong modeling assumption for a mild exchangeability assumption.

3.4 Conformal wraps any base score

Conformal never inspects the model; it only transforms a score. Swap the score and you get different, better-shaped intervals:

That last row is a general, textbook fact worth stating plainly: conformal calibration can be applied on top of any raw uncertainty score, including an ensemble's spread. The ensemble's std is a raw, usually-miscalibrated signal; feeding |y − mean| / std through the conformal quantile rescales it so the band actually hits 1−α. It is one ordinary use of conformal among many — nothing special about that particular score.

3.5 Conformalized Quantile Regression (CQR)

Plain split conformal produces a constant-width band (±q everywhere), which is wasteful when uncertainty varies by hour. CQR fixes this by conformalizing a quantile model instead of a mean model. You already have q̂_lo and q̂_hi (from pinball-loss quantile regression). Define the score:

   s_i = max( q̂_lo(x_i) − y_i ,  y_i − q̂_hi(x_i) )

     y_i BELOW the band  ->  q̂_lo − y_i  > 0   (missed low by that much)
     y_i ABOVE the band  ->  y_i − q̂_hi  > 0   (missed high by that much)
     y_i INSIDE the band ->  both terms < 0     (s_i is negative = slack)

Take q as the ⌈(n+1)(1−α)⌉-th smallest of these sᵢ, then output [q̂_lo − q, q̂_hi + q]. If the original quantiles were too narrow, q > 0 widens them to hit coverage; if they were too wide, q < 0 actually tightens them. CQR keeps the shape (adaptive width from the quantile model) and adds the guarantee (from conformal).

3.6 The time-series catch — and the fixes

Everything above assumes exchangeability. Time series usually violate it: today's load is correlated with yesterday's (autocorrelation), and the distribution drifts (seasons, economic growth, new EV load). Shuffling rows destroys nothing that matters for i.i.d. data, but here the order carries information, so naive random-split conformal can quietly lose coverage.

Standard fixes:

ACI in one line: track realized coverage as new actuals arrive and nudge the effective α_t each step (α_{t+1} = α_t + γ·(α − err_t)), so the interval self-corrects toward the target even as the series drifts. It trades the exact finite-sample guarantee for a long-run coverage guarantee that survives non-exchangeability — the right tool for a live forecasting service.

---

4. The math

4.1 The conformal quantile, worked (split conformal, mean model)

Calibration set of n = 10 points, target α = 0.10 (want ≥ 90% coverage). Absolute residuals sᵢ = |yᵢ − ŷᵢ| (in MW), already sorted:

   sorted scores:  42, 55, 63, 71, 88, 96, 110, 125, 140, 210
   index:           1   2   3   4   5   6    7    8    9   10

   k = ceil( (n+1)(1 - alpha) ) = ceil( 11 * 0.90 ) = ceil(9.9) = 10
   q = the k-th smallest score = the 10th = 210 MW

So the calibrated margin is q = 210 MW. For a new point with ŷ = 8400:

   interval = 8400 +/- 210 = [ 8190 , 8610 ] MW    (guaranteed >= 90% coverage)

Note the edge case: with n = 10 and 90%, k = 10 = n, so q is the max score — the tightest interval the sample can certify at 90%. Want a non-degenerate (non-max) index? Use more calibration data. With n = 99, k = ⌈100·0.9⌉ = 90, so q is the 90th of 99 scores — leaving 9 above it, a much tighter and smoother margin. More calibration data → tighter certified intervals.

4.2 Why the (n+1) and ceiling give the guarantee (intuition)

Think of the new test score s_new dropped among the n calibration scores. Under exchangeability, s_new is equally likely to land in any of the n+1 rank positions. The interval misses only if s_new exceeds q. By choosing q at rank ⌈(n+1)(1−α)⌉, at most α fraction of the n+1 ranks sit above it, so:

   P( miss ) = P( s_new > q )  <=  alpha    ->    P( cover ) >= 1 - alpha

No distribution appears anywhere — only the ranking of exchangeable scores. That is the entire magic.

4.3 CQR, worked

Quantile model gives, on three calibration points, [q̂_lo, q̂_hi] and the actual y:

   point   q_lo   q_hi   actual y   s = max(q_lo - y, y - q_hi)
     1     7900   8900     8400     max(7900-8400, 8400-8900) = max(-500,-500) = -500  (inside)
     2     8100   9100     9500     max(8100-9500, 9500-9100) = max(-1400, 400) =  400  (above)
     3     7600   8600     7300     max(7600-7300, 7300-8600) = max(  300,-1300)=  300  (below)

Negative s = the actual sat inside with room to spare; positive s = it poked out by that much. Collect all sᵢ, take q = the ⌈(n+1)(1−α)⌉-th smallest, and adjust:

   new interval = [ q_lo - q , q_hi + q ]

   if q > 0  -> original quantiles were too NARROW, widen them (restore coverage)
   if q < 0  -> original quantiles were too WIDE,   tighten them (save width)

CQR thus corrects both under- and over-coverage while keeping the adaptive (hour-varying) width that plain ±q cannot.

---

5. Real code

A reusable SplitConformal wrapper around any scikit-learn-style regressor, plus a CQR sketch, each reporting achieved PICP on a held-out test split.

"""Split conformal + CQR for hourly load forecasting.

Split conformal wraps ANY point regressor and guarantees marginal coverage >= 1-alpha
under exchangeability. CQR wraps a quantile model for adaptive-width intervals.
For time series we split by TIME (no shuffling) to respect ordering.
"""
import numpy as np

rng = np.random.default_rng(1)

# ---- data: same style as the probabilistic-forecasting article ---------------
N = 4000
hour = rng.integers(0, 24, N)
temp = rng.normal(20, 8, N)
weekend = rng.integers(0, 2, N)
base = (7000 + 900 * np.sin((hour - 8) / 24 * 2 * np.pi)
        + 25 * (temp - 20) ** 2 / 3 - 400 * weekend)
y = base + rng.normal(0, 200, N) + rng.exponential(120, N)
X = np.column_stack([hour, temp, weekend])

# TIME-ORDERED split: train | calibration | test  (NEVER shuffle a time series)
i_tr, i_cal = 2600, 3300
Xtr, ytr = X[:i_tr], y[:i_tr]
Xcal, ycal = X[i_tr:i_cal], y[i_tr:i_cal]
Xte, yte = X[i_cal:], y[i_cal:]


def conformal_q(scores, alpha):
    """The k-th smallest score, k = ceil((n+1)(1-alpha)); clip index into range."""
    n = len(scores)
    k = int(np.ceil((n + 1) * (1 - alpha)))
    k = min(k, n)                                  # if k>n the sample can't certify -> use max
    return np.sort(scores)[k - 1]


# ---- 1. Split conformal around a point regressor -----------------------------
from sklearn.ensemble import GradientBoostingRegressor

f = GradientBoostingRegressor(n_estimators=300, learning_rate=0.05, max_depth=3)
f.fit(Xtr, ytr)

cal_scores = np.abs(ycal - f.predict(Xcal))        # nonconformity = |y - yhat|
q = conformal_q(cal_scores, alpha=0.10)            # margin for 90% coverage

yhat_te = f.predict(Xte)
lo, hi = yhat_te - q, yhat_te + q
picp = np.mean((yte >= lo) & (yte <= hi))
mpiw = np.mean(hi - lo)
print("Split conformal  q=%.0f MW  PICP=%.3f  MPIW=%.0f MW" % (q, picp, mpiw))
# PICP lands ~0.90 by construction (guaranteed >= 0.90 under exchangeability).


# ---- 2. CQR: conformalize a quantile model for ADAPTIVE width -----------------
def make_q(tau):
    return GradientBoostingRegressor(loss="quantile", alpha=tau,
                                     n_estimators=300, learning_rate=0.05, max_depth=3)

qlo_m = make_q(0.05).fit(Xtr, ytr)
qhi_m = make_q(0.95).fit(Xtr, ytr)

# CQR nonconformity score on the calibration set
clo, chi = qlo_m.predict(Xcal), qhi_m.predict(Xcal)
cqr_scores = np.maximum(clo - ycal, ycal - chi)    # max(qlo - y, y - qhi)
q_cqr = conformal_q(cqr_scores, alpha=0.10)        # can be NEGATIVE (tightens)

tlo, thi = qlo_m.predict(Xte), qhi_m.predict(Xte)
lo2, hi2 = tlo - q_cqr, thi + q_cqr                # adjust both edges
picp2 = np.mean((yte >= lo2) & (yte <= hi2))
mpiw2 = np.mean(hi2 - lo2)
print("CQR              q=%+.0f MW  PICP=%.3f  MPIW=%.0f MW" % (q_cqr, picp2, mpiw2))
# CQR also hits >= ~0.90, but its width ADAPTS by hour instead of being constant.
# Whether MPIW ends up smaller than plain split conformal depends on how good the base
# quantile model is; the win is the adaptive shape (tight at calm hours, wide at peaks),
# not a guaranteed narrower average.

# NOTE: raw (uncalibrated) quantile bands from the sibling article typically
# UNDER-cover (~0.80-0.85). After conformal calibration, PICP >= 0.90. That gap
# closing is the whole point.

The load-bearing details: k = ceil((n+1)(1-alpha)) with the min(k, n) clip is the exact quantile rule; the split is time-ordered (no shuffle) to respect the series; and q_cqr printed with a sign shows it can go negative and tighten an over-wide quantile band.

---

6. Real-world example

Scenario: a live day-ahead load-forecasting service that must not lie about its intervals.

Your team ships a "90% interval" API that a grid operator uses to size reserves. Legal and operations both need the 90% to be real — under-coverage means blackouts, over-coverage means wasted money.

You compare three configurations on the last 700 hours (the held-out test window):

   Reading the table:
     - Raw quantiles claim 90% but deliver 81% -> under-reserved 9 evenings in 100 -> risk.
     - Split conformal restores the guarantee but pays with a constant, fat band.
     - CQR keeps the guarantee AND narrows the band at calm hours (3 a.m.), widening it
       only at volatile hours (6 p.m. peak) where uncertainty is genuinely high.

   Business impact: switching raw -> CQR removed the 9% coverage shortfall (the blackout-risk
   tail) while keeping MPIW ~13% tighter than blunt split conformal -> less reserve over-buy.

Then the drift check. Two months later a heatwave arrives; realized coverage on the newest week slips to 0.86. Because load is not exchangeable across seasons, the fixed q from spring under-covers summer. The fix is ACI: let the service update q online each hour from recent coverage, so the band re-widens automatically as the heatwave pushes errors up — restoring ~0.90 without a retrain. That is the time-series caveat turning into an operational safeguard.

---

7. Interview questions companies actually ask

Q1 [easy] (Amazon, utilities) "What problem does conformal prediction solve?"
  A It turns a CLAIMED coverage into a GUARANTEED one. Instead of trusting a model's '90%'
    band, you hold out a calibration set, measure the model's real errors, and pad predictions
    by the right empirical quantile so the interval covers >= 1-alpha of outcomes — distribution-
    free, for any base model.

Q2 [easy] (Google, Meta) "What is a nonconformity score?"
  A A number measuring how 'surprising' a point is under the model — classically the absolute
    residual s = |y - yhat|. Big score = the point doesn't conform to what the model expected.
    Conformal only ever looks at these scores, never the model internals, which is why it wraps
    anything.

Q3 [medium] (forecasting teams) "Walk me through split conformal for a 90% interval."
  A Split data into train and calibration. Fit f on train. On calibration compute s_i =
    |y_i - f(x_i)|. Set q = the k-th smallest score with k = ceil((n+1)(0.9)). For any new x
    output f(x) +/- q. Under exchangeability, coverage >= 90%.

Q4 [medium] (Amazon, Google) "Why the (n+1) and the ceiling? Why not just the 90th percentile?"
  A The new test point is itself an exchangeable draw, so you reason about its rank among n+1
    scores (n calibration + 1 new). ceil((n+1)(1-alpha)) picks the rank that caps the miss
    probability at alpha. The plain 90th percentile of n points slightly UNDER-covers; the
    (n+1)/ceiling correction gives the exact finite-sample guarantee.

Q5 [medium] (grid ops, energy) "Split conformal gives a constant-width band. When is that bad,
   and what's the fix?"
  A Constant +/-q wastes width when uncertainty varies by hour (calm night vs volatile peak). Fix
    is CQR: conformalize a QUANTILE model using s = max(qlo - y, y - qhi). You keep the adaptive,
    hour-varying width of the quantile model and add the coverage guarantee, usually at smaller
    MPIW.

Q6 [medium] (ML platforms) "Can conformal calibrate an ensemble's uncertainty?"
  A Yes — conformal wraps ANY raw score. Feed it s = |y - mean(x)| / std(x), where std is the
    ensemble's disagreement, and it rescales that heuristic spread so the band actually hits
    1-alpha. Raw ensemble spread is usually overconfident; conformal fixes the scale. It's just
    one ordinary choice of nonconformity score.

Q7 [hard] (Amazon, Google, utilities) "What assumption does the guarantee need, and why is it a
   problem for time series?"
  A EXCHANGEABILITY — the joint distribution is invariant to reordering (weaker than i.i.d.).
    Time series break it: autocorrelation (today depends on yesterday) and drift (seasons,
    growth). Naive random-split conformal can then silently lose coverage.

Q8 [hard] (grid ops, live forecasting) "How do you keep coverage under drift in production?"
  A Respect ordering: time/block split, or block bootstrap to preserve autocorrelation. For
    ONGOING drift use Adaptive Conformal Inference (ACI): update the effective alpha each step,
    alpha_{t+1} = alpha_t + gamma*(alpha - err_t), so q widens when recent coverage drops and
    tightens when it's too high. You trade the exact finite-sample guarantee for a long-run one
    that survives non-exchangeability.

Q9 [hard] (Meta, Amazon) "Does conformal make your point forecast better?"
  A No. Conformal never changes yhat — it only calibrates the INTERVAL around it. A bad base
    model gives correct coverage but WIDE intervals (MPIW blows up). Coverage is guaranteed;
    SHARPNESS still depends on model quality. So you still invest in a good base model.

Q10 [medium] (utilities) "How does calibration-set SIZE affect the interval?"
  A More calibration data -> a smoother, tighter certified quantile. With n=10 at 90% you're
    forced to the MAX score (k=10=n); with n=99 you use the 90th of 99, leaving 9 above it — a
    much tighter margin. Too little calibration data forces conservative (wide) intervals.

Q11 [hard] (research-leaning) "Split conformal vs full/transductive conformal — tradeoff?"
  A Full conformal refits the model for every candidate label — strongest guarantee, no data
    split, but O(retrain) per prediction: infeasible for big models. Split (inductive) conformal
    fits ONCE and reuses a held-out calibration set — cheap and deployable, at the cost of
    'spending' some data on calibration. In practice split conformal is what ships.

---

8. When to use / tradeoffs

   USE conformal prediction when:
     ✓ you must PROMISE a coverage level (SLA, safety, regulator, reserve sizing)
     ✓ your raw '90%' band doesn't actually cover 90% (measure PICP first!)
     ✓ you want a guarantee that holds for ANY base model / distribution
     ✓ you can spare a held-out calibration set the model didn't train on

   PICK the variant:
     • split conformal (|y-yhat|)  -> simplest; constant-width band
     • CQR (max(qlo-y, y-qhi))      -> adaptive width; usually smaller MPIW at same coverage
     • normalized (/sigma or /std)  -> width scales with local/ensemble uncertainty
     • ACI (online alpha update)    -> production time series under drift

   HONEST LIMITS:
     ✗ guarantee needs EXCHANGEABILITY — time series break it (autocorrelation, drift)
     ✗ coverage is MARGINAL (over the whole test set), not guaranteed for every subgroup/hour
     ✗ conformal fixes COVERAGE, not SHARPNESS — a weak base model -> correct-but-WIDE bands
     ✗ you 'spend' data on calibration; too little -> conservative (wide) intervals
     ✗ split conformal's constant width is wasteful where uncertainty varies -> use CQR
     ✗ heavy distribution shift beyond mild drift can defeat even ACI

   RULE OF THUMB: build the sharpest base band you can (quantile regression), then CQR-calibrate
   it on a TIME-ordered holdout; deploy with ACI if the series drifts.

---

9. Summary + related articles

• Conformal prediction turns a claimed interval into a guaranteed one, distribution-free.
• Split conformal: fit on train, score sᵢ = |yᵢ − ŷᵢ| on a calibration set, set q = the ⌈(n+1)(1−α)⌉-th smallest score, output ŷ ± q.
• Under exchangeability, coverage is 1−α ≤ P(cover) < 1−α + 1/(n+1) — mildly conservative, tightening with more calibration data.
• Conformal wraps any base score: mean (|y−ŷ|), quantile (CQR, max(q̂_lo−y, y−q̂_hi)), locally-scaled, or an ensemble-spread score — the last is one ordinary, textbook use, not a special construction.
• CQR keeps the adaptive width of a quantile model and adds the guarantee, usually at smaller MPIW than constant-width split conformal.
• Time series break exchangeability — fix with time/block splits, block bootstrap, or adaptive conformal (ACI) that updates q as coverage drifts.
• Conformal calibrates coverage, not sharpness — you still need a good base model for narrow bands.

Related: Probabilistic Forecasting & Prediction Intervals · Forecast Evaluation & Backtesting · Time-Series Forecasting & Uncertainty — Interview Questions

Resources

• Vovk, Gammerman & Shafer — Algorithmic Learning in a Random World (foundational text) — https://link.springer.com/book/10.1007/b106715
• Angelopoulos & Bates — A Gentle Introduction to Conformal Prediction — https://arxiv.org/abs/2107.07511
• Romano, Patterson & Candès — Conformalized Quantile Regression (CQR) — https://arxiv.org/abs/1905.03222
• Gibbs & Candès — Adaptive Conformal Inference under distribution shift (ACI) — https://arxiv.org/abs/2106.00170
• MAPIE (model-agnostic conformal library for Python) — https://mapie.readthedocs.io/
• Tibshirani et al. — Conformal prediction under covariate shift — https://arxiv.org/abs/1904.06019