Skip to content

Concept: Conformal Modes

What it is

Conformal prediction attaches honest uncertainty intervals to point forecasts by calibrating against past residuals:

residual  r  =  obs − sim
half-width   =  quantile_α( |r| )
interval     =  [sim − half-width,  sim + half-width]

The non-trivial design choice is which residuals to pool. The residual_mode field answers this. Four modes were introduced in PR #361 as part of the CalibrationResult refactor. Each produces one persistable CalibrationResult sidecar; predict_interval dispatches on which internal dict field is non-None.

Where it lives in the code

Symbol Location Notes
ResidualMode (type alias) models/meta_models/types.py:16 Four-string Literal
_MODE_DISPATCH models/meta_models/conformalise.py:571 Dict keying each mode to its recipe
apply_conformal (experiment overload) conformalise.py:584 Dispatches through _MODE_DISPATCH
CalibrationResult.predict_interval conformalise.py:360 Applies half-widths to sim_yield
fit_and_save_all_configured stages/run_meta_models.py:85 Fits + saves every mode in config.postprocess.conformalise
primary_calibration stages/run_meta_models.py:119 Returns calibration for config.forecast.residual_mode

Key invariants

  • Exactly one of per_init_md, per_year, pooled is non-None on any given CalibrationResult (invariant at conformalise.py:106).
  • fit_and_save_all_configured runs before the fold loop in postprocess_experiment (run_meta_models.py:168) so all calibrations are complete before build_rows is called.
  • postprocess.conformalise (list) controls which sidecars are written; forecast.residual_mode (single mandatory field since PR #372) selects which is applied at runtime. The axes are independent: a forecast can request a mode absent from postprocess.conformalise — get_or_fit_calibration (run_meta_models.py:100) fits it on demand.

Mode comparison table

Mode What is pooled When preferred Calibration-set size Bias risk
hindcast_oos_per_init_date OOS residuals grouped by "MM-DD" init label Default; captures season-of-year uncertainty gradient n_folds × n_inits ÷ n_labels per label Low — honest OOS
hindcast_oos_per_year Walk-forward bootstrap; fold k uses folds < k Diagnosing calibration over time; per-year CI evolution Grows from 0; earliest fold → NaN Low — honest OOS
hindcast_oos_fully_pooled All (year, init_date) OOS residuals Baseline; robust for small calibration sets n_folds × n_inits (full pool) Low — loses seasonal structure
in_sample_pooled Production fold train_preds residuals Fallback when no CV folds exist n_train_years × n_inits High — biased narrow; in-sample

in_sample_pooled emits logger.warning labelling intervals as biased narrow (conformalise.py:546). The validate_residual_mode gate (run_forecast.py:91) rejects OOS modes when no CV folds exist, making it the only legal choice for fit_production-only run_dirs.

PR #361 worked-example numbers (95% half-width, kg/ha, corn)

hindcast_oos_per_init_date[01-01]: 309
hindcast_oos_per_init_date[04-08]: 305
hindcast_oos_per_init_date[07-08]: 422   ← wider mid-season
hindcast_oos_per_init_date[10-15]: 309
hindcast_oos_fully_pooled:         347
hindcast_oos_per_year[2020]:       NaN   ← no prior residuals
hindcast_oos_per_year[2021]:        83   ← only one prior year
hindcast_oos_per_year[2022]:       330
hindcast_oos_per_year[2023]:       317
hindcast_oos_per_year[2024]:       304

The seasonal gradient in per_init_date (309 → 422 → 309) is the key reason it is preferred: intervals widen correctly at mid-season when accumulated weather information is most uncertain.

predict_interval dispatch (conformalise.py:360)

  • per_init_md set → circular calendar interpolation on init_md; non-exact hits emit logger.warning (conformalise.py:279).
  • per_year set → exact-hit or per_year_fallback reduce on fold_year (conformalise.py:319); linear interpolation intentionally absent (adjacent fold-years have non-comparable residual sets).
  • pooled set → broadcasts single half-width; ignores both kwargs (conformalise.py:395).

Per-mode parquet sidecar layout

{run_dir}/conformal/{residual_mode}.parquet

e.g. runs/20260505_corn_usa/conformal/hindcast_oos_per_init_date.parquet. Canonical path via calibration_path(run_dir, mode) (run_meta_models.py:60). Long-format schema:

fold_year:        Int32      (NA for per_init_date)
fold_init_md:     category   (NA for per_year / pooled)
level:            float64
half_width_kg_ha: float64
n_residuals:      int64
residual_mode:    category
method:           category
commodity:        category

PR #361 live demo (corn, per_init_date): Total rows: 210 = 42 init_mds × 5 levels. The commodity_ prefix that previously appeared in sidecar filenames was removed in PR #361.

How it interacts with the pipeline

During hindcast POSTPROCESS, fit_and_save_all_configured writes one sidecar per configured mode. During forecast POSTPROCESS, primary_calibration loads the primary sidecar and applies it in build_rows (residuals.py:52). See residual_modes.md for the mandatory forecast.residual_mode field and validate_residual_mode gate.

Pitfalls and historical bugs

  • per_year NaN for earliest fold: No prior residuals → NaN half-widths. strict=False returns (nan, nan) silently.
  • commodity_ prefix removed in PR #361: All consumers must use calibration_path() rather than hard-coding the path string.
  • to_frame() empty-row guard (PR #372): Before PR #372, a misconfigured CalibrationResult raised an opaque KeyError: 'fold_year'. PR #372 added a ValueError with a diagnostic message at conformalise.py:187.

PRs and commits

PR Relevance
PR-361 Introduced all four modes, CalibrationResult, mode-keyed sidecars, _MODE_DISPATCH
PR-372 Made forecast.residual_mode mandatory; extracted ResidualMode to types.py; added to_frame() guard

Open questions

  • per_year_fallback is persisted in all modes for round-trip consistency but is only consulted in per_year mode. The design rationale is not documented.
  • MapieConformalRegressor (conformalise.py:645) provides a feature-conditioned calibration pathway unused in the main pipeline; its relationship to CalibrationResult is undocumented.