Skip to content

Concept: Residual Modes

What it is

forecast.residual_mode is a mandatory string field on ForecastConfig that declares which pool of past errors the forecast pipeline should use when calibrating prediction intervals. It was made mandatory in PR #372, replacing a previous implicit assumption that calibration would always use postprocess.conformalise[0].

Separating the two axes — which sidecars to pre-compute during hindcast (postprocess.conformalise) vs which to apply at forecast time (forecast.residual_mode) — means changing the live forecast's calibration strategy does not require re-running the full hindcast. The field also gives validate_residual_mode a machine-readable declaration to check before any compute begins.

The four invariant strings

Defined as a Literal in models/meta_models/types.py:16:

ResidualMode = Literal[
    "hindcast_oos_per_init_date",
    "hindcast_oos_per_year",
    "hindcast_oos_fully_pooled",
    "in_sample_pooled",
]

These four strings are the complete and exhaustive set. No other value is accepted by pydantic, by the _MODE_DISPATCH dict (conformalise.py:571), or by validate_residual_mode. They double as the filename stems of conformal sidecar parquets: {run_dir}/conformal/{residual_mode}.parquet.

For the recipe, calibration-set size, and bias risk of each string see conformal_modes.md.

Where it lives in the code

Symbol Location Notes
ResidualMode models/meta_models/types.py:16 Literal type alias; leaf module, no package imports
ForecastConfig.residual_mode config.py:603 Required field; no default value
validate_residual_mode stages/run_forecast.py:91 Gate function; first call in run_forecast.run()
_OOS_MODES stages/run_forecast.py:82 frozenset{"hindcast_oos_per_init_date", "hindcast_oos_per_year", "hindcast_oos_fully_pooled"}

ResidualMode was extracted from conformalise.py into types.py in PR #372 to break a circular import: config.py would otherwise import from conformalise.py, violating the single-direction DAG in DESIGN.md (config → utilities → domain services).

validate_residual_mode

def validate_residual_mode(experiment: ExperimentResult, mode: ResidualMode) -> None:

run_forecast.py:91

First call inside run_forecast.run() (run_forecast.py:161). Raises FileNotFoundError with an actionable next-step message for three failure cases:

  1. Neither CV folds nor production fold present — "run make hindcast (preferred) or cli run fit-production (fast, narrow CIs)."
  2. OOS mode + no CV folds (mode in _OOS_MODES) — "run make hindcast here, or change config to in_sample_pooled."
  3. in_sample_pooled + no production fold — "run cli run fit-production first."

Integration-tested by an 18-case parametrised matrix at tests/integration/commodity_hindcast/test_forecast_residual_mode_validation.py: 16 cases covering has_cv_folds × has_production × residual_mode plus 2 message-content checks. Real ExperimentResult.from_run_dir, no mocks.

Key invariants

  • forecast.residual_mode is required on ForecastConfig with no default (config.py:603); omitting it from a YAML with a forecast: block raises a pydantic ValidationError at config load time.
  • validate_residual_mode MUST be the first call in run_forecast.run() — fail fast before any S3/feature I/O (PR #372 lesson).
  • cli run forecast --config <yaml> was removed in PR #372; all forecast invocations require --run-dir.

The no-backwards-compat decision

PR #372 applied the project's no-backwards-compatibility policy: residual_mode has no default value. All four production YAML configs (corn_usa, soybeans_usa, soybeans_bra, wheat_usa) were updated to declare residual_mode: "hindcast_oos_per_init_date", preserving prior behaviour.

Old run_dirs whose config_resolved.yaml predates PR #372 cannot be replayed against the new code — those run_dirs were already producing crashes for any forecast that needed calibration (KeyError: 'fold_year' deep in pandas). A missing field at config load time is a one-line pydantic error; the previous failure was two minutes of compute followed by an opaque exception.

Architecture decision rationale

Before PR #372 the primary calibration mode was implicitly postprocess.conformalise[0]. This conflated the question of which modes to materialise (hindcast time) with which to apply (forecast time). A consumer could not change the live CI strategy without editing the list and re-running hindcast, and there was no explicit field to audit.

forecast.residual_mode makes the selection explicit, mandatory, and independently changeable. If the sidecar is absent, get_or_fit_calibration (run_meta_models.py:100) fits it on demand — but only after the gate has confirmed the underlying residuals exist.

How it interacts with the pipeline

forecast.residual_mode is read in three places in the forecast path:

  1. validate_residual_mode at run_forecast.py:161 — gate.
  2. primary_calibration(result, ci_levels) at run_meta_models.py:119 — load or fit the calibration.
  3. build_rows at residuals.py:52 — apply predict_interval to national rows.

Pitfalls and historical bugs

  • Pre-PR-372 silent crash: make forecast EXPERIMENT_KEY=corn_usa (no RUN_DIR) crashed with KeyError: 'fold_year' after two minutes. validate_residual_mode moved the failure to the door.
  • cli run forecast --config removed: The shortcut that triggered the pre-gate crash was deleted in PR #372; forecasts now always require a --run-dir.

PRs and commits

PR Relevance
PR-361 Introduced ResidualMode and the four string values; postprocess.conformalise[0] was the implicit primary
PR-372 Made field mandatory; added gate; extracted ResidualMode to types.py; removed --config shortcut

Open questions

  • _OOS_MODES at run_forecast.py:82 is a hand-maintained frozenset. A fifth mode added to ResidualMode would require a manual update — there is no compile-time guard.
  • A stale sidecar parquet from a pre-PR-361 run_dir would pass validate_residual_mode and then fail on load. The gate checks residual existence, not schema version.