Skip to content

ADR-003: forecast.residual_mode is mandatory

Status: Accepted (retroactively documented 2026-05-08) Date: 2026-05-05 (PR #372 merge commit befbba58, "feat(commodity_hindcast): require forecast.residual_mode + gate forecast on run_dir compatibility") Authors: ai-tommytf (per git log on PR #372)

Context

The forecast pipeline calibrates prediction intervals from past residuals; which residuals it reads is governed by ForecastConfig.residual_mode (config.py:642). Four modes are defined as a Literal in models/meta_models/types.py:16: three OOS modes that read walk-forward CV residuals (hindcast_oos_per_init_date, hindcast_oos_per_year, hindcast_oos_fully_pooled) and in_sample_pooled that reads the production fold's train_preds.parquet. The modes have a strict trust gradient — OOS modes give honest, out-of-sample half-widths; in_sample_pooled gives narrow CIs because the model has already seen those rows (wiki/commodity_hindcast/concepts/residual_modes.md, wiki/commodity_hindcast/concepts/conformal_modes.md).

Before PR #372 (wiki/commodity_hindcast/sources/prs/PR-372.md) the field was optional with a silent default. A run_dir produced by cli run fit-production alone (no walk-forward CV) would still serve a forecast, silently picking up the default mode and shipping intervals from the wrong calibration. Wrong-mode intervals would land in delivery before anyone noticed.

Decision

forecast.residual_mode is a required ResidualMode field on ForecastConfig with no default (market_insights_models/src/commodity_hindcast/config.py:642). The forecast entry point gates every run on a compatibility check between the YAML choice and what is on disk in the run_dir:

  • validate_residual_mode(experiment, mode) is the gate (stages/run_forecast.py:91-140). It refuses runs where neither walk-forward CV folds nor a production train_preds.parquet exist; refuses OOS modes when no CV folds exist; refuses in_sample_pooled when no production fold exists. Each branch raises FileNotFoundError with an explicit next-command-to-run (run_forecast.py:122-139).
  • The gate is called on the first line of run() after loading the experiment, before any feature build or disk write (stages/run_forecast.py:161).
  • _OOS_MODES is a frozenset[ResidualMode] enumerating the three modes that need CV folds (run_forecast.py:82); in_sample_pooled requires experiment.production.train_preds_path.exists() (run_forecast.py:117).
  • Companion fix in CalibrationResult.to_frame() (models/meta_models/conformalise.py:188-198) raises ValueError rather than silently writing an empty sidecar when the mode handler returns no residuals — defensive belt-and-braces if the gate is bypassed.
  • ResidualMode was lifted out of conformalise.py into models/meta_models/types.py:16 to break a circular import.

Consequences

Positive

  • Forecast cannot run if residual_mode is missing from YAML; pydantic rejects the config at load time (config.py:642, no default).
  • Mode/run_dir incompatibility is caught before any feature build, so the user gets a fast error pointing at the next command rather than a deep KeyError from inside pandas after a slow forecast walk (run_forecast.py:96-99).
  • Mode mismatch between hindcast (which produces residuals) and forecast (which consumes them) becomes impossible at runtime — every hindcast/ forecast pairing is now an explicit YAML choice.
  • The empty-frame guard in to_frame() prevents an unusable sidecar from ever being persisted, even if a future caller side-steps the gate (conformalise.py:188-198).

Negative

  • _OOS_MODES is a hand-maintained frozenset that can drift from the ResidualMode Literal (run_forecast.py:82 vs models/meta_models/types.py:16). Any new value added to the literal that is not also added to _OOS_MODES would silently pass through the validation gate when no conformal sidecar exists — Risk 5 in the risk register, captured under "Architectural Drift Items" in wiki/commodity_hindcast/LINT_REPORT.md line 133.
  • Existing YAML configs without an explicit forecast.residual_mode entry fail to load and need a migration to add the field.
  • The error messages bake in CLI command names (make hindcast, cli run fit-production); if those entry points are renamed the gate copy must be updated alongside.

Neutral

  • validate_residual_mode is a pure function on (ExperimentResult, ResidualMode); it can be called from tests without spinning up a pipeline.
  • The integration test tests/integration/commodity_hindcast/test_forecast_residual_mode_validation.py is named in the docstring as the single source of truth for the contract (run_forecast.py:103-105); the docstring and tests must move in lockstep.

Alternatives considered

Keep optional with a default mode. Rejected — that is the failure PR

372 fixed: silent wrong-mode intervals shipped to delivery.

Derive _OOS_MODES from typing.get_args(ResidualMode) minus {"in_sample_pooled"}. Recommended in wiki/commodity_hindcast/LINT_REPORT.md:133 as the drift fix, with a unit test pinning the derivation. Not done in PR #372; the literal frozenset was kept for readability at the call site. [PLACEHOLDER: confirm rationale with PR-372 author — readability vs drift-resistance trade-off was not stated in the commit body.]

Promote ResidualMode from Literal to enum.Enum. Would give iteration and exhaustiveness checks for free but breaks YAML readability (members would need .value) and has wide blast radius across CalibrationResult, sidecars, and docs. Out of scope for the PR-372 patch.

Verification

  • Forecast invoked against a YAML missing residual_mode → pydantic ValidationError at config load (config.py:642, required field).
  • Forecast invoked with an OOS mode against a run_dir lacking CV folds → SystemExit via FileNotFoundError at run_forecast.py:161 before any feature build.
  • Forecast invoked with in_sample_pooled against a run_dir lacking a production fold → SystemExit via FileNotFoundError at run_forecast.py:161.
  • CalibrationResult.to_frame() invoked on an all-empty calibration → ValueError at conformalise.py:188-198.
  • Integration test: tests/integration/commodity_hindcast/test_forecast_residual_mode_validation.py (named as contract anchor in run_forecast.py:103-105).

References

  • market_insights_models/src/commodity_hindcast/stages/run_forecast.py:82 — _OOS_MODES frozenset
  • market_insights_models/src/commodity_hindcast/stages/run_forecast.py:91-140 — validate_residual_mode function
  • market_insights_models/src/commodity_hindcast/stages/run_forecast.py:161 — gate call site at start of run()
  • market_insights_models/src/commodity_hindcast/config.py:642 — residual_mode: ResidualMode required field
  • market_insights_models/src/commodity_hindcast/models/meta_models/types.py:16 — ResidualMode literal (moved here in PR-372 to break circular import)
  • market_insights_models/src/commodity_hindcast/models/meta_models/conformalise.py:188-198 — to_frame() empty-rows guard
  • wiki/commodity_hindcast/entities/ForecastConfig.md
  • wiki/commodity_hindcast/sources/prs/PR-372.md
  • wiki/commodity_hindcast/concepts/residual_modes.md
  • wiki/commodity_hindcast/concepts/conformal_modes.md
  • wiki/commodity_hindcast/LINT_REPORT.md:133 — _OOS_MODES drift item
  • PR #372 (merge commit befbba58, 2026-05-05): "feat(commodity_hindcast): require forecast.residual_mode + gate forecast on run_dir compatibility"