Skip to content

Entity: Fold

Definition

A walk-forward cross-validation fold identified by fold_label: str. Numeric labels (e.g. "2020") denote "train on years < 2020, test on year 2020"; the literal "production" denotes the no-holdout fit on all available data up to feature_end_year. fold_label is both a conceptual identifier and a filesystem key — there is no Fold class. The concept is carried by HindcastSlice.fold_label and HindcastSlice.cutoff.

Kind

Value object (fold_label: str field on the frozen HindcastSlice dataclass). No dedicated class. The fold_label string is the single canonical identity; the cutoff date is always derived from it.

Source of truth

market_insights_models/src/commodity_hindcast/lib/results/results_slice.py:150 — HindcastSlice.cutoff property: numeric labels → date(int(fold_label), 1, 1); "production" → date(cfg.feature_end_year + 1, 1, 1). run/experiment_protocol.py:110 — ExpandingFoldGenerator emits str(test_year) as fold_label for each test year, plus a "production" fold added by run_walk_forward. config.py:483 — ExperimentProtocolConfig.test_years: list[int] enumerates the years that become fold labels.

Key attributes / structure

Attribute Type Notes
fold_label str Identity key; e.g. "2020" or "production"
cutoff date (derived) date(int(fold_label), 1, 1) for numeric; date(feature_end_year + 1, 1, 1) for production
Filesystem path directory component run_dir/preds/{experiment_key}/{fold_label}/ and run_dir/models/{experiment_key}/{fold_label}/
Training window derived All data with year < int(fold_label) (expanding window)
Test data derived Data with year == int(fold_label) for numeric folds; no held-out test for production

Cardinality: One per test year in ExperimentProtocolConfig.test_years, plus one "production" fold per commodity run. Typical hindcast runs have 5–15 numeric folds depending on feature_start_year and test_years config.

Cutoff semantics: The "cutoff" term (from Nixtla / Prophet / Hyndman FPP3) is the moment dividing known past from predicted future. For numeric folds it is Jan 1 of the test year. For the production fold it is Jan 1 of the year after feature_end_year — the first day on which the model could issue a live forecast.

Lifecycle

Created: ExpandingFoldGenerator.generate_folds() (run/experiment_protocol.py:135) yields one (fold_label, train_data, test_data, year_data, references_fold) tuple per entry in test_years. The "production" fold is added by run_walk_forward() after the walk-forward loop completes.

Consumed: - run_experiment(fold_label, ...) (run/experiment_protocol.py:22) fits a detrender + regressor for the fold and writes train_preds.parquet under preds/{experiment_key}/{fold_label}/. - HindcastSlice is constructed with fold_label as its identity; HindcastSlice.cutoff derives the date boundary on demand. - Dashboard (app/_chart_helpers.py:71) maps numeric fold labels to human-readable display labels (e.g. "2020" → "2020 — Jan Final"). - ExperimentResult.production (lib/results/run_result.py:175) locates the "production" fold via production_hindcast_slice().

Destroyed: The fold label is an immutable directory name; artefacts persist until the run_dir is deleted.

Relationships to other entities

  • SeasonYear — encoded as — numeric fold labels are str(season_year); cutoff = date(season_year, 1, 1) is the training/test split boundary
  • Commodity — scoped by — fold directories are namespaced by experiment_key = {commodity}_{country_code.lower()}
  • Yield — evaluated against — OOS yield predictions in the test year (year == int(fold_label)) are compared to observed yield to compute skill metrics
  • InitDate — scoped within — all init dates for a test season year fall within one fold's test window; conformal calibration modes may group by init date or pool across folds

Concepts and pipelines that touch this entity

PRs and commits

  • PR-361 — Introduces CalibrationResult with four residual_mode values, including hindcast_oos_per_year which groups residuals by fold year; clarifies how fold-level OOS residuals are aggregated for conformal calibration
  • PR-339 — Structural refactor that canonicalised fold_label as the filesystem directory component and HindcastSlice as the fold handle; removed the earlier steps/-layer fold representation

Open questions

  • fold_label is a plain str with no enforced schema; a misconfiguration setting a non-integer, non-"production" label would not be caught until HindcastSlice.cutoff raises a ValueError at runtime.
  • The "production" fold's cutoff is derived from feature_end_year at access time; if feature_end_year changes between runs but the run_dir is reused, the cutoff date will silently shift.
  • There is no fold-level metadata file recording what test_years configuration produced a given run_dir; this must be reconstructed from config_resolved.yaml.
  • The dashboard's fold-label display mapping (app/_chart_helpers.py) is coupled to a fixed set of season-DOY labels; weekly-grid fold labels that do not appear in the map fall back to _fold_label_fallback, which may produce uninformative display strings.