Skip to content

Source: Orchestration & Configuration

Overview

The orchestration subsystem is the user-facing entry layer that transforms a YAML experiment config and CLI flags into a rooted, reproducible run_dir. Users enter via the commodity-hindcast Click CLI (or the Makefile dev wrappers); the CLI resolves the config into an ExperimentConfig (a pydantic-settings model), gates execution behind preflight path checks, and then delegates to stage modules. The run/ sub-package provides the three lower-level building blocks consumed by stages/run_hindcast.py: a walk-forward fold runner (runner.py), the fold-generation and per-fold fit/predict protocol (experiment_protocol.py), and the preflight gate library (preflight.py).

Modules

cli.py (~637 lines)

Purpose: Click entrypoint that exposes end-to-end pipeline workflows and stage-level re-run commands.

Public surface:

  • cli (click.Group, cli.py:161) — root entrypoint; invoking with no subcommand prints help.
  • run (click.Group, cli.py:171) — sub-group for multi-stage workflows.
  • run features (cli.py:188) — run_features_cmd: builds fit.parquet + pred.parquet.
  • run hindcast (cli.py:212) — run_hindcast_cmd: walk-forward hindcast (FIT + POSTPROCESS + EVALUATE + DELIVER).
  • run fit-production (cli.py:337) — run_fit_production_cmd: production model fit only.
  • run forecast (cli.py:477) — run_forecast_cmd: point-in-time forecast against an existing run_dir.
  • run forecast-features (cli.py:381) — run_forecast_features_cmd: materialise forecast indices + build forecast features.
  • run forecast-predict (cli.py:427) — run_forecast_predict_cmd: predict + postprocess + deliver using pre-built forecast features.
  • run all (cli.py:269) — run_all_cmd: features → hindcast → forecast → optional export in one call.
  • run export (cli.py:516) — run_export_cmd: model-output export from an existing run_dir.
  • postprocess (cli.py:535) — stage-level re-run: aggregate + bias correction + conformal CIs.
  • evaluate (cli.py:552) — stage-level re-run: metrics + optional plots.
  • plots (cli.py:581) — stage-level re-run: regenerate PNGs only, with --only/--skip filter flags.
  • predict (cli.py:609) — stage-level re-run: generate forecast using the production model.
  • deliver (cli.py:625) — stage-level re-run: client-facing CSVs.

Key functions:

  • run_features_cmd() (cli.py:188) — calls preflight_paths_for_features, then build_features.
  • run_hindcast_cmd() (cli.py:212) — delegates entirely to stages.run_hindcast.run.
  • run_forecast_cmd() (cli.py:477) — delegates to stages.run_forecast.run; requires --run-dir, --season-year, --init-date.
  • run_all_cmd() (cli.py:269) — sequential four-stage orchestrator; logs per-stage elapsed time.
  • _prepare_config() (cli.py:112) — sets COMMODITY_HINDCAST_CONFIG env var then instantiates ExperimentConfig() via BaseSettings so the env/YAML source chain fires correctly (calling model_validate would bypass settings_customise_sources).
  • _resolve_experiment_config_path() (cli.py:87) — accepts a bare stem (corn_usa) and resolves it to configs/corn_usa.yaml under the package root.
  • _validate_export_options() (cli.py:128) — enforces PIPELINE_RUN_ID and MODEL_INGESTION_PATH env vars when --export is set.

Key type: AnyPathParam (cli.py:36) — custom click.ParamType that validates paths via cloudpathlib so s3:// URIs accepted at the CLI don't fail the built-in click.Path existence check (added in PR #345).

External dependencies: click, cloudpathlib.AnyPath, loguru

Notes: INPUT_DATA_DIR is mandatory; require_input_data_dir() raises RuntimeError when unset. --config defaults to configs/corn_usa.yaml. Stage commands (postprocess, evaluate, plots, predict, deliver) operate on an existing run_dir argument and do not need --config.

config.py (~902 lines)

Purpose: All pydantic(-settings) config classes for the experiment; the single source of truth for every path, commodity parameter, model choice, and output schema option.

Public surface:

  • ExperimentConfig (config.py:611) — root BaseSettings class; resolves INPUT_DATA_DIR → data_root and fills derived paths.
  • CommodityConfig (config.py:274) — season calendar, builder registry, feature/target columns, climo/weather windows, delivery unit.
  • ModelConfig (config.py:491) — detrend strategy, regression estimator, fit-aggregation level, sample weights.
  • ExperimentProtocolConfig (config.py:483) — CV strategy string, test_years, production cumulative threshold.
  • BiasCorrectorConfig (config.py:528) — kind (none | coverage), lookback window.
  • PostprocessConfig (config.py:545) — bias_corrector, ordered conformalise tuple (primary mode first).
  • ForecastConfig (config.py:579) — raw_obs_filepath, materialised_climo_filepath, residual_mode, runtime-injected init_date.
  • DeliveryConfig (config.py:432) — model_public_name, ci_levels, enforce_ci_narrowing, drop_frozen_tail.
  • BaseBuilderConfig (config.py:168) — typed base for builder sub-configs.
  • Builder discriminated union (config.py:250) — YieldsBuilder | StressBuilder | ClimoBuilder | WeatherBuilder | NDVIBuilder, discriminated on type.

Key functions / validators:

  • require_input_data_dir() (config.py:50) — mandatory env-var read; raises RuntimeError when INPUT_DATA_DIR is unset.
  • resolve_data_path() (config.py:82) — anchors relative paths at data_root; passes through s3:// URIs unchanged.
  • resolve_nested_config() (config.py:124) — model_validator helper that loads a nested YAML or passes through an inline dict/model instance.
  • ExperimentConfig._fill_defaults_from_data_root() (config.py:795) — fills raw_dir, features_dir, models_dir, preds_dir, run_dir_base from data_root when unset.
  • ExperimentConfig._resolve_data_paths() (config.py:815) — walks every ResolvablePath field via _iter_resolvable_fields and resolves against data_root.
  • ExperimentConfig.settings_customise_sources() (config.py:861) — resolution order: CliSettingsSource > env > YamlConfigSettingsSource > init defaults.
  • ExperimentConfig.init_dates_for() (config.py:706) — returns the single forecast.init_date in forecast mode, or the full hindcast_init_season_doys grid otherwise.
  • experiment_config_to_yaml_safe_dict() (config.py:888) — serialises the config for config_resolved.yaml and MLflow; handles CloudPath/Path via a JSON fallback.

Notes: ExperimentConfig uses SettingsConfigDict(cli_parse_args=False) (config.py:630) to prevent pydantic-settings from consuming sys.argv — Click handles CLI flags independently (see issue #264). Builder type discriminators are auto-injected from the YAML dict key by _inject_builder_type_from_key (config.py:354). run_dir_base and all derived output paths are set at load time; stages/run_hindcast.py::_create_run_root mutates models_dir and preds_dir in-place to point inside the fresh run_root.

run/runner.py (~139 lines)

Purpose: Walk-forward hindcast orchestrator; loops over folds and accumulates per-fold predictions in memory before emitting a single parquet write per fold.

Public surface:

  • run_walk_forward() (runner.py:27) — accepts (config, data_fold_generator), calls run_experiment per fold, then calls _predict_fold_rolling and writes year_data.parquet + walk_forward_preds.parquet once per fold.

Key design note: Walk-forward deliberately bypasses stages.run_predict.run_predict because that writer has blind-overwrite persistence — sequential calls would destroy earlier init_date rows. Instead, _predict_fold_rolling (runner.py:86) accumulates positional masks on an in-memory copy of year_data (county-major order, matching upstream pred.parquet) and emits one write per fold. The county-major row order must be preserved verbatim because the manifest oracle hashes raw bytes.

External dependencies: loguru, numpy, pandas, treefera_market_insights.shared.utils.dataframes.write_dataframe

run/experiment_protocol.py (~148 lines)

Purpose: Defines the per-fold fit/predict protocol (run_experiment) and the expanding-window fold generator (ExpandingFoldGenerator).

Public surface:

  • run_experiment() (experiment_protocol.py:22) — fits detrender + imputer + regressor for one fold label, predicts on train data, persists train_preds.parquet under preds_dir/<experiment_key>/<fold_label>/.
  • AbstractFoldGenerator (experiment_protocol.py:92) — ABC with abstract generate_folds() yielding (fold_label, train_data, test_data, year_data, references_fold) tuples.
  • ExpandingFoldGenerator (experiment_protocol.py:110) — concrete walk-forward implementation; iterates ExperimentProtocolConfig.test_years, slicing fit_df[year < test_year] for training and pred_df[year == test_year] for prediction. Also yields a references_fold: dict[str, DataFrame] keyed by spec.name for per-fold reference-data lookup.

Notes: Test-time predictions are NOT produced or persisted by run_experiment; downstream consumers derive the test slice on demand from walk_forward_preds at the latest init_date of the fold year (via HindcastSlice.load_test_slice). The references_by_harvest parameter to ExpandingFoldGenerator.__init__ (experiment_protocol.py:119) accepts an empty dict as the "no reference data" sentinel — folds still yield but references_fold is empty.

run/preflight.py (~208 lines)

Purpose: Path-existence gate library; provides one Check-producing function per pipeline stage. run_preflight raises SystemExit on any critical failure.

Public surface:

  • Check (preflight.py:21) — dataclass: name, passed, message, critical.
  • check_path_exists() (preflight.py:30) — wraps AnyPath.exists() (cloud-safe); returns a critical Check.
  • run_preflight() (preflight.py:42) — iterates checks, logs results, raises SystemExit on the first critical failure.
  • preflight_paths_for_features() (preflight.py:110) — checks all ResolvablePath inputs, skipping stress filepath when assemble_stress_from_indices is set (that parquet is produced during this stage).
  • preflight_paths_for_hindcast() (preflight.py:59) — checks check_data_exists entries plus fit.parquet and pred.parquet under features_dir.
  • preflight_paths_for_resolvable_inputs() (preflight.py:77) — mechanically walks _iter_resolvable_fields(config) so adding a new ResolvablePath field automatically extends preflight coverage.
  • preflight_paths_for_forecast_features() (preflight.py:132) — resolvable inputs + canonical pred.parquet.
  • preflight_paths_for_forecast_predict() (preflight.py:147) — resolvable inputs + per-(season_year, init_date) forecast features parquet + production model artefacts (detrender.pkl, feature_fill_values.parquet).
  • preflight_paths_for_forecast() (preflight.py:183) — union of the two forecast sub-stage check sets.
  • preflight_paths_for_export() (preflight.py:205) — resolvable inputs only.

Notes: _skip_stress_filepath_preflight() (preflight.py:93) uses duck-typing on the owner model to avoid a circular import with commodity_hindcast.config.

Makefile

Purpose: Developer convenience wrapper around the commodity-hindcast CLI; anchors all invocations at the repo root.

Key variables:

  • REPO_ROOT — resolved via git rev-parse --show-toplevel; cd $(REPO_ROOT) is prepended to every CLI invocation so relative config paths resolve correctly.
  • CONF_FLAG — expands to --config <path> when EXPERIMENT_KEY is set (e.g. make hindcast EXPERIMENT_KEY=corn_usa); unexport EXPERIMENT_KEY prevents it leaking into the subprocess env where pydantic-settings might misinterpret it.
  • INIT_DATE / SEASON_YEAR — default to current UTC date/year via date -u; can be overridden per invocation.

Targets: sync, lint, format, test, features, features-force, hindcast, fit-production, forecast, postprocess, evaluate, plots, deliver, explore, help

Notes: forecast requires RUN_DIR to be set explicitly (no default) — callers must run make hindcast or make fit-production first to produce one.

Cross-references to wiki pages

Relationships to other subsystems

  • run_features_cmd calls features/run.build_features and run.preflight.preflight_paths_for_features.
  • run_hindcast_cmd delegates to stages/run_hindcast.run, which in turn calls run/runner.run_walk_forward and run/experiment_protocol.run_experiment.
  • run_forecast_cmd delegates to stages/run_forecast.run.
  • stages/run_hindcast._create_run_root creates the timestamped run_dir under config.run_dir_base, mutates config.models_dir and config.preds_dir in place, and writes config_resolved.yaml.
  • ExperimentConfig is consumed by every stage module and all builder/model code as the single config authority.
  • ExperimentConfig.data_root is sourced from INPUT_DATA_DIR; relative ResolvablePath fields in builder configs resolve against it via resolve_data_path.
  • YAML configs live in configs/<commodity>_<iso3>.yaml (e.g. configs/corn_usa.yaml); the resolved snapshot is written to <run_dir>/config_resolved.yaml for reproducibility.