Skip to content

Pipeline: Postprocess

Purpose

The postprocess stage (META-MODELS) bridges raw county-level walk-forward predictions and the delivery layer. It has two orthogonal responsibilities: conformal calibration (attaching honest prediction intervals from walk-forward residuals) and bias correction (shifting national yield estimates to account for uncovered geographies). Both are fit once per experiment and persisted so forecast, delivery, and diagnostic consumers can load without recomputing. The entry point is postprocess_experiment(run_dir, *, included_geo_identifiers) in stages/run_meta_models.py:138.

Inputs

Input Path Format Producer
Per-fold walk_forward_preds.parquet {run_dir}/preds/{experiment_key}/{fold_label}/walk_forward_preds.parquet Parquet run_predict.write_walk_forward_outputs
config_resolved.yaml (via ExperimentResult) {run_dir}/config_resolved.yaml YAML run_hindcast._create_run_root
NASS county panel Loaded via load_nass_county_panel_yield_area(config) Parquet External — NASS download
included_geo_identifiers Passed as frozenset[str] kwarg In-memory run_hindcast._persist_included

Outputs

Output Path Format Consumer
Conformal calibration sidecars {run_dir}/conformal/{residual_mode}.parquet Parquet (long-format) run_forecast, run_deliver, diagnostics via CalibrationResult.load
Per-fold bias_corrector.pkl {run_dir}/preds/{experiment_key}/{fold_label}/bias_corrector.pkl (via period.bias_corrector_path) Pickle run_deliver, forecast delivery
national.parquet {run_dir}/postprocessed/national.parquet Parquet run_deliver.deliver_experiment, run_diagnostics.evaluate_experiment, dashboard

Step-by-step flow

  1. Load experiment handle — ExperimentResult.from_run_dir(run_dir) reads config_resolved.yaml and lazily discovers fold artefacts on disk (run_meta_models.py:158).

  2. Load NASS panel conditionally — load_nass_county_panel_yield_area(config) is called only when config.postprocess.bias_corrector.kind != "none"; otherwise an empty pd.DataFrame() is assigned (run_meta_models.py:163–166).

  3. Fit all configured calibrations — fit_and_save_all_configured(result, ci_levels) iterates config.postprocess.conformalise in declaration order (run_meta_models.py:168). For each mode: a. apply_conformal(result, ci_levels, residual_mode=mode) dispatches to the appropriate residual recipe via _MODE_DISPATCH (conformalise.py:571). b. cal.save(calibration_path(run_dir, mode)) serialises as long-format parquet at {run_dir}/conformal/{mode}.parquet (run_meta_models.py:77–78). The first-listed mode is the primary calibration; its half-widths populate national.parquet and delivery CSVs.

  4. Load primary calibration — primary_calibration(result, ci_levels) calls get_or_fit_calibration for config.forecast.residual_mode (run_meta_models.py:119–135). This mode may or may not be in postprocess.conformalise — if it was not pre-fitted, it is fit on demand here.

  5. Fold loop — _all_folds(result) returns CV folds sorted by numeric label, then appends the production fold when its walk_forward_preds.parquet exists (run_meta_models.py:199–204). For each fold: a. period.load_walk_forward_preds() — load the fold's county-level predictions (run_meta_models.py:174). b. Skip if empty with a warning (run_meta_models.py:175–177). c. aggregate_weighted_frame(rolling, ["sim_yield_kg_ha", "obs_yield_kg_ha"], level="ADM0", group_cols=["year", "init_date"], area_col="area_harvested_ha") — area-weighted ADM0 aggregation; unweighted means are forbidden (DESIGN.md) (run_meta_models.py:179–185). d. build_bias_corrector(config.postprocess.bias_corrector) — factory dispatching on kind (run_meta_models.py:187). e. bias_corrector.fit(nass_panel, included_geo_identifiers, test_year) — for CoverageBiasCorrector, computes bias_y = (area_out / area_total) × (yld_in − yld_out) per lookback year, reduces to scalar via configured reduction_method (run_meta_models.py:188). f. bias_corrector.save(period.bias_corrector_path) — pickle beside the fold (run_meta_models.py:189). g. build_rows(national_by_init, bias_corrector, calibration, commodity) — per-(year, init_date) row builder (residuals.py:22): applies bias_corrector.apply_national(sim_raw), calls calibration.predict_interval(sim_post_bias, fold_year=..., init_md=...), emits lower_{pct} / upper_{pct} columns. h. Append rows to national_rows list (run_meta_models.py:191).

  6. Write national parquet — write_dataframe(pd.DataFrame(national_rows), str(national_path)) (run_meta_models.py:195). Production fold's NaN obs_yield_kg_ha flows through build_rows without affecting calibration (calibrations were fitted in step 3 before this loop).

Mermaid flow diagram

flowchart LR
    WFP["walk_forward_preds.parquet\nper CV fold + production"]
    ER["ExperimentResult.from_run_dir\nrun_meta_models.py:158"]
    CALIB["fit_and_save_all_configured\nrun_meta_models.py:85\n→ conformal/{mode}.parquet"]
    PCAL["primary_calibration\nrun_meta_models.py:119\n(forecast.residual_mode)"]

    subgraph LOOP["Fold loop — _all_folds(result)\nrun_meta_models.py:172"]
        AGG["aggregate_weighted_frame\n(area-weighted ADM0)"]
        BC["build_bias_corrector.fit(...)\nbias_corrector.save(...)"]
        BR["build_rows\n(bias-correct + CI attach)\nresiduals.py:22"]
        AGG --> BC --> BR
    end

    WFP --> ER
    ER --> CALIB
    CALIB --> PCAL
    PCAL --> LOOP
    ER --> LOOP

    NAT["postprocessed/national.parquet\n(year, init_date, sim, obs, lower_*, upper_*)"]
    LOOP --> NAT

Residual-mode dispatch

The residual_mode value in config.forecast.residual_mode selects which calibration sidecar governs CI widths. Four modes are defined in models/meta_models/types.py:16:

Mode Sidecar key Half-width structure Typical use
hindcast_oos_per_init_date conformal/hindcast_oos_per_init_date.parquet per_init_md: dict["MM-DD", dict[float, float]] — one set per calendar init-date, pooling years Default; captures season-of-year uncertainty gradient
hindcast_oos_per_year conformal/hindcast_oos_per_year.parquet per_year: dict[int, dict[float, float]] — one set per fold-year, bootstrap accumulation Reflects evolving uncertainty as evidence accumulates
hindcast_oos_fully_pooled conformal/hindcast_oos_fully_pooled.parquet pooled: dict[float, float] — single scalar set Baseline / debugging
in_sample_pooled conformal/in_sample_pooled.parquet pooled: dict[float, float] — from train_preds.parquet Fallback when no CV folds exist; biased narrow

Separation of axes: postprocess.conformalise controls which sidecars are written during hindcast; forecast.residual_mode controls which is applied at runtime. A forecast can request a mode not in postprocess.conformalise — get_or_fit_calibration fits it on demand (run_meta_models.py:100–116).

Invariants and contracts

DESIGN.md artefact contract (wiki Clause 34; actual DESIGN.md:73, verbatim):

"POSTPROCESS reads every fold → postprocessed/{experiment_key}_national.parquet."

Note: the actual path in the code is {run_dir}/postprocessed/national.parquet (without the {experiment_key} infix, consistent with {run_dir} already encoding the experiment key in its timestamped directory name).

DESIGN.md on area-weighted aggregation (verbatim):

"Clause — Area-weighted aggregation: unweighted mean of yields is forbidden."

Enforced at aggregate_weighted_frame (run_meta_models.py:179–185).

DESIGN.md Clause 35 (verbatim):

"included_geo_identifiers: frozenset[str]; required kwarg at every level; never optional, never falls back to test-fold geo."

postprocess_experiment signature declares it as *-only kwarg: run_meta_models.py:138.

CalibrationResult is persisted (not transient) — the domain model's AGGREGATES.md:214 incorrectly described it as transient. It has save (conformalise.py:215) and load (conformalise.py:224) methods backed by parquet sidecars.

Failure modes and recovery

Symptom Likely cause Recovery
ValueError: to_frame would produce zero rows from CalibrationResult.save No CV folds with valid residuals — only production fold present Run full hindcast to generate CV folds, or use in_sample_pooled mode
FileNotFoundError when loading bias_corrector.pkl at delivery Postprocess was not re-run after a re-fit Re-run cli postprocess {run_dir}
RuntimeError: bias_kg_ha before fit CoverageBiasCorrector.bias_kg_ha accessed without calling .fit() Bug in caller — always call .fit() before .bias_kg_ha or .apply_national()
national.parquet empty or missing folds All fold walk_forward_preds.parquet files are empty Check run_predict stage; verify included_geo_identifiers is non-empty
CI widths are unexpectedly narrow in_sample_pooled mode active — in-sample residuals underestimate OOS uncertainty Switch forecast.residual_mode to an OOS mode; re-run cli postprocess
hindcast_oos_per_year mode yields KeyError at forecast time New forecast season_year not in calibrated fold-year dict Mode uses per_year_fallback ("mean" or "max") on miss — check config

Cross-references

PRs that materially changed this stage

  • PR #361 — introduced four ResidualMode variants and per-mode sidecar parquets under {run_dir}/conformal/; postprocess.conformalise list replaces the previous single-mode configuration.