Module 19 — Validation Engineering for ML Models

Part IV · Machine Learning · Priority 🎯 Core · Status: Draft v0.1 Prerequisites: Module 2, Module 5, Module 18

Overview

Validation engineering is the craft that turns a promising backtest into evidence — and it is the discipline that separates research from gambling. Anyone can fit a model to the past; almost nobody can prove the fit wasn’t luck, because financial labels overlap, returns echo their own recent history, and every hyperparameter sweep quietly buys lottery tickets (Module 2’s multiple-testing trap). Your counterpart on the other side of this trade is your own future self, holding a deployed model whose “Sharpe 2.8” was really the maximum of two hundred noisy attempts.

House position, stated bluntly: a mediocre model with bulletproof validation beats a brilliant model with shuffled K-fold. The mediocre model’s measured edge is approximately real, so it survives contact with live money; the brilliant model’s edge is partly an artifact of the measurement process itself. This module builds the measurement process you can trust — purged folds, walk-forward retrains, combinatorial path ensembles, overfitting statistics, and the drift monitors that tell you when yesterday’s evidence expires.

🔧 For your pipeline: nested walk-forward + untouched-holdout discipline ✅ already exceeds industry norms — treat this module as audit-and-extend, not greenfield. The gaps it closes: (1) a CPCV path ensemble so your OOS Sharpe stops being a single split-dependent point estimate and gains split-variance + PBO; (2) deflated Sharpe accounting for your Optuna v12 trial counts (your ledger already logs them — this makes the count bite); (3) PSI/KS drift dashboards fed by your schema-v6 snapshots, so feature drift is visible before P&L drift; (4) a formal parity triage checklist co-owned with Module 24.

How It Works

Figure: the full validation life cycle. Research-side machinery produces a defensible verdict; live-side monitors decide when that verdict expires and feeds the loop back to a fresh snapshot.

flowchart TD
    snap[(Config + data snapshot<br/>fingerprinted)]
    folds[Purged + embargoed folds<br/>walk-forward plan]
    inner[Inner purged CV<br/>tune hyperparameters]
    outer[Outer walk-forward<br/>OOS evaluation]
    cpcv[CPCV ensemble<br/>many OOS paths]
    gates{PBO + deflated Sharpe<br/>clear the gates?}
    live[Live trading with drift monitors<br/>PSI / KS on features and scores]
    alarm{Drift alarm or live-vs-backtest<br/>divergence flagged?}
    triage[Triage: parity, fills,<br/>regime, selection artifact]
    retrain[Retrain on fresh snapshot<br/>or retire the route]

    snap --> folds
    folds --> inner
    inner --> outer
    outer --> cpcv
    cpcv --> gates
    gates -->|"pass"| live
    gates -->|"fail: redesign or kill"| folds
    live --> alarm
    alarm -->|"yes"| triage
    alarm -->|"no: keep watching"| live
    triage --> retrain
    retrain -->|"new snapshot"| folds

    classDef data fill:#e8f0fe,stroke:#4a86e8
    classDef process fill:#f3f3f3,stroke:#888
    classDef decision fill:#fff4d6,stroke:#d6a300
    classDef ok fill:#e6f4ea,stroke:#2e7d32

    class snap data
    class folds,inner,outer,cpcv,triage,retrain process
    class gates,alarm decision
    class live ok

How to read this:

• Everything downstream of the snapshot node inherits its fingerprint: a verdict is only valid for the exact data it scored (reproducibility closure at the end of this module).
• The two amber decisions are the module’s two exams: gates ask “did the edge survive honest measurement?”, alarm asks “has the world moved since?”
• The dashed-by-construction loop back from retrain to folds is deliberate: a retrained model is a new candidate and must re-enter validation, never skip it.

Why Shuffled K-Fold Lies

Standard scikit-learn KFold(shuffle=True) assumes rows are independent coin flips. Financial rows are neither independent nor interchangeable, and two properties break shuffling outright:

1. Overlapping labels. Your triple-barrier label at bar t (Module 18) resolves by walking bars t+1 … t+h. Its outcome is computed from future prices — including, potentially, prices that belong to the test window.
2. Autocorrelation. Returns cluster: today’s volatility echoes yesterday’s, news shocks propagate for hours. Even rows with disjoint labels share information across a boundary.

Shuffling grants training rows access to test-window information through both channels. Tiny worked example — 24 bars, 4 folds of 6 bars each, label horizon h = 6 bars, third fold selected as test (zero-based indexing throughout):

Bar 11 is the smoking gun: its label is decided by bars 12–17. If a random shuffle assigns bar 11 to training, the model fits a mapping to an outcome derived from the test set’s prices — then gets graded on that same test set. It is booking revenue before delivery (Module 5’s bias catalogue, B1 wearing a CV costume).

The honest accounting: purge removes training rows 6–11 (labels touch the test span); embargo adds a 2-bar dead zone after, removing rows 18–19. Training shrinks from 18 rows to 10 — bars 0–5 and 20–23. That contraction is not waste; it is the price of an unbiased verdict. Later recipes mechanize this cut.

Validation Recipes

Recipe 1 — Purged K-Fold With Embargo

Two operations, one generator:

• Purging — delete training samples whose label span overlaps the test span. If labels look h bars ahead, any training row within h bars before the test block is contaminated and goes.
• Embargo — delete an additional dead zone after the test block, sized to the serial-dependence length. It kills the leakage that labels alone cannot explain: vol clustering and shock echoes.

# Purged K-fold with embargo: contiguous test blocks in time order;
# training rows deleted wherever their labels reach toward the test span.
import numpy as np

def purged_kfold_splits(n, n_folds, label_horizon, embargo_bars=0):
    """Yield (train_idx, test_idx) pairs; test blocks never overlap."""
    edges = np.linspace(0, n, n_folds + 1).astype(int)
    for k in range(n_folds):
        t0, t1 = edges[k], edges[k + 1]          # test block = bars[t0:t1)
        left_cut = max(t0 - label_horizon, 0)    # purge: labels crossing in
        right_cut = min(t1 + embargo_bars, n)    # embargo: dead zone after
        train_idx = np.r_[0:left_cut, right_cut:n]
        yield train_idx, np.arange(t0, t1)

splits = list(purged_kfold_splits(120, 6, label_horizon=12, embargo_bars=4))
for train_idx, test_idx in splits[:2]:
    print(f"test {test_idx[0]:3d}-{test_idx[-1]:3d} | train {len(train_idx)}")
# -> test   0- 19 | train 96     (fold 0: embargo only, no earlier data)
# -> test  20- 39 | train 84     (8 purged left + 4 embargoed right)

Embargo sizing rule of thumb: at least the label horizon, extended to cover your series’ dependence memory (vol clustering at M5 typically decays within a day; start there and inspect fold-boundary agreement for signs of residual seams). This recipe is the sklearn-compatible building block behind github.com/eslazarev/purged-cross-validation, which wraps it as a drop-in CV object; canonical treatment is López de Prado, Advances in Financial Machine Learning, ch. 7.

Recipe 2 — Walk-Forward Retraining Pipelines

Purged K-fold evaluates; walk-forward simulates the life you’d actually live: fit on the past, trade the future, roll, repeat. This is Module 5’s validation-ladder rung 2 — this recipe mechanizes it end to end.

Three engineering decisions:

• Retraining cadence vs regime drift. Refit too rarely and the model lags the market’s rotation; too often and each fit sees too little data while churn multiplies operational risk. Anchor cadence to your regime clock — your Student-t HMM already refits monthly, so Meta refits keyed to the same registry replay keep regime labels and model weights in sync.
• Warm-start is for production, never for scoring. Continuing a boosted tree from the previous fold’s weights saves minutes and quietly couples folds — fold k’s “independent” verdict inherits fold k−1’s parameter choices. Score every validation fold cold; warm-start only the live refresh path.
• Inner loop tunes, outer loop judges. Inside each outer walk-forward train window, run a second purged CV (plus the Optuna study) to pick hyperparameters; evaluate the chosen configuration once on the untouched outer test block. Nested structure is what prevents hyperparameters from being tuned on the data that grades them. Your pipeline already does this — the extension below adds the statistics that grade the whole procedure.

Recipe 3 — Combinatorial Purged Cross-Validation (CPCV)

A single walk-forward pass produces exactly one out-of-sample path. Whatever number falls out, you cannot tell edge from split-luck: would a different test window have said something else? You never learn this from one draw.

CPCV fixes it by enumeration. Split the timeline into N contiguous groups; hold out k groups at a time in every possible combination; train on the rest (purged + embargoed). With N = 6 groups and k = 2 held out:

• Combinations: C(6,2) = (6 · 5) / 2 = 15 train/test arrangements — hand-checkable.
• Each group serves as test in C(5,1) = 5 combinations, so every bar accumulates 5 independent OOS forecasts.
• Stitching forecasts by occurrence rebuilds 15 · 2 / 6 = 5 complete, full-length OOS paths — five alternative histories, hence a distribution of Sharpe ratios instead of a point.

Scale the knobs and paths multiply: C(10,5) = 252 combinations → 126 paths; C(16,8) = 12,870 → 6,435. Dozens to hundreds of paths is the working range.

# CPCV combination census: how many honest OOS estimates do we get?
from itertools import combinations
from math import comb

n_groups, n_test_groups = 6, 2     # 6 time blocks, hold out 2 at a time
n_combos = comb(n_groups, n_test_groups)
print(f"C({n_groups}, {n_test_groups}) = {n_combos} test combinations")
# -> C(6, 2) = 15

oos_paths = n_combos * n_test_groups // n_groups
print(f"paths = {n_combos} x {n_test_groups} / {n_groups} = {oos_paths}")
# -> paths = 15 x 2 / 6 = 5 full-length OOS backtest paths

for held_out in list(combinations(range(n_groups), n_test_groups))[:3]:
    train_on = tuple(g for g in range(n_groups) if g not in held_out)
    print(f"test {held_out} <- train {train_on}")

Why this measures selection variance: the five rebuilt paths differ only in which slices served as test. Same strategy, same data, same rules — so the spread of their Sharpes isolates the component of your headline number that is pure split-luck. Report the median path and the spread, not the single walk-forward figure. Ready-made implementations: fynance.data.combinatorial_purged_cv with signature combinatorial_purged_cv(T, n_groups=6, n_test_groups=2, purge=0, embargo=0), Stefan Jansen’s ml4t_diagnostic CPCV docs, and the eslazarev reference implementation.

💡 Idea: CPCV converts “what did my one backtest say?” into “what does my backtest procedure say, across every way it could have been sliced?” The first is an anecdote; the second is a statistic.

Recipe 4 — Backtest Overfitting Statistics: PBO and Deflated Sharpe

Probability of backtest overfitting (PBO) grades your selection process, not one strategy. Procedure: take your candidate configurations (Optuna trials, feature sets, barrier multiples); within each CPCV combination, rank them by in-sample performance; locate the in-sample champion’s rank in that combination’s OOS results; PBO is the fraction of combinations where the champion lands in the bottom half OOS. Read it like this:

Deflated Sharpe ratio discounts an observed Sharpe by how hard you looked: the number of trials, the variance of trial Sharpes, and the return distribution’s skew/kurtosis. The full formula lives in Bailey & López de Prado (AFML ch. 11–14); the operating discipline is a trial-count haircut:

# Trial-count haircut: qualitative tiers standing in for the full DSR math
HAIRCUT_TIERS = [              # (trials up to ..., discount multiplier)
    (1,   1.00),               # one pre-registered hypothesis: face value
    (10,  0.85),               # a small, honest sweep
    (50,  0.60),               # a feature/target scan
    (200, 0.35),               # an Optuna study's worth of trials
]

def discounted_sharpe(observed_sharpe, n_trials):
    cap, mult = next((c, m) for c, m in HAIRCUT_TIERS if n_trials <= c)
    return observed_sharpe * mult

print(f"Sharpe 2.80 @   1 trial -> {discounted_sharpe(2.80,   1):.2f}")
print(f"Sharpe 2.80 @  50 trials -> {discounted_sharpe(2.80,  50):.2f}")
print(f"Sharpe 2.80 @ 200 trials -> {discounted_sharpe(2.80, 200):.2f}")
# -> 2.80 / 1.68 / 0.98   ('world-class' melts into noise territory)

Worked example, hand-checkable against the table: your route-local study reports best-of-200-trials Sharpe 2.8. Applying the tiers:

Trial registry discipline is what makes either statistic computable: every trial gets a ledger ID stamped onto every result row it produced (Module 0/Module 3 continuity — your JSONL experiment ledger is exactly this). A backtest number that cannot name its trial ID, trial siblings, and dataset fingerprint does not exist for reporting purposes.

📌 Convention: stamp trial_count, ledger_id, and dataset_fingerprint into every saved result object. Future-you computing a deflated Sharpe next quarter will thank present-you for making N retrievable.

Live-Versus-Backtest Divergence Forensics

Live tracking below backtest expectations is a diagnosis waiting to happen, not a verdict on the edge. Work the suspects in cost order — cheapest to exonerate first:

1. Feature-parity mismatch (check first — it is the most common). Replay identical historical bars through the backtest feature code and the live inference path; diff outputs to machine precision. Classic culprits: broker GMT offset shifting session encodings, off-by-one bar indexing (forming vs closed bar), float32 casts in the export path truncating float64 features. Module 24 owns the bit-level parity harness; this module owns the reflex to reach for it first.
2. Fill and cost differences. Swap the live fills, realized spreads, slippage, and swaps into the backtest engine and rerun. If divergence closes, the strategy is fine and execution is the finding — route to Module 22.
3. Regime shift. Compare the HMM state mix of backtest windows vs the live window. A model validated mostly on oscillating states deployed into a breakout expansion should degrade — that is drift (next section), not defect.
4. Selection artifact. Check the trial registry: was the deployed configuration the best of hundreds? Recompute the deflated Sharpe and, if CPCV paths exist, check whether live performance sits inside the path distribution. Live behaving like the worse paths is expected; live below all paths is a red flag worth escalating.

Only after all four come back clean should you conclude “the edge decayed.” Most “edge decay” is actually arithmetic, timestamps, or luck of the draw.

Drift Monitoring: PSI, KS, and the Retrain-or-Retire Decision

A validated model is a claim about a distribution — and distributions migrate. Population stability index (PSI) measures how far a feature’s live distribution has wandered from its training reference: bin the feature using quantiles frozen on the training snapshot, compute each bin’s share in both populations, and aggregate the divergence:

# Population Stability Index: training reference window vs live window
import numpy as np

def population_stability_index(reference, live, n_bins=10):
    """Quantile bins frozen on the training sample, divergence summed."""
    quantiles = np.linspace(0, 1, n_bins + 1)[1:-1]
    cuts = np.unique(np.quantile(reference, quantiles))   # flat-safe bins
    edges = np.concatenate(([-np.inf], cuts, [np.inf]))
    ref_share = np.histogram(reference, edges)[0] / len(reference)
    liv_share = np.histogram(live, edges)[0] / len(live)
    ref_share = np.clip(ref_share, 1e-6, None)   # no log-of-zero blowups
    liv_share = np.clip(liv_share, 1e-6, None)
    ratio = liv_share / ref_share
    return float(np.sum((liv_share - ref_share) * np.log(ratio)))

rng = np.random.default_rng(42)
train_window = rng.normal(0.00, 1.0, 5_000)   # schema-v6 training snapshot
live_window  = rng.normal(0.35, 1.1, 2_000)   # same feature, paper feed
print(f"PSI = {population_stability_index(train_window, live_window):.2f}")
# -> PSI ~= 0.3   (>0.25 action band: investigate, queue a retrain)

Industry rule-of-thumb bands (credit-scoring-model convention; see machinelearningplus’s PSI guide):

Alternatives and complements:

• Kolmogorov–Smirnov two-sample test — scipy.stats.ks_2samp(reference, live) checks whether two empirical distributions plausibly share one continuous CDF. Sensitive to subtle shifts; on thousands of M5 bars it flags almost everything, so use it as a ranking signal alongside PSI’s banded thresholds.
• Prediction-distribution monitoring. Track the live histogram of model scores, not just features. Score drift precedes P&L drift: when the score distribution flattens or slides toward your acceptance threshold, the model is telling you its beliefs moved weeks before the equity curve confirms.

Retrain-vs-retire rules, keyed to the signals above:

Your schema-v6 snapshots are purpose-built telemetry for this: they capture the feature vector at inference time, so the reference-vs-live PSI computation is a join, not a instrumentation project.

⚠️ Pitfall: PSI bands are conventions, not laws. Bin count changes the number (more bins → higher PSI), so freeze the binning with the model version, and monitor stationary transforms — PSI on a rolling z-score behaves; PSI on a raw price level alarms forever.

Reproducibility Closure

Every verdict above decomposes silently unless the measurement is repeatable: pin environments (lockfiles, not “whatever uv resolved last Tuesday”), fingerprint datasets so the exact snapshot a fold scored is recoverable from your content-addressed cache DAG (Module 3), seed every source of randomness, and attach a model card per experiment — config hash, data fingerprint, trial count, ledger ID. A validation result without these is a rumor. The pipeline habit: if you cannot re-run the experiment bit-identically, you did not validate anything; you took a screenshot.

Testing It Honestly

Fixture test — construct leakage and catch it. The test below builds labels with a known horizon overlap and measures, per splitting scheme, what fraction of training rows carry label spans touching the test rows (i.e., rows sharing price outcomes with the test set). Shuffled K-fold should fail catastrophically; purged K-fold must come back clean. Wire it into CI as a regression fixture: if a refactor of the split generator ever reintroduces seam overlap, this catches it before your research does.

# Fixture: prove shuffled K-fold hands training rows the test answers.
# Label of row t resolves from bars t+1..t+h; overlap == shared prices.
import numpy as np
from sklearn.model_selection import KFold

n, horizon = 300, 24
label_span = [(t + 1, min(t + horizon, n - 1)) for t in range(n)]

def avg_leak(splits):
    rates = []
    for train_idx, test_idx in splits:
        lo, hi = test_idx.min(), test_idx.max()
        bad = sum(label_span[t][0] <= hi and label_span[t][1] >= lo
                  for t in train_idx)
        rates.append(bad / len(train_idx))
    return np.mean(rates)

shuffled = KFold(5, shuffle=True, random_state=0).split(np.zeros((n, 1)))
purged = purged_kfold_splits(n, 5, label_horizon=horizon, embargo_bars=6)
print(f"shuffled: {avg_leak(shuffled):.0%} of train rows see test labels")
print(f"purged:   {avg_leak(purged):.0%} of train rows see test labels")
# -> shuffled: 100%   purged: 0%

Read the numbers once and the abstraction dies: with 24-bar labels scattered over 300 bars, essentially every training row a shuffled fold draws touches the test window’s price history. Purging drives it to exactly zero by construction.

Ledger-integrity check. The trial registry is only as good as its enforcement — run this gate before any result leaves research:

# Ledger-integrity gate: a result without provenance does not exist
import json, pathlib

lines = pathlib.Path("experiments/ledger.jsonl").read_text().splitlines()
trials = {json.loads(line)["trial_id"] for line in lines if line.strip()}
report = json.loads(pathlib.Path("results/final_report.json").read_text())

assert report["dataset_fingerprint"] == report["expected_fingerprint"], (
    "verdict was scored against a stale data snapshot -- rescore it")
strays = [r["run_id"] for r in report["runs"]
          if r["optuna_trial_id"] not in trials]
assert not strays, f"unregistered trials reached the report: {strays}"
print(f"ledger ok: {len(report['runs'])} runs trace to registered trials")

Validation-specific pitfalls:

Hands-On Project

Deliverable: src/validation/ utilities (purged splits, CPCV paths, PSI monitor) + docs/research/edge-inflation-audit.md + a Streamlit drift dashboard wired to paper-feed snapshots.

Project A — Retrofit purged CV + walk-forward, quantify edge inflation.

1. Wrap purged_kfold_splits as a sklearn-compatible CV object; confirm your nested walk-forward already purges (audit, don’t assume — check the resume-cache boundary logic).
2. Build a CPCV path ensemble (N = 6, k = 2 to start) on your existing meta_dataset frames for one route (R0 is the cheapest).
3. Compute per-path net Sharpe; report median, spread, and worst path beside your current single-path number.
4. Run PBO across your Optuna v12 trial population; compute the trial-count haircut on your headline Sharpe.
5. Write the inflation memo: how much of the historically claimed edge was validation artifact?

Project B — Drift monitor prototype.

1. Pick your top-20 Meta features by permutation importance.
2. From schema-v6 paper-feed snapshots, compute daily PSI vs the training reference for each; render a heatmap dashboard with the banded thresholds colored.
3. Add score-distribution monitoring (live score histogram vs training histogram) and a weekly ks_2samp ranking column.
4. Define and commit the retrain-vs-retire playbook as a decision table in the dashboard’s sidebar.

Acceptance criteria:

• Purged CV fixture test (shuffled ~100% leak vs purged 0%) passes in CI.
• CPCV ensemble yields ≥ 5 full-length OOS paths; per-path Sharpes reported with spread.
• PBO computed over the full Optuna trial population; headline Sharpe shown deflated by trial count.
• Edge-inflation memo states the single-path number, the path-median, and the delta in plain language.
• PSI dashboard covers ≥ 20 features with banded thresholds; score-drift panel included.
• Every reported number carries ledger_id + dataset_fingerprint; ledger-integrity gate passes.

Key Takeaways

• Validation engineering is the difference between research and gambling: a mediocre model with bulletproof validation outranks a brilliant model with shuffled K-fold, because only the first number is real.
• Shuffled K-fold is disqualified on financial data: overlapping labels let training rows read test-window prices, and autocorrelation leaks through the seams even where labels don’t reach.
• Purging removes training samples whose label spans touch the test span; embargo adds a dead zone after it. Together they convert K-fold from a lie into a measurement.
• One walk-forward path is one draw — CPCV enumerates C(N,k) test-group combinations and rebuilds dozens of complete OOS paths, exposing split-variance that a single path conceals.
• PBO grades the selection process: if the in-sample champion lands bottom-half OOS in ~half the combinations, your tuning is a coin flip. Deflated Sharpe makes trial count bite: best-of-200 Sharpe 2.8 is Sharpe ~1.0 once the search effort is paid for.
• Ledger IDs and dataset fingerprints are load-bearing: unprovenanced results are rumors, and drift monitors expire every verdict on a schedule you don’t control.
• Live divergence is triaged in order — parity, fills, regime, selection artifact — because most “edge decay” is timestamps, arithmetic, or luck, not alpha death.
• PSI/KS monitors on features and scores give weeks of warning before P&L drift confirms it; your schema-v6 snapshots make the dashboards a join, not a project.

References

• Marcos López de Prado — Advances in Financial Machine Learning (Wiley, 2018): chs. 7–8 (purged K-fold, embargo, CPCV, feature/label leakage), chs. 11–14 (backtest overfitting, PBO, deflated Sharpe, synthetic-data evaluation)
• Eslazarev — purged-cross-validation: sklearn-compatible PurgedKFold / combinatorial purged CV reference implementation
• Fynance — fynance.data.combinatorial_purged_cv: combinatorial_purged_cv(T, n_groups=6, n_test_groups=2, purge=0, embargo=0)
• Stefan Jansen (ml4t_diagnostic) — CPCV method docs: path reconstruction walkthrough with worked combinations
• MachineLearningPlus — Population Stability Index (PSI): PSI construction and the 0.1 / 0.25 industry bands
• Bailey & López de Prado — The Deflated Sharpe Ratio (SSRN 2460551): the full trial-aware Sharpe correction
• Internal continuity: Module 2 multiple-testing trap · Module 5 validation ladder · Module 18 triple-barrier labels · Module 24 parity harness
• Next in sequence: Module 20 — Deep Learning for Trading