Module 17 — Honest Validation: Purged Splits and Uncomfortable Baselines

Part IV · Finance Regime Practice · Status: Draft v0.1 Scope: Purged OOS splits, jump-model and threshold baselines · Evidence status: hypothesis-under-test · Prerequisites: Modules 15, 16

Overview

A router that looks good on the bars it was tuned on has shown you nothing. Honest validation means the router meets bars it never saw, with a gap that stops information leaking across the boundary, and against baselines that are trying to beat it.

This module builds that test bench on the shared synthetic series. You split with a purge and an embargo, you line up a jump-model-style router and a one-parameter threshold router next to the HMM, and you learn why gross curves lie.

How It Works

Validation has three jobs, and skipping any one of them flatters the router. The split decides which bars count as unseen. The embargo decides how much padding separates tuning from testing. The baselines decide whether the HMM earns its complexity.

You will see one uncomfortable pattern in honest comparisons. Descriptive regime charts look smooth and persuasive, while cost-inclusive out-of-sample horse races look flat or defensive. Both facts stay on the page, and the mechanics-versus-edge gate resolves them.

flowchart TD
    A[Full 2000 bars] --> B[Train block]
    A --> C[Purge gap]
    A --> D[Validation block]
    A --> E[Embargo gap]
    A --> F[Test block OOS]
    B --> G[Fit all routers]
    G --> H[Run same costs]
    H --> I[Compare net numbers]

How to read this: data flows left to right through gaps that block leakage, then every router faces the same frozen test block with the same cost bill. No router gets its own private test set.

Purged Splits and Embargoes

A plain split is not enough when your labels smooth over bars. Walk-forward refits, rolling volatilities, and dwell rules all smear information across neighboring bars, so a bar sitting exactly on the boundary still whispers about its neighbor.

The fix has two named parts. The purge drops training bars whose label window overlaps the test block. The embargo drops a cushion of bars after each test block starts, so the autocorrelation of volatility cannot carry the answer across. On daily data with a 500-bar refit window, a 20-bar embargo is a reasonable frozen choice.

import numpy as np

n_bars = 2000
train_end = 1000
embargo = 20  # frozen before measurement
purge = 20  # drop bars whose windows touch test
valid_start = train_end + purge + embargo
valid_end = 1500
test_start = valid_end + purge + embargo
test_end = n_bars
print("train:", 0, train_end, "valid:",
      valid_start, valid_end, "test:", test_start, test_end)

The worked reading is exact arithmetic, not estimation: train covers bars 0 to 1000, validation covers 1040 to 1500, and the test block covers 1540 to 2000. You tune thresholds and dwell lengths on train, pick one frozen configuration on validation, and report once on test.

You refit the HMM only on bars the split allows. Parameters estimated on the test block never route the test block, because that is the subtler leak Module 15 warned about. The threshold router and the jump-style router below obey the same boundary, or the comparison is rigged.

[!IMPORTANT] Freeze the split points, the purge, and the embargo before you fit anything. Moving the boundary after seeing the test number is cherry-picking with extra steps.

The Two Adversarial Baselines

Your HMM router must beat two opponents that want it to lose. The first is the threshold router: hold long unless trailing realized volatility sits above its own trailing percentile, then step aside. It has one parameter, it trades rarely, and you can explain it in a sentence.

The second is the jump-model-style router. A statistical jump model fits states the way k-means fits clusters, then adds an explicit jump penalty that charges every state switch during fitting. A large penalty forces persistent regimes directly, instead of detecting flicker first and calming it with a dwell rule afterward. Nystrup and coauthors report this family beating HMM routing out-of-sample net of costs over decades of data (arXiv:2402.05272, retrieved 2026-09-17), which makes it the strongest adversarial evidence in this curriculum. You summarize their method here, not their exact numbers, because your series is synthetic and theirs is not.

rng = np.random.default_rng(42)
ret_series = rng.normal(0.0, 0.01, size=n_bars)
lookback = 60  # frozen before measurement
real_vol = np.zeros(n_bars)
for bar in range(lookback, n_bars):
    window = ret_series[bar - lookback:bar]
    real_vol[bar] = float(np.std(window))
cut = float(np.quantile(real_vol[lookback:train_end], 0.8))
thresh_on = np.ones(n_bars, dtype=int)
thresh_on[lookback:] = (real_vol[lookback:] < cut).astype(int)
thresh_pos = np.zeros(n_bars, dtype=int)
thresh_pos[1:] = thresh_on[:-1]  # next-open fills only

The code above is the full threshold opponent: trailing vol, one frozen percentile cut from train only, next-open fills. Its worked reading is illustrative, not a finding: expect it to trade roughly one half to one third as often as the raw HMM router on the seeded series.

jump_penalty = 0.5  # frozen: explicit cost per switch at fit
jump_state = np.zeros(n_bars, dtype=int)
centers = np.array([0.005, 0.02])  # calm and hot vol centers
for bar in range(lookback, n_bars):
    vol_now = real_vol[bar]
    dist = np.abs(vol_now - centers)
    switch_cost = np.array([0.0, jump_penalty * 0.001])
    switch_cost[1 - jump_state[bar - 1]] += 0.0
    score = dist + (jump_penalty * 0.001
                    * (np.arange(2) != jump_state[bar - 1]))
    jump_state[bar] = int(np.argmin(score))

The jump-style loop above is a teaching sketch in numpy, not the published estimator: nearest vol center wins unless the jump penalty defends the current state. Its lesson is structural, and you should read it that way. Persistence belongs inside the fit, not bolted on after.

[!NOTE] The threshold router is the baseline to beat, not a straw man. If your HMM cannot beat one transparent parameter net of costs, the HMM adds complexity without paying for it.

Why Gross Curves Lie

Gross curves lie through two channels, and this module makes you name both. The first channel is leakage: smoothed states, full-sample parameters, or a moved split boundary all let the future vote. The second channel is cost blindness: a router that flips twice as often pays twice the bill, and the gross curve hides the invoice.

You resolve the curriculum’s central contradiction here in text, not by hiding either side. Descriptive regime studies, including vendor-style demos, plot full-sample state paths against market history and look persuasive. Honest out-of-sample comparisons in the Bulla lineage and the Nystrup jump-model comparison show the defensive version: regime overlays plausibly trim volatility and drawdown, but they do not generate alpha net of costs. Both statements stay. The mechanics-versus-edge gate resolves them: the first proves regimes describe history, the second tests whether they predict profitably, and only the second can carry an edge claim.

Your realistic prior going into Module 18 is therefore defensive, not profitable. Expect a good honest outcome to read as lower realized vol or a shallower drawdown at similar-or-lower net return, never as a new source of alpha. Anything stronger needs evidence this curriculum has not seen.

[!WARNING] Any chart without a cost line and a frozen split label is motivation or a leakage specimen. Do not read it as a result, and do not let anyone present it as one.

Testing It Honestly

Your test question is comparative: on the frozen test block, with identical costs, does the HMM-plus-dwell router beat both baselines net of costs? You preregister the full comparison before running: same bars, same next-open fills, same placeholder spread-plus-fee rate printed in line with every number.

You report switches, gross pnl, cost bill, and net pnl per router, plus realized vol and maximum drawdown depth as the defensive statistics. A win on net alone with worse drawdown is a partial win you label as such. A loss on net is reported as a loss, even if the gross curve looked exciting.

Failure handling is explicit. If the threshold router wins net of costs, you write that the HMM complexity did not pay on this series. If the ranking flips when you double the placeholder spread, you write that the result is cost-fragile. Failed comparisons are findings, and Module 18 depends on your honesty here.

Hands-On Project

Run the three-way comparison on the shared synthetic series and write the verdict in one honest paragraph. Everything below is a method demonstration on synthetic data, with no claim about real markets.

Project card v0.1 for this experiment. Identity: synthetic seeded series, np.random.default_rng(42), about 2000 bars, generated 2026-09-17. Horizon and clock: daily decision bar, close signals, next-open fills, long-or-cash only. Execution: hypothetical next-open mid minus half-spread, zero latency, no partials. Costs: half-spread 0.0005 plus fee 0.0002 per switch, both PLACEHOLDER assumptions printed with every net number. Information: HMM refits use train-plus-validation bars only for test routing; threshold cut from train only; jump penalty frozen before validation. Risk: unit exposure, no leverage, no shorting. Validation: train 0 to 1000, validation 1040 to 1500, test 1540 to 2000, purge plus embargo 20 bars each, one frozen configuration into test. Acceptance: HMM must beat both baselines net of costs on test to claim anything beyond defensive behavior; otherwise report the loss.

Your deliverable is the four-number table per router plus the one-paragraph verdict. Extend it once: move the test boundary 100 bars earlier and check whether the ranking survives. If it does not, your result was boundary-lucky, and you say so.

Key Takeaways

• Purge plus embargo stops label smoothing and vol memory leaking across splits.
• The threshold router is a one-parameter opponent that trades rarely and explains itself.
• Jump models penalize switches inside the fit, which is why they are the hardest baseline.
• Gross curves hide leakage and the cost invoice, so only net-of-cost OOS numbers count.
• Descriptive charts and honest horse races can disagree, and the edge gate resolves them.
• The realistic prior is defensive: less vol or drawdown, not new alpha.
• A baseline win over the HMM is a finding you report, never a failure you hide.

References

• Nystrup et al., statistical jump models for regime identification, arXiv:2402.05272 — Edge evidence / adversarial: strongest OOS net-of-cost comparison favoring jump models over HMM routing; method summarized, exact numbers not repeated; retrieved 2026-09-17; https://arxiv.org/abs/2402.05272
• Bulla et al. 2011, HMM timing lineage — Edge evidence: early regime-timing results that motivate the defensive-not-profitable prior; read as motivation, not proof; retrieved 2026-09-17; https://doi.org/10.1007/s10614-010-9239-1
• Pohle et al. 2017, selecting the number of states, arXiv:1701.08673 — Mechanics: why state-count and penalty choices must be frozen before validation; retrieved 2026-09-17; https://arxiv.org/abs/1701.08673
• hmmlearn GaussianHMM documentation — Mechanics: the HMM estimator under test in the comparison; retrieved 2026-09-17; https://hmmlearn.readthedocs.io/
• Exclusion log: QuantInsti and Medium-style HMM trading tutorials were rejected as evidence (no out-of-sample or cost methodology; background only).