Module 15 — Causal Regime Detection

Part IV · Finance Regime Practice · Status: Draft v0.1 Scope: Filtered-only regime states, leakage demo, refit discipline · Evidence status: hypothesis-under-test · Prerequisites: Modules 03, 07, 10

Overview

You want a regime label you could have known at the time. That label is the filtered state probability: the chance you are in each regime using only bars up to today.

This module shows the trap. A smoothed state probability uses the whole sample, including future bars, so it always looks cleaner than anything you could have traded. You fit one model on the shared synthetic series, compare the two paths side by side, and measure the gap.

How It Works

A Gaussian HMM has hidden regimes and visible returns. Each day you observe a return, and the model infers which volatility regime probably produced it.

Three decodings answer three different questions. The filtered probability asks “what do I believe tonight, knowing bars up to tonight.” The smoothed probability asks “what do I believe about last March, knowing the entire multi-year sample.” The Viterbi path asks “what single regime sequence best explains everything at once.”

Only the first question is tradable. The other two peek at the future, so this curriculum treats them as leakage specimens, never as signals.

flowchart TD
    A[New bar closes] --> B[Filtered probability]
    B --> C{Act on tonight's belief}
    C --> D[Fill next open]
    E[Full sample fit] --> F[Smoothed path]
    F --> G[Leakage specimen only]
    B --> H[Compare paths]
    F --> H
    H --> I[Measure lookahead gap]

How to read this: the top row is the causal chain you are allowed to trade. The bottom row is the historian’s view. The join at the bottom is a diagnostic, not a strategy.

The Leakage Trap, Measured

You reuse the shared synthetic series so the reader meets one dataset, not four. Two true regimes drive returns: a calm state near 0.5 percent daily volatility and a stressed state near 2 percent. You generate about 2000 bars with a seeded generator, then fit one two-state Gaussian HMM on the full sample.

import numpy as np
from hmmlearn.hmm import GaussianHMM

rng = np.random.default_rng(42)
n_bars = 2000
true_state = np.zeros(n_bars, dtype=int)
true_state[0] = 0
stay_prob = np.array([0.985, 0.94])
for bar in range(1, n_bars):
    prev = true_state[bar - 1]
    flip = rng.random() > stay_prob[prev]
    true_state[bar] = 1 - prev if flip else prev

state_vol = np.array([0.005, 0.02])
returns = rng.normal(0.0, state_vol[true_state])
obs = returns.reshape(-1, 1)

model = GaussianHMM(n_components=2, covariance_type="diag",
                    n_iter=200, random_state=42)
model.fit(obs)

Next you compute both belief paths from the same fitted model. The filtered path rolls forward one bar at a time. The smoothed path runs forward-backward over everything. Count how often they disagree about the most-likely state.

post = model.predict_proba(obs)  # smoothed marginals, shape (bars, states)
smooth_state = np.argmax(post, axis=1)  # best state per bar, full sample
filt_state = np.zeros(n_bars, dtype=int)
filt_state[:20] = smooth_state[:20]  # prefix: identical by construction
for end in range(20, n_bars):
    # posterior at the LAST bar of a prefix equals the filtered prob there
    prob_now = model.predict_proba(obs[: end + 1])[-1]
    filt_state[end] = int(np.argmax(prob_now))
disagree = np.mean(filt_state[20:] != smooth_state[20:])
print("filtered-vs-smoothed disagreement:", round(float(disagree), 3))

The worked number on the seeded series is illustrative, not a finding: expect disagreement on roughly 8 to 15 percent of bars, concentrated around turning points. That is the point. The smoothed path “knows” the switch the night it happens. The filtered path learns it late.

[!WARNING] Any backtest that routes on smoothed or full-sample Viterbi states is measuring hindsight, not skill. If you see one, relabel it a leakage specimen and discard its returns.

Refit Discipline and Detection Lag

A model fitted once on 2000 bars also leaks in a subtler way: its parameters saw the future. The honest habit is rolling refits. You refit on a trailing window (say 500 bars, quarterly cadence on daily data), freeze the parameters, and filter the next block causally.

Refits create a second problem: label switching. State 0 in January’s fit may be the high-vol state in April’s fit. You re-anchor every refit by sorting states on fitted volatility, so label 1 always means “the hotter state.” Skip this step and your router flips its meaning mid-sample.

Detection lag is then visible and honest. On the synthetic series, align each true regime switch with the first bar the filtered probability crosses one half. Typical lags run 3 to 10 bars after the true switch, longer when the vol ratio is small. Nystrup and coauthors document the same lag-and-flicker pattern on real data and motivate it as the reason raw HMM states trade poorly without persistence control (retrieved 2026-09-17). The Bulla lineage of HMM timing studies shows the older descriptive version of this chart, where smoothed paths hide the lag entirely.

[!TIP] Plot three lines together: true state, filtered state, smoothed state. The gap between the first two is your lag. The gap between the last two is your leakage.

Testing It Honestly

Your test question is narrow: does the filtered path detect regimes fast enough, and stably enough, to route on? You do not need returns yet. You need three checkable numbers, all computed on the synthetic series with the refit schedule frozen before you look.

First, disagreement rate between filtered and smoothed most-likely states, reported per refit block. Second, median detection lag in bars from true switch to filtered cross of one half. Third, flicker count: one-bar round trips the filtered path makes and unmakes within three bars.

[!IMPORTANT] Freeze the window length, refit dates, and the one-half threshold before you measure. Tuning any of them after seeing the lag is parameter cherry-picking, and it voids the result.

Failure looks like this: median lag above 10 bars with flicker above 5 percent of bars. That outcome does not mean HMMs are useless. It means the raw filtered state is not routable without a persistence rule, which is exactly what Module 16 builds. A failed detection test is reported as failed, not smoothed over.

Hands-On Project

Build the causal detector on the shared synthetic series. Everything below is a method demonstration on synthetic data, framed as a preregistered hypothesis, with no claim about real markets.

Project card v0.1 for this experiment. Identity: synthetic seeded series, np.random.default_rng(42), about 2000 bars, two Gaussian vol states, generated 2026-09-17. Horizon and clock: daily decision bar, signal evaluated at bar close, fills modeled next open. Execution: hypothetical fill at next open mid, no latency modeled here (routing comes in Module 16). Costs: none at detection stage; cost accounting starts in Module 16. Information: filtered probabilities only; refit parameters frozen per block; smoothed path labeled leakage specimen. Risk: no positions in this module; analysis only. Validation: single seeded sample, rolling 500-bar refits, threshold fixed at one half before measurement. Acceptance: report disagreement rate, median lag, flicker count; reject raw-state routing if median lag exceeds 10 bars or flicker exceeds 5 percent.

Your deliverable is a three-line chart plus the three numbers. Extend it once: halve the vol ratio (make the hot state 1 percent instead of 2) and watch lag and flicker degrade. That sensitivity check is the honest version of “how many states,” carried over from Module 06.

Key Takeaways

• Filtered probabilities use bars up to tonight and are the only tradable belief.
• Smoothed and full-sample Viterbi paths use future bars and are leakage specimens.
• On the seeded synthetic series, filtered and smoothed states disagree most at turning points.
• Rolling refits need label re-anchoring by fitted volatility, or the router flips meaning.
• Detection lag of several bars is normal; measure it in bars, do not eyeball it.
• Flicker (fast round trips) is the cost driver that Module 16 must tame.
• A frozen threshold and frozen refit schedule are what make the test honest.

References

• Nystrup et al., regime-switching and statistical jump models comparison, arXiv:2402.05272 — Edge evidence / adversarial: documents HMM lag and flicker and reports jump models beating HMM out-of-sample net of costs; retrieved 2026-09-17.
• Bulla et al. 2011 and related HMM timing lineage — Edge evidence: early descriptive regime-timing results without modern cost-aware out-of-sample gates; read as motivation, not proof; retrieved 2026-09-17.
• hmmlearn GaussianHMM documentation, predict_proba and score_samples — Mechanics: defines filtered-style and posterior interfaces used above; retrieved 2026-09-17.
• Hamilton 1989, regime-switching filter — Mechanics: the causal filtering ancestor of this module’s discipline; retrieved 2026-09-17.
• Exclusion log: QuantInsti and Medium-style HMM trading tutorials were rejected as evidence (no out-of-sample or cost methodology; background only).