The Learning Library
Contents

Module 10 — Diagnostics: Is Your HMM Actually Working

Part II · Tooling & Evaluation · Status: Draft v0.1 Scope: Decoding choices, residual checks, and honest validation · Prerequisites: Modules 04, 06, 07

Overview

A fitted HMM always produces states, probabilities, and a likelihood. None of those numbers certifies that the model works. This module gives you the three checks that do: decode with the right objective, inspect residuals for leftover structure, and validate prediction out of sample with costs and causality respected.

You run all three on the shared synthetic series. You see Viterbi and posterior decoding disagree on purpose, pseudo-residuals expose a deliberately overfit model, and a four-state fit beats a two-state fit in sample while losing one step ahead. That last result is the chapter’s moral: likelihood is not prediction quality.

How It Works

Diagnostics operate at three levels. Decoding asks which state path to report for a fitted model. Residuals ask whether the fitted model left predictable structure on the table. Validation asks whether the model predicts fresh bars it never trained on, under causal discipline.

Each level has its own failure mode. Wrong-objective decoding reports a path optimized for the wrong question. Skipped residual checks bless a model whose emissions miss the data shape. In-sample validation with smoothed states and no costs manufactures edge that evaporates live. You address them in that order.

flowchart TD
    A[Fitted HMM] --> B[Decode with right objective]
    B --> C[Check pseudo-residuals]
    C --> D[Validate one step ahead]
    D --> E[Causal OOS verdict]
    B --> F[Viterbi or posterior]
    C --> G[Normal-looking noise?]
    D --> H[K=2 vs K=4 live]

How to read this: the spine A-B-C-D-E is the mandatory sequence. The side branches name the decision each step forces, and you do not advance past a failed check by relabeling it.

Viterbi Paths Versus Posterior Probabilities

Decoding answers two different questions, and Rabiner (1989) defines both. The Viterbi path is the single jointly most likely state sequence: the best whole story. The posterior (filtered or smoothed) probabilities give the most likely state at each time step individually: the best per-day call. Optimizing one does not optimize the other.

The distinction bites at turning points. The per-day posterior can flip to stress on an isolated wild return, because that day alone looks stressed. The Viterbi path can keep the calm label through the same day, because a one-day round trip pays two unlikely transitions and the joint story prefers continuity. Neither is wrong; they optimize different objectives.

import numpy as np
from hmmlearn.hmm import GaussianHMM

# Shared series setup, identical to Module 07.
rng = np.random.default_rng(42)
n_bars = 2000
trans_true = np.array([[0.97, 0.03], [0.08, 0.92]])
state_vol = np.array([0.005, 0.02])
hidden = np.zeros(n_bars, dtype=int)
returns = np.zeros((n_bars, 1))
for day in range(1, n_bars):
    hidden[day] = rng.choice(2, p=trans_true[hidden[day - 1]])
    returns[day, 0] = rng.normal(0.0, state_vol[hidden[day]])
model = GaussianHMM(
    n_components=2, covariance_type="diag", n_iter=100,
    tol=1e-4, random_state=42, reg_covar=1e-4,
).fit(returns)

# Two objectives on the same fit: joint path vs per-day call.
viterbi_path = model.predict(returns)
posterior = model.predict_proba(returns)
post_call = posterior.argmax(axis=1)
print("disagreement days:", int((viterbi_path != post_call).sum()))
print("disagreement rate:", round(
    float((viterbi_path != post_call).mean()), 4))

Expect a small but nonzero disagreement concentrated near regime boundaries. If the two agreed everywhere, one of them would be redundant. Report which decoder you used every time you show states: Viterbi for the single best path, posteriors for per-day uncertainty, and never smoothed quantities inside a live-decision claim.

[!WARNING] Smoothed posteriors and Viterbi paths computed on the full sample peek at the future. They describe history. Only filtered probabilities may enter a performance number.

Residuals and the Overfit Demonstration

A well-fit model should leave nothing predictable behind. Pseudo-residuals, introduced for HMMs in Zucchini, MacDonald and Langrock (2016, 2nd ed), turn each observation into a uniform-then-normal score given its past: roughly, where today’s return falls inside the model’s own one-step-ahead forecast distribution. Under a correct model those scores look like plain noise.

The implementation sketch is short. For each day, compute the forecast distribution from filtered state probabilities times emission densities, evaluate today’s return inside it to get a uniform score, then map that score through the standard normal quantile. Plot the resulting series and its histogram: a good fit shows no trend, no volatility clustering, and a bell-shaped histogram.

# Seeded demo: K=4 overfits in sample, loses one step ahead.
from scipy.stats import norm

# Split causally: first 1500 bars train, last 500 test.
train, test = returns[:1500], returns[1500:]
fit2 = GaussianHMM(
    n_components=2, covariance_type="diag", n_iter=100,
    tol=1e-4, random_state=42, reg_covar=1e-4,
).fit(train)
fit4 = GaussianHMM(
    n_components=4, covariance_type="diag", n_iter=100,
    tol=1e-4, random_state=42, reg_covar=1e-4,
).fit(train)
print("train K=2:", round(float(fit2.score(train)), 1))
print("train K=4:", round(float(fit4.score(train)), 1))
print("test K=2:", round(float(fit2.score(test)), 1))
print("test K=4:", round(float(fit4.score(test)), 1))

Read the four numbers as the chapter’s central exhibit. The four-state fit should score higher on the training bars and at or below the two-state fit on the held-out bars: extra states memorized training noise and paid for it live. That is the likelihood-versus-prediction gap in one seeded comparison, and it is why Module 06 refuses to pick state counts by in-sample likelihood.

[!NOTE] This comparison is causal: both models train on bars the test never enters. Any version using full-sample fits or smoothed states would leak and prove nothing.

Cross-Validation and Refit Instability

Standard cross-validation breaks twice on HMMs. First, labels permute across folds: state 0 in fold one can be state 1 in fold two, so averaging per-state parameters across folds without alignment is meaningless. Align by sorted spread (or an assignment match) before any comparison, or compare fold likelihoods rather than parameters.

Second, likelihoods are comparable only at fixed state counts. A K=4 fold score and a K=2 fold score live on different model sizes, and the larger model wins by capacity before it wins by skill. Compare within K across folds for stability, and across K only with a penalized or held-out criterion from Module 06.

The rolling-window refit is the diagnostic practitioners actually run. Fit K=2 on windows ending at successive dates, align labels by spread, and track the two spreads plus the transition diagonal across windows. Stable stretches with occasional jumps describe markets honestly; spreads that swing wildly window to window describe an unidentified model.

Check Do this Failure signal
Label alignment Sort by spread first Parameters “jump” spuriously
K comparison Held-out or penalized only Bigger K always wins
Refit stability Rolling windows, one K Spreads swing wildly

[!IMPORTANT] Any validation that trains on future bars, decodes with smoothed states, or omits transaction costs is a motivation sketch, not evidence. Part IV enforces the causal, cost-inclusive version.

Testing It Honestly

Your honesty checklist for this module has five lines. Decoder named (Viterbi versus posterior, filtered versus smoothed). Residual plot attached with the normal-quantile mapping stated. Train/test split with dates and no overlap. Per-K likelihoods reported separately for train and test. Reproduction card with data recipe, library version, and seed 42.

The expected checkable outputs on the shared series: nonzero but small Viterbi-versus-posterior disagreement near boundaries, K=4 training likelihood above K=2 with test likelihood at or below, and rolling K=2 spreads that stay near 0.005 and 0.02 without wild swings. Miss all three and the code is suspect; hit all three and the diagnostics are working.

Hands-On Project

Run the full diagnostic battery on the shared series and write the verdict.

  1. Fit K=2 with seed 42 and record Viterbi-versus-posterior disagreement days.
  2. Build pseudo-residuals for the K=2 fit and check the noise-plot behavior.
  3. Run the K=2 versus K=4 train/test comparison and record all four likelihoods.
  4. Refit K=2 on three rolling windows; align by spread and record spread drift.
  5. Write one paragraph: which decoder your next analysis uses and why, plus what the K=4 result taught you.

Keep the four likelihoods on the reproduction card. They are the smallest honest exhibit in the curriculum: proof that you can make likelihood rise while prediction gets worse.

Key Takeaways

  • Viterbi optimizes the joint path; posteriors optimize per-day calls; report which you used.
  • Pseudo-residuals map each bar through its own forecast distribution; leftover structure means misfit.
  • In-sample likelihood can rise while one-step-ahead prediction falls; the K=4 demo proves it.
  • Cross-validation needs label alignment, and likelihoods compare only within fixed K.
  • Rolling refits separate honest regime drift from identification failure.
  • Smoothed states and cost-free curves never enter a performance claim.
  • Decoder, residuals, split, per-K scores, and seeds all go on the card.

References