The Learning Library
Contents

Module 06 — How Many States?

Part I · Foundations & the Inference Engine · Status: Draft v0.1 Scope: AIC/BIC fitting on the shared series and a pragmatic selection protocol · Prerequisites: Modules 03, 04, 05

Overview

Every HMM fit starts with a choice you cannot dodge: how many hidden states. Too few and you smear distinct behaviors into one blur. Too many and the spare states memorize outliers, as Module 05 showed.

This module fits one through five Gaussian states to the shared synthetic series, whose truth you know is two. You compute real information criteria, watch them disagree, and learn why. The headline comes from Pohle et al. (2017): extra states often absorb misspecification and outliers, so treat them as lack-of-fit diagnostics, not discoveries.

How It Works

Information criteria trade fit against complexity. Each one starts from the training log-likelihood, then subtracts a penalty that grows with the parameter count. Akaike’s criterion penalizes lightly and tends to keep extra states. The Bayesian criterion penalizes with the log of the sample size and tends toward fewer states.

Count the parameters before you trust either number. A Gaussian HMM with K states in one dimension carries roughly K start values, K-by-K transitions, plus K means and K spreads. Every added state buys a full new row of transitions plus its own curve, so the penalty must earn its keep. When the criteria disagree, they are telling you the fit-complexity trade has no clean winner.

A concrete parameter count

Walk through K=2 versus K=3 to feel the penalty. At two states you estimate about 4 transitions plus 2 means plus 2 spreads, near 8 effective parameters before start values. At three states that rises to about 9 transitions plus 3 means plus 3 spreads, near 15. On 2000 bars the Bayesian penalty per parameter is the log of 2000, about 7.6, so the third state must buy roughly 50 points of log-likelihood just to break even. Small likelihood gains past K=2 therefore fail the Bayesian bar by construction.

flowchart TD
    A[Fit K 1 to 5 with restarts] --> B[Score AIC and BIC]
    B --> C{Criteria agree on K?}
    C -->|Yes| D[Check interpretability]
    C -->|No| E[Inspect extra states]
    E --> F[Outlier sponge or real regime?]
    F --> D
    D --> G[Confirm out of sample]

How to read this: A-B is computation, C-F is diagnosis, and G is the gate that decides.

[!NOTE] Likelihood is not prediction quality. A higher training score can mean a better model or a better cheat. Module 05 taught you to tell them apart.

Fitting K=1..5 on the Shared Series

You rebuild the shared synthetic series from Module 03 without contamination, then fit one through five states with restarts and print both criteria. The parameter count drives the penalty term in the snippet.

import numpy as np
from hmmlearn.hmm import GaussianHMM

# Shared series recipe (Module 03): seed 42, ~2000 bars, true K=2.
rng = np.random.default_rng(42)
n_bars = 2000
trans_mat = np.array([[0.97, 0.03], [0.08, 0.92]])
hidden = np.zeros(n_bars, dtype=int)
returns = np.zeros(n_bars)
for day in range(1, n_bars):
    hidden[day] = rng.choice(2, p=trans_mat[hidden[day - 1]])
    vol = 0.005 if hidden[day] == 0 else 0.02
    returns[day] = rng.normal(0.0, vol)
data = returns.reshape(-1, 1)

for n_states in [1, 2, 3, 4, 5]:
    fit = GaussianHMM(
        n_components=n_states, covariance_type="diag",
        n_iter=100, tol=1e-4, random_state=42, reg_covar=1e-4,
    )
    fit.fit(data)
    score = float(fit.score(data))
    # Free parameters: transitions + means + variances (+ starts).
    n_params = n_states * (n_states + 2)
    n_points = len(data)
    aic = 2 * n_params - 2 * score
    bic = np.log(n_points) * n_params - 2 * score
    print(n_states, round(aic, 1), round(bic, 1))

Run this yourself; the printed numbers are your real evidence for this dataset. In illustrative runs the Bayesian criterion bottoms out at two states while Akaike keeps improving toward three or four, and scores beyond the truth improve only marginally. Treat that pattern as illustrative, not as a measured finding from your machine. Your checkable claim is the shape: a large jump from one to two states, then flattening gains with disagreement at the top.

Reading the printed row like a reviewer

Each printed row carries three facts: the state count, the Akaike value, and the Bayesian value. Lower wins within each column, but the columns disagree on purpose because their penalties differ. Your review sentence should name both winners explicitly, for example that the Bayesian column favors two states while the Akaike column favors three. A memo that reports only the friendlier column is hiding the disagreement your reader needs to see.

Why Extra States Absorb Misspecification

Pohle et al. (2017) give the mechanism behind the disagreement. Real data always violates the model somewhere: heavier tails than Gaussian, lingering autocorrelation, a few wild days. Each spare state soaks up one violation. One becomes the outlier sponge, another becomes the slow-drift sponge, and the criteria reward the better fit even though no new behavior was discovered.

Diagnose before you celebrate. Inspect each extra state’s weight, spread, and assigned days. A state with near-zero weight and a handful of scattered days is the Module 05 collapse wearing a better score. A state that owns long contiguous blocks with a distinct spread may be real structure worth keeping. Contiguity plus interpretability separates regimes from sponges.

The two questions that settle most disputes

First, does the extra state persist across restarts with different seeds. A sponge appears under some seeds and vanishes under others, while a genuine regime reappears every time with a similar spread and weight. Second, does the extra state survive the variance floor from Module 05. A state that exists only when spreads may collapse is a numerical artifact, and no criterion value rehabilitates it.

Signal Sponge pattern Real-regime pattern
State weight Near zero Substantial share
Assigned days Scattered spikes Long blocks
Spread Extreme or tiny Distinct but sane
Restart stability Appears and vanishes Persists across seeds

[!IMPORTANT] An extra state is guilty until proven innocent. Demand weight, contiguity, restart stability, and a story you can state in one sentence.

The pragmatic selection protocol in full

You now own every step, so here it is as one checklist. Fit each candidate state count with multiple restarts and keep the best score per count. Compute both criteria from those winners and note where they disagree. Inspect every extra state’s weight, contiguity, and restart stability with the table above. Confirm the surviving candidate on held-out bars. The state count that passes all four gates is your answer, and a count that fails any gate is a finding you report, not a failure you bury.

Testing Your Implementation

You verify four things. First, the one-state fit lands its single spread between the two true values, reproducing your Module 03 misfit note. Second, the two-state fit recovers sorted spreads near 0.005 and 0.02. Third, gains beyond two states are small relative to the one-to-two jump. Fourth, at least one higher-K fit shows a sponge signature under the table above.

Add a restart-stability check to make the verdict stick. Refit the K=3 model under three different seeds and confirm whether the third state’s spread and weight reproduce or wander. A wandering third state is the strongest evidence that K=2 is your honest answer for this series.

Hands-On Project

Produce the selection memo you would defend in review. Fit K=1..5 with three seeds each, keep the best score per K, and compute both criteria from the best runs.

  1. Record AIC and BIC per K from your own machine into a small table.
  2. Label each state in the K=3 and K=4 winners as sponge or candidate regime.
  3. Hold out the final 400 bars, score each fitted model there, and note whether the winner changes.
  4. Write your verdict in three sentences: chosen K, why the criteria disagreed, and what out-of-sample check would overturn you.
  5. Repeat the K=3 fit with the variance floor removed and note whether the third state turns into a Module 05 collapse. If it does, your selection memo just gained its strongest paragraph.

Bring this memo to Part II. The diagnostics there give you posterior decoding and residual checks that either corroborate your chosen K or reopen the case, and a written memo makes that update a revision instead of a restart.

Carry one sentence from this module into every future selection debate. The criteria advise, the diagnostics corroborate, but only restart-stable, interpretable, out-of-sample-checked states survive. Everything else is a sponge with a good score.

Key Takeaways

  • AIC penalizes lightly and favors richer models; BIC penalizes harder.
  • On the shared series the truth is two states; criteria often disagree above it.
  • Extra states absorb outliers and misspecification rather than discovering regimes.
  • Diagnose spare states by weight, contiguity, spread sanity, and restart stability.
  • The pragmatic protocol is restarts, interpretability, then out-of-sample.
  • A failed selection is a finding; report it instead of hiding it.
  • Pohle et al. (2017) is the anchor citation for states-as-diagnostics.

References