The Learning Library
Contents

Module 05 — Degenerate Solutions: When a State Collapses

Part I · Foundations & the Inference Engine · Status: Draft v0.1 Scope: Covariance collapse, why it happens, and the mitigation checklist · Prerequisites: Modules 03, 04

Overview

Sometimes a Gaussian HMM fit looks too good. One state claims a handful of observations, its spread shrinks toward zero, and the likelihood shoots toward infinity. You have not discovered a precise new regime. You have watched a state collapse onto noise.

This module shows you that failure on purpose. You fit three states to data built with two, watch one state isolate a few points, and learn the mitigations that keep spreads honest. The pathology is classical: Bishop (2006, section 9.2.2) documents the same singularity for Gaussian mixtures, and it bites Gaussian HMMs the same way.

How It Works

A Gaussian curve rewards tight fits. When a state’s assigned points cluster very close together, shrinking the spread raises the density at those points without bound. In the limit of zero spread on a single point, the likelihood contribution goes to infinity. The optimizer is doing its job. The model specification let it cheat.

Outliers invite this cheat. A few extreme returns far from the bulk give the extra state an easy job: sit on the outliers, shrink, and collect an enormous likelihood bonus. The remaining states then fit the bulk as before, so nothing else in the score pushes back. That is why over-specified fits on heavy-tailed data collapse so reliably.

The one-dimensional intuition you can check by hand

Take eight injected spikes near 0.09 against a bulk whose spreads are 0.005 and 0.02. A state centered at 0.09 with a spread of 0.001 assigns each spike an enormous density, because every spike sits within a fraction of one spread from the center. Widen that spread to 0.02 and the same spikes sit several spreads out, where the density is orders of magnitude smaller. The optimizer sees that gap and keeps shrinking. Nothing in raw likelihood objects, which is why the guard rails in the next section must come from you.

flowchart TD
    A[Fit 3 states to 2-state data] --> B[Extra state finds outliers]
    B --> C[Spread shrinks toward zero]
    C --> D[Likelihood explodes upward]
    D --> E[Restart protocol picks the cheat]
    E --> F[Apply floor and refit]

How to read this: A through E is the failure chain, and F is the exit. Every mitigation in this module intervenes at C or earlier.

[!WARNING] Never select models by raw training likelihood alone when state counts differ. The collapsed fit almost always wins that contest, and the prize is a broken model.

How to spot the collapse in a results table

You do not need a plot to catch this failure. Print three columns per state: weight, mean, and spread. The collapsed state announces itself as the row with a weight under 0.05 and a spread an order of magnitude below the rest. If two rows look identical except that one owns almost no data, the same verdict applies: the spare state found nothing real and shrank to justify itself.

Make this table your reflex after every Gaussian fit with more than two states. It takes one print statement and it catches the most expensive silent failure in this part of the curriculum. Module 06 turns the same table into a state-count diagnostic.

Building the Collapse on Purpose

You reuse the shared synthetic series recipe from Module 03, then contaminate it slightly and overfit with three states. The contamination stands in for the real-world outliers that heavy-tailed return data always contains.

import numpy as np
from hmmlearn.hmm import GaussianHMM

# Shared recipe (Module 03): seed 42, sticky 2-state vols.
rng = np.random.default_rng(42)
n_bars = 2000
trans_mat = np.array([[0.97, 0.03], [0.08, 0.92]])
hidden = np.zeros(n_bars, dtype=int)
returns = np.zeros(n_bars)
for day in range(1, n_bars):
    hidden[day] = rng.choice(2, p=trans_mat[hidden[day - 1]])
    vol = 0.005 if hidden[day] == 0 else 0.02
    returns[day] = rng.normal(0.0, vol)
# A few injected spikes stand in for real-world outliers.
returns[rng.choice(n_bars, size=8, replace=False)] += 0.09
data = returns.reshape(-1, 1)

# Deliberate overfit: three states for two-state data, no floor.
greedy = GaussianHMM(
    n_components=3, covariance_type="diag",
    n_iter=100, tol=1e-4, random_state=7,
)
greedy.fit(data)
print("spreads:", np.round(np.sqrt(greedy.covars_.ravel()), 5))
print("weights:", np.round(greedy.get_stationary_distribution(), 3))

In illustrative runs one fitted spread lands one to two orders of magnitude below the rest while its state weight sits near zero, isolating roughly the injected spikes. Treat those magnitudes as illustrative, not as measured findings from your machine. The checkable pattern is structural: one tiny-weight state with a tiny spread plus a soaring training score.

Floors, Regularizers, and Tied Covariances

Three mitigations break the cheat, and you should try them in order. First, set a variance floor through the library’s regularizer so no spread can reach zero. In hmmlearn this is the reg_covar knob, which adds a small constant to each diagonal covariance entry.

# Same 3-state fit with a variance floor via reg_covar.
guarded = GaussianHMM(
    n_components=3, covariance_type="diag",
    n_iter=100, tol=1e-4, random_state=7, reg_covar=1e-4,
)
guarded.fit(data)
print("guarded spreads:", np.round(np.sqrt(guarded.covars_.ravel()), 5))
print("guarded score:", round(float(guarded.score(data)), 1))

Second, simplify the covariance shape. Diagonal covariances already help in one dimension, and tying covariances across states removes the extra spread parameter entirely when your regimes plausibly share one. Third, reduce the state count and let Module 06 adjudicate: a state that only survives with a near-zero spread is not a regime.

Choosing a floor value without superstition

A floor of 1e-4 on the variance means spreads below 0.01 are strongly resisted, which suits daily returns whose genuine calm spread is 0.005 in variance terms near 2.5e-5. Wait: that floor would bind on the true calm state itself, so for this series a gentler floor near 1e-6 is the honest default and 1e-4 is the stress test. The general rule is to set the floor one to two orders of magnitude below the smallest spread you would believe, then confirm your fitted spreads sit comfortably above it.

Mitigation What it does When you reach for it
Variance floor Blocks zero spreads via reg_covar Always; your default guard
Diagonal covariance Removes cross-term freedom Multivariate observations
Tied covariance One shared spread for all states Small data or similar scales
Fewer states Removes the spare state Tiny-weight state persists

Only the floor addresses the singularity directly, so it stays on even when you also simplify shapes or cut states. The shape choices reduce the optimizer’s room to maneuver, and cutting states removes the spare mouth to feed. Used together, the three layers turn collapse from a routine surprise into a rare event that your results table still catches.

[!TIP] Start every Gaussian HMM fit with reg_covar at 1e-4 or above. You can relax it later as an experiment, never as the default.

Testing Your Implementation

You verify four things. First, the unguarded 3-state fit shows the collapse signature: one state weight near zero with a spread far below the other two. Second, the guarded refit removes that signature and all spreads sit in a sane band. Third, the guarded training score is lower than the collapsed score, proving that raw likelihood rewards the cheat. Fourth, your restart spread from Module 04 narrows once the floor is on.

Add one more check that catches silent misconfiguration. Print the effective spreads as percentages and ask whether each is economically legible: a daily spread of 0.0001 percent is a data error or a collapse, never a market regime. Numbers that cannot survive that sentence do not belong in your report.

Hands-On Project

Run the collapse-and-rescue experiment and write it up as a one-page lab note. Fit the contaminated series with three states and no floor, record the spreads and weights, then refit with the floor and record the same.

  1. Run the collapse snippet and save the three spreads and weights.
  2. Run the guarded snippet with two floor values, 1e-6 and 1e-4, and compare.
  3. Try covariance_type values diag and tied and note which kills the tiny state.
  4. Write your mitigation checklist settings for the rest of the curriculum.
  5. Rerun the unguarded fit under three extra seeds and count how often the collapse lands on the same observations. A collapse that jumps between spike groups is opportunistic, and that sentence belongs in your lab note.

File the lab note where you will actually reread it. Module 06 will hand you extra states that look respectable, and the collapse signature table from this module is the fastest way to unmask the ones that are not.

One last habit separates professionals from tutorial readers. Every time you publish a fitted Gaussian HMM, state the floor value beside the spreads, exactly as you state the seed beside the score. A spread without its floor is an uninterpretable number, because the reader cannot tell a defended estimate from an undefended one.

Key Takeaways

  • A Gaussian state can drive its spread to zero and its likelihood to infinity.
  • Outliers plus spare states are the standard trigger for collapse.
  • The signature is a near-zero weight state with a near-zero spread.
  • Raw training likelihood rewards the cheat, so never select on it alone.
  • Variance floors via reg_covar are your default guard.
  • Diagonal or tied covariances remove freedom the cheat needs.
  • Bishop (2006, section 9.2.2) documents the same singularity for mixtures.

References