Module 09 — statsmodels Markov Switching: The Econometrics Sibling

Part II · Tooling & Evaluation · Status: Draft v0.1 Scope: Switching regressions, MS-AR, and the FRED recession example · Prerequisites: Modules 03, 07

Overview

hmmlearn treats each observation as a draw from a state-specific curve. Econometrics often wants more: the regime can shift the regression line itself, the variance, or the dynamics linking today to yesterday. That is the Markov-switching family, and statsmodels implements it.

You fit a switching-variance model to the shared synthetic series and recover the two volatility levels from inside the econometrics interface. You then learn where Markov-switching autoregression differs from a plain Gaussian HMM. You close with the institutional example: the St. Louis Fed’s published smoothed recession probabilities.

How It Works

statsmodels puts switching models under tsa.regime_switching. MarkovRegression switches the intercept, slope coefficients, or variance of a regression across hidden states. MarkovAutoregression adds regime-dependent dependence on past values. Estimation runs the Hamilton filter forward and the Kim smoother backward, which are the econometrics names for the forward pass and forward-backward smoothing you already know.

The vocabulary maps one to one. Filtered probabilities use data up to today and match the scaled forward pass. Smoothed probabilities use the whole sample and match forward-backward.

Predicted and forecasted probabilities step the transition matrix forward without new data. Only filtered quantities are causal; smoothed ones peek at the future.

flowchart TD
    A[Pick switching spec] --> B[Hamilton filter forward]
    B --> C[Filtered prob today]
    C --> D[Kim smoother backward]
    D --> E[Smoothed prob full sample]
    C --> F[Causal use only]
    E --> G[History description only]

How to read this: the left branch C-to-F is what live decisions may use. The right branch E-to-G describes history after the fact and must never enter a performance claim.

Switching Variance on the Shared Series

The shared series has zero mean in both states and differs only in spread, so the honest spec switches the variance while holding the mean fixed. That is switching_variance=True with switching_mean disabled (the default constant-only regression has no switching intercept unless you ask for it).

The snippet below regenerates the Module 03 series and fits a two-regime switching-variance model. Residual standard errors per regime are the econometrics name for the two fitted spreads, so you compare them against 0.005 and 0.02 exactly as before.

import numpy as np
from statsmodels.tsa.regime_switching import MarkovRegression

# Shared series: seed 42, 2000 bars, vols 0.005 and 0.02.
rng = np.random.default_rng(42)
n_bars = 2000
trans_true = np.array([[0.97, 0.03], [0.08, 0.92]])
state_vol = np.array([0.005, 0.02])
hidden = np.zeros(n_bars, dtype=int)
returns = np.zeros(n_bars)
for day in range(1, n_bars):
    hidden[day] = rng.choice(2, p=trans_true[hidden[day - 1]])
    returns[day] = rng.normal(0.0, state_vol[hidden[day]])

# k_regimes=2 with switching variance recovers the two spreads.
ms_model = MarkovRegression(
    returns, k_regimes=2, switching_variance=True,
)
ms_fit = ms_model.fit(em_iter=50, search_reps=3, disp=False)
print(ms_fit.summary())
print("regime sigmas:", np.sort(ms_fit.sigma2**0.5))

How to read the output: sort the two sigmas before comparing, for the same label-switching reason as hmmlearn. Expect one near 0.005 and one near 0.02. The search_reps argument runs several EM starts and keeps the best, which is the multi-restart discipline from Module 04 expressed as one keyword.

Filtered state probabilities come from ms_fit.filtered_marginal_probabilities and smoothed ones from ms_fit.smoothed_marginal_probabilities. Plot both and confirm the filtered line lags the smoothed line at turning points. That visible lag is the cost of causality, and Part IV charges you for it.

[!TIP] Start every switching spec with the plainest version that can express the effect: variance-only switching here. Add switching means or slopes only when the plain spec demonstrably fails.

MS-AR: Where the Regime Lives

A plain Gaussian HMM draws each return from a fixed per-state curve, so regimes differ only in level or spread. A Markov-switching autoregression lets the regime change the dynamics: today’s mean depends on yesterday’s value with a slope that itself switches.

Use MS-AR when calm and stressed markets differ in persistence or reversal rather than just noisiness. A calm drift state has a slope near one on yesterday’s level; a snapping-back stress state has a smaller or negative slope. statsmodels expresses this as MarkovAutoregression with an order and a switching_ar flag.

The practical consequence is estimation cost. Each extra switching slope multiplies the parameter count and the local-optima surface, so MS-AR needs more data and more starts than a vol-switching fit on the same series. Do not reach for it until the plain switching-variance residuals show leftover autocorrelation inside regimes.

[!WARNING] Smoothed MS-AR state plots look deceptively clean because each point borrows future information. Show them as history description, and evaluate decisions only on filtered probabilities.

The Institutional Example: FRED Recession Probabilities

The St. Louis Fed publishes smoothed US recession probabilities as FRED series RECPROUSM156N, maintained in the lineage of the Chauvet (1998) Markov-switching model of the business cycle. Each observation is the model’s probability that the economy was in recession in that month, estimated from the full sample available at publication.

This example earns its place for two reasons. First, it proves Markov switching is production econometrics infrastructure, not a teaching toy: a central data institution ships its output monthly. Second, it demonstrates the honest labeling this curriculum demands. The series is smoothed, so it describes history with hindsight; the Fed presents it as a dating device, not as a tradeable forecast.

# Sketch: fetch and plot the published smoothed series.
# Replace the CSV path with your local FRED download.
import numpy as np

# FRED series RECPROUSM156N: smoothed recession probability.
fred = np.genfromtxt("RECPROUSM156N.csv", delimiter=",",
                     names=True, dtype=None, encoding="utf-8")
print("rows:", len(fred))
print("columns:", fred.dtype.names)
# Check: values in [0, 1]; spikes align with known recessions.

Read the series with causal discipline. Peaks near one mark months the model later classified as recession, after seeing what came next. A live recession call would need the filtered counterpart estimated with data up to that month only, and it would lag the smoothed peaks. That lag is not a flaw in the model; it is the price of not peeking.

[!NOTE] Trading on smoothed probabilities is lookahead by construction. Part IV excludes smoothed states from every performance number for exactly this reason.

Testing It Honestly

Pin statsmodels alongside NumPy and record the retrieval date, because optimizer defaults and starting-value routines move across releases. Seed the data generator with 42 and record search_reps and em_iter, since switching fits are restart-sensitive by design.

Verify four outputs on the shared series: convergence reported by the fitter, sorted sigmas straddling 0.01 (one below, one above 0.012), a stressed-regime weight clearly between zero and one, and filtered probabilities that lag smoothed ones at transitions. A fit that matches sigmas but assigns 99 percent of days to one regime has described the average, not the switching.

Your reproduction card mirrors Module 07 with one addition: the exact switching spec (which coefficients switch, the AR order, the number of search repetitions). Without the spec, the card cannot reproduce the fit.

One last discipline carries into Part IV. Whenever you plot switching states for a market audience, label the panel filtered or smoothed in the title itself, not in a footnote. Readers skim figures first, and an unlabeled smoothed panel will be misread as a tradeable signal.

Hands-On Project

Fit the econometrics sibling and compare it against the hmmlearn baseline.

1. Regenerate the shared series and fit the switching-variance snippet above.
2. Record sorted sigmas, the converged flag, and filtered versus smoothed plots.
3. Fit hmmlearn GaussianHMM from Module 07 on the same array and compare spreads.
4. Download RECPROUSM156N from FRED, plot it, and mark where a filtered estimate would lag each peak.
5. Write the reproduction card including the full switching spec.

Success means the two libraries agree within a few basis points on the shared series, and you can state in one sentence why the FRED series is description rather than signal.

Stretch goal once the baseline matches: fit a switching-mean variant and watch it chase noise, since the true means are both zero. That failed fit previews Module 06’s warning about states absorbing lack of fit, this time inside the econometrics interface.

Key Takeaways

• MarkovRegression switches intercepts, slopes, or variance; MarkovAutoregression adds regime-dependent dynamics.
• Hamilton filter means forward pass; Kim smoother means forward-backward smoothing.
• Switching variance recovers the shared series spreads near 0.005 and 0.02.
• MS-AR puts the regime in the dynamics, at the cost of more parameters and starts.
• FRED RECPROUSM156N shows Markov switching as maintained institutional infrastructure.
• Smoothed probabilities describe history; only filtered probabilities may inform decisions.
• Every switching fit ships with its full spec on the reproduction card.

References

• Hamilton, J. D. (1989). A new approach to the economic analysis of nonstationary time series. Econometrica, 57(2). https://www.jstor.org/stable/1912559 — role: Mechanics. Retrieved 2026-09-17.
• statsmodels developers. Markov switching models documentation (MarkovRegression, MarkovAutoregression, Hamilton filter, Kim smoother). https://www.statsmodels.org/stable/regime_switching.html — role: Mechanics. Retrieved 2026-09-17.
• Chauvet, M. (1998). An econometric characterization of business cycle dynamics with factor structure and regime switching. Int. Economic Review, 39(4). https://www.jstor.org/stable/2527348 — role: Mechanics. Retrieved 2026-09-17.
• Federal Reserve Bank of St. Louis. FRED series RECPROUSM156N (smoothed US recession probabilities). https://fred.stlouisfed.org/series/RECPROUSM156N — role: Mechanics. Retrieved 2026-09-17.
• Exclusion: the Scribd upload of the Baum et al. paper — rejected as a secondary copy; Project Euclid and journal sources govern origination claims.