Module 12 — Speech and Audio: End-to-End Won Recognition; Alignment Kept the HMM

Part III · Modern Practice Across Domains · Status: Draft v0.1 Scope: From HMM-GMM recognition to end-to-end ASR and surviving alignment tools · Verdict: legacy-but-alive · Prerequisites: Module 00

Overview

A forced aligner takes audio plus its known transcript and returns when each phone or word occurs. This is a different job from recognition, which must guess the words from audio alone. The HMM lost the guessing job and kept the timing job, and this module shows you why both outcomes were rational.

Recognition moved in three steps: HMM-GMM systems, then hybrid HMM-DNN systems, then end-to-end models built on CTC, RNN-T, and self-supervised pretraining such as wav2vec 2.0 (Baevski et al., arXiv:2006.11477). What survived is alignment tooling: the Montreal Forced Aligner, which trains Kaldi-style HMM-GMM models under the hood (MFA docs v3.4.1, retrieved 2026-09-17), and k2/icefall, the next-generation Kaldi ecosystem with differentiable weighted finite-state transducers (project sites, retrieved 2026-09-17).

In this module you trace the arc, meet the two surviving tools, and run a toy forced alignment. You learn the structural reason one task demanded neural accuracy at scale while the other only needed cheap, extensible alignment.

Keep the two tasks separated in your head at all times. Every confused argument about HMMs in speech mixes them up: someone cites aligner vitality to defend recognizers, or cites recognizer defeat to dismiss aligners. This module’s verdict covers each task separately, and so should you.

How It Works

Recognition asks which word sequence best explains the acoustics, with no transcript to lean on. Alignment asks where each unit of a given transcript sits in time, which constrains the search enormously. The known transcript turns an open vocabulary problem into a narrow timing problem, and narrow problems admit simpler machinery.

How to read this: boxes are modeling eras in time order, arrows show succession, and the lower branch shows the alignment niche that stayed HMM-based after recognition moved on.

flowchart TD
    G[HMM-GMM systems] --> H[Hybrid HMM-DNN]
    H --> E[End-to-end CTC RNN-T]
    E --> W[wav2vec 2.0 pretraining]
    G --> A[Forced alignment niche]
    H --> A
    A --> M[MFA Kaldi HMM-GMM]
    A --> K[k2 icefall differentiable FST]

The top row is the accuracy race that neural models won. The lower branch is the timing task where transcript constraints do most of the work. Notice the arrows into the niche come from both HMM eras: alignment inherited the tooling without needing the accuracy race to continue.

[!NOTE] A weighted finite-state transducer (FST) is a graph that maps input sequences to output sequences with scored arcs. Decoding speech means finding the best path through composed graphs for acoustics, pronunciation, and language (project documentation, retrieved 2026-09-17).

From HMM-GMM to End-to-End: How Recognition Was Lost

The HMM-GMM era modeled each phone state with a Gaussian mixture over acoustic features and decoded with Viterbi search through pronunciation and language graphs. It was principled, trainable on modest data, and genuinely productive for decades. Hybrid HMM-DNN systems then replaced the mixture likelihoods with neural network posteriors and bought a large accuracy jump while keeping the HMM skeleton.

End-to-end training removed the skeleton too. CTC and RNN-T train sequence models without frame-level labels or explicit HMM states, and self-supervised pretraining in the wav2vec 2.0 style (arXiv:2006.11477) learns representations from raw audio before any transcription is seen. At industrial data scale this combination beat every HMM-based recognizer on accuracy, and accuracy is the only metric the recognition market pays for.

Pretraining deserves a second look because it changed the data economics. Earlier systems needed transcribed audio for everything; wav2vec-style pretraining learns from raw audio first and needs far less transcription to reach production accuracy. That removed the one constraint, label scarcity, that had historically protected simpler models.

You should not read this as HMMs being “wrong.” Gaussian mixtures were simply a weaker acoustic model than deep networks, and frame-state scaffolding was a crutch that labels plus compute made unnecessary. When abundant labeled audio meets a metric that rewards raw accuracy, the flexible model wins. That is the whole obituary.

The hybrid stage deserves emphasis because it isolates the cause of death. Hybrids kept the HMM skeleton and only swapped mixtures for networks, yet gained most of the accuracy jump. That tells you the mixtures were the binding constraint, not the Markov assumptions. End-to-end training then removed scaffolding that better models no longer needed.

What Survived: Forced Alignment with MFA and k2

The Montreal Forced Aligner trains speaker-adapted HMM-GMM acoustic models in the Kaldi tradition and uses them to align audio to transcripts (MFA docs v3.4.1, retrieved 2026-09-17). It survives because alignment for a new language or dialect needs trainable models on small corpora, not a giant pretrained recognizer. A linguist with a few hours of transcribed field recordings can train an aligner; they cannot train wav2vec 2.0.

The k2 toolkit and its icefall recipes are the next-generation Kaldi ecosystem: differentiable finite-state acceptors and transducers with CTC and LF-MMI training (project sites, retrieved 2026-09-17). k2 keeps the graph-based alignment machinery but makes it differentiable and GPU-friendly. Think of it as the alignment tradition modernized rather than replaced.

The practical consequence is a two-tier world you should navigate deliberately. Working linguists aligning field recordings live in the MFA tier, where training on hours of audio just works. Researchers building new sequence pipelines live in the k2 tier, where alignment graphs compose with neural training. Both tiers are alive, documented, and dated, which is more than the recognition side can say for any HMM method.

The toy below is forced alignment in miniature: a fixed three-phone transcript decoded over five frames, where each frame carries a cost per phone. Real MFA learns those costs from audio; your toy takes them as given.

# Toy forced alignment: transcript SIL AH T over 5 frames.
transcript = ["SIL", "AH", "T"]
frame_cost = [  # costs stand in for neg log emissions
    {"SIL": 0.1, "AH": 2.0, "T": 3.0},
    {"SIL": 0.5, "AH": 1.0, "T": 3.0},
    {"SIL": 2.0, "AH": 0.2, "T": 2.0},
    {"SIL": 3.0, "AH": 1.5, "T": 0.3},
    {"SIL": 3.0, "AH": 2.0, "T": 0.1},
]

def align_topology(costs, script):
    # Stay on current phone or advance; order fixed.
    best = {0: (costs[0][script[0]], [script[0]])}
    for frame in costs[1:]:
        nxt = {}
        for pos, (total, path) in best.items():
            for advance in (0, 1):
                new = min(pos + advance, len(script) - 1)
                cand = total + frame[script[new]]
                if new not in nxt or cand < nxt[new][0]:
                    nxt[new] = (cand, path + [script[new]])
        best = nxt
    return best[max(best)][1]
print(align_topology(frame_cost, transcript))

['SIL', 'SIL', 'AH', 'T', 'T']

The output shows early frames preferring silence, middle frames preferring AH, and late frames preferring T. Constraining the path to transcript order is what makes alignment cheap; recognition cannot assume the order and must search it.

[!TIP] If you need timings for a new corpus, reach for MFA first and read its v3.4.1 docs on training custom acoustic models. If you need differentiable graph machinery inside a neural pipeline, look at k2/icefall recipes instead.

Why It Survives or Died

Speech recognition demanded accuracy at industrial scale, where labeled data and compute let neural models win outright, while forced alignment needed cheap, language-extensible, trainable timing on small corpora, where Kaldi-style HMM tooling was already good enough and no accuracy race funds its replacement.

Testing It Honestly

Re-verify the verdict with three dated checks. First, open the MFA documentation channel and confirm maintained releases around v3.4.1 describe Kaldi-style HMM-GMM training (retrieved 2026-09-17 here; use your own date). Second, open the k2 and icefall project sites and confirm active recipes using differentiable FST/FSA machinery with CTC or LF-MMI training. Third, look for any dated production announcement of an HMM-based recognizer beating end-to-end systems on a current benchmark; this curriculum’s 2026-09-17 pass found none, which is what keeps the flagship task historical and the niche alive.

Watch for the common exaggeration in both directions. “HMMs are dead in speech” ignores maintained aligners linguists use daily. “HMMs still compete in ASR” confuses hybrid history with present production. The verdict is legacy-but-alive precisely because both slogans are wrong.

A further check separates users from spectators. Search linguistics paper methods sections for MFA citations with dates: applied papers that actually aligned corpora are production evidence for the niche. This curriculum’s 2026-09-17 pass found that pattern holding, which confirms the niche is worked daily rather than merely maintained. Date every check you run so your verdict carries its own expiry.

File this module’s method for reuse: the three dated checks above are the template you will apply to any speech tool you meet later, including ones this curriculum never surveyed.

Hands-On Project

Align a one-minute recording of your own voice reading a ten-word sentence using MFA’s pretrained path, then retrain or adapt on three repetitions and compare boundary shifts. Reproduction card: data source is your own recording (note microphone and sample rate); tool version is the MFA release you installed (record it, expected v3.4.1 family); random seed is fixed where the docs expose one; expected output is word boundaries within about 50 ms across runs on clean audio.

Then run the toy aligner above on perturbed costs and find the perturbation size that flips a boundary. Write one paragraph connecting the two experiments: why does transcript constraint make both the real tool and the toy robust to cost noise that would destroy an open-vocabulary recognizer? Your deliverable is the MFA output, the toy experiment, and the paragraph.

If MFA installation fights you, the fallback still teaches the lesson. Run the toy on three hand-made cost matrices (clean, noisy, adversarial) and record where each boundary lands. The deliverable is the same paragraph, now grounded in controlled noise instead of real audio, and the conclusion about transcript constraint survives the substitution.

Key Takeaways

• Speech split into two tasks: open-vocabulary recognition and transcript-constrained alignment.
• The recognition arc ran HMM-GMM to hybrid HMM-DNN to CTC/RNN-T and wav2vec 2.0 (arXiv:2006.11477).
• End-to-end models won recognition because accuracy at scale is the only metric that market pays for.
• MFA survives on Kaldi-style HMM-GMM training for low-resource alignment (docs v3.4.1, retrieved 2026-09-17).
• k2/icefall modernize graph-based alignment with differentiable FST/FSA and CTC/LF-MMI (project sites, retrieved 2026-09-17).
• Transcript constraint is why alignment stays cheap while recognition stays neural.
• No dated production evidence shows HMM recognizers competing today, so the flagship task is not alive.
• Re-verify with release channels plus one honest search for counter-evidence.

References

• Baevski et al., “wav2vec 2.0: A framework for self-supervised learning of speech representations,” arXiv:2006.11477 (Survey; end-to-end era marker).
• Montreal Forced Aligner documentation, v3.4.1 release channel (Survey; retrieved 2026-09-17).
• k2 and icefall project sites and recipe collections (Survey; retrieved 2026-09-17).
• Rabiner, “A tutorial on hidden Markov models,” Proceedings of the IEEE 77(2) 1989 (Mechanics; HMM-GMM decoding background).
• Exclusion log: no tertiary tutorial was admitted as status evidence; recognition claims rest on the paper and project channels above.