Module 20 — Deep Learning for Trading

Part IV · Machine Learning · Priority 🎯 Core · Status: Draft v0.1 Prerequisites: Module 17, Module 18, Module 19

Overview

Deep learning adds a family of models that learn their own features from raw sequences — stacks of neural layers that turn bars and windows into representations no hand-written indicator captures. Where they earn their keep, the mechanism is representation: compressing noisy, high-dimensional streams into a few informative numbers. Your counterpart on the other side of this trade is the quant who shipped a bigger network on the same candles and mistook in-sample memorization for foresight; everything in this module exists to keep you out of that seat.

House position, deliberately calibrated: generalization — not architecture — is the bottleneck. The 2018–2025 survey consensus frames deep learning in trading chiefly as a feature and representation engine whose outputs — forecasts, embeddings, regime probabilities — feed trading and risk layers you already trust, rather than acting as an end-to-end signal oracle. Gradient-boosted trees remain brutally competitive on engineered tabular features (Module 17); neural networks earn their complexity premium where raw sequence structure matters: volatility paths, multi-scale patterns, regime shape.

Concretely, this module activates your dormant feat/lstm-model branch around three experiments — an LSTM volatility forecaster feeding the Meta model, a sequence encoder replacing hand-crafted slope/distance features, and an HMM state-count ablation — each wired into the purged-validation machinery of Module 19 and the ONNX export path toward Module 24.

Evidence Grounding: Read This Before Building

🧪 Evidence: arXiv 2309.11400 (Transformers versus LSTMs for electronic trading) ran controlled benchmarks on limit-order-book tasks and found LSTM-based models better and more robust on difference-sequence prediction — price movements and differences, which is what traders actually forecast. Transformers showed only a limited advantage on absolute price sequences. A cautionary, well-controlled result that deflates the “attention eats everything” narrative.

Contrasting studies in the IEEE and ScienceDirect literature show Transformers exceeding LSTMs precisely where long-range dependencies dominate the task. Neither camp is wrong; the honest summary is task-dependent — and in both camps, tuned GBMs on engineered tabular features remain brutally competitive (Module 17’s bake-off protocol is how you find out which camp your own task belongs to).

Read these three rows together and the module’s agenda writes itself: try sequence models where sequence structure lives (volatility, windows), keep the GBM as judge and host, and let purged out-of-sample P&L — never architecture fashion — award the trophy.

How It Works: The Representation-Engine Pattern

The practical winning architecture is a hybrid stack: your existing causal features keep flowing to the GBM path exactly as today; a neural path learns additional representations from the same windows; both merge into the Meta feature vector; one calibrated probability drives sizing and filtering; and nothing reaches the live platform without passing the export-parity gate.

Figure: the representation-engine architecture. Bars enter once; two model families extract complementary structure; the merged vector feeds the Meta model you already govern with Module 19 machinery.

flowchart TD
    bars[(Market data<br/>M5 and H4 bars)]
    feats[Causal features<br/>scaled on train fold only]
    gbm[GBM path<br/>route-local XGBoost Meta]
    rep[LSTM representation path<br/>vol forecast + embeddings]
    merge[Merged feature vector<br/>base features + NN outputs]
    meta{Meta model on<br/>triple-barrier labels}
    prob[Calibrated probability]
    sizing[Position sizing and<br/>regime filtering]
    gate{ONNX export gate:<br/>bit-parity verified?}
    live[(ONNX artifact for<br/>MT5 EA / serving stack)]
    block[Block export:<br/>fix parity first]

    bars --> feats
    feats --> gbm
    feats --> rep
    gbm --> merge
    rep --> merge
    merge --> meta
    meta --> prob
    prob --> sizing
    sizing --> gate
    gate -->|"yes: parity holds"| live
    gate -->|"no: float or feature drift"| block

    classDef data fill:#e8f0fe,stroke:#4a86e8
    classDef process fill:#f3f3f3,stroke:#888
    classDef decision fill:#fff4d6,stroke:#d6a300
    classDef risk fill:#fde8e8,stroke:#c0392b
    classDef ok fill:#e6f4ea,stroke:#2e7d32

    class bars,live data
    class feats,gbm,rep,merge,prob process
    class meta,gate decision
    class sizing ok
    class block risk

How to read this:

• The neural path is a column supplier, not a replacement: rep feeds merge alongside the GBM path, so a disappointing LSTM costs you one column, not the system.
• The amber diamond on the right is Module 24’s parity harness moved to design time — float and feature mismatches die before deployment, not in the EA.
• Every arrow left of meta runs inside Module 19’s purged folds; there is no modeling step that escapes the causality contract.

💡 Idea: think of the LSTM as a junior analyst who reads raw windows and files a one-line memo (“next-day vol looks elevated”), while the GBM stays the portfolio manager who actually decides. Firing the analyst never helps; promoting the analyst to PM without an audit trail is how accounts die.

The Neural Model Zoo for Market Data

MLPs on Tabular Features

A multilayer perceptron is the plainest neural net: weighted sums, nonlinearities, done. On pure numeric tabular features it usually matches, rarely beats, a tuned gradient-boosted tree — its distinctive trick is handling categorical inputs via embeddings: small learned lookup tables that map each category to a dense vector, letting the model discover that two sessions behave alike instead of treating them as unrelated dummy columns.

Worked example — session embeddings vs one-hot, hand-checkable:

Five sessions one-hot encoded add five columns to every row and declare all session pairs equally different. An embedding table stores just 40 numbers (plus 2 × 8 = 16 for two symbols) yet learns which sessions trade alike — similarity comes free with training. Meaning: representations for pennies, which is why embeddings survive even when the rest of the network doesn’t.

Low signal-to-noise data demands the full regularization kit: dropout (randomly mute neurons during training), weight decay (shrink weights toward zero), and normalization layers (steady the activations). The training loop below shows all of it plus the non-negotiable: early stopping decided on a purged validation tail, never on training loss.

# Training loop: early stopping on the PURGED validation tail decides when
# the network stops learning signal and starts memorizing noise.
import numpy as np, torch, torch.nn as nn

model = VolForecaster(n_features=X_tr.shape[2], dropout=0.3)   # defined below
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)
loss_fn = nn.HuberLoss()                    # robust to vol-spike outliers
rng = np.random.default_rng(42)
best_val, patience, wait = float("inf"), 10, 0
for epoch in range(100):
    model.train()
    order = rng.permutation(len(X_tr))      # shuffle SAMPLES, never bar order
    for b in range(0, len(order), 256):
        idx = torch.from_numpy(order[b:b + 256])
        opt.zero_grad()
        loss = loss_fn(model(X_tr[idx]), y_tr[idx])
        loss.backward(); opt.step()
    model.eval()
    with torch.no_grad():
        val_loss = float(loss_fn(model(X_va), y_va))
    if val_loss < best_val - 1e-4:          # improvement must be meaningful
        best_val, wait = val_loss, 0
        torch.save(model.state_dict(), "lstm_best.pt")
    else:
        wait += 1
        if wait >= patience: break          # stop before noise memorization

⚠️ Pitfall: shuffling samples is fine — each window is a self-contained tensor. Shuffling or rescaling inside the time dimension, or fitting the scaler on the full sample before splitting, is leakage wearing a neural costume (Module 19 rules apply unchanged).

Sequence Models: LSTM and GRU

An LSTM (long short-term memory) maintains a gated memory cell: at every timestep, forget gates discard stale context, input gates admit new information, and output gates expose only the useful part. Intuition: a rolling journal where a disciplined editor decides each bar what to keep, what to cross out, and what’s worth reporting upward. A GRU is the same idea with fewer gates — cheaper, often equal on financial horizons.

Two design choices dominate outcomes:

• Windowing — the lookback length is the model’s working memory. Too short and regimes are invisible; too long and each sample carries mostly dead history. Start at the horizon multiples your features already use (Module 18).
• Sequence-to-scalar vs sequence-to-sequence — mapping a window to one number (next-horizon vol, next-bar direction) is far more robust than reconstructing a whole future path. Forecast the thing you’d act on.

The builder honors purged boundaries explicitly; the definition that follows is deliberately tiny:

# Windowed sequence dataset: one sample = LOOKBACK bars ending at bar e,
# predicting volatility realized over the NEXT HORIZON bars after e.
import numpy as np

def build_windows(features, realized_vol, lookback=64, horizon=12):
    """X: (n, lookback, k) input windows; y: forward vol; ends: last bar."""
    ends = np.arange(lookback, len(realized_vol) - horizon)
    X = np.stack([features[e - lookback:e] for e in ends])
    y = realized_vol[ends + horizon]         # label sits AFTER the window
    return X.astype("float32"), y.astype("float32"), ends

X_all, y_all, ends = build_windows(feat_matrix, realized_vol)

# Purged boundary: keep a training sample only if its whole footprint —
# window AND label span — clears the validation opening (Module 19 purge).
val_open = int(0.8 * len(y_all))
is_train = ends + HORIZON_DEFAULT <= val_open
is_val = (ends >= val_open) & (ends < int(0.9 * len(y_all)))
X_tr, y_tr = X_all[is_train], y_all[is_train]   # scaler fits HERE only
X_va, y_va = X_all[is_val], y_all[is_val]

# Compact seq-to-scalar forecaster: a window of bars in, one number out.
import torch.nn as nn

class VolForecaster(nn.Module):
    """L x k feature window -> positive next-horizon volatility estimate."""
    def __init__(self, n_features, hidden=32, layers=1, dropout=0.2):
        super().__init__()
        self.lstm = nn.LSTM(n_features, hidden, layers, batch_first=True,
                            dropout=dropout if layers > 1 else 0.0)
        self.head = nn.Sequential(
            nn.Linear(hidden, 16), nn.ReLU(),
            nn.Linear(16, 1), nn.Softplus())   # vol > 0 by construction
    def forward(self, x):                       # x: (batch, lookback, k)
        out, _ = self.lstm(x)                   # (batch, lookback, hidden)
        return self.head(out[:, -1, :])         # last timestep -> scalar

Statefulness pitfalls. The LSTM above is stateless: each sample restarts the hidden state from zero, so training batches stay independent and inference reproduces training conditions exactly. A stateful variant carries hidden state across samples — seductive for online use, poisonous for parity: a hidden state bridging unrelated samples (overnight gaps, refit boundaries) creates a live model that literally cannot be reproduced from recorded inputs, breaking Module 24’s replay tests. Rule: stateless training, window-recomputed inference; reset state only across deliberate session boundaries, and log it when you do.

Transformers for Time Series

Attention, in plain words: every timestep asks which past steps matter. Instead of a recurrent chain squeezing history through one memory cell, each position computes relevance scores against every other position and takes a weighted blend — the model learns, per situation, whether yesterday or three weeks ago is the reference that counts. Positional encodings tell the model where each value sits, and for irregular financial time they should encode real clock features (sinusoidal hour-of-day, day-of-week) rather than naive step counts, because weekends and session gaps make bar index ≠ market time.

The practical refinement is patching (PatchTST-style): chunk the series into short patches and let attention operate over patches, so the transformer stops treating every tick as a word in a very long sentence. Worked example, hand-checkable:

Thirty-one meaningful tokens instead of 256 noisy ones: cheaper attention and each token carries local shape rather than a single point. Meaning: patching is why modern time-series transformers are even affordable — but per the evidence section, affordability ≠ superiority on difference-sequence tasks, so benchmark honestly.

Compute-cost honesty: a PatchTST run costs roughly an order of magnitude more training time than the LSTM here, for an advantage that appears only on some tasks. Budget it like any other experiment — through the trial ledger, with a kill criterion.

CNNs: Learned Chart-Pattern Scanners

A 1D convolution slides a small learned template (kernel) across a price/volume window and fires where the template matches — a pattern detector you didn’t have to draw by hand. Stack kernels of widths 4, 8, 16 and pool the responses: the layer stack becomes receptive to shapes at multiple scales, the way a trader’s eye combines micro- and macro-structure. As feature extractors ahead of your GBM they’re cheap and occasionally useful; as standalone predictors they see only as far as their receptive field.

The 2D branch — rendering series as images (Gramian Angular Fields, recurrence plots) and applying image CNNs — stays a curiosity in this curriculum: visually striking, computationally indulgent, and with no convincing incremental evidence over the 1D features it was built from. Look at the pictures if you like; spend your GPU-hours elsewhere.

Representation Learning and Hybrid Stacks

An autoencoder squeezes many features through a narrow bottleneck and reconstructs them — the compressed middle layer is a low-dimensional embedding of “what today looks like,” analogous to a one-line journal summary of a hundred-ledger-entry day. A denoising autoencoder trains on corrupted inputs and must reconstruct the clean original, forcing it to learn structure rather than copy noise; those robust embeddings double as anomaly detectors (reconstruction error spikes on novel regimes).

The hybrid pattern is the practical winner reported across the 2018–2025 literature: embeddings + GBM beats either alone. The network finds smooth structure trees can’t express; the tree decides what’s tradable without inheriting the network’s calibration weaknesses.

For uncertainty, two workhorse tools:

• Deep ensembles — train N copies differing only in seed; disagreement among them estimates uncertainty. You get this almost free from the seed-ensembling habit below.
• MC dropout — keep dropout active at inference, average several stochastic passes; variance across passes is a cheap uncertainty band.

Both produce a confidence number that plugs straight into sizing: shrink position size as model disagreement widens — the neural cousin of the uncertainty discounts in Module 23.

Probabilistic Regime Models: Mixtures and HMMs

A Gaussian mixture model clusters feature space into soft states — “today is 70% quiet-regime, 30% transitional” — with no notion of time order. A Hidden Markov Model adds the missing piece: a transition matrix scoring how likely each state is to persist or hand off to another, which is exactly what regimes do. Bull/bear/high-vol become latent states inferred from returns and volatility, consumed two ways:

• As features — state probabilities join the Meta feature vector (the representation-engine pattern again).
• As switches — the state gates strategy families outright: reversion-on in quiet, momentum-on in expansion — Module 15’s master switch, now probabilistic.

🔧 For your pipeline: your production system already runs a forward-only Student-t HMM, 3 states, monthly refits, live registry replay — so treat this section as audit-and-extend, not greenfield. The 3-state Student-t configuration is a design choice, not a law of nature: run the ablation below under your existing monthly-refit causality contract. Tools: hmmlearn for HMM variants; arch for the GARCH-family comparator.

The experiment table that operationalizes this module on your feat/lstm-model branch:

Every run in all three experiments registers in the experiment ledger with a pre-registered hypothesis before training starts — the neural zoo gets no exemption from governance.

Training Craft for Financial Data

Financial training sets misbehave in three specific ways, each with a standing countermeasure:

• Tiny effective sample sizes. Overlapping triple-barrier labels mean 100k rows may carry far fewer independent bets (Module 18’s uniqueness weighting applies verbatim). Networks sized for ImageNet will memorize; keep them small and lean on early stopping.
• Class imbalance. Genuine breakouts are rare events; weighted losses (or focal-style down-weighting of easy negatives) stop the model from achieving 95% accuracy by never predicting the interesting class.
• Non-stationarity. Any fit ages. Retrain on the cadence your regime clock dictates — walk-forward loops reusing Module 19’s split generator verbatim, exactly as the GBM path already does.

# Walk-forward retraining: Module 19's purged split generator reused
# verbatim — the LSTM obeys the same causality contract as the XGB baseline.
from mtm.validation import purged_kfold_splits    # Module 19, Recipe 1

def sharpe_for_seed(seed):
    torch.manual_seed(seed)                   # init + dropout masks differ
    oos = np.full(len(vol_target), np.nan)
    folds = purged_kfold_splits(len(vol_target), n_folds=6,
                                label_horizon=HORIZON_DEFAULT,
                                embargo_bars=LOOKBACK)
    for train_idx, test_idx in folds:
        model = fit_early_stopped(X[train_idx], vol_target[train_idx])
        oos[test_idx] = predict_vol(model, X[test_idx]).ravel()
    pnl = economic_pnl(oos, realized_vol, cost_pips=SPREAD_PIPS)
    return sharpe_ratio(pnl)

report = pd.Series({s: sharpe_for_seed(s) for s in [7, 42, 2024]})
print(report.describe().round(2))   # median and spread decide, never max
# -> mean 1.17, std 0.21  : seed-variance comparable to the edge itself

Ensemble across seeds and windows: average forecasts from several seeds (uncertainty estimate included) and optionally several window lengths. Then hold the line on evaluation: economic P&L — never accuracy — selects architectures. A direction-classifier at 53% accuracy may print money or bleed it depending entirely on which 53% (Module 6 owns the verdict math).

Export Path: PyTorch to ONNX to MT5

The road to production runs through ONNX: a portable computation-graph format that the MQL5 runtime executes natively (Architecture A) or your Python serving stack consumes (Architecture B — your current hardened setup). Three disciplines make exports boring:

1. Float32 everywhere. PyTorch defaults to float32, but NumPy-side preprocessing quietly produces float64; cast before export and in the caller. A silent float64-into-float32 handoff corrupts numerics in ways that pass smoke tests and fail parity audits.
2. Dynamic batch axes, so the same artifact serves one bar or a rebatch of hundreds.
3. Visual verification in Netron — open the exported graph, confirm input names/shapes/opset before anything ships to the parity harness.

# ONNX export: float32 tensors only, dynamic batch axis, verified in Netron.
model.load_state_dict(torch.load("lstm_best.pt"))
model.eval()                               # freeze dropout/batchnorm behavior
example = torch.randn(1, LOOKBACK, X_all.shape[2], dtype=torch.float32)
if example.dtype != torch.float32:         # paranoia clause: cast discipline
    example = example.to(torch.float32)
with torch.no_grad():
    torch.onnx.export(
        model, example, "vol_forecaster.onnx",
        input_names=["features"], output_names=["vol_forecast"],
        dynamic_axes={"features": {0: "batch"},
                      "vol_forecast": {0: "batch"}},
        opset_version=17)
# Caller-side rule: features.astype("float32") upstream, every single time.

The exported artifact then faces the golden-dataset parity test from Module 24: identical rows through Python and the runtime must agree bit-for-bit before any capital sees it.

Testing It Honestly

Known-answer fixture first

Before market data touches the network, prove the plumbing on a problem with a known answer:

# Sanity fixture: a network that cannot fit a sine wave is broken before it
# ever sees a candlestick — windowing bugs hide behind market noise.
import numpy as np

t = np.arange(2000, dtype="float32")
wave = np.sin(2 * np.pi * t / 40.0).reshape(-1, 1)
X_w, y_w, _ = build_windows(wave, wave.ravel(), lookback=32, horizon=1)
# Train 30 epochs -> expect validation HuberLoss < 1e-3 and R^2 > 0.99;
# anything worse means the window builder or loader is lying to you.

Representation-quality probe

Embeddings claim to capture market structure; test the claim directly. Take known regime labels (your HMM states, or Module 15’s ATR buckets) and ask whether embeddings cluster by regime using the silhouette score — a cluster-cohesion measure running from −1 (scattered) to +1 (cleanly separated):

# Probe: do learned embeddings cluster by known regime labels?
from sklearn.metrics import silhouette_score

regime = regime_labels[ends]                 # one label per window
score = silhouette_score(X_emb, regime, sample_size=5000, random_state=7)
null = silhouette_score(X_emb, rng.permutation(regime),
                        sample_size=5000, random_state=7)
print(f"silhouette {score:+.3f} vs null {null:+.3f}")
# -> +0.11 vs +0.00 : real but modest structure — probe result, not proof

Interpretation discipline: the score must clearly beat the shuffled-label null, and even then it certifies the representation, not profitability. Modest positive structure feeding a GBM is a normal, healthy result; a spectacular score on market data usually means leakage found a way in.

Seed-ensemble variance report

Every headline number ships with its seed distribution: at minimum three seeds, median and range reported beside any mean (the walk-forward snippet prints exactly this). Decision rule written down in advance: if seed-to-seed spread rivals the edge itself, the model isn’t ready — widen ensembles, shrink the network, or retire the experiment.

Hands-On Project

Three projects, executed in order on the feat/lstm-model branch. All inherit the house contracts: purged folds from Module 19, ledger registration per run, net-of-cost verdicts.

Project 1 — LSTM Vol-Forecaster Feeding the Meta-Labeler

Build the seq-to-scalar vol forecaster (definition above), train it walk-forward with early stopping, and inject its forecast as one new Meta context column (experiment E1). Measure uplift against the current route-local Meta on identical CPCV paths — the representation-engine pattern, minimal disruption to your contract.

Acceptance criteria:

• Sine-wave fixture passes (val loss < 1e-3) before any market data is loaded
• Scaler statistics fitted on train folds only; full-sample scaling absent from the codebase path
• Walk-forward retrains call Module 19’s split generator unmodified; no fold shares scaler or early-stop data
• Uplift reported as delta net-of-cost P&L + PR-AUC on ≥ 5 CPCV paths, median and spread
• Every training run registered in the ledger with seed, config hash, and dataset fingerprint

Project 2 — PatchTST vs LightGBM Head-to-Head

Run both models on identical multi-timeframe feature windows (same folds, same embargo, same costs) and produce a written why-the-winner-won memo: which structures did the winner exploit (long memory? patch-level shapes?), where did the loser leave money, and does the evidence-grounding section’s prediction hold for your data?

Acceptance criteria:

• Identical feature matrices and purged folds for both contenders; differences documented
• Both models evaluated on economic P&L, never classification metrics alone
• Memo cites the concrete mechanism of victory/defeat with at least one diagnostic plot per model
• Compute cost (wall-clock, memory) logged alongside results — cost-benefit stated in the memo
• Ledger entries for every trial, including the abandoned hyperparameter attempts

Project 3 — HMM State-Count and Emission Ablation

Refit your forward-only HMM across state counts k∈{2..5} and Gaussian vs Student-t emissions under the monthly-refit causality contract (experiment E3). Score each configuration on out-of-sample regime persistence and on downstream P&L when its states route strategy permissions à la Module 15.

Acceptance criteria:

• Refit cadence and replay mechanics byte-identical to the production registry path
• Grid covers k∈{2..5} × {Gaussian, Student-t}; each cell scored OOS, no in-sample cherry-picking
• Winner compared against the 3-state Student-t incumbent with ledger-registered deltas
• State-rotation check performed: state identities tracked across refits, flips flagged
• Written recommendation states keep/change/extend — with the trial count paid honestly per Module 2

Key Takeaways

• Deep learning’s proven role in trading is feature and representation engine: its outputs feed trading and risk layers — end-to-end signal oracles remain unsubstantiated by the 2018–2025 evidence.
• Generalization, not architecture, is the bottleneck; arXiv 2309.11400 found LSTMs more robust than Transformers on difference-sequence tasks, and GBMs still rule engineered tabular features.
• Embeddings turn categoricals into learned geometry for pocket change — 5 sessions × 8 dims = 40 parameters versus 5 dead dummy columns.
• Purged boundaries apply to neural pipelines with zero relaxation: windows, scalers, and early stopping all obey Module 19; a stateful LSTM that bridges unrelated samples breaks live reproducibility.
• Patching makes time-series transformers affordable (256 steps → 31 patches, ~68× less attention), but affordability is not superiority — benchmark against LightGBM before believing.
• Uncertainty is a deliverable: seed ensembles and MC dropout produce confidence bands that shrink position sizes exactly when the model admits it’s guessing.
• Selection on economic P&L, fixtures before market data, silhouette probes for representations, seed-variance beside every headline — honesty habits transfer from Modules 5/19 unchanged.

References

• arXiv — Transformers versus LSTMs for Electronic Trading (2309.11400): controlled LOB benchmark; LSTM robustness on difference-sequence prediction
• WisdomChain Insights — Deep Learning & Reinforcement Learning in Algorithmic Trading 2018–2025: survey grounding the representation-engine consensus
• Yuqi Nie et al. — PatchTST: A Time Series is Worth 64 Words (arXiv 2211.14730): patching, channel independence, positional encodings for series
• Ian Goodfellow, Yoshua Bengio, Aaron Courville — Deep Learning (MIT Press): canonical chapters on MLPs, regularization, representation learning
• PyTorch — torch.onnx.export docs and the export tutorial: dynamic axes, opsets, float32 discipline
• Netron: visual inspection of exported ONNX graphs before deployment
• hmmlearn: Gaussian/GMM-emission HMMs for the k-ablation · arch: GARCH-family comparator from Module 15
• Internal continuity: Module 17 GBM baselines · Module 18 windows and labels · Module 19 purged machinery · Module 15 regime routing · Module 24 ONNX-in-EA parity
• Next in sequence: Module 21 — Reinforcement Learning for Trading