Module 17 — The ML Toolbox Beyond XGBoost

Part IV · Machine Learning · Priority 🎯 Core · Status: Draft v0.1 Prerequisites: Module 2, Module 5

Overview

XGBoost is one point in a large toolbox — the best-known member of a family of gradient-boosted trees, surrounded by faster siblings, calmer cousins (forests), humbler ancestors (regularized linear models), and specialists worth knowing. Your repo currently runs a route-local Meta-XGB and calls its pipeline lgbm_pipeline while never importing LightGBM; this module fixes that gap.

Why does breadth matter? Because every model class carries an inductive bias — a built-in assumption about what patterns it prefers to find. Boosted trees assume axis-aligned, piecewise-constant structure; linear models assume weighted sums; forests average away variance. When you know each model’s bias, choosing one becomes an engineering decision matched to your data’s shape — not whatever won last month’s Kaggle. Model choice stops being fashion and starts being fit.

The framing throughout is practical: everything here is judged in a same-folds bake-off against your existing route-local Meta-XGB baseline — identical purged folds from Module 19, identical features, verdict by net-of-cost P&L in Phase-5-style replays, never PR-AUC alone (Module 2 explains why single metrics lie).

The Gradient-Boosting Family Head-to-Head

All three libraries train an additive team of small trees, where each new tree predicts the errors still left over. The differences are entirely about how trees are grown, how fast rows are scanned, and how categories are handled — in plain words:

• XGBoost grows trees level-wise (whole depth at a time) and regularizes with L1/L2 penalties on leaf weights. Balanced, conservative, battle-tested.
• LightGBM grows trees leaf-wise (always split the leaf that reduces loss most) on histogram bins of features instead of exact values. It adds GOSS — keeping the large-gradient (poorly-fit) samples plus a sample of easy ones — and EFB, which bundles sparse features so they share one column. Result: typically 5–10× faster training and far lower memory than XGBoost on wide feature sets.
• CatBoost trains on random permutations of the data with ordered boosting: each new tree is validated against data it did not fit, which prevents the target leakage that classic boosting quietly accumulates. Its signature trick is native categorical encoding via ordered target statistics — each row’s category value is replaced by the label average computed only from earlier rows, so a label never leaks into its own features. Trees are symmetric (“oblivious”): every node at a depth uses the same split condition, which acts as built-in regularization.

Comparison table

🧪 Evidence: The TALENT benchmark (>300 tabular datasets, McElfresh et al., NeurIPS 2023) found tree-based ensembles top real-world AutoML leaderboards — especially for regression tasks. The Grinsztajn et al. study (111 datasets, NeurIPS 2022) reached the sharper conclusion: gradient-boosted trees match or beat deep learning on typical tabular data, because tables lack the spatial/image priors neural nets exploit.

So the deep-learning detour (Module 20) is optional curiosity; this family is where your edge lives. Within the family, pick per-route by measurement, not reputation.

Practical laws

1. Always early-stop. Fix n_estimators high and let validation loss decide the round count — every library supports it natively.
2. Learning rate ≈ 0.05 beats 0.1/0.3 given early stopping. Small steps find lower validation minima; the stopping rule pays the extra compute.
3. num_leaves is LightGBM’s complexity knob, not max_depth. Because growth is leaf-wise, depth caps lie about model size; keep num_leaves < 2^max_depth.
4. Prefer native categoricals over one-hot. One-hot explodes dimensionality and starves each dummy column of support.
5. Use monotonic constraints for economic sanity. A higher Meta z-score must never lower predicted win probability; constrain the relationship and the model can’t learn nonsense your risk desk would reject.
6. Watch the train/validation gap as the overfit alarm. Train AUC 0.90 / validation 0.58 means memorization, not skill — same warning sign as walk-forward divergence.

Worked example: why learning rate 0.05 wins

Early stopping compares validation log-loss after every round and keeps the best. Watch two rates race on the same folds:

Hand-check the step budgets: the lr 0.30 model took 80 coarse steps before degrading; the lr 0.05 model took 600 fine steps. Final score 0.6838 vs 0.6899 looks small, but on noisy financial labels every bit of validation loss matters — and the shape is the tell: slow descent to a late minimum is what honest learning looks like, while early-peak-then-decay curves mean the model grabbed noise first.

# LightGBM: leaf-wise growth makes num_leaves -- not max_depth -- the dial
import lightgbm as lgb

cat_cols = ["route", "regime_state", "session"]
for col in cat_cols:
    X_train[col] = X_train[col].astype("category")   # native categoricals

model = lgb.LGBMClassifier(
    n_estimators=2000,          # ceiling only -- early stopping decides
    learning_rate=0.05,         # slow steps reach lower validation loss
    num_leaves=31,              # complexity knob; keep < 2^max_depth
    min_data_in_leaf=200,       # financial noise demands fat leaves
)
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    eval_metric="aucpr",
    callbacks=[lgb.early_stopping(100)],   # patience: 100 flat rounds
)
print(model.best_iteration_)     # -> e.g. 617 of 2000 allowed rounds

# CatBoost: ordered boosting + ordered target statistics fight leakage by
# construction -- each row's categorical encoding sees only PRIOR rows of
# a random permutation, so labels never leak into their own features.
from catboost import CatBoostClassifier, Pool

cat_cols = ["route", "regime_state", "session"]   # low-cardinality fits well
train_pool = Pool(X_train, y_train, cat_features=cat_cols)  # raw strings ok
val_pool = Pool(X_val, y_val, cat_features=cat_cols)

model = CatBoostClassifier(
    iterations=2000,            # ceiling only -- stopping decides
    learning_rate=0.05,
    depth=6,                    # symmetric trees: one split per level
    verbose=False,
)
model.fit(train_pool, eval_set=val_pool, early_stopping_rounds=100)
print(model.get_best_iteration())    # -> e.g. 480

The Bagging Counterpoint: Forests

Random forests and extra trees take the opposite bet: instead of sequentially correcting errors (boosting), grow many deep trees in parallel on bootstrap samples with random feature subsets, then average. Averaging decorrelated trees cuts variance without inflating bias — the ensemble’s opinion is steadier than any single tree’s. Extra trees push decorrelation further by cutting splits at random thresholds instead of searching for the best one.

A forest gives you a free gift boosted models cannot: out-of-bag (OOB) estimates. Each tree never saw ~37% of the rows, so those rows serve as a built-in validation set — no fold machinery needed for a rough generalization read.

On noisy financial tabular data, forests usually trail boosted trees: boosting actively chases residual structure while bagging just averages, and when signal-to-noise is tiny, averaging preserves the blur. But forests earn their place two ways: as stable meta-labelers (their averaged probabilities move less between refits — valuable when your selector retrains monthly), and as variance-reduced second opinions whose disagreement with a GBM is itself diagnostic information.

Linear Baselines You Must Beat

Before any GBM gets promoted, it must beat the boring family: ridge, lasso, and elastic-net — linear regressions/classifications with a penalty shrinking coefficients toward zero. Ridge shrinks smoothly; lasso shrinks some coefficients to exactly zero, giving built-in feature selection; elastic-net blends both, handling correlated features better than pure lasso.

Add a calibrated logistic regression — logistic outputs remapped so predicted probabilities match observed frequencies (see calibration section below) — and you have the strongest cheap baseline available.

Why do linear models matter here? Fewer ways to memorize. A flexible learner can carve the feature space into thousands of regions that happen to fit one regime’s noise; a linear model has exactly as many knobs as features. When a regime shift hits, the flexible learner’s memorized regions are all wrong at once, while the linear model’s few coefficients degrade gracefully. Empirically, linear models often survive regime shifts better than flexible learners — they cannot chase structure they lack the capacity to represent.

House law: a challenger GBM must beat the calibrated elastic-net baseline by a real margin, not a noise margin (the one-standard-error rule below quantifies “real”). A 0.001 PR-AUC win is fashion; only net-P&L separation across walk-forward windows justifies the added complexity.

# Elastic-net logistic: the humility benchmark every GBM must beat clearly
from sklearn.linear_model import LogisticRegressionCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import TimeSeriesSplit
label_horizon = 60                    # bars your label looks ahead (M18)
splits = TimeSeriesSplit(n_splits=5, gap=label_horizon)
enet = make_pipeline(
    StandardScaler(),                   # penalties are scale-sensitive
    LogisticRegressionCV(
        Cs=20,
        cv=splits,
        penalty="elasticnet",
        solver="saga",
        l1_ratios=[0.1, 0.5, 0.9],      # ridge <-> lasso blend, CV-chosen
        scoring="average_precision",
        max_iter=5000,
    ),
)
enet.fit(X_train, y_train)
coefs = enet[-1].coef_.ravel()
print(f"surviving features: {(coefs != 0).sum()} / {coefs.size}")

Other Classical Learners Worth Knowing

Support vector machines find the separating boundary with the widest safety margin; only the borderline points define it. Their era was “few rows, many features” (n << features) — relevant if you ever classify rare events on wide engineered sets, but they scale poorly to your row counts and give probabilities only via awkward post-hoc fits.

k-nearest neighbors votes by committee of the most similar historical bars. It teaches the curse of dimensionality viscerally: past a few dozen features, “nearest” neighbors are nearly equidistant and the vote degenerates toward noise — the standing argument for ruthless feature selection rather than feature hoarding.

Gaussian processes fit distributions over functions and hand back explicit uncertainty bands with every prediction — genuinely useful for uncertainty-aware regression on small data (think: modeling a route’s thin trade-count months). Quadratic compute cost confines them to hundreds-to-low-thousands of rows.

Naive Bayes multiplies per-feature likelihoods assuming independence — wrong almost always, yet serviceable for event-flag style features (news-day flag × session flag) where a crude but stable probability beat is enough and speed matters more than finesse.

Stacking & Blending

Stacking combines a level-1 team of heterogeneous base models — ideally different families, since their errors differ — by feeding their predictions into a level-2 meta-learner that learns when to trust whom. Blending is the same idea with a simpler (held-out split) instead of full cross-validated generation.

The iron rule: the meta-learner trains on out-of-fold predictions ONLY. If base models predict their own training rows, their overconfident in-sample scores teach the meta-learner to trust them far more than live behavior will justify — the stack inherits and amplifies leakage.

And the purged stacking rule compounds it: the OOF predictions themselves must be generated with purged folds (Recipe 1). If fold boundaries leak, leakage enters twice — once inside each base model, once through the stack’s training matrix — and the combined backtest looks best precisely where it cheats most.

Diversity sources worth engineering deliberately:

• Different algorithm families — a GBM, a linear model, a forest see different structure in the same frame.
• Different feature views — price-action-only vs regime-only vs session-only subsets per base model.
• Different seeds/windows — same architecture retrained on shifted windows; cheap stability through averaging.

# Purged stacking skeleton: level-2 sees out-of-fold predictions ONLY
from sklearn.base import clone
import numpy as np

def oof_predictions(base_models, X, y, purged_folds):
    """One column per base model; every row scored by unseen-fold models."""
    oof = np.zeros((len(X), len(base_models)))
    for tr_idx, te_idx in purged_folds:          # folds pre-purged (M19)
        for j, model in enumerate(base_models):
            fitted = clone(model).fit(X.iloc[tr_idx], y.iloc[tr_idx])
            oof[te_idx, j] = fitted.predict_proba(X.iloc[te_idx])[:, 1]
    return oof

stack_X = oof_predictions([xgb_clf, lgbm_clf, enet_logit, rf_clf],
                          X_meta, y_meta, purged_folds)
level2 = LogisticRegression(max_iter=2000).fit(stack_X, y_meta)

Probability Calibration

Your sizing thresholds consume probabilities — filter trades below p = 0.55, size positions by Kelly on p (Module 18). Uncalibrated tree probabilities wreck that chain: boosted trees push scores toward 0/1, so a “0.70” prediction may win only 52% of the time. Feed that into threshold logic and you take oversized bets on inflated confidence.

Two standard repairs:

• Platt scaling — fit a sigmoid remapping of raw scores to outcomes; robust when calibration data is scarce.
• Isotonic regression — a free-form monotonic staircase; more accurate with plenty of data, prone to overfitting with little.

Both must be fit on out-of-sample folds only — calibrating on training predictions just learns the model’s own optimism back. Sanity-check with a reliability curve: bucket predictions, compare predicted rate vs observed win rate per bucket.

# Calibration on OUT-OF-FOLD scores only -- never on training predictions
from sklearn.calibration import IsotonicRegression, calibration_curve

raw_val = gbm.predict_proba(X_val)[:, 1]   # uncalibrated OOS scores
iso = IsotonicRegression(out_of_bounds="clip")
iso.fit(raw_val, y_val)                    # learn remap from OOS data
prob_cal = iso.predict(gbm.predict_proba(X_test)[:, 1])

frac_pos, mean_pred = calibration_curve(y_test, prob_cal,
                                        n_bins=8, strategy="quantile")
for mp, fp in zip(mean_pred, frac_pos):    # reliability table
    print(f"predicted {mp:.2f} -> observed {fp:.2f}")
# Diagonal-ish rows mean thresholds can trust these probabilities;
# "predicted 0.60, observed 0.42" wrecks Kelly-based sizing silently.

Model Selection Protocol

Three disciplines keep the zoo honest:

Nested cross-validation. The outer loop delivers the verdict; the inner loop does the tuning. Tuning inside the loop that also reports performance inflates results — the inner winner is selected because it looked good on that data. Module 19 builds the machinery; here, accept the rule: never let the same folds choose and certify a model.

One-standard-error rule. After outer-loop scoring, take the simplest model whose mean score sits within one standard error of the best. Complex models win by noise constantly; the 1-SE band filters luck. Concretely: if CatBoost nets 12.4 ± 3.1 bps/day and elastic-net nets 10.9 ± 2.8, they are tied — keep the linear model.

Simplicity bonus. Between otherwise-equal candidates prefer fewer hyperparameters, fewer moving parts, cheaper retraining. Every knob is another lottery ticket in the multiple-testing trap.

⚠️ Pitfall: Leaderboard-style selection — trying N configurations and promoting whichever scored highest — is data snooping wearing a lab coat. With 50 trials, the best score is mostly selection luck; register trials in a ledger and deflate expectations (Module 2) before believing any champion.

For Your Pipeline: The Bake-Off

🔧 For your pipeline: run the bake-off on your existing route-local meta_dataset frames — R0/R1/R2 separately, never pooled — under IDENTICAL purged folds from Module 19. Contestants: your current XGBoost vs LightGBM vs CatBoost vs elastic-net logistic vs calibrated random forest. Judge by net-selector P&L in Phase-5-style replays, never PR-AUC alone. Two prior expectations to test: CatBoost’s ordered target statistics suit your low-cardinality route/regime categoricals; ONNX export maturity favors XGBoost/LightGBM when Module 24 deployment paths arrive — a model you cannot ship is a model you should not select.

Figure: the bake-off feeds five contestants identical purged folds, calibrates each on out-of-fold scores, stacks them, then lets net-cost replays — not leaderboard metrics — deliver the verdict under the one-standard-error rule.

flowchart TD
    folds[(Shared purged folds<br/>identical for all models)]
    xgb["XGBoost<br/>baseline to beat"]
    lgbm["LightGBM<br/>leaf-wise + GOSS"]
    catb["CatBoost<br/>ordered target stats"]
    enet["Elastic-net logistic<br/>linear humility bench"]
    rf["Random forest<br/>bagging second opinion"]
    cal[Per-model calibration<br/>fit on OOF scores only]
    stack[OOF stacking layer<br/>level-2 meta-learner]
    replay{Net-of-cost PnL replay<br/>Phase-5 style}
    pick[Select champion<br/>one-SE rule]
    kill[Kick back to research<br/>no promotion]

    folds --> xgb
    folds --> lgbm
    folds --> catb
    folds --> enet
    folds --> rf
    xgb --> cal
    lgbm --> cal
    catb --> cal
    enet --> cal
    rf --> cal
    cal --> stack
    stack --> replay
    replay -->|"clearly beats linear"| pick
    replay -->|"within noise of linear"| kill

    classDef data fill:#e8f0fe,stroke:#4a86e8
    classDef process fill:#f3f3f3,stroke:#888
    classDef decision fill:#fff4d6,stroke:#d6a300
    classDef risk fill:#fde8e8,stroke:#c0392b
    classDef ok fill:#e6f4ea,stroke:#2e7d32

    class folds data
    class xgb,lgbm,catb,enet,rf,cal,stack process
    class replay decision
    class kill risk
    class pick ok

How to read this:

• One shared fold object fans out to every contestant — differences in verdicts come from models, never from data splits.
• Calibration sits between raw models and the stack because sizing consumes probabilities, not rankings.
• The amber decision judges by net-of-cost P&L; “beats linear” means outside the one-SE band, not merely ahead on average.

Testing It Honestly

The linear-truth fixture. Build a synthetic dataset where the true signal is known to be linear — labels generated from a fixed weighted sum of features plus noise. On that fixture, elastic-net should match or beat every GBM, because there is no piecewise structure for trees to exploit. If a boosted model “wins” on the linear-truth fixture, do not celebrate: suspect leakage or a broken fold boundary letting it peek. This is a five-minute tripwire that has caught many a too-good bake-off.

# Fixture: linear truth -- elastic-net must match/beat the GBMs here
rng = np.random.default_rng(42)
n_rows, n_feats = 20_000, 12
X_syn = rng.normal(size=(n_rows, n_feats))
true_weights = rng.normal(scale=2.0, size=n_feats)      # linear by design
logits = X_syn @ true_weights / np.sqrt(n_feats)
y_syn = rng.binomial(1, 1 / (1 + np.exp(-logits)))
# Run the full bake-off on (X_syn, y_syn) with the same purged folds.
# Expected: enet within noise of best GBM. GBM clearly ahead => leak hunt.

Runtime/memory logging discipline. Every sweep report carries wall-clock seconds and peak RSS next to every score. Speed differences are part of the verdict: a LightGBM config that ties XGBoost on P&L but retrains in a tenth of the time changes your monthly-refit economics; record time.perf_counter() deltas and resource.getrusage().ru_maxrss (or psutil) in the trial ledger so the trade-off is visible years later.

Pitfalls the zoo invites:

Hands-On Project

Deliverable: docs/research/bakeoff_ml_toolbox.md + a reusable src/experiments/bakeoff.py runner registered in your experiment ledger.

Project A — the bake-off on YOUR current dataset:

1. Load each route-local meta_dataset frame (R0/R1/R2) separately; build ONE shared purged-fold iterator per route from Module 19.
2. Fit five contestants per fold: current XGBoost configuration (frozen as baseline), LightGBM, CatBoost, elastic-net logistic, calibrated random forest. Register EVERY configuration — including the baseline rerun — in the trial ledger before running.
3. Calibrate each on OOF scores; evaluate by net-selector P&L in Phase-5-style replays with full cost model; report PR-AUC alongside but never alone.
4. Apply the one-SE rule across walk-forward windows to name per-route winners; write the verdict memo including runtime/memory columns.

Project B — calibrated ensemble blend:

1. Blend the best GBM + calibrated logistic + RF via purged stacking (level-2 logistic on OOF predictions only).
2. Measure blend-vs-best-single stability across walk-forward windows: window-by-window net-Sharpe deltas, worst-window behavior, and whether the blend ever falls below the best single model for two consecutive windows.

Acceptance criteria:

• All five contestants ran under byte-identical purged folds per route (fold indices committed).
• Every configuration appears in the trial ledger with timestamp, params, seed, and runtime/memory stats.
• Verdict metric is net-of-cost selector P&L from Phase-5 replays; PR-AUC reported secondary.
• One-SE rule applied across windows; ties resolved toward the simpler model, documented explicitly.
• Stack trained exclusively on OOF predictions; fold-purge audit attached.
• Calibration reliability table included per contestant (predicted vs observed by decile).
• Linear-truth fixture passed: elastic-net within noise of best GBM on synthetic linear data.
• Blend stability table shows per-window deltas; conclusion states whether blending earned its complexity.

Key Takeaways

• XGBoost is one point in a toolbox; knowing each model’s inductive bias turns model choice into engineering matched to your data’s structure, not fashion.
• LightGBM buys 5–10× speed with leaf-wise growth, histograms, GOSS, and EFB; CatBoost buys leakage resistance with ordered boosting and ordered target statistics; both deserve a same-folds shot at your Meta-XGB baseline.
• Benchmarks agree: on tabular data — especially regression — tree ensembles top deep learning; spend tuning effort inside the GBM family before wandering.
• Always early-stop at lr ≈ 0.05; treat the train/validation gap as the overfit alarm and monotonic constraints as economic sanity rails.
• The calibrated elastic-net baseline is the humility benchmark: a GBM earns promotion only by beating it by a real margin across walk-forward windows — linear models survive regime shifts precisely because they have fewer ways to memorize.
• Stacking works only on purged out-of-fold predictions; in-fold scores compound leakage instead of diversity.
• Uncalibrated tree probabilities poison threshold-based sizing — calibrate on OOS scores and check the reliability table before trusting any p-value in a sizing rule.
• Nested CV delivers verdicts, inner loops tune; the one-SE rule and simplicity bonus convert lucky champions into durable ones.

References

• PythonDataBench — Gradient Boosting in Python: XGBoost vs LightGBM vs CatBoost (2026) (family comparison, speed/memory numbers)
• NPBlue — ML frameworks overview: XGBoost, LightGBM, CatBoost (algorithmic differences in plain terms)
• LightGBM documentation — Parameters and features (leaf-wise growth, GOSS, EFB, num_leaves)
• CatBoost documentation — Ordered boosting and categorical features (ordered target statistics, oblivious trees)
• scikit-learn — Probability calibration (Platt vs isotonic, reliability diagrams)
• McElfresh et al. — When Do Neural Nets Outperform Boosted Trees on Tabular Data? (TALENT), NeurIPS 2023 (>300-dataset benchmark; tree ensembles lead)
• Grinsztajn, Oyallon, Varoquaux — Why do tree-based models still outperform deep learning on tabular data?, NeurIPS 2022 (111 datasets; GBMs match or beat DL)
• Next in sequence: Module 18 — Feature Engineering & Labeling for Financial ML