Module 5 — Backtesting Methodology

Part II · Research Infrastructure · Priority 🎯 Core · Status: Draft v0.1 Prerequisites: Module 2, Module 4

Overview

A backtest answers a counterfactual: if this system had run at time t, knowing only what was knowable then, what would it have earned? Every bias in existence is a way of cheating that counterfactual — letting the future leak backward, pretending costs away, or quietly selecting survivors.

This module builds the machinery that keeps the counterfactual honest: an explicit catalogue of biases with defenses, architecture choices between simulation styles, a layered validation ladder, and special treatment for low-timeframe backtesting, where generic engines lie most convincingly.

House position: your production stack already validates with nested walk-forward and untouched holdouts — this module formalizes why those work, adds the layers they lack (Monte Carlo perturbation, reconciliation discipline), and equips the research side to the same standard.

Where Biases Enter

Figure: every arrow in a backtest is a place where the future can leak backward or reality can be rounded off. Auditing means walking these arrows one by one.

flowchart TD
    universe[Universe selection<br/>survivorship lives here]
    data[(Historical data<br/>restatement & lookahead)]
    signal[Signal computation<br/>uses close of bar t?]
    execution[Execution model<br/>fills, spreads, slippage]
    accounting[P&L accounting<br/>sizing, compounding]
    report{Reported result}

    universe --> data --> signal --> execution --> accounting --> report

    classDef risk fill:#fde8e8,stroke:#c0392b
    classDef process fill:#f3f3f3,stroke:#888
    classDef decision fill:#fff4d6,stroke:#d6a300
    classDef data fill:#e8f0fe,stroke:#4a86e8

    class data data
    class universe risk
    class signal,execution,accounting process
    class report decision

How to read this:

• Each node names its signature disease; the catalogue below gives each one a demo, a defense, and a habit.
• The red node is first on purpose: survivorship corrupts everything downstream before a single line of strategy code runs.

The bias catalogue

B1 demonstrated in four lines — the classic same-close cheat:

# WRONG: signal uses close[t], P&L books close[t] -> close[t+1] move,
#        but `signal` was computed AFTER seeing close[t]... which is fine,
#        EXCEPT engines often shift by zero by accident. Prove it:
import numpy as np
close = np.array([100., 101., 103., 102., 104.])
sig   = (close > np.roll(close, 1)).astype(float)   # up-bar => long next? NO:
pnl_naive = sig * np.diff(close, prepend=close[0])  # leaks! sig[1]=long earns
                                                    # the bar that SET it.
pnl_honest = np.roll(sig, 1) * np.diff(close, prepend=close[0])  # shifted

Meaning: the difference between pnl_naive and pnl_honest is pure time-travel profit. In tiny examples it’s obvious; inside a 200-feature pipeline it hides until live trading collects it back.

B5 quantified once more, because it decides strategies: with the Module 1 spec sheet, a 0.8-pip all-in round trip against a 5-pip average target spends 16% of gross edge per trade — before slippage tails. Any M1–M15 system whose gross expectancy isn’t a multiple of its cost line is dead on arrival (Module 8 runs this arithmetic as a filter).

Architecture: Three Engines, Three Personalities

Use them together: vectorized for search, event-driven for verdicts, Strategy Tester as the execution-reality anchor. When two engines disagree on the same system with the same costs, the discrepancy is the finding — usually an intrabar assumption or a timestamp convention.

Low-Timeframe Backtesting Requirements

Generic backtest advice fails below H1. Four requirements make M1–M15 simulations trustworthy:

1. Intrabar path ambiguity. An OHLC bar tells you prices touched extremes, not their order. Stops and targets both inside one bar are unresolvable — resolve pessimistically:

def fill_stop_or_target(bar_high, bar_low, stop, target, side):
    """Conservative OHLC rule when both barriers sit inside one bar."""
    hit_stop   = bar_low <= stop if side == "long" else bar_high >= stop
    hit_target = bar_high >= target if side == "long" else bar_low <= target
    if hit_stop and hit_target:
        return "STOP"          # assume the bad one happened first
    return "TARGET" if hit_target else ("STOP" if hit_stop else None)

Figure: requirement 1 drawn out — one wide M5 bar can touch both barriers, and OHLC never records which touch came first.

How to read this:

• Navy dashed = long filled at the signal close 2410.20; green dotted target pays +1.30, red dotted stop costs −0.70.
• The 13:15 bar reaches the target on top and breaks the stop underneath — both touches happened; their order is unknowable from OHLC.
• The conservative rule books the loss. An optimistic engine books +1.30 here instead — that gap is exactly inflated backtest edge.

2. Spread injection by clock. Apply the hourly median/p90 curves from Module 4; a flat-spread assumption flatters exactly the hours you’ll trade worst.
3. Slippage and requotes as distributions, not constants — sample from demo-fill measurements (Module 22); stops get the fat tail.
4. Swap on any hold crossing rollover, triple-Wednesday included (Module 1).

The Validation Ladder

One backtest is a claim. The ladder turns claims into evidence, each rung killing a distinct failure mode.

Figure: climb only after surviving the previous rung. Skipping rungs doesn’t save time — it relocates the discovery of failure to live trading, where it costs money.

flowchart TD
    r1[Rung 1 In-sample backtest<br/>all-in costs, honest fills]
    gate1{Positive net<br/>expectancy?}
    r2[Rung 2 Walk-forward<br/>re-fit through time]

    r1 --> gate1
    gate1 -->|"yes"| r2
    gate1 -->|"no"| kill[Kill and autopsy]
    r2 --> gate2{OOS holds up?<br/>IS-vs-OOS gap small?}
    gate2 -->|"no"| kill
    gate2 -->|"yes"| r3[Rung 3 Untouched holdout<br/>scored ONCE]
    r3 --> r4[Rung 4 Monte Carlo<br/>shuffle/bootstrap trades]
    r4 --> gate4{Edge survives<br/>sequence luck?}
    gate4 -->|"no"| kill
    gate4 -->|"yes"| r5[Rung 5 Paper trade<br/>on MT5 demo]
    r5 --> promote[Earns a size-disciplined trial]

    classDef process fill:#f3f3f3,stroke:#888
    classDef decision fill:#fff4d6,stroke:#d6a300
    classDef ok fill:#e6f4ea,stroke:#2e7d32
    classDef risk fill:#fde8e8,stroke:#c0392b

    class r1,r2,r3,r4,r5 process
    class gate1,gate2,gate4 decision
    class promote ok
    class kill risk

How to read this:

• Rungs are ordered by cost of failure: statistical death is free; live death is not.
• Rung 3 exists only because it is never used for selection — a holdout consumed during tuning is just more in-sample (Module 0’s house law).
• Rung 5 compares live telemetry against backtest expectations; divergences route to drift forensics (Modules 19, 24).

Walk-forward done right

Walk-forward re-runs history the way you’d actually have lived it: fit on the past, trade the future, roll forward. Two design choices matter:

• Expanding window (train on everything so far): assumes the edge is broadly stationary. Sliding window (fixed-length recent past): assumes regimes rotate and old data misleads. Choose by belief about your family’s stability — trend systems often prefer sliding; factor systems expanding.
• The purge gap: if labels look N bars into the future, training samples whose labels reach past the boundary contaminate the test fold. Insert an N-bar dead zone between train end and test start (plus a small embargo after). This is the label-horizon leakage fix — the reason naive K-fold is disqualified outright: shuffling grants training access to the future, and autocorrelated series make every fold leak.

def walk_forward_splits(n, train_len, test_len, label_horizon):
    """Yield (train_idx, test_idx) honoring chronological order + purge."""
    test_start = train_len
    while test_start + test_len <= n:
        train_end = test_start - label_horizon      # <- the purge gap
        yield (
            range(0, train_end),
            range(test_start, test_start + test_len),
        )
        test_start += test_len                      # sliding variant

A practical alarm rides on this rung: in-sample vs out-of-sample divergence. Sharpe 3 in-sample and 0.5 out-of-sample is not a tuning disappointment — it’s the fingerprint of memorized noise, and the expected-size warning sign is roughly “OOS less than half of IS”.

Monte Carlo: attacking sequence luck

Even a genuinely positive-expectancy trade list had one particular ordering. Reshuffling the same trades answers: how much of my drawdown story was order, not edge?

import numpy as np

def mc_drawdown(trade_pnls, n_sims=2_000, rng=np.random.default_rng(7)):
    """Distribution of max drawdown under shuffled trade order."""
    dd = []
    for _ in range(n_sims):
        eq = np.cumsum(rng.permutation(trade_pnls))
        peak = np.maximum.accumulate(eq)
        dd.append((peak - eq).max())
    return np.percentile(dd, [50, 90, 99])

# If the 99th percentile DD is 3x your backtest DD, your capital plan
# was fitted to one lucky ordering. Size for the distribution instead.

Parameter robustness: plateaus, not peaks

When sweeping parameters, prefer regions where neighbors perform similarly (plateaus) over isolated spikes (peaks). A peak is usually the noise fitting itself; a plateau survives perturbation. Concretely: heatmap neighboring values, require the chosen cell’s ring of neighbors to stay within ~30% of its performance, and cap total knob count — every added parameter is another lottery ticket in the Module 2 multiple-testing trap.

Testing It Honestly

The module’s ritual is engine reconciliation: run the identical system — same data window, same costs, same sizing — through two independent engines, then explain every difference line by line. Expected sources of small deltas: intrabar ordering assumptions, spread-instant conventions, swap timing. Unexpected deltas: timezone shifts, off-by-one signal alignment, hidden default costs. Write the reconciliation memo; unexplained residuals mean one engine is lying and you must find which.

Hands-On Project

Deliverable: docs/research/reconciliation_m5_breakout.md + reusable validation utilities in src/validation/.

Tasks:

1. Take one deliberately simple system — Donchian(20) breakout on XAUUSD M5/H1, fixed fractional sizing.
2. Implement it twice: vectorized (any library) and event-driven hand-rolled (~100 lines is enough).
3. Run both through the ladder’s rungs 1–4 with the Module 4 cost curves; log every metric.
4. Export the same system to MT5 Strategy Tester (every-tick real ticks if available) and reconcile all three equity curves; produce a delta table with explanations per line item.
5. Run the trade-shuffle Monte Carlo and parameter-ring robustness check; record drawdown percentiles and neighbor-cell ratios alongside the headline results.
6. Deliberately introduce each bias B1/B5 once, quantify the inflation, then revert — the memo shows both numbers.

Acceptance criteria:

• Three engines agree within stated tolerance or every residual is explained in writing.
• Purge gap present in all walk-forward splits (label horizon respected).
• Monte Carlo p99 drawdown reported next to the naive single-path number.
• Bias-inflation section quantifies B1 and B5 effects in currency terms.
• Utilities (split generator, conservative fills, MC harness) carry unit tests.

Key Takeaways

• A backtest is a counterfactual claim; the entire discipline exists to keep the future from leaking into the past and reality from being rounded off.
• Six biases account for most inflated backtests — walk the five arrows of the pipeline diagram whenever results look too good.
• Random K-fold is disqualified on financial data outright; walk-forward with a purge gap sized by the label horizon is the minimum honest scheme, and sliding-vs-expanding is an explicit bet about regime stability.
• Below H1, add conservative intrabar fill rules, clock-varying spreads, sampled slippage, and swap — otherwise the engine flatters precisely your worst hours.
• The validation ladder orders evidence cheaply: kill ideas statistically before killing them financially; the untouched holdout works only while it stays untouched.
• Monte Carlo converts “my max drawdown was X” into “X sits at the 60th percentile of plausible orderings” — size capital against the distribution.
• Reconciling independent engines is the closest thing backtesting has to unit tests: disagreements are findings, not annoyances.

References

• Quant Lessons — Walk-forward vs k-fold (counterfactual framing; why shuffling fails; leakage taxonomy)
• QuantEngines — Cross-validation for trading models (walk-forward/purged-K-fold/CPCV implementations)
• Alphaware — Preventing overfitting: walk-forward vs purged K-fold (when to combine both; IS/OOS divergence alarms)
• Marcos López de Prado — Advances in Financial Machine Learning, ch. 11–12 (purged CV, embargo, backtest overfitting probability)
• Robert Carver — Systematic Trading, part C (slow-and-steady system validation philosophy)
• tradingstrategy.ai — Backtesting & research methodology docs (layered validation framework reference)
• Next in sequence: Module 6 — Performance Measurement & Risk Analytics