Module 6 — Performance Measurement & Risk Analytics

Part II · Research Infrastructure · Priority ⭐ Recommended · Status: Draft v0.1 Prerequisites: Module 2, Module 5

Overview

A backtest produces a number; measurement turns it into knowledge. This module makes you fluent in every standard performance metric — computed once by your own hand so no vendor report can intimidate you afterward — and then packages them into a reusable tear-sheet generator that every later module plugs into.

Two themes run throughout:

1. Every metric has a failure mode. Sharpe misleads on skewed P&L, win rate misleads on asymmetric payoffs, CAGR misleads across volatility regimes. Fluency means knowing what each number hides.
2. Context is the metric. “Sharpe 2” is meaningless alone — against what timeframe, cost model, trial count, and regime? The generator enforces context by construction (Module 2’s annualization discipline).

The Metric Family Tree

Figure: where every common metric lives. The generator implements all four branches; reading order in any report is top-to-bottom.

flowchart TD
    root[Strategy P&L stream]

    root --> ret[Return metrics<br/>CAGR, expectancy,<br/>profit factor, hit rate]
    root --> risk[Risk metrics<br/>volatility, drawdown,<br/>Ulcer index]
    ret --> ratio[Risk-adjusted ratios<br/>Sharpe, Sortino, Calmar]
    risk --> ratio
    root --> bench[Benchmark-relative<br/>alpha/beta, capture]

    classDef data fill:#e8f0fe,stroke:#4a86e8
    classDef process fill:#f3f3f3,stroke:#888

    class root data
    class ret,risk,ratio,bench process

How to read this:

• Return and risk metrics are computed independently, then combined into ratios — which is why a ratio can look fine while its ingredients scream.
• Benchmark-relative analysis matters less for CFD trading (no true passive alternative) than for equities; keep the machinery but feed it a sensible yardstick (e.g., buy-and-hold of the same symbol, or cash).

Return Metrics That Matter

From a trade list (or bar P&L series), five numbers carry most of the story:

import numpy as np

trades = np.array([120., -80., 150., -80., 90., -80., 200., -80.])

wins, losses = trades[trades > 0], trades[trades < 0]
hit_rate     = len(wins) / len(trades)                    # -> 0.50
avg_win      = wins.mean()                                # -> 140.00
avg_loss     = abs(losses.mean())                         # -> 80.00
expectancy   = trades.mean()                              # -> +30.00/trade
profit_factor= wins.sum() / abs(losses.sum())             # -> 1.75

Hand-check: 4 winners averaging +140 and 4 losers at −80 → total +240 over 8 trades = +30 per trade. A 50% hit rate feels mediocre; with 1.75 payoff asymmetry it’s a healthy edge. Anyone quoting win rate without payoff structure is selling something.

⚠️ Pitfall: At high frequencies these metrics distort. Two thousand M1 trades at +2 pips expectancy each looks superb until you realize the estimate’s confidence interval is wider than the edge itself, and one bad slippage month erases a quarter of trades’ profits. High-frequency systems need more statistical discipline, not less (Module 8).

Risk Metrics: Where Survival Lives

Returns tell you what you made; risk metrics tell you what you endured. Compute them on the equity curve, not the trade list:

eq = np.array([100_000, 108_000, 103_000, 94_000, 101_000, 109_000])

peak       = np.maximum.accumulate(eq)
drawdown   = eq / peak - 1                  # series of underwater depths
max_dd     = drawdown.min()                 # -> -12.96%
dd_end     = int(np.argmax(drawdown == max_dd))
dd_start   = int(np.argmax(eq[:dd_end+1] == peak[dd_end]))
duration   = dd_end - dd_start              # bars spent recovering starts here

Hand-check: peak 108k → trough 94k gives 94/108 − 1 ≈ −13.0%. Max drawdown is the deepest hole; duration is time underground; both matter because deep-and-long holes trigger de-gearing rules and human capitulation long before arithmetic ruin.

The Ulcer index extends this idea — it’s the average underwater depth rather than just the worst, penalizing strategies that constantly sit in holes even if their worst moment was mild:

ulcer_index = float(np.sqrt(np.mean(drawdown ** 2)) * 100)   # in % units
# Low UI + similar CAGR = smoother ride = higher sustainable leverage.

Downside deviation (used by Sortino below) measures volatility counting only losses — the intuition being that upside surprises don’t threaten survival.

Risk-Adjusted Ratios — and Their Failure Modes

def sharpe(returns_per_bar, bars_per_year):
    """Classic Sharpe; annualization per Module 2 assumption table."""
    r = returns_per_bar
    return r.mean() / r.std() * np.sqrt(bars_per_year)

def sortino(returns_per_bar, bars_per_year, target=0.0):
    downside = returns_per_bar[returns_per_bar < target] - target
    dd_dev   = np.sqrt((downside ** 2).mean())
    return returns_per_bar.mean() / dd_dev * np.sqrt(bars_per_year)

def calmar(cagr: float, max_dd_frac: float) -> float:
    return cagr / abs(max_dd_frac)          # growth per unit of worst pain

Failure modes to memorize:

Meaning: ratios are for ranking similar-shaped strategies, not for absolute verdicts. Always print the ingredients (return distribution stats, drawdown series) beside the ratio.

Benchmark-Relative Analysis

Even directionless CFD books deserve a yardstick. The minimal kit: regress strategy returns on a benchmark’s (buy-and-hold of the underlying, or cash) to decompose results into beta (exposure you’re paid for) versus alpha (excess beyond exposure); add up/down capture percentages for plain-language reporting. For market-neutral systems (pairs, meta-filtered books) benchmark-relative numbers mostly confirm neutrality claims — useful as a lie detector, not a goal.

Rolling Diagnostics & Regime Slicing

Aggregate numbers average away the only information that matters for survival: when the system made and lost money. Standard slices to compute for every study:

• By calendar period — yearly/monthly tables expose decay and regime dependence instantly.
• By session window — Asia/London/NY split (Module 13); most intraday edges live in one window and die in others.
• By volatility regime — quiet vs expansion buckets via an ATR percentile (Module 15).
• By spread regime — cheap-costing hours vs expensive ones, using the Module 4 curves.

import polars as pl

# trades: Polars frame with columns time_utc, pnl, session, vol_regime
summary = (
    trades.group_by("session")
    .agg(
        n=pl.len(),
        net_pnl=pl.col("pnl").sum(),
        expectancy=pl.col("pnl").mean(),
        hit_rate=(pl.col("pnl") > 0).mean().round(3),
    )
    .sort("expectancy", descending=True)
)
print(summary)

Meaning: a system whose entire expectancy comes from one session or one year is a conditional system — either restrict it to its habitat or reject it. Aggregate-only reports hide exactly this.

Building Your Tear Sheet

Assemble everything into one function that consumes a trade ledger (or equity curve) and emits the full report. Use quantstats for instant coverage, then extend:

import quantstats as qs

# returns: pandas Series of per-bar strategy returns, datetime-indexed
qs.reports.full(returns, title="R042 — XAUUSD M5 breakout",
                benchmark=None)     # HTML/PDF-ready tear sheet in one call

Custom additions the library lacks (and your generator should implement):

1. Cost-decomposition view — gross vs net P&L, spread/commission/swap shares (Module 22 closes the loop).
2. Trial-count stamp and ledger ID printed on every sheet (deflation discipline made automatic).
3. Regime-slice table from the block above.
4. Live-vs-backtest divergence panel once real fills exist (Module 24).

Testing it honestly

Before trusting the generator, validate it against known-answer fixtures: synthetic series with hand-computable metrics (constant +10 per trade; a single −20% cliff; alternating win/loss patterns). If your code can’t reproduce textbook answers on textbook inputs, no real-world number it prints deserves trust.

Hands-On Project

Deliverable: src/analytics/tearsheet.py — the curriculum-wide reporting tool — plus its test suite.

Tasks:

1. Implement all metrics from this module (return block, risk block, ratios, regime slices) as pure functions over a typed trade/equity frame.
2. Wire quantstats for standard visuals; add the four custom panels above (cost decomposition may stub inputs until Module 22).
3. Build three known-answer fixture tests: flat +10/trade list, one-cliff equity curve, alternating 1.75-payoff pattern — assert exact expected values.
4. Generate sheets for the Module 5 reconciliation runs and attach them to that memo.
5. Enforce context printing: no sheet renders without timeframe, annualization assumption, sample window, and ledger ID stamped on it.

Acceptance criteria:

• All metrics match hand-computed fixture values exactly.
• Generator rejects empty/too-short inputs (< 30 trades) with named errors.
• Every rendered sheet carries its context header automatically.
• Regime slice reproduces the session table from raw trades within tolerance.
• One command regenerates the full sheet from a ledger file path.

Key Takeaways

• Win rate without payoff structure is marketing; expectancy times frequency minus costs is the actual business model.
• Drawdown has two axes — depth and duration — and capital plans must respect the distribution of both, not the single realized path.
• Ratios rank similarly-shaped strategies; they never certify absolute quality, and annualized Sharpe comparisons across frequencies are fiction unless normalized (Module 2).
• Every aggregate hides its regime composition; session, volatility, and cost-regime slices turn “it works” into “it works here, under these conditions”.
• A tear sheet is infrastructure, not decoration: fixtures-tested metrics, automatic context stamps, ledger IDs, and room for live-vs-backtest panels make honesty the default output format.

References

• QuantStats — documentation (tear-sheet generation used above)
• quantopian — pyfolio / empyrical (metric definitions reference implementations)
• Ernest Chan — Quantitative Trading, 2nd ed., ch. 3–4 (backtest metric interpretation pitfalls)
• Robert Carver — Systematic Trading, ch. 3–4 (standardized returns and why raw P&L misleads)
• Bailey & López de Prado — The Sharpe Ratio Efficient Frontier (Sharpe estimation error context)
• Next in sequence: Part III begins — Module 7 — The Frequency Spectrum