Module 21 — Reinforcement Learning for Trading

Part IV · Machine Learning · Priority 🎯 Core · Status: Draft v0.1 Prerequisites: Module 5, Module 8, Module 19

Overview

Reinforcement learning (RL) trains an agent by letting it act, charging it for the consequences, and nudging its behavior toward whatever earned the most reward. On paper this is the dream trading technology: unlike supervised models that predict a label and stop, an RL agent optimizes the whole trading loop — entries, exits, sizing — and can internalize transaction costs and drawdown pain directly inside its objective. That promise is iffy: it holds only if the environment the agent trains in is honest. The strongest survey consensus through 2025 is blunt — end-to-end RL trading demonstrations rarely survive contact with real costs, because most were trained in frictionless fantasy worlds (arXiv 2106.00123; WisdomChain 2018–2025 review).

Where RL plausibly helps this practitioner is narrow and specific: execution scheduling, position-sizing policies stacked on top of fixed signals, and meta-parameter adaptation — not inventing signals from scratch. Signal generation remains the job of Modules 8, 17, and 18; RL is a candidate upgrade for the layers around them. This module teaches you to frame trading as an MDP, build an environment that cannot lie, and evaluate an agent with the same purged-path discipline you already apply to every other model.

🧪 Evidence: the 2018–2025 literature review converged on “prediction ≠ trading” in its strongest form for RL: higher simulated returns almost always evaporate once realistic spreads, latency, and slippage enter the step function. Papers that report surviving agents share one trait — honest environments — not exotic algorithms (WisdomChain survey).

How It Works: Trading As An MDP

An RL problem is a Markov Decision Process (MDP) — in plain words: at each tick of a clock, the agent sees a state, picks an action, receives a reward, and lands in a next state. Repeat thousands of times until behavior improves. Nothing in that loop requires mathematics to understand; it requires design choices, and those choices decide everything.

Two of these deserve emphasis. First, the state: features alone are not a state. Your existing Meta-labeling stack knows today’s features; a trading agent additionally needs to know its current exposure and where it stands in the session — otherwise “close the position” and “open one” are indistinguishable acts. Second, the episode: an episode is one pass over a walk-forward window from Module 5, started at a sampled historical date, played bar by bar until flat or expired. Training shuffles episode start dates; it must never shuffle the bars inside an episode.

Reward Design Is The Whole Game

The reward function is your one chance to tell the agent what “good trading” means, and every weakness in it becomes a behavior. Credit raw PnL and the agent learns to chase gross pips and ignore the pain incurred earning them. Survey evidence consistently finds risk-adjusted rewards outperform raw PnL rewards: per-step Sharpe-style contributions and drawdown penalties produce policies that survive out-of-sample, because the reward finally matches the thing you actually optimize for in production.

The workhorse pattern used throughout this module: each step’s reward is the account-return change, minus a tax on how far equity sits below its running peak:

# Risk-adjusted reward: per-step account return minus a drawdown tax
# The same +0.30R win scores positive at a fresh peak, negative underwater
DD_PENALTY_WEIGHT = 0.5   # lambda: how harshly underwater equity is taxed

def risk_shaped_reward(step_return, equity_now, running_peak):
    """All inputs are decimal fractions of equity (0.003 = +0.30%)."""
    drawdown = max(running_peak - equity_now, 0.0) / running_peak
    return step_return - DD_PENALTY_WEIGHT * drawdown

# Hand check 1: +0.30R winner at R = 1% risk, closing 2% below the peak
r1 = risk_shaped_reward(step_return=0.003, equity_now=98.0,
                        running_peak=100.0)
print(round(r1, 4))   # -> -0.007  (a raw-PnL agent books this as +0.003)

# Hand check 2: the identical win booked at a fresh equity high
r2 = risk_shaped_reward(step_return=0.003, equity_now=100.3,
                        running_peak=100.0)
print(round(r2, 4))   # -> 0.003   (clean win keeps full credit)

Meaning: the same winning trade earns opposite signs depending on the equity context it happens in. A raw-PnL agent sees +0.003 either way and happily repeats trades that grind out marginal wins from deep drawdowns. The shaped agent experiences −0.007 and learns to stand aside. Watch the two reward designs disagree on the identical two-step scenario:

• Setup: risk R = 1% of equity per trade, λ = 0.5, episode opens at a fresh peak.
• Policy GRIND takes both candidates: loses 1.0R (−0.0100), then wins 1.3R (+0.0130).
• Policy PATIENT takes neither: 0.0000, 0.0000.

The raw-PnL agent ranks GRIND above PATIENT and learns churn; the shaped agent ranks PATIENT above GRIND and learns selectivity. Neither policy knew anything — the reward function decided what got learned. This is why reward design dominates algorithm choice in practice.

Rewards also differ in density. Dense rewards (credit every bar, e.g., mark-to-market change) give the agent frequent feedback but invite hacking of the shaping terms themselves. Sparse rewards (credit only at position close) are harder to learn from but nearly impossible to game. The pragmatic middle: dense per-step mark-to-market credit plus the drawdown tax, settled honestly at exit.

💡 Idea: treat the reward function like a compensation contract for an employee. Whatever metric you pay out on, the employee will optimize — including the letter-not-spirit loopholes. Audit a reward the way you’d audit a bonus scheme: ask “what is the laziest way to farm this number?”

The Closed Loop

Figure: the agent-environment training loop (top cycle) runs against an honest simulator whose costs live inside the step function; the offline branch (bottom) stores logged episodes, retrains, and gates deployment on the CPCV path machinery from Module 19.

flowchart TD
    mkt[(Candidate streams<br/>from Module 8)]
    state[State builder<br/>features + position<br/>+ session clock]
    policy[PPO policy network]
    act{Action<br/>hold / size / exit}
    env[Environment step<br/>honest costs inside]
    rew[Reward:<br/>return minus<br/>drawdown tax]
    buf[(Replay buffer<br/>logged episodes)]
    train[Offline training<br/>on replay buffer]
    gate{CPCV gate:<br/>beats rules on<br/>path distribution?}
    live[Deploy behind<br/>rule-based guardrails]
    redo[Iterate reward<br/>or state design]

    mkt --> state
    state -->|"observation"| policy
    policy --> act
    act -->|"order intent"| env
    env --> rew
    rew -->|"next observation"| state
    env -->|"episode log"| buf
    buf --> train
    train -->|"updated weights"| policy
    train --> gate
    gate -->|"pass"| live
    gate -->|"fail"| redo
    redo -->|"redesign"| state

    classDef data fill:#e8f0fe,stroke:#4a86e8
    classDef process fill:#f3f3f3,stroke:#888
    classDef decision fill:#fff4d6,stroke:#d6a300
    classDef ok fill:#e6f4ea,stroke:#2e7d32

    class mkt,buf data
    class state,env,rew,train,redo process
    class act,gate decision
    class policy,live ok

How to read this:

• The top cycle is the experience loop: state → action → costly step → risk-shaped reward → next state. Every arrow crossing into env passes through the cost model — there is no cost-free path anywhere in the diagram.
• The bottom branch is the learning loop: episodes land in a replay buffer, training consumes them offline, and updated weights re-enter the cycle. Separating experience from learning lets you audit what the agent saw.
• The amber gate is the exit condition: deployment is granted only when the policy beats rule-based exits across the CPCV path distribution — median and spread — never on one flattering backtest.

The Algorithm Tour In Plain Words

Three families cover essentially every trading RL paper. None requires the formulas to reason about; each answers a different question about how behavior gets improved.

Value-based — DQN and its variants. Deep Q-Networks learn a worth table: “in this kind of state, how much is each action ultimately worth?” The agent then simply picks the highest-worth action. Because worth is attached to a discrete menu, DQN fits problems where actions are naturally enumerable — hold/exit decisions, sizing ladders. Its practical weakness is overestimating worth on actions it has barely tried; variants (Double DQN, dueling heads) patch this. A 2024 Scientific Reports study of a multi-level deep Q-network on Bitcoin is representative of the family’s current form: discretized, level-based actions made learning stable on a 24/7 tape, with results still acutely sensitive to fee assumptions — the environment honesty theme again (Nature s41598-024-51408-w).

Policy-gradient — PPO and A2C. Instead of valuing actions, these directly tune behavior itself — nudging the probability of each action up or down according to how its outcomes scored. PPO (Proximal Policy Optimization) adds a seatbelt: each update is clipped so the policy cannot swing violently on one lucky batch, which makes it the de facto default for tabular-and-timescale problems like ours — stable enough to train on a laptop, forgiving of imperfect reward scales. A2C is the same idea without the seatbelt; simpler, touchier.

Continuous-control actor-critics — DDPG, SAC, TD3. These output continuous numbers (e.g., target position 0.37 lots) rather than picking from a menu, pairing an actor (proposes the action) with a critic (scores it). They shine in genuine continuous-control domains, but they are sample-hungry, hyperparameter-sensitive, and their precision is mostly wasted at retail scale where the broker’s volume grid quantizes everything anyway (Module 1).

Environment Engineering — Where These Projects Succeed Or Die

The algorithm is commodity software; the environment is your actual product. An RL environment is honest when the step function charges the agent exactly what live trading would — at the moment live trading would charge it.

• Transaction costs inside the step function. Spread, commission, and slippage reduce the reward at fill time within step() — never in post-processing. An agent that experiences gross prices learns gross behavior.
• Spread dynamics by session. Apply the hourly median/p90 spread curves from Module 4. A flat-spread env flatters precisely the Asian-roll hours where you will trade worst.
• Slippage as a distribution, not a constant. Sample from measured demo-vs-live fill studies (Module 22); stops get the fat tail.
• Partial fills. Large ladder rungs must sometimes fill partially — otherwise the agent learns size with no penalty and meets rejection live.
• Swap accrual. Any hold crossing rollover pays swap, triple-Wednesday included, straight from the Module 1 spec sheet.
• Realistic action latency. The action chosen on bar t’s close executes at bar t+1’s open, through the spread. Same-bar-close execution is time travel (Module 5’s B1 bias wearing an RL costume).

Build this table for your own env and fill the last column with your measurements before trusting any training run. Rows you cannot measure are rows where your agent is being lied to.

Why Published Demos Fail

Read RL trading papers with this checklist in hand; most demos die on at least one item.

1. Frictionless fills. Zero spread or a constant pip charge; the agent discovers arbitrage that is really just uncharged rent. Costs decide before signals do (Module 7).
2. Full-sample normalization leakage. Scalers and normalizers fitted on the entire history — including the evaluation segment — leak the future into the state (Module 19’s purge discipline applies unchanged).
3. In-sample-only evaluation. The “result” is training performance. No held-out window, no walk-forward, nothing an auditor could sign.
4. Reward hacking. The agent farms the reward’s loopholes instead of trading well:
   • Avoidance hack: drawdown penalty without a floor → the agent learns to never trade, scoring zero forever and beating every losing alternative. Fix: floors, or compare against the flat-baseline reward explicitly.
   • Churn hack: dense shaping bonuses per trade → the agent trades maximally to harvest the bonus, bleeding spread each time. Fix: net-of-cost rewards only, bonuses smaller than the round trip.
   • Latency hack: reward credited on the bar the action was decided → the agent collects profit for moves it could not have captured. Fix: settle on the fill bar, always.
5. Action-space mismatch with the broker. Actions imply 0.185 lots; the broker accepts 0.01 steps. Orders rejected live invalidate the entire learned policy — snap actions to the grid inside the env (Module 1).

⚠️ Pitfall: an RL result that cannot name its cost model, its normalization boundary, and its evaluation protocol is a demo, not evidence. Those three sentences belong in every abstract — and their absence predicts failure better than any architecture choice.

Practical Niches, Ranked

Ranked by expected value for this practitioner: smallest action space and easiest honest environment first. The pattern is deliberate — each niche shrinks what the agent must discover and reuses components you already trust.

1. Exit-policy learner on fixed entries. Entries come from the proven Module 8 routes; the agent owns only hold/scale/exit each bar. Smallest menu, richest state, and the environment reduces to managing one open candidate — the honest-env problem at its easiest.
2. Position-size controller over existing Meta probabilities. The Meta model from Module 18 already outputs calibrated probabilities; the agent maps probability-plus-context to a sizing rung. Bet-sizing logic becomes learned instead of fixed, while signal quality stays borrowed.
3. Execution scheduler slicing parent orders. Decide how much of a parent order to release each bar, minimizing arrival-price shortfall — the retail approximation of TWAP/Almgren-Chriss thinking. Direct bridge to Module 22.
4. End-to-end signal + policy from raw features. The agent generates entries itself. Explicitly a research lottery: this is the mode the survey literature fails at hardest, and it competes against your already-validated primary routes with none of their accumulated evidence. Fund it only with money you’d spend on lottery tickets.

Building It In Python

Stack: Gymnasium for the environment API, Stable-Baselines3 for PPO, your own event-driven simulator as the substrate. Polars/NumPy conventions from Module 3 apply to the frames feeding the env.

The environment wraps your simulator — it does not reimplement pricing logic. Skeleton of the wrapper pattern:

# Gymnasium env skeleton: your Module 5 simulator becomes the environment
# costs live INSIDE step() -- never in a post-processing pass afterwards
import gymnasium as gym
from gymnasium import spaces
class BreakoutExitEnv(gym.Env):
    def __init__(self, bars, costs, n_feat):   # bars: OHLC/spread/clock cols
        super().__init__()
        self.bars, self.costs = bars, costs
        self.action_space = spaces.Discrete(3)  # HOLD/SCALE_DOWN/EXIT_NOW
        self.observation_space = spaces.Box(
            float("-inf"), float("inf"), (n_feat,), dtype="float32")
    def reset(self, seed=None, options=None):
        super().reset(seed=seed)
        self.t = self._sample_window_start()   # episode = one WF window
        self.pos = self._fill_entry_next_bar() # entry at NEXT bar open
        return self._observe(), {}
    def step(self, action):
        fill = self.costs.sample_fill(self.t + 1, action)  # spread + slippage
        pnl = self._mark_to_market(fill)       # costs bite RIGHT HERE
        rew = risk_shaped_reward(pnl, self._equity_state())
        trunc = self.t >= len(self.bars) - 1   # 128-bar expiry wall
        return self._observe(), rew, self._closed(action), trunc, {}

Training uses PPO across purged walk-forward splits — the Module 19 boundaries plug in directly, and evaluation episodes append to the CPCV path ensemble:

# PPO across purged splits: train inside folds, judge on OOS folds only
from stable_baselines3 import PPO
from stable_baselines3.common.monitor import Monitor

path_sharpes = []
for train_idx, test_idx in purged_wf_splits(bars, horizon=128):
    env = Monitor(BreakoutExitEnv(bars.iloc[train_idx], costs, N_FEAT))
    model = PPO("MlpPolicy", env, seed=42, verbose=0)
    model.learn(total_timesteps=200_000)

    oos = Monitor(BreakoutExitEnv(bars.iloc[test_idx], costs, N_FEAT))
    obs, _ = oos.reset(options={"candidate_stream": stream_id})
    done = False
    while not done:
        action, _ = model.predict(obs, deterministic=True)
        obs, _, term, trunc, _ = oos.step(int(action))
        done = term or trunc
    path_sharpes.append(net_sharpe(oos.get_episode_rewards()))

# Verdict: compare path_sharpes DISTRIBUTION vs the rule-based baseline's
# distribution (median + spread) -- never on a single lucky backtest.

When the niche is sizing rather than exits, actions become ladder rungs and the mapper enforces the broker grid from Module 1:

# Discrete sizing ladder: policy picks rung k; code snaps to broker grid
VOLUME_MIN, VOLUME_STEP, VOLUME_MAX = 0.01, 0.01, 5.0  # Module 1 spec sheet
LADDER = [0.0, 0.25, 0.50, 0.75, 1.0]  # action k -> share of base lots

def ladder_to_lots(action_k, base_lots):
    """Discrete action -> broker-legal size; rung 0 means stay flat."""
    fraction = LADDER[action_k]
    if fraction == 0.0:
        return 0.0                    # flat is a legitimate order too
    snapped = round(fraction * base_lots / VOLUME_STEP) * VOLUME_STEP
    return round(max(VOLUME_MIN, min(snapped, VOLUME_MAX)), 2)

for k in range(len(LADDER)):
    print(k, "->", ladder_to_lots(k, base_lots=0.40))
# -> 0 -> 0.0 | 1 -> 0.1 | 2 -> 0.2 | 3 -> 0.3 | 4 -> 0.4
# Unsnapped, rung 2 on a 0.37-lot base requests 0.185 lots -- a volume
# MT5 rejects outright, at the worst possible moment.

Ecosystem pointers, with one warning each:

• Gymnasium (docs) — the reset()/step() API convention used above; return contract is (obs, reward, terminated, truncated, info).
• Stable-Baselines3 (docs) — shipped PPO/A2C/DQN/SAC implementations; Monitor wrappers supply the episode statistics your gates consume.
• FinRL (arXiv 2504.02281) — ecosystem of ready-made trading envs with comparative agent results; useful as scaffolding and as a cautionary benchmark set, not as evidence of tradable edge.
• Recommended architecture: wrap your own Module 5 event-driven simulator as the gym env. Costs, fills, and swap are already honest there; adopting someone else’s env restarts that battle from zero.

🔧 For your pipeline: you already own a hardened event-driven replay with journal telemetry. Wrap the phase_5 trailing-backtest-engine semantics as a Gymnasium env: reset() loads a candidate stream, step() advances one bar through the existing fill/cost machinery, info carries the journal fields. First experiment: PPO exit-policy vs the deterministic R2 route exits (profit-take 1.7 / stop 1.30 ATR multiples / 128-bar expiry) on IDENTICAL candidate streams, reward = net-Sharpe contribution per step window (per-step net return divided by the rolling step-volatility estimate). Success = the policy beats rule-based exits on the CPCV path distribution from Module 19 — median and spread — not on a single backtest.

Testing It Honestly

Three rituals, in escalating order. Skipping any of them converts training time into self-deception.

Environment unit tests. Script a known action sequence through step() and assert the exact expected P&L including every cost line — spread at that hour, slippage draw, swap if the hold crossed rollover. These tests pin the simulator’s semantics; when a refactor changes a fill rule, they fail loudly instead of silently retraining your agent on a different world.

The random-agent baseline. Run an untrained/random agent through many episodes. It MUST lose approximately the expected negative edge after costs. If randomness profits, your environment leaks — look for look-ahead in observations, costs charged on the wrong side of the fill, or rewards credited before settlement. Fix the leak before a single gradient descends.

Policy-vs-rules on held-out CPCV paths only. The comparison that matters — learned policy against the incumbent rule-based exits — runs exclusively on held-out combinatorial-purged paths (Module 19). Report the path distribution (median, spread, worst path), register the trial count, and let the ledger decide promotion. Evaluating on training episodes is not a weak version of testing; it is not testing.

Hands-On Project

Deliverable: src/rl/ (env + training script + tests) and docs/research/rl-exit-policy-m8.md with the verdict memo.

Task: train PPO on the Module 8 breakout environment — R2 route candidates on M5 gold, entries fixed, measured session spreads loaded from the Module 4 curves — with reward = net-Sharpe contribution per step (drawdown-taxed as in this module). Compare against the fixed-rule exits (profit-take 1.7 / stop 1.30 ATR multiples / 128-bar expiry) on identical candidate streams. Close with a behavioral autopsy: export the learned policy’s exit decisions and ask what it actually discovered — did it rediscover something shaped like your stops? Does it exit earlier in high-spread sessions? Findings about behavior matter more than the headline Sharpe delta.

Acceptance criteria:

• Env unit tests pass: scripted action sequences reproduce exact expected P&L including spread, slippage, and swap lines.
• Random-agent baseline loses approximately the expected negative edge after costs (a profitable random agent blocks everything until explained).
• All training windows sit inside purged folds; evaluation touches only held-out CPCV paths.
• PPO exit-policy compared vs fixed R2 barriers on identical candidate streams, distributions reported side by side.
• Behavioral autopsy written: what the policy learned, mapped against barrier geometry and session clocks.
• Verdict registered in the experiment ledger with the honest trial count paid per Module 2.

Key Takeaways

• RL is policy optimization that can internalize costs and risk — but only through an honest environment; the survey consensus is that end-to-end RL trading demos rarely survive contact with real frictions.
• Reward design outweighs algorithm choice: risk-adjusted rewards (per-step Sharpe contribution, drawdown taxes) demonstrably outperform raw-PnL credit, and every reward loophole becomes a learned behavior.
• The state must include position and session clock; the episode is a purged walk-forward window — shuffling inside episodes is leakage.
• DQN fits discrete menus (exits, sizing ladders); PPO is the stable default for sequential decisions; continuous actor-critics are wasted precision on a broker’s volume grid.
• Environment engineering is the project: costs inside step(), session spreads, sampled slippage, partial fills, swap, next-bar execution — each row of the sim-to-real table needs a measurement source.
• Rank niches by honesty-feasibility: exit policies first, sizing controllers second, execution schedulers third, end-to-end signals never (research lottery).
• The random-agent test is the cheapest lie detector you own: if randomness profits, the environment leaks and nothing trained on it means anything.
• Deployment is gated on CPCV path distributions, not single backtests — the same bar every other model in Part IV cleared.

References

• arXiv 2106.00123 — review of deep reinforcement learning in quantitative trading: taxonomy, reward-design evidence, and the cost-sensitivity pattern
• WisdomChain Insights — Deep Learning & Reinforcement Learning in Algorithmic Trading 2018–2025: survey grounding the “end-to-end RL rarely survives costs” consensus
• Scientific Reports (Nature) — Multi-level deep Q-network for Bitcoin trading (s41598-024-51408-w): discrete level-based actions; fee-assumption sensitivity
• arXiv 2504.02281 — FinRL ecosystem: comparative agent/environment benchmark results
• Gymnasium documentation — environment API conventions (reset/step, termination vs truncation)
• Stable-Baselines3 documentation — shipped PPO/DQN/SAC, Monitor wrappers, reproducibility flags
• Internal continuity: Module 1 volume grid and swap spec · Module 4 spread curves · Module 5 event-driven substrate · Module 8 breakout candidates · Module 18 Meta probabilities for sizing niches · Module 19 purged/CPCV machinery
• Next in sequence: Part V begins — Module 22 — Transaction Costs, Slippage & Execution