Is my backtest just luck? Shuffle test and data-mining check
Before going live, make your agent shuffle your backtest trades to see the drawdown you should really expect, and check whether your 'edge' beats rules mined from random data.
What they did
Four posts this week showed how a good-looking backtest can mislead. Derek reshuffled the order of his backtest trades many times. StatOasis mined 5,000 rules on SPY and compared the best with the best rule mined on zero-edge data. BacktestingMax ran 24 day-trading strategies against simply holding the index. Bernard the Autotrader shared a live book with a high win rate.
How it went
The posters' claims: Derek's backtest drawdown was 12%, but the median shuffled drawdown was 15% and the 95th percentile 26%, so he sizes and sets his stop-trading rule around 26%. StatOasis' best mined SPY rule made 11.2% a year, but the best rule on zero-edge data made 6.4%, so the real edge is closer to 4.8%. BacktestingMax: with $10,000 in 2024–25 after costs, holding ended at $14,350 versus $13,540 for the best day-trading strategy, which was ahead on only 52 of 502 days. Bernard: 257 live trades, 70% winners, yet −2.46% net on $100K, because the average winner was +0.02% and the average loser −0.08%.
Sources:
Worth knowing
- Your backtest's max drawdown is one lucky ordering of trades. Plan for the bad orderings.
- If you tried many rules, the best one looks good partly by chance. Compare it against noise.
- Win rate means little without the average win and loss.
- This is an education prompt: it audits, it doesn't trade.
The prompt
You are my backtest auditor. Do NOT place any trades.
Input: my backtest trade list [paste or attach CSV: date, P/L per trade] and how many rule variations I tried: [number].
1. Shuffle test: randomly reorder the trades [1,000] times. Report the median and 95th-percentile max drawdown versus my original drawdown. Recommend a position size so the 95th-percentile drawdown stays under [20]% of my account, and a 'stop trading and review' level.
2. Mining check: generate [same number of variations I tried] random-entry strategies with the same holding period and trade count on the same data. Report the best random result. Is my strategy clearly better?
3. Baseline: compare with buy-and-hold of [the traded asset] over the same period, and count on how many days my strategy's equity was ahead.
4. Win/loss shape: win rate, average win, average loss, and the break-even win rate implied by them.
End with a plain-English verdict: 'looks robust', 'needs more testing' or 'probably luck', with reasons.
SAFETY DEFAULTS (keep unless you deliberately change them):
- Analysis only: never place orders while running this audit.
- Before every order, show me: symbol, side, size, order type, limit price, stop, and the rule that triggered it. Then WAIT for my "approve".
- Hard limits: max [$500] per trade, max [3] open positions, stop all trading for the day after a loss of [$300].
- Log every signal, decision and fill to [a running log / spreadsheet] with timestamps.To go fully autonomous: (1) run on paper for at least [4–8 weeks] or [30+ trades]; (2) delete the 'WAIT for my approve' line and keep the hard limits; (3) switch to live keys or account. On Robinhood, turn off trade approvals in your agent settings, but note that some orders can still require in-app approval or acknowledging a pre-trade alert, so unattended execution isn't guaranteed for every order. On IBKR, standard AI Integration asks for approval on every order. Its new Autonomous Trading mode (eligible individual non-margin accounts, after 10 approved trades) lets the AI trade a funded AI Partition without per-order approval. Never remove the daily loss limit.