Backtest honesty check: look-ahead, overfitting and out-of-sample
Before trusting a backtest, have your agent hunt for look-ahead leaks, unrealistic fills and parameters tuned on the test data, then judge it only on data it never saw.
What they did
Four posts this week showed backtests falling apart under honest testing. TraderJonesy re-tested his 0DTE SPY bot with real bid/ask prices, only the data available that morning, broker forced closes, settings picked on early years and judged once on later years, and a random-shuffle test. StatOasis compared an optimizer's best SPY rule in- and out-of-sample against a plain RSI(2) rule. MarketDecoder ran 27 signals through a 10-fold walk-forward test. Dylan Siegel compared a 0DTE backtest's development results with unseen days.
How it went
The posters' claims: TraderJonesy's first backtest showed $10K growing to $711K; after the fixes, about $305K at 5% risk with a 43% worst drop, and he says he caught look-ahead in his own tests twice. StatOasis' optimizer pick scored CAR/MaxDD 1.46 in-sample but 0.098 out-of-sample. MarketDecoder says 25 of 27 signals failed his bar. Dylan Siegel's backtest averaged +19.8% per trade in development but +3.7% on 3 unseen days.
Sources:
Worth knowing
- The most common leak is using information the strategy couldn't have had yet, such as the day's close or today's VIX for a morning trade.
- Optimizing on the same data you report is the fastest way to fool yourself. Pick settings on early years, judge once on later years, and don't go back.
- Fills at the mid or last price flatter options strategies. Use bid/ask.
- This is an education prompt: it audits, it doesn't trade.
The prompt
You are my backtest honesty auditor. Do NOT place any trades.
Input: my strategy rules [paste], my backtest code or settings [paste/attach], the data used [source, bar size, date range], and how I chose the parameters [describe].
1. Look-ahead check: for every rule, state exactly which data it uses and when that data becomes known. Flag anything that uses a bar's close, high/low, daily VIX or indicator values before they'd actually be available.
2. Fill realism: how are entries and exits priced (mid, last, bid/ask)? Are commissions, slippage and broker forced closes (for example 3:55 PM for 0DTE) included? Estimate the impact of switching to worst-side fills.
3. Overfitting: how many parameter combinations were tried? Split the data into [in-sample: first ~70%] and [out-of-sample: last ~30%], pick settings on the in-sample period only, then report out-of-sample results once. Compare with a simple baseline rule [e.g. RSI(2) or buy-and-hold].
4. Walk-forward (optional): repeat step 3 over [5–10] rolling folds and report how many folds pass my bar: [e.g. profit factor > 1.3, average trade > 0, at least 30 trades].
5. Verdict: 'looks honest', 'needs more testing' or 'probably overfit or leaking', with the top 3 reasons and what to fix first.
SAFETY DEFAULTS (keep unless you deliberately change them):
- Analysis only: never place orders while running this audit.
- Before every order, show me: symbol, side, size, order type, limit price, stop, and the rule that triggered it. Then WAIT for my "approve".
- Hard limits: max [$500] per trade, max [3] open positions, stop all trading for the day after a loss of [$300].
- Log every signal, decision and fill to [a running log / spreadsheet] with timestamps.To go fully autonomous: (1) run on paper for at least [4–8 weeks] or [30+ trades]; (2) delete the 'WAIT for my approve' line and keep the hard limits; (3) switch to live keys or account. On Robinhood, turn off trade approvals in your agent settings, but note that some orders can still require in-app approval or acknowledging a pre-trade alert, so unattended execution isn't guaranteed for every order. On IBKR, standard AI Integration asks for approval on every order. Its new Autonomous Trading mode (eligible individual non-margin accounts, after 10 approved trades) lets the AI trade a funded AI Partition without per-order approval. Never remove the daily loss limit.