Elite tier
Backtest Detective
Backtest Detective hands you a strategy report that may have flaws planted in it: a cherry-picked window, costs left out, order blocks marked in hindsight, filters fitted to the sample. You flag the flawed lines against four desk rules, then set the average R you would expect out of sample.
Free, with a verified email address. An account saves your progress, and floors and tiers open as you level up.
What it teaches
Reading a backtest the way a sceptic would.
- Hindsight bias: zones that could only be drawn after the move.
- Cherry-picked windows, and tests that leave out part of the data.
- Costs: spread and slippage left out of a test.
- Overfitting: too many fitted parameters for the number of trades. See does SMC work.
How a round plays
- A report appears as seven lines: test window, trades, win rate, average R, costs, zones and filters. Four desk rules are listed under it.
- Tap every line that breaks a rule. There may be none, and a good-looking result is not a flaw by itself.
- Check. The Lens rings the flawed lines and the desk re-runs what can be measured again.
- Then set the average R you would expect out of sample: start from the re-run, take off the typical cost if none was charged, and halve it if the filters are overfit.
- A run is five reports with two questions each and takes about five minutes.
How it is graded
Each report has a fixed set of planted flaws. You are graded on flagging them without flagging results that are fine, and on an out-of-sample figure within 0.05R of the one the stated desk rule gives. The limit on fitted parameters and the halving for overfit filters are rules of thumb, and the game says so.
The honest note
The reports are invented for the game. The flaws in them are the ordinary ones, and most strategy results shared online have never been checked for any of them. A backtest is a ceiling, not a forecast.