Every quantitative researcher has produced a backtest with a Sharpe ratio above 3 that turned out to be worthless. The uncomfortable part is that the backtest was not wrong — the code ran correctly, the data was clean, the equity curve was real. The problem is that the number reported was never an estimate of future performance. It was an estimate of how hard the researcher had searched.
Selection bias has a closed form
The core result is simple enough to be uncomfortable. If you test N independent strategies that all have zero true edge, the maximum observed Sharpe ratio across those trials grows roughly with the square root of the logarithm of N. Test twenty variations of a moving-average crossover and the best one will look decent. Test two thousand and the best one will look extraordinary.
This means the headline Sharpe of a strategy is uninterpretable without knowing the size of the search that produced it. A Sharpe of 1.8 from a single pre-registered hypothesis and a Sharpe of 1.8 from a parameter sweep over ten thousand configurations are entirely different claims about the world. Almost no research process tracks the second number.
A practical checklist
Formal deflation methods exist and are worth using, but the majority of the damage in real research pipelines comes from a handful of concrete, checkable mistakes.
- Count your trials and write the number down. Including the ones you abandoned, including the ones from last month on the same dataset. If you cannot state N, you cannot interpret your result.
- Audit for look-ahead in your data, not your code. Point-in-time correctness is the issue: restated fundamentals, survivorship-filtered universes, index membership applied retroactively. Your code can be perfect and still see the future because the vendor’s database does.
- Model costs before you model alpha. Half-spread, market impact scaled to your participation rate, borrow cost, and a realistic fill assumption. A strategy that is only profitable at the mid price is not a strategy.
- Check turnover against the holding period you claim. A signal with a one-day decay half-life and a stated monthly rebalance is measuring something other than what you think.
- Use combinatorial cross-validation rather than a single hold-out. One out-of-sample period is one draw. Once you have looked at it twice, it is in-sample.
The hold-out set problem
Everyone knows to reserve an out-of-sample period. Far fewer research processes actually protect it. The failure mode is gradual: a strategy fails out-of-sample, the researcher adjusts a filter and re-runs, and the hold-out silently becomes a second training set. After a dozen iterations there is no untouched data left, and no record that this happened.
Out-of-sample data is a consumable resource. Every time you look at it, you spend some. Budget it like capital.
The organisational fix is more effective than any statistical one: log every backtest run automatically, with its parameters and its result, to a store the researcher cannot edit. This makes the trial count observable and it makes hold-out consumption visible. Teams that do this find their reported Sharpe ratios drop noticeably — and their live-versus-backtest tracking error improves by more than the drop.
Priors beat statistics
The most reliable filter is not a test statistic. It is the question of whether there is a plausible economic reason for the effect to exist and to persist: someone with a non-economic motive on the other side of the trade, a structural constraint forcing a flow, a risk genuinely being borne. A strategy with a coherent story and a modest Sharpe survives contact with live markets far more often than a spectacular one with no explanation.
That is not a rigorous criterion, and it will occasionally cause you to discard something real. It will more often stop you from allocating capital to a pattern that exists only in the shape of your search.