Multiple Testing in Backtests: Why the Best Strategy May Be a False Winner

Multiple Testing in Backtests: Why the Best Strategy May Be a False Winner

বাংলায় পড়ুন

Evergreen trading tutorial: The examples below are hypothetical and simplified. Statistical evidence from historical data does not guarantee future performance. This is educational information, not personalised advice or a trading recommendation.

A trader tests 100 strategy variations and keeps the one with the best equity curve. The result may look disciplined, profitable and statistically convincing. It may also be the luckiest member of a weak group.

This is the multiple-testing problem. Every extra market, indicator, lookback, stop, filter and date range gives chance another opportunity to produce an attractive result. If the research process reports only the winner, the backtest can materially overstate the evidence for a real edge.

Learning objective

By the end of this guide, you should be able to:

• explain why testing many variants increases false discoveries;
• count the main choices that create hidden trials;
• separate research, validation and final test data;
• apply simple multiple-comparison safeguards; and
• deploy an unproven strategy with a smaller, explicit risk budget.

One test versus many tests

Suppose a statistical test uses a 5% significance threshold. Under its assumptions, a strategy with no real edge still has a 5% chance of appearing significant in one test.

If 100 independent no-edge variants are tested, the probability that at least one appears significant is:

1 − (1 − 0.05)¹⁰⁰ = about 99.4%

Real strategy variants are rarely independent; moving-average lengths of 19 and 20 days are highly related. That means 99.4% is not a universal answer for a real research project. The calculation illustrates the direction of the problem: the probability of finding a lucky winner rises as the number of opportunities rises.

The relevant number of trials is not just the final strategies saved in a spreadsheet. It includes abandoned ideas, markets screened, timeframes viewed and parameters changed after seeing results.

Where hidden trials appear

A backtest may quietly multiply its tests through:

• 20 lookback lengths;
• 5 stop-loss distances;
• 4 profit-taking rules;
• 10 markets;
• 3 timeframes;
• several start and end dates; and
• repeated filters added after inspecting losing periods.

Testing every combination would create 20 × 5 × 4 × 10 × 3 = 12,000 variants before date changes and manual judgement. Selecting the best Sharpe ratio from that search and presenting it as one planned hypothesis hides the selection process.

A worked hypothetical example

Assume a researcher builds 100 variations of a daily breakout strategy. Each uses the same asset class but changes lookback, volatility filter and exit rule. The best variation reports:

• annualised return: 18%;
• maximum drawdown: 9%;
• Sharpe ratio: 1.6; and
• nominal p-value: 0.03.

If this had been the only pre-specified test, a p-value of 0.03 might be treated as evidence under a 5% threshold. It was selected from 100 trials, so the 0.03 cannot be interpreted as if no selection occurred.

One very conservative illustration is a Bonferroni adjustment:

Adjusted threshold = 0.05 ÷ 100 = 0.0005

The selected result's 0.03 does not pass that threshold. Bonferroni can be overly conservative when tests are correlated, but it demonstrates why the original 5% cutoff is too generous after a large search.

The researcher then freezes the complete rule and evaluates it on an untouched chronological holdout period. Suppose the holdout shows:

• annualised return: 4%;
• maximum drawdown: 14%;
• Sharpe ratio: 0.3; and
• costs remove most of the remaining profit.

The strategy did not necessarily “stop working”. The original winner may have been an inflated estimate produced by selection. The holdout provides a less biased view of what happened after the rule was fixed.

A step-by-step research process

1. Write the economic or behavioural hypothesis first

Explain why the edge might exist and who is expected to pay for it. “This parameter produced the highest return” is a result, not a hypothesis.

Examples of testable reasoning include delayed reaction to information, compensation for providing liquidity or persistent institutional constraints. A plausible story does not prove an edge, but it narrows arbitrary searching.

2. Record every research choice

Maintain a trial ledger containing:

• markets and date ranges screened;
• parameters tested;
• filters added or removed;
• cost assumptions;
• missing-data treatment;
• rejected versions; and
• the metric used to select the winner.

This “researcher degrees of freedom” record makes the real search size visible.

3. Split data chronologically

A practical structure can include:

1. Research set: develop the basic idea;
2. Validation set: choose among a limited number of pre-defined alternatives; and
3. Final holdout: evaluate the frozen rule once.

Time series should normally keep chronological order. Randomly shuffling observations can leak future regimes into the past and ignore serial dependence.

4. Limit and adjust comparisons

Pre-register a small family of sensible parameters instead of searching every value. Where formal inference matters, consider a method designed for multiple comparisons, such as family-wise error control, false-discovery-rate control or specialised backtest-selection tests.

No adjustment repairs bad data, look-ahead bias or unrealistic execution. It addresses only part of the research problem.

5. Use walk-forward testing

Repeatedly train on past data and test on the next unseen block. This reveals whether parameters remain stable as regimes change. Our walk-forward guide explains the process.

Walk-forward testing still creates multiple observations and requires honest aggregation. Do not discard the windows that failed.

6. Include realistic implementation

Model spread, commission, slippage, borrow availability, funding, futures rolls, taxes where relevant and delays between signal and execution. A tiny statistical edge can disappear when implementation is added.

Our backtesting guide covers look-ahead, survivorship bias and trading costs.

7. Freeze before final evaluation

After the holdout is opened, do not keep editing the strategy against it. Every change converts part of the holdout into training data. If a change is necessary, document it and obtain a new untouched test period.

From research evidence to position risk

Even a strategy that survives validation remains uncertain. Deployment size should reflect that uncertainty.

Suppose a £20,000 account chooses a hypothetical initial risk budget of 0.25% for a newly validated setup:

£20,000 × 0.25% = £50 planned cash risk

If entry is £30 and invalidation is £28.75, the distance is £1.25:

£50 ÷ £1.25 = 40 units before costs.

This sizing example does not make the backtest valid. It limits the consequence of being wrong while live evidence accumulates. If slippage or gaps are material, theoretical size should be lower.

Increasing size immediately because a backtest had a high Sharpe ratio compounds selection risk. A better process defines what live sample, cost and drawdown evidence would justify scaling—and what would trigger a reduction.

Parameter stability matters more than one peak

A robust strategy should not require one exact parameter to work. Plot performance across neighbouring settings. A broad plateau is usually more credible than a single sharp optimum surrounded by failure.

For example, if lookbacks from 18 to 24 days produce similar results but 21 days is slightly best, that is more reassuring than a system where only 21 works and 20 or 22 loses heavily. Stability is not proof, but fragility is useful warning evidence.

Common mistakes

1. Counting only the winner as one test. Rejected variants still influence selection.
2. Using the same data for discovery and proof. The result is optimised and evaluated on the same noise.
3. Changing the rule after opening the holdout. The holdout stops being unseen.
4. Reporting one Sharpe ratio without the search size. The number ignores selection.
5. Trying many start dates until the curve looks smooth. Date selection is another trial.
6. Ignoring failed markets or timeframes. Selective reporting inflates confidence.
7. Using unrealistic costs. A small edge may exist only before implementation.
8. Preferring the sharpest parameter peak. It may be the most overfit point.
9. Scaling quickly after one live win. A tiny sample cannot validate the backtest.

Practical checklist

• The economic or behavioural hypothesis is written before the search.
• All markets, parameters, filters and date ranges are logged.
• Selection metric and acceptance criteria are pre-defined.
• Research, validation and final holdout periods are chronological and separate.
• The final rule is frozen before the holdout is opened.
• Multiple comparisons are limited or adjusted appropriately.
• Neighbouring parameter stability is examined.
• Failed variants and windows are retained in the report.
• Costs, slippage, data quality and execution delays are realistic.
• Initial live risk is small and scaling rules are defined in advance.

The Money Man View

The best backtest in a large search is guilty until proven on unseen data. A credible research report explains how many opportunities chance had to win, preserves failed trials and separates discovery from evaluation. Simpler rules, fewer tests, stable neighbouring parameters and modest initial live risk usually provide more useful evidence than one spectacular historical curve.

Educational risk disclosure: Backtests are hypothetical and can lose money in live trading. Historical relationships can change, and statistical adjustments cannot remove execution, liquidity or model risk. The figures above are simplified examples, not recommendations. This article was prepared with AI-assisted research and writing under The Money Man editorial process.