Parameter Sensitivity: Test Whether Your Trading Strategy Is Robust
Last saved:
Executive Summary
- ✦Small changes in settings can expose fragile historical results.
- ✦A three-by-three parameter grid contains nine related configurations, not nine independent confirmations.
- ✦The hypothetical best return of 18% falls to 1% or −2% with neighbouring stop settings.
- ✦Inspect costs, drawdowns and trade counts, then freeze the selection rule before independent evaluation.
A strategy that works with a 20-day setting but collapses at 19 or 21 days deserves investigation before capital. Parameter sensitivity asks how much the result changes when a rule's inputs change modestly. It helps distinguish a broadly usable idea from one unusually lucky historical setting.
Learning objective: build a small sensitivity table, identify an isolated performance peak, and decide what needs independent validation. This is a research lesson, not a trading signal.
1. Separate the trading idea from its settings
The idea might be that persistent trends can continue. The settings could include a breakout lookback and an initial stop distance. A parameter is an adjustable input; changing it may alter entries, exits, trade count and exposure.
QuantConnect's parameter documentation warns that impressive historical performance which is sensitive to small parameter changes can indicate overfitting. That is a warning, not proof: genuine market thresholds and limited samples can also create uneven results.
Before testing, write down the mechanism, the parameters worth varying and the result that would make you reject the idea. Avoid inventing new filters each time a disappointing result appears.
2. Build a controlled comparison
Choose a modest, economically sensible range around the intended settings. For this exercise, compare breakout lookbacks of 18, 20 and 22 completed daily bars, with initial stops at 1.8, 2.0 and 2.2 times ATR. ATR is an average range measure; keep its calculation fixed across runs.
Use the same instrument universe, historical period, starting capital, execution assumptions and cost model. Keep the position-sizing rule fixed too: if it sizes to a fixed risk budget, a wider stop should change quantity according to that same rule. Do not confuse changing risk exposure with finding a better signal.
A three-by-three grid contains nine configurations. Log all nine, including failures. It is a research comparison, not nine independent pieces of evidence.
3. Worked hypothetical example: the suspicious centre
The table below is invented for teaching, not a backtest we ran. Each cell represents total return over the same hypothetical period, after modelled costs and before tax; the figures are not annualised.
| Breakout lookback | Stop: 1.8 × ATR | Stop: 2.0 × ATR | Stop: 2.2 × ATR |
|---|---|---|---|
| 18 daily bars | +2% | +3% | +1% |
| 20 daily bars | +1% | +18% | −2% |
| 22 daily bars | 0% | −1% | +2% |
Starting with a hypothetical £10,000, the central setting ends at £11,800. Keeping the 20-bar lookback but moving the stop to 1.8 × ATR ends at £10,100; moving it to 2.2 × ATR ends at £9,800.
That is a 17-percentage-point drop from +18% to +1%, and a 20-percentage-point drop from +18% to −2%. The eight neighbouring cells range from −2% to +3%. The centre is an isolated peak, so the next step is to inspect trade-level differences and data handling—not declare 2.0 × ATR the ideal stop.
Were the profits concentrated in one trade? Did different parameters change the order of entry and exit? Did a gap or ambiguous candle change a simulated fill? Our same-candle backtest lesson explains why execution assumptions can create apparent precision.
4. Examine a region, then test beyond it
A broad area of acceptable results is more reassuring than one spike, but a plateau can still be overfitted. Inspect drawdown, trade count, turnover and exposure alongside returns. A smooth grid of three-trade samples is weak evidence; neighbouring configurations may also share most of their trades.
Choose your selection rule using development data, freeze it, then evaluate on a later period that did not influence the choice. Backtrader's own guide cautions that optimisation can make an unsound idea appear profitable only on the tested dataset. Our walk-forward guide covers sequential evaluation. If you revise the strategy after seeing the evaluation period, treat that period as used research data.
Common mistakes
- Reporting only the highest-return cell while hiding the rest.
- Expanding the search repeatedly until a pleasing plateau appears.
- Ignoring costs, drawdown or changes in position size.
- Treating nearby settings as independent confirmation.
- Assuming stability in one market regime guarantees stability in another.
The Money Man View and practical checklist
For a daily strategy, review sensitivity during scheduled research—not after every losing trade. A stable region justifies further testing; an unexplained spike calls for investigation. If the effect disappears with realistic costs or on untouched data, the claim of robustness is weakened.
Before accepting a configuration, check:
- The hypothesis and parameter ranges were recorded before comparing results.
- All configurations used consistent data, risk rules and costs.
- Returns, drawdowns, trade counts and exposure were reviewed together.
- Exceptional trades and fill assumptions were inspected.
- The selection rule was frozen before independent evaluation.
- Failures were retained in the research log, without promises of future profit.
The Money Man uses AI-assisted research and writing. This is general trading education, not personalised investment advice. All example returns are hypothetical; no strategy performance or trade execution is claimed. Backtests cannot guarantee future results, and capital is at risk. The AI-generated cover is a conceptual illustration, not a data visualisation.