Overfitting
Tuning settings until they fit past ups and downs that won't repeat.
3 min readLesson 56 of 57Interactive
In short
Try enough settings and one will look brilliant by luck. The test is whether neighboring settings also do well, and whether it holds up on a period you didn't tune it on.
Why it matters
A backtest can be made to look brilliant. Tweak the settings enough times and some combination will have dodged every past crash and caught every rally. The trouble is that it learned the past's particular noise, not anything that will happen again.
This is the most common way backtests mislead people, including professionals. Knowing how it happens is what turns a backtester from a machine for fooling yourself into a useful tool.
How it works
Past prices mix two things: patterns that might persist, and random noise that won't. Every setting you try is a new chance to fit the noise. With one setting, a good result is some evidence. After trying hundreds, the best result is mostly a measure of how many you tried.
There are two good defenses. First, look at the neighborhood: if 50 and 200 days work well, 45 and 210 should work nearly as well. A strategy whose results collapse when a setting moves a little was probably tuned to luck. Second, test out of sample: choose the settings on one period, then check them on a different period you didn't look at.
Fewer moving parts also help. Each extra rule or setting gives the strategy more freedom to fit the past, and makes a lucky fit more likely.
A parameter sweep, which runs the same backtest over a whole grid of settings, is the right tool for the first check and the wrong tool for picking a winner. Use it to see how sensitive a strategy is to its settings, not to find the one that did best.
A worked example
You test a crossover with 10 choices of fast average and 10 choices of slow average. Suppose none of them has a real edge, and each has a 5% chance of looking great by pure luck.
- Combinations tested: 10 × 10 = 100.
- Expected lucky winners: 100 × 5% = 5 settings that look great.
- If the results were independent, the chance that at least one looks great is 1 − 0.95¹⁰⁰ ≈ 99.4%.
- Settings close to each other give similar results in practice, so the true chance is lower, but the point stands.
Result: Testing 100 settings almost guarantees a winner, even when no setting has any edge. The best cell alone proves very little; a broad area of good results is far more convincing.
Try it
The best setting, then the next two years
Years 1–2: tuning
- 5 days+1.8%
- 10 days−0.8%
- 15 days+1.7%
- 20 days+20.3%
- 25 days+24.3%
- 30 days+11.2%
- 35 days+9.1%
- 40 days+8.4%
- 45 days+4.5%
- 50 days+5.9%
- 55 days+3.7%
- 60 days+4.1%
Years 3–4: the test
- 5 days−9.1%
- 10 days−10.5%
- 15 days−11.5%
- 20 days−12.4%
- 25 days−15.5%
- 30 days−22.2%
- 35 days−10.2%
- 40 days−8.5%
- 45 days−3.8%
- 50 days−7.8%
- 55 days−8.9%
- 60 days−8.7%
Best while tuning
25 days
+24.3%
Same setting, next 2 years
−15.5%
Its rank in the test
11 of 12
The best setting in years one and two, a 25-day fast average, returned +24.3%; in years three and four it returned −15.5%, ranking 11 of 12.
A 100-day slow average with 12 fast settings, on generated prices. Try other markets to see how often the winner repeats.
How to read it
- One standout cell in the sweep
- A warning sign. A single great setting surrounded by poor ones is probably luck.
- A broad area of good results
- More convincing. Many nearby settings working suggests the strategy isn't hanging on one exact number.
- Great in one period, poor in the next
- The classic sign of overfitting: the settings fit the first period's ups and downs, which didn't repeat.
In the real world
- Researchers have found that many market patterns reported in academic studies weakened noticeably after they were published. Some had been partly luck, and others were traded away once more investors knew about them.
- Famous joke correlations make the point: people have found past stock returns that lined up with things like Super Bowl winners or butter production in faraway countries. Search enough data and coincidences are guaranteed.
- Professional quant firms set aside data they never use for tuning, limit how many variations they test, and are skeptical of backtests that look too smooth.
Common mistakes
- Picking the best cell in a sweep and expecting it to keep winning. The best past setting rarely stays best.
- Testing on the same period you tuned on. A strategy has to be checked on data it hasn't seen.
- Adding rules until the backtest looks perfect. Each extra rule makes it easier to fit noise.
- Trusting a very high Sharpe ratio or a tiny max drawdown from a backtest. Results that look too good usually are.
See it in MarketLab
Test a range of settings →
Opens a crossover backtest on Apple. Under the results, use Test a range of settings to sweep both averages and see whether good results form a broad area or a lone spike.
Backtest →
Change the start and end dates to check settings you chose on one period against another.
Machine learning sandbox →
The Machine learning tab tests models walk-forward against a shuffled-label baseline, so you can see how often an apparent edge is luck.
Test yourself
Optional: 4 quick questions on the words, using the idea, and how it plays out in real markets.
Loading your answers…