Luck or skill?
Checking a result against random versions of the same strategy, and against reshuffles of its own days, to see how much luck could explain.
5 min readLesson 17 of 57
In short
A strategy that beat 95% of random versions making the same kind of choices has some evidence behind it; one that beat 60% looks like chance. A bootstrap range shows how far its Sharpe ratio could have landed with the same kind of days in a different order.
Why it matters
A strategy beat the market over five years. Was it a good idea, or a lucky draw? From one backtest you can't tell: markets are noisy enough that plenty of strategies with no edge at all will beat a benchmark over any given stretch, just as some coin-flippers throw five heads in a row.
The honest way to judge a result is to ask what luck alone would have produced. If random choices of the same kind usually did as well, the strategy hasn't shown anything yet. Only when it beats nearly all of them is there something worth a closer look.
How it works
The first yardstick is a set of random versions of the same strategy. They take the same kind of decisions as the real one, but at random. For a timing strategy, like a moving-average crossover or custom rules, each random version holds each stock for as many separate stretches, and as many days in total, as the real one did, but at random dates. For a picking strategy, like momentum rotation, risk parity or optimised weights, each one rebalances on the same days with the same weights, but hands them to the stocks in a random order.
The result is a percentile: the share of random versions the real run beat. Beating 71% means luck alone did at least as well 29% of the time, which is what chance often gives. Beating 97% is harder to explain by luck, though one period is still only one period. Strategies that make no choices, like buy and hold, dollar-cost averaging or Rebalancing, have nothing to randomise, so they get no random versions.
The second yardstick is a bootstrap: a range for the Sharpe ratio. Take the run's own daily returns, cut them into blocks of 21 trading days, and draw blocks at random, with repeats, to build a new history of the same length. Do that 2,000 times and work out the Sharpe ratio each time. The middle 90% of those values is the range the result could plausibly have landed in with the same kind of days in a different order.
Blocks matter because markets have moods. Calm weeks tend to follow calm weeks, and stormy ones follow stormy ones. Shuffling single days would break those stretches up and make the range look narrower than it should. The benchmark is reshuffled on the same days, so you also see how often the strategy's Sharpe ratio came out ahead of the benchmark's.
Both checks assume you ran one strategy. If you tried twenty settings and kept the best, its percentile overstates the evidence, which is the problem of Overfitting; the parameter sweep's deflated Sharpe ratio is built to allow for that.
Share beaten = random versions that did worse ÷ all random versions; chance by luck = 100% − share beaten
How unusual the run is next to strategies that made the same kind of choices at random.
A worked example
A moving-average crossover on one stock made +34% over five years, with a Sharpe ratio of 0.90. Its Was it luck? check runs 1,000 random versions and 2,000 reshuffles.
- Random versions: 710 of the 1,000 made less than +34%, so it beat 710 ÷ 1,000 = 71% of them. Luck alone did at least as well 100% − 71% = 29% of the time.
- Blocks: five years is about 1,260 trading days, or 1,260 ÷ 21 = 60 blocks of 21 days. Each reshuffle draws 60 blocks at random and works out a new Sharpe ratio.
- Range: sort the 2,000 Sharpe ratios. The low end of the middle 90% is the 2,000 × 5% = 100th from the bottom, here 0.10, and the high end is the 1,900th, here 1.70.
- Many tries: had you tested 20 settings with no edge at all, the chance that at least one beats 95% of its random versions is 1 − 0.95²⁰ ≈ 64%.
Result: Beating 71% of random versions is what chance gives about three times in ten, and a Sharpe ratio that could be anywhere from 0.10 to 1.70 is too uncertain to lean on. The fair verdict: no evidence of skill yet, and none against it.
How to read it
- Beat 95% or more
- Luck alone rarely does that well, so the choices may have mattered. Check it holds on another period before trusting it.
- Beat 75% to 95%
- Better than most random versions, but luck does this often enough to be suggestive, not convincing.
- Beat 25% to 75%
- What chance typically gives. No sign the strategy's choices added anything.
- Sharpe range includes the benchmark's, or zero
- With the same kind of days in another order, the result could easily have been no better. Treat the edge as unproven.
In the real world
- Burton Malkiel wrote in 1973 that a blindfolded monkey throwing darts at the stock pages could pick a portfolio as well as the experts. Researchers have since found that random portfolios often match or beat the indexes they're drawn from, which is why random strategies make a fair yardstick.
- S&P's persistence reports find that funds in the top quarter one year rarely stay there: only a small share are still in the top quarter a few years later, about what luck would predict.
- The bootstrap was introduced by statistician Bradley Efron in 1979, and the block version for data that comes in streaks followed in the late 1980s. Quant researchers use both to put ranges around backtest results instead of quoting a single number.
Common mistakes
- Treating one good backtest as proof. Beating the market once is something many strategies with no edge do.
- Reading a 71st percentile as "71% likely to be skill". It means luck alone did as well 29% of the time, not that the strategy is probably good.
- Checking only the best of many settings. Its percentile looks impressive because you picked it; that's Overfitting.
- Treating the bootstrap range as a forecast. It only reshuffles the past, so a future unlike any stretch in it isn't covered.
See it in MarketLab
Was it luck? on a crossover →
Opens a crossover backtest on Apple. Under the results, press Was it luck? to see the share of random timings it beat, a histogram of their returns and a 90% range for its Sharpe ratio.
Was it luck? on momentum →
A picking strategy: here the random versions rebalance on the same days but give the top weights to stocks chosen at random.
Was it luck? for your own account →
The Portfolio page's Review tab compares your real paper trading with 1,000 random traders who keep your habits: the same number of trades and holding periods at random dates, or your dates with random stocks.
Test yourself
Optional: 4 quick questions on the words, using the idea, and how it plays out in real markets.
Loading your answers…