Testing

How to avoid overfitting a backtest

By Koen van Goch · Updated

You avoid overfitting by starting from a logical reason for the edge, keeping rules and parameters few, checking that nearby parameter values give similar results, and confirming the system on data you didn't use to build it. An overfit backtest describes the past beautifully and predicts nothing. If a result looks too good, it usually is.

What is overfitting (curve fitting)?

Overfitting means tuning a system so closely to historical data that it captures noise instead of a real, repeatable behaviour. Every rule and parameter you add gives the backtest more freedom to fit the past. The result is an equity curve that looks smooth in testing and falls apart with real money.

Warning signs of an overfit backtest

  • Many rules or filters, each added because it "improved the results".
  • One parameter value works brilliantly while its neighbours don't. For example, a 47-day lookback works but 40 and 55 don't.
  • Very high win rates, or a very smooth equity curve with shallow drawdowns. Real systems have ugly periods, and a backtest that hides them is usually too good to be true.
  • You can't explain why the system should make money.
  • Results depend on a handful of trades or one specific year.
  • The test ignores commissions, slippage or delisted stocks.

Checks that catch it

  • Logic first. Write down why the edge should exist before you look at results.
  • Few rules, round numbers. Prefer a 50-day or 100-day average to a "perfect" 63.
  • Parameter stability. Test a range of values. A robust system shows a broad plateau of decent results, not a single sharp peak. Pick a value from the middle of that plateau, not the best one.
  • Out-of-sample data. Hold back years or markets you never look at during design, and test on them once at the end.
  • Different markets and regimes. Does it behave sensibly in bull, bear and sideways periods, and on similar markets?
  • Realistic costs. Include commissions, slippage and realistic fills. For stocks, use survivorship-bias-free data.
  • Monte Carlo or trade shuffling. Resample trades to see the range of drawdowns you could realistically face, not only the single historical path.
  • Live or paper tracking. Compare live results with the backtest's expectations before scaling up.

The data-mining trap

Testing thousands of rule combinations and keeping the best is the fastest way to overfit. With enough tries, something will look great by chance. That applies just as much when an AI tool does the searching. Use AI to write and check code, not to hunt for patterns.

Robust beats optimised

An optimised system has the best backtest. A robust system has a slightly worse backtest that changes little when you nudge it. Robust systems are the ones you can still trust after a bad month. Choose robustness. The same thinking runs through how to build your own trading system.

Illustration, not real data. Each square is one combination of two parameter settings; brighter means better backtest results.

FAQ

Common questions

How much data do I need to avoid overfitting?

Enough to include different market regimes (bull, bear, sideways) and enough trades for the results to mean something. More rules need more data. Fewer rules need less.

Is walk-forward analysis enough?

It helps, but it isn't a guarantee. Walk-forward testing can itself be overfit if you keep re-running it with different settings. Combine it with logic, few parameters and stability checks.

Why are my live results worse than my backtest?

Usually a mix of overfitting, unrealistic costs or fills, survivorship bias in the data, and simply being in a different market regime. Some gap between live and backtest is normal. A large, persistent gap means something in the test was too optimistic.

What's an out-of-sample test?

A test on data that was never used, or looked at, while designing the system, such as the most recent years or a different set of markets. It's only valid the first time. If you tweak and test again, it becomes in-sample.

Can optimisation ever be useful?

Yes, to understand how sensitive a system is to its parameters. Use it to find stable regions, not the single best value.

Want someone to stress-test your backtest before you trade it?

That's a core part of every programme. Request a free 30-minute introductory call.