Today, we'll discuss one of the main reasons why algorithmic traders are often unsuccessful in actual trading, even though their test results are perfect: curve fitting.
In this post, we will take a quick dive into what curve fitting is and how it is manifested, looking at one particular example for additional clarity on the matter.
First things first – definitions.
What is curve fitting?
It’s basically the process of creating a trading strategy that is overly optimized to fit historical data. This involves tweaking a trading model to the point where it performs exceptionally well on past data but fails to generalize to new, unseen data.
Curve fitting is often associated with backtesting, where traders test their strategies using historical market data to evaluate their potential effectiveness.
Why does it happen?
The answer is diverse, as curve fitting comes in all shapes and sizes. There are countless ways to curve-fit the algorithm to past data so that it will only perform well on the data it has seen, but there are only a few ways to get it right.
One of the most frequent ways it happens is by having too many core parameters and optimizing all of them together. The solution to this is to have simpler models that are complex just enough to capture the essence of existing market inefficiencies but not too complicated to avoid curve fitting. It may be counterintuitive, but often, a simpler system is a better one.
The gist is that each part of your trading model should capture real, existing market inefficiencies. A good trader should understand how each part of the trading system connects to the underlying market effects it is designed for. If some part of your system captures noise instead, it will lead to curve fitting.
How can one avoid curve fitting?
The simplest answer is that each part of the trading system should capture an existing market inefficiency in a robust way. Let’s clarify this with a simple example.
Suppose we are designing a simple recommendation system that tells us when to buy the S&P 500 index and when to sell it. We’ve taken the last 10 years of data and noticed one particularity – in 8 years out of 10, June closed at a loss. Suppose the probability of closing the month at a loss is 50%. In this case, the probability of having 8 negative months out of 10 is 45/1024, or about 4.4%. If we have a 95% probability threshold to consider something an inefficiency, we satisfy it as 100-4.4=95.6>95%
The calculation is correct, but the main mistake here is that we have also looked through each other month, and only June had 8 negative years out of 10. If we consider the fact that we studied data from all 12 months, the probability of having 8 negative months out of 10 increases significantly to 1-(1-0.044)^12=41.7% . So, we can have 8 negative months out of 10 simply by chance! And the “not trading in June” criteria will likely capture random noise, not an existing market inefficiency.
In reality, even having 10 Junes or Octobers closed at a loss in a row is not that improbable. Almost always, taking a particular month out of trading hurts more than it helps. But that’s just an illustration of a more general case. In trading, there are all kinds of “random noise” mistakes one can make, and very often, using logic, reason, and common sense is the best way to avoid them
There are various ways of dealing with curve fitting: cross-validation, out-of-sample testing, walk-forward analysis, regularization, and many other complicated trading terms.
The most important one, at least from our point of view, is using sound logic when developing the system to make sure that each part of it captures an existing market tendency.
For example, the logic used when developing the Algocrat AI's proprietary trading system.
Hope you find this helpful!
Best regards,
The Algocrat AI Team
