Before jumping in to create and use a trading strategy, it’s crucial to understand the traps in backtesting. Be aware of the different biases and errors that can sometimes give misleading or overly optimistic results. In this blog, we’ll shed light on some of the most common biases that can affect trading strategies, and share how to minimize their impact.
The Common Biases in Trading Strategies:
- Survivorship Bias
- Hindsight Bias
- Sample Bias
- Selection Bias
- Look Ahead Bias
- Recency or Market Condition Bias
- Curve Fitting and Data Mining
Survivorship Bias
Survivorship bias occurs when backtesting is conducted only on assets that have survived until now, neglecting those that were delisted , as well as those that performed poorly and lost prominence over the years. This can make your strategy look better than it actually is.
To steer clear of survivorship bias, it’s crucial to approach backtesting as if you had absolutely no idea what the future would bring at the time you placed each trade. You should imagine that you had no knowledge of which coins would succeed or fail in the long run. You should adopt a systematic approach that selects assets based on the data and information available at that specific time.
For example, if you backtest your strategy using only the top 10 coins currently, you may miss many coins that were once in the top 10 but have since disappeared from the market. These may include coins like LUNA, FTT, EOS, BSV, and others, which might have faded out due to being delisted or losing popularity. However, they would have been part of your consideration if you had been trading or conducting backtests during their heyday.
Here’s an illustrative example with our Trend Catcher strategy:
The blue graph represents the backtesting of the Trend Catcher strategy on the current top 10 assets, hence this backtest suffers from survivorship bias. We only tested on the assets that eventually became successful, naturally leading to an impressive performance. The backtest started when these assets were relatively unknown, and now they are among the top 10.
The green line, on the other hand, presents the accurate method of backtesting, and the only realistic trading scenario given the information available at that time. Trades were only made on the top 10 assets of that period. It’s a pretty big difference, no? 4 times less profit, 15% more drawdown, profit factor is almost half. This leads us directly to our next bias.
Hindsight Bias
Hindsight bias is the trap of thinking past events were more predictable than they really were. It leads to overconfidence and could make traders believe they can handpick coins for their strategies, convinced that certain coins were destined to perform well, and disregarding the ones that didn’t. This mindset neglects the unpredictable nature of the market, and such selective analysis can lead to biased results.
Consider an example of a simple Bollinger Band breakout strategy implemented on Ethereum (ETH) since 2017.
Now, compare it with the same strategy applied on IOTA:
Both of these coins were in the top 10 at the start of 2018. However, people often boast about their single-coin strategies on assets like ETH that have consistently trended well. At the same time, they completely ignore results of the same strategies on coins that have largely faded into obscurity, like IOTA. To ensure a robust strategy, it’s crucial to test it across a wide universe of multiple coins.
To sum it up, the Bollinger Band strategy in this case wasn’t always the best fit when you use it on a lot of different assets. Sure, if we knew which coins would do the best in the next five years, we’d put all our money into them. But knowing that beforehand is just hindsight bias. Unfortunately, we can’t see the future when we’re trading in real time. This leads us again directly to our next bias!
Sample Bias
Similar to survivorship bias, sample bias occurs when certain coins are intentionally included or excluded from the test based on their past performance. This cherry-picking of data skews the test results and leads to misleading conclusions.
For instance, traders might be tempted to favor coins that fit well with their strategy, including them in the backtest while ignoring the underperformers, simply because they’d spoil their backtest results.
However, this makes absolutely no sense, as we have already learned that we should always backtest as if we had no idea what the future brings. So excluding coins from your backtest to get good results is just ridiculous, as who knows what you will trade in the future. Definitely in crypto, those coins come and go so quickly.
To avoid sample bias, it’s essential to maintain an unbiased and systematic approach to selecting assets for backtesting.
Selection Bias
Selection bias is a significant concern when there are more trading opportunities than available capital, as it introduces discretion in choosing which trades to execute. This undermines the systematic nature of backtesting.
For instance, if you only have the budget for 10 trades but receive 20 signals, you must implement a systematic approach to select the signals to act upon. Failure to inform your backtest about the 10 trades you would have picked will lead to inaccurate results.
A trader’s reply on a Tweet perfectly illustrates the impact of selection bias: “I’ve completed my daily mean reversion strategy. Funny thing happened. I used the ATR to score my entries. Whilst amending some exit rules I switched it off accidentally and the backtest profit went up 50%. Needless to say, the strategy no longer has a setup score.”
This real-life incident strongly emphasizes the importance of understanding selection bias and its effects on strategy outcomes. It underscores the critical need to adopt a systematic approach when selecting trades. Selection bias can genuinely deceive your results, and you might not even be aware of its influence.
In the case of our Momentum Catcher strategy, there are considerably more potential setups or signals than we actually use.
Look-Ahead Bias
Look-ahead bias is a problem that can occur in backtesting when future data is used to make trade decisions. This can lead to inaccurate results, because the model is learning from information that it wouldn’t have had access to when it was making trade decisions.
This bias is commonly referred to as “repainting” particularly seen on the platform TradingView.
Recency or Market Condition Bias
For example, a strategy that performed well during a bull market may not perform as well during a bear market. If you test your short strategy on the bear market of 2022, you will obviously make money. But what if you test it also on the bull market before? Will it ruin you? Markets change, your strategies need to be able to cope with those changes. Don’t think they will stay the same.
Curve Fitting and Data Mining
Think of a scatter plot with lots of data points. Curve-fitting is like drawing a line that best matches these data points
So, how does curve-fitting relate to trading?
Imagine a situation where you are testing a strategy with two indicators. For example, a moving average and an RSI indicator. You don’t think ahead about what situation you are trying to capture in the market with these indicators. You just play around with the settings of the indicators for so long until you really like the equity curve. That’s why this problem is called curve fitting. What’s wrong with it? The likelihood is high that the resulting strategy isn’t a stroke of genius, but rather an inevitable outcome of tailoring it to past data.
You can think of data mining very similarly to curve fitting. It just looks more professional. That’s why it’s also trickier. In a testing software or an add-on for Tradingview, you set up for example 5 different indicators and let brute computing power find the best combinations in the past. Such a strategy almost always leads to failure in live trading. If you don’t know what you are doing, never do datamining!
Both of the above methods are very often used by beginner traders and are very dangerous. Amateur traders often feel that they have done basically everything they can to find the best combination of indicators and their parameters. They have backtest results that are so good, they could win an Oscar for “Best Performance in a Historical Setting”. But they don’t realize that they are trading future price action. Not the past.
Conclusion
In algorithmic trading, backtesting is vital but it’s not foolproof. Several biases can trick us into believing we’ve struck gold with a killer strategy. Survivorship, hindsight, sample, selection, look-ahead, and recency bias together with curve fitting and data mining – they’re all potential traps. To truly create an effective strategy, it’s not enough to just know these biases exist; understanding and mitigating their impact is paramount. So, backtest with prudence.
Crypto has a very small data set, so it is very crucial to be careful with backtesting. Imagine that on the limited data, you are cherry-picking data to get good results, you are just fooling yourself. If you’re tweaking your parameters to fit this short historical data, you’re not creating a sound strategy, you’re just setting yourself up for a fall. Backtesting is a dangerous tool in the hands of a fool.
