Weekly Perspectives: The Shortcomings of Backtesting 🔭 🔬

In this week's lesson certified risk manager and frequent contributor, Michael Toma shares the discrepancies between real trading and backtested results.

 



There were times where I spent more effort backtesting than I would sleep. As traders, we’ve all done it. Waking up at 4 am with a strategy you heard about and simply can’t get it out of your head.

The mission; backtest, backtest and more backtest. Some of the popular trading platforms now have pretty cool backtesting tools but for me, if the strategy didn’t seem too complicated, I would dive in with pen and pad and backtest it old school. I can’t tell you how many times I witnessed a successful backtest only to quickly fail in a live environment.

How can that be? We’re talking the holy grail of setups only to make its way to the trash bin along with the backtesting scribble.

So how do risk managers play a role in the system building process?

Since the process can ultimately be the supporting driver of putting live dollars in action, auditing a strategy is an essential part of the risk identification process.  

This gap between backtested vs. actual results is relatively easy to identify but can be a challenge to assess the cause of the differential.

This frustrating gap often comes from the inability to execute the method in a live environment. It’s so easy to scroll through your charts on a Saturday morning with coffee in hand but throw it in a live environment and it’s an entirely different ballgame, hence the opportunity risk.

In addition, one of the more common discrepancies is a result of confirmation bias.  Simply put, the mind tends to identify information that validates our own point of view. In trading, that means winning trades.  

One of my first money management gigs was with a successful businessman but failing trader who decided to outsource his trading account to execute his ‘winning’ strategy.  He had a thoroughly backtested track record but admitted he didn’t have the time, execution discipline nor ability to take losses so in comes yours truly.

We walked thru hundreds of days of his backtested results, which were outstanding, to say the least. Mostly focused on the ES futures with a scary 1-minute chart and it was evident early on that he was not identifying all the losing trades; confirmation bias at its finest.

He was so obsessed that his ‘system’ worked that we went live anyway, even after a not so rosy backtest. I was able to reduce his losses dramatically using protective stops but in the end, his holy grail wasn’t as holy as he had thought.   

The guy with the big account didn’t cater to my confirmation bias risk speech, particularly after seeing red in his live account, so we quickly parted ways. It was a frustrating part of my risk management career but taught me the valuable importance of backtesting and the flaws that come with it. So what can we learn from it:

  1. Be as unbiased as possible. Confirmation bias is a human condition but if we approach our testing knowing such bias exists, we tend to be more aware of all outcomes. You’re only hurting yourself if you don’t.
  2. Use technology as a validator. Go for it with pen and paper but try running it thru your platform’s backtesting feature. You may be quite surprised (and possibly alarmed) of the outcome. 
  3. Short time frames will increase execution risk.  A big problem with the 1-minute method was the entry had to be so precise with the resulting gap being impacted merely by the 1-tick bid/ask differential. ‘Print-no-fill’ risk also contributed to the backtest vs. live gap.  
  4. Keep it simple. Having multiple rules and filters will fuel confirmation bias and ultimately an inaccurate backtest. Try running it with just the core framework to see if that works and then add your conditional filters. Better yet, perform it with a fellow trader with each exchanging their own strategy. It’s amazing what a second set of eyes will identify and you may have just been handed your future go-to strategy and be able to help a fellow trader as well.

Always remember that this is your business and your setups are in essence your employees. Some will outperform and you will promote, others fail and must be let go. 

Look at backtesting as the interview process knowing that some of the biggest mistakes we make as traders happen before we even push the button.

Stay connected with news and updates!

Join our mailing list to receive the latest news and updates from our team.
Don't worry, your information will not be shared.

Subscribe