If you've been looking into trading futures, commodities, forex, or stocks, you've probably been exposed to backtest results. These results purportedly show what a trading method would have done in the past, had you followed it. They are one reason why the US government requires the phrase "Past performance is no guarantee of future results" when discussing trading systems or approaches.
Backtests are hypothetical, and may or may not have been actually traded with real money. It is even possible that it would be impossible to make the trades shown in a backtest.
As an experienced system developer, in five minutes with my Tradestation testing software, I can create a system in any market with a backtest that would blow you away it would look so good. But, it would fail going forward, practically guaranteed! Trust me, many developers create systems exactly this way, and then try to sell you their "secret."
So, why are backtests so unreliable? Four reasons come to mind.
1) Optimization - Most testing software has an optimization feature that will pick the best set of parameters based on the past data. Most developers abuse this feature. What worked best in the past is highly unlikely to be the best in the future. Consequently, by falling for these overoptimized backtests, you think you are buying a Mercedes, but in reality, you'll get a Yugo.
2) Hindsight Bias - It is difficult to create a system without "peeking" at the data. Since one can't peek at future data, a developer who does this during development is, in effect, cheating. Many times, people do not even realize they are doing this; it can be that subtle of a mistake.
3) Software Limitations - The software itself has limitations that allow unrealistic or unachievable fills. For example, systems with market on close orders very likely include unrealistic fills, since the order might be sent after the market is closed (and never filled), but the software still thinks it got filled.
4) No Real-Time Performance - Developers post their own backtest results, make them appear like they are real, and have no independent, real-time verification of their results. Can you really trust backtests from the same person trying to sell you the system?
What's the solution to this backtest dilemma? Simple. If you are dealing with a CTA (commodity trading advisor), hedge fund, or mutual fund, make sure you see actual, audited real-time records. If you are dealing with a system developer, make sure the results you see are independently checked and verified, preferably with the results based on real-money (not demo) accounts. I list two of these Web sites at the bottom.
It is easy to be seduced by extraordinary looking backtest results. Just remember that those results might not be real.
Web sites such as WorldCupAdvisor.com and Collective2.com are just two of many Web sites that offer third-party verification of trading results.
By Kevin Davey of KJTradingSystems.com
