The mining of the historical record by high-frequency trading algorithms is making the future increasingly hard to predict, writes senior editor Igor Greenwald.

“There’s no clarity, so everyone is watching technicals,” I heard a market analyst say on TV the other day. He was talking about the here and now—this week. But it’s been the same for the last dozen years.

The 20th century saved its best bull market for last, and at its tail end things couldn’t have been any clearer.

Stocks were the asset class that created wealth. Bonds were for old folks. Inflation was dead, and the business cycle nearly abolished. The dollar was the most sought after commodity around the world, tangible proof that Pax Americana was the natural order of things.

In retrospect, Long-Term Capital Management was only the canary in the coal mine. The star-studded and star-crossed hedge fund made a mint for several years after its founding in 1994, by exploiting underappreciated historical relationships in asset prices.

Its success was widely imitated on Wall Street—so when crisis hit in 1998, it was trapped in a crowded, leveraged trade largely of its own devising, with the exits blocked by all the imitators trying to get out.

It was an early demonstration of the financial version of the observer effect, wherein the act of observation alters the phenomenon being observed, and therefore the expected outcome. LTCM’s highlighting of market inefficiencies hardly made them obsolete, but it did make the markets behave in ways that the fund’s legendary traders and Nobel laureates had failed to anticipate.

Fast-forward 13 years, a half-dozen market bubbles, and a bailout or two later. Buy-and-hold is dead and buried, replaced by sell-and-never-again for some, while others pursue the increasingly arcane trading strategies that were until recently the exclusive province of professional arbitrageurs.

With volatility high by historical standards, we’ve seen an explosion of option trading, such that the TV is choc-a-bloc with talk of straddles and iron butterflies. Technical analysis is all the rage, and any self-respecting housewife can tell you all about Fibonacci retracements and Bollinger bands.

We command more computing power than ever, and quite a lot of it has been channeled into trawling the historical trading record for actionable ideas that can be turned into a profit by high-frequency trading algorithms.

It’s a secular trend that LTCM helped pioneer, and it’s been reinforced by the past decade of range-bound market action. Why? Because it’s hard to make money on the fundamentals out there, whereas almost anyone who’s anyone can buy themselves a couple of quants.

In short, the volatility and apparent lack of long-term progress by equities have turned us all into a bunch of fruit flies—living for the day, with little in the way of fundamental conviction about the market.

How did I do today? What’s he done in the last year? What are the technical setups for tomorrow?

Many of those who used to take a longer view checked out in 2008, after a hard look in the rear-view mirror, and haven’t been back. This is now a thinner market dominated by traders, which only serves to heighten volatility and turn off more conservative investors, in a self-reinforcing process.

Futurists have for years been kicking around the concept of singularity, the notion that the accelerated rate of technological and socio-economic change will eventually make even the near future more or less completely unpredictable. This is sometimes seen as the likely outcome of the development of super-intelligent computers that might outthink us into irrelevance.

I tend to think the singularity might already be at hand—brought about by nothing more exotic than some relatively dumb computer programs scalping pennies per share, and humans staring hard at all the charts.

The stock market does not reward group-think; in fact, it tends to punish it rather severely over time. So the more computing power we harness in examining the historical record, the less likely it is that the past will be any sort of a guide to the future.

That’s almost the definition of singularity. Everyone in the market can’t be right. And if everyone in the market is looking at the historical data (and they are, they are), then it’ll eventually prove no more useful than the Dogs of the Dow, the January effect, and any number of other apparent patterns that only hold until they’re disseminated widely enough.

Back in the good old days, commodities traded largely between producers and consumers based on the prevailing dynamics of supply and demand. But that era seems to have passed.

From a Reuters story: “Commodity traders are increasingly blaming computer-driven activity for a series of anomalous price movements ranging from quick blips in natural gas and cocoa to deeper, longer-lasting jolts like the one that shook oil on May 5.”

Such trading makes up just 15% of the volume in commodities today, but could double in the next few years, according to the article.

One of the ways traditional investors have protected themselves from excess volatility in the past was by setting stop-loss orders, instructions to sell should a security reach (usually fall to) a specific price, one typically based on technical analysis.

But as the Reuters article above points out, and as last May’s “flash crash” might suggest, computer programs already exist to sniff out such sell-stops ( a task made easier, in fact, by their use of common technical points of reference) and trigger them before taking the prices in the opposite direction. Absent sellers beware.

Here are my takeaways from all of the above:

  • We need to learn, if we can, to put less trust in the past as a reliable predictor of the future, most particularly past price action and some of the most widely known tenets of technical analysis
  • With more and more of the market’s volume devoted to short-term scalping, perhaps it’s time to take a longer, more fundamental view. This at least gives us a chance to use our intuitive, non-linear reasoning—safe from mimicry by computers, for now
  • I’m not ready to buy and hold, but stop-loss orders may not be your friend if all your friends are also setting them at predictable price points
  • Beware the market inefficiency you read about. All the computer programs have known about that one for a while, and they’re watching.