Predicting the future is for suckers

I spent the weekend thinking through a trading strategy dubbed by a Wall St. Journal reporter as the “Common Sense” (CS) trading strategy (see Part I). It turns out that common sense was a disaster when tested against historical data. The original formula was to buy or sell 5% of your cash or share value whenever the market moved 5%. I will refer to these levels as the market threshold and the aggression level. Using that formula, the CS strategy faithfully sold shares at market peaks and bought in the market valleys, but it sold off too many shares over time and paid out too much money in taxes. It was substantially worse than a “Buy and Hold” (B&H) strategy that bought on the low points and never sold.

Knowing that the shortcomings of the original CS trading strategy are incurring taxes and holding too much money in cash, can it be improved? Two things come to mind: search out better set of values for when and how much to buy/sell and alter the algorithm to buy more aggressively than it sells. Raising the market threshold will cause the strategy to trade less and lowering the % of assets to buy and sell will mean lower taxable income. To prove that the results are not a fluke of timing, four different historical time periods were used. Computing power is cheap so I plowed through hundreds of combinations of inputs to find the most profitable over four different time periods.

Start Date End Date Length Significance

1/1/06

12/31/12

7 years Short term

8/11/87

12/31/12

25 years Before the 1987 crash

10/19/87

12/31/12

25 years After the 1987 crash

1/1/50

12/31/12

63 years Long term

Buy and Hold is hard to beat. Starting with the longest term, 1/1950 through 12/2012, the B&H strategy earned $855,902. The best CS configuration returned 45% less money or $472,129. Ouch. The best performing version of the CS strategy was to buy and sell when the market moved wildly, by 20% and to sell very small percentages of your holdings each time, around 1%. This meant that the minimal amount of value was lost to taxes. The best B&H on the other hand spent 100% of its free cash the first time the market moved 2% and never traded again.

Graph.1950.Total

For fun I chose to medium length periods of time on either side of the great 1987 crash. The Before graph started in August of 1987 and the After graph was from October 1987. The results are very similar, except as you would expect, buying at a low point right after a market crash earned more money overall. The best performing inputs were nearly identical. B&H performed best when it bought aggressively after the first market move and never traded again. CS performed best when it bought or sold 1% when the market moved 20%.

Before the 1987 crash (August)Graph.1987B.Total

After the 1987 crash (October):Graph.1987A.Total

Now for the short term where things get interesting. The tables turn thanks the most recent market crash. The CS strategy came out on top, but it reinforces why this is a poor trading strategy for most people. I will just say that the CS strategy gets lucky here:

Graph.2006.Total

Unlike the longer time periods, the best performing versions of the CS strategy in the short run were very aggressive. When the market moved 20%, it bought and sold 100% of its positions. This is what you would do if you had a crystal ball. Sell everything when the market is high, and buy back when the market is low. The pink shaded sections are the times where the CS strategy owned zero shares of stock. So why is this bad? It is a strategy that counts on huge volatility to be successful and historically the markets just don’t fluctuate that much. If the market entered a period of calm sustainable growth, you would be caught with your money on the sidelines earning no return. It is the reason why historically the CS strategy works best when it makes very small moves. So unless you can predict the future and you *know* that there is a major market crash coming, you can’t win by timing the market with this strategy.

At the start of this post I mentioned that there might be a second way to improve the CS strategy, by buying and selling at different rates. That might get around the tax issue while preventing too much money from sitting around in cash. But this post is getting long, so I will come back to that another day.

Here is the source code and data if you want to play yourself:  CS.Trading.Strat.Source

Debunking the “Common Sense” trading strategy

Most small investors are not that good at predicting the financial future. I am certainly awful at it and I worked at a trading company for five years. I had a front row seat watching how the big guys run a trading firm today and I can tell you that it takes a lot of specialist knowledge, technology, and money. My job was to run the technology. It is a world that is far removed from the tools and timescale of everyday people.

For investors who don’t have a supercomputers and 10Gb links to the New York Stock Exchange, how do you know when to make trading decisions? A writer for the Wall St. Journal wrote an article a year or two back claiming that a “common sense” trading strategy was the right move for the average Joe. He claimed that investors should buy whenever the market fell by 5% and sell whenever it rose by 5%. The intuition was clear, this model would force people to buy low and sell high.

But does it work? I built a model to test this trading strategy. The original article was a little fuzzy on some of the key details such as how much of your wealth should you move when the market crosses the 5% threshold, so I assumed that to be 5% as well.

Here are the key assumptions for the “Common Sense” (CS) trading strategy:

  • Buy or sell when the market falls/rises by 5%
  • Every sale results in 15% of your gains going to long-term capital gains taxes
  • Every sale is 5% of the value of held shares
  • Every purchase uses 5% of free cash
  • With an initial $10,000, $7K is invested on day 1 and $3k is held in cash
  • Investment was started on 1/3/2006

To compare this against a “Buy and Hold” (BH) trading strategy

  • Buy when the market falls by 5% from the last high
  • Every purchase uses 5% of free cash
  • Never sell
  • With an initial $10,000, $7K is invested on day 1 and $3k is held in cash
  • Investment was started on 1/3/2006

Here are two graphs of the S&P 500 index with the transactions of the two strategies overlaid. The CS strategy has both purchases and sales, the B&H has only purchases. Both strategies make purchases when the market has fallen by 5%. A glance at the graphs confirms that both strategies are doing pretty well at buying during market low points.

First the CS graph. Notice that it Buys (blue) when the market falls and Sells (red) on the way up. It does a great job of hitting all the peaks on the graph:

Graph.2006.CS.1 The B&H has the same pattern of purchases, but never sells any shares:Graph.2006.BH.1

So how did they do? The “common sense” trading strategy is a disaster. Buy and Hold finished with $11,586.81 and “common sense” ended up losing money, ending with only $9,711.43 of the original $10,000. That is $1,875 or 16% worse than the B&H strategy over the seven years.

Final Earnings

What happened? Three things are going wrong. First, long-term capital gain taxes suck out a ton of your profits. At 15% of your gains, every sale nibbles away at your purchasing power. The CS strategy paid $1,021 in taxes over the seven years, which is a majority of the performance difference between the two strategies. In the seven years, that $1,021 would have grown by 6.5% to $1,088. Over more time both the money you don’t pay in taxes plus the 6.5% in growth really get big.

Second, the CS algorithm is too risk-averse. It moves too much money out of investments and into cash. Since the market has generally risen for the last 70+ years, there are more selling events than buying events. So the number of shares owned decreases over time as the cash begins to equal the value of the shares you own. You can see in this graph how the B&H strategy ends up with 2.6x more shares. When you money sits in cash, it does not grow.

Graph.2006.Shares.Owned

Lastly, as you purchase new shares over time, you increase the average purchase price of the shares you hold. Since the price of stocks has continued to rise over time, in the future when the market dips by 5%, shares will still cost more then they did on day one. Buying the small dips does not help because they are very rarely deep enough to lower the average cost of your purchases. What seems like a perfectly good idea turns out to be horrible in real life. But are there ways to save the CS strategy? I will give this some more thought and follow up with another post.

Here is the python source code and data if you want to play. CS.Trading.Strat.Source

Python pipeline with gap detection and replay using zeromq

I needed a way to process a ton of information recently. I had a bunch of systems that I could use, each with wildly different levels of resources. I needed to find a way to distribute work to all of these CPU’s and gather results back without missing any data. The answer that I came up with was to create a processing pipeline using zeromq as a message broker. I abstracted the process into three parts: task distribution, processing, and collection.

Since Python has a global interpreter lock, I needed to distribute my process using something other than threads. I turned to ZeroMQ since they have a robust message system. One of the ZeroMQ message patterns, the pipeline, was a great jumping off point for my needs. But the pipeline didn’t have any sort of tracking or flow control. I added a feedback loop to keep track of the total items sent and gap detection. The idea is that all work to be processed will have a sequential key. There are three pieces of the app, Start, Middle, and Finish. Start and Finish will use the sequential keys to understand where they are in the process and to replay any missing items.

There can be gotchas. If you don’t set the timing correctly on the apps, you might start replaying long processing jobs too quickly and create a process storm. But for the type of work I was doing it was fairly easy to set a reasonable set of times.

Pipeline Files