Quickies #1: Overfitting and EWMAC forecast scalars

 I'm now in full book writing mode, so I don't have the time to do full blog posts. Instead I plan to do a series of quick posts where I share some research I did for the book. Cynically, there is also a chance it will encourage you to buy the book, as long as I don't overshare like one of those movie trailers that gives away the plot and includes all the best action scenes.

Overfitting - A pictorial guide

Here's a slide I use a lot:

Although we know intuitively that complexity improves in sample performance, and at some point makes it worse, can we visualise it?

Let’s assume I think that the pattern of up and down price movements over the last 64 business days will predict the return in the following month (about 20 business days). I can analyze these patterns by:

    • Using one period: just looking at the last 64 days of trading and seeing if the price moved up or down. Because of weekends and holidays this is about three months.

    • Using two periods: Looking at the first 32, and second 32 days of trading. And now you see why I'm using 64 days.

    • Four periods: Examining the first 16 days, second 16 days, third 16 days, and then the final 16 days.

In theory I could end up looking at each day individually, but in practice I will stop at 16 periods of four days which turns out to be sufficiently interesting (For four periods there are 2^4 = 16 possible states, at 2^64 it would just be silly). For my analysis I see what the market did on average after each pattern of daily returns. For example, if I’m using two 32 day returns for my patterns, then there are four possibilities: up and up, up and down, down and up; and down and down. I can then see whether the market went up or down after twenty days in each of these four states.

Subsequently to trade this method I will then work out which state the market is currently in, and then buy if that state has led to the market going up by more than average in the past. This is a long only book, and hence I can't short if I expect the market to go down by more than average. Instead I'll just stay flat.

The plot shows the results of performing this exercise with Microsoft. I’d get the same kind of picture with pretty much any instrument – this is one occasion where it’s not necessary to use a huge pool of data to get robust results. Each line is a different number of periods.

Unfortunately we can’t actually realize these high levels of profit. The problem is that we are in sample fitting the method to all the available price history. So now we follow the usual procedure to get out of sample testing: fit the method to the earlier part of the data, and then test that calibrated method on later data periods. Note we might see a state we haven't seen before, in which case we forward fill from the last known state.

 Let’s see what happens if I fit my method using 16 periods of four days on all history up to 2015, and then test it on the last ten years.

The black line uses the original procedure; we take all the price history and calibrate our method accordingly. It does very well, but as I’ve already noted we couldn’t get this performance in reality. The dark gray line is only fitted up to the start of 2015. It’s performance is very similar to the black line up to then, which isn’t surprising as the methods are virtually the same and share around 70% of the data used to fit them. 

However after 2015 when it is trading on price history it hasn’t seen, it’s performance radically deteriorates. Another useful experiment is to see how well the method does on different instruments. The light gray line shows what happens if use the 2015 Microsoft method to subsequently trade Apple (AAPL). You can clearly see that the performance is terrible. This method is too closely fitted to the returns of Microsoft pre-2015; it does not perform well in later years, especially with a different stock.

What happens if I do the same procedure with a much simpler model, which just looks at the entire 64 day period? Since we know stock prices trend over periods of a few months, we can already guess what this fitted method looks like. It will buy if the price has gone up in the last 64 days, otherwise it won’t do anything. 

As the figure shows, the fitted method ends up looking exactly the same regardless of whether it was built using price history up to 2015 or for the entire period. Consequently, the dark grey line is exactly overlaid on top of the black line. You can also see that this very simple method does very well on Apple stock, which it hadn’t seen before. 

Calculating not measuring forecast scalars for EWMAC

My previous books have been full of estimated forecast scalars for my various trading rules. Many of these follow a square root of time rule. So for example, the scalars for EWMACN, 4N using daily standard deviation of returns to normalise are around 15 for 2,8, 10 for 4,16 and so on. We get the next scalar by dividing by sqrt(2). Once you estimate 2,8 you can derive the other numbers easily.

That's great, but how do we get the 'seed' value for different starting multipliers other than 2,8? I started by generating a full table of multipliers for all powers of 2:

The rows are fast, the columns are slow. Since fast<slow for trend following the bottom diagonal is empty. We need to calculate the numbers in the top row. Note that this should also work for any value of S EWMAC 2,S; but it's easier to do it in power of 2 space for fitting.

I used..... chatgpt! Yes I just pasted in the numbers and asked it to fit. It used a power law distribution, and it even gave me with some prompting when I asked it to include an intercept the required python code. It's not a perfect fit because empirical data, but as it says in the footnote of my book that all this effort was done for: 

To calculate the scaling factor for any value of f=2 and a given value of s, use the formula 2.2+(184÷s^1.25) for the appropriate value of s. 

The fitting gets gradually worse as we get longer values of S, but that makes sense as things like the long bias in returns tends to have more of an effect.