Breaking Badly: finding the structural breaks in parameter estimates

 Here's a nice picture from a lovely book written by a top bloke:

It shows the cumulative p&l from different speeds of momentum over time (for portfolios containing 102 instruments) over 50 years of data. Notice how the two fastest speeds (2&4) get worse in the second half of the sample. I've called the line #2 here the 'second most famous hockey stick graph in history'. It certainly looks like something changed in 1990. 

This is important. If we're optimising portfolios of such things we only want to consider data that is relevant, but we also want as much data as possible for statistical significance. Now if I were a simpleton I'd do this by looking at graphs like that and going 'aha i only need to use data after 1990'. As a simpleton I don't use capital letters. But I am a big fan of not doing in sample fitting, even of meta parameters like this; and I am an even bigger fan of doing things automatically which means not wading through thousands of graphs like that (since there are thousands of SR estimates in my forecast p&l space, plus a good chunk of correlations).

So we need an automatic way of identifying such breaks. Fortunately this is not a new problem as you will know if, like me, you did undergraduate econometrics. Finding structural breaks is an entire industry. We need two things: a test for how likely it is that a break has occured between two sub-samples A and B. And an algorithim for going through all the options of A and B

And in case you haven't realised this is the seventh post in my summer 2026 series on portfolio optimisation.

What parameters

The first question to think about is what parameters we're going to apply this process to. I do two kinds of optimisation:

• Forecast weights
• Instrument weights

And in both cases I have estimates of SR (one per asset) and correlations ([N^2-N]/2 for N assets). I haven't really looked at instrument weight optimisation yet in this series, and there are some wrinkles there so I'm going to park that for now. That just leaves the SR for a forecast (which remember is a pairing of a trading rule and an instrument), and the correlation of such forecasts within an instrument.

Now I am going to ignore correlations in this post. As I discussed in an earlier post, although correlations are relatively unstable in the short term, they are unlikely to have secular trends like SR. And it's quite easy to deal with this by just using a relatively long lookback to estimate them, probably with an ewma on the correlation estimate.

What test

This is quite an easy one, compared to the world of econometrics and linear regression where we have to do such nonsense as a Chow test. Given two sub-samples A and B, to find out if they have different sample means we just do an independent t-test: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html

An important characteristic of such tests is they are only likely to be significant if there is sufficient data. If sample A or sample B is too small in size, it's unlikely we'll find a signficant difference in the means.

Typical values for critical values in t-tests are 1% and 5%; I will also check out 10% later.

Since all the p&l streams in a risk targeted trading system will have the same expected standard deviation in the long run, I can legitmately treat this as a test to see if the SR are different if I adjust the samples so they have an identical standard deviation. 

(If any ex-students of mine are reading this, you should remember this from week 3 of the course!)

What algo / procedure

OK let's make this concrete. Suppose we have 50 years of data as in the original graph above, and we're currently wondering if there were one or more structural breaks. We don't want to have less than five years data to do our estimation. Note these numbers are appropriate for my context, but not in a domain with faster trading, higher SR, and faster alpha decay. We proceed as follows:

• We compare year 1 with years 2 to 50. Since year 1 is very small it's unlikely we'll find a break; but if we do then we do a split (see below).
• If no split has occured, we compare years 1-2 with years 3-50. Again if we find a break, we split.
• If no split has occured, we compare years 1-3 with years 4-50...
• ...
• If no split has occured, we compare years 1-45 with years 46-50. If we still don't find a break, then we use the entire period for estimation (since the period 47-50 will only have four years, we terminate here).

Now what if a split occurs at some point? Then we restart the process, but this time without the pre-split data. Suppose for example a split had occured in year 20 (which is 1992 in the original graph). Then:

• We compare year 21 with years 22 to 50. If we find a break, then we split again.
• If no split has occured, we compare years 21-22 with years 23-50. Again if we find a break, we split.
• If no split has occured, we compare years 21-23 with years 24-50...
• ...
• If no split has occured, we compare years 21-45 with years 46-50. If we still don't find a break, then we use the entire period from years 21-50 for estimation.

You get the idea. This procedure is quite quick and easy to run; and notice we're identifying any break that exceeds a certain threshold rather than finding the most likely break as we would do with a QLR type test. 

The main downside of it is that it won't identify breaks that reverse. For example if the world is in regime A, then regime B, then regime A again then ideally we'd estimate our parameters using both regime A's. But the above test will eithier use only the final regime A, or the last two regimes, or possible all three regimes depending on whether there is a significant difference at the appropriate point(s). There are fancy things we could do to deal with this, but I feel these are corner cases and life is too short.

An example

Let's use a concrete example. This is the performance of the momentum4 rule on CORN. I've chosen it because we already know momentum4 has a structural break, and CORN has plenty of history. Just for fun, before reading on, see if you can identify where the algo finds the break here (there is exactly one break - this isn't a trick question).

Some code

This is probably the first post in this series where it's been practical to actually include the code, since there isn't much of it, nor are there are any dependencies apart from getting the returns:

from copy import copy
from typing import List, Callable

import numpy as np
import pandas as pd
from scipy.stats import  ttest_ind

BUS_DAYS_IN_YEAR = 256
import matplotlib.pyplot as plt

MIN_NUMBER_OF_YEARS = 5

def identify_and_plot_breaks(all_returns: pd.Series, CV: float =0.01):
    breaks_as_dict = identify_all_breaks(all_returns)
    breaks_as_df = pd.DataFrame(breaks_as_dict)
    breaks_as_df = breaks_as_df.bfill(axis=1)
    breaks_as_df.cumsum().plot()
    plt.show(block=True)


def identify_all_breaks(all_returns: pd.Series, CV: float):
    ## returns a dict, turn into a dataframe and you can plot
    returns_to_consider= copy(all_returns)
    returns_to_consider = returns_to_consider.dropna()
    broken_list = identify_all_breaks_recursively(returns_to_consider=returns_to_consider, list_of_returns_broken_off=[],
                                                   CV=CV)
    broken_list.reverse()
    broken_dict = dict([
        (idx, value) for idx, value in enumerate(broken_list)
    ])

    return broken_dict

def identify_all_breaks_recursively(returns_to_consider: pd.Series, list_of_returns_broken_off: List, CV: float) -> List:
    years_in_returns = how_many_years_approx(returns_to_consider)

    for i in range(years_in_returns):
        year_idx=i+1
        first_sample, second_sample = split_sample_after_n_years(returns_to_consider, year_idx)
        if len(second_sample)<(MIN_NUMBER_OF_YEARS*BUS_DAYS_IN_YEAR):
            break
        is_broken_here = test_a_break(first_sample, second_sample, CV=CV)
        if is_broken_here:
            list_of_returns_broken_off.append(first_sample)
            return identify_all_breaks_recursively(
                second_sample, list_of_returns_broken_off=list_of_returns_broken_off,
                CV=CV
            )
        else:
            continue

    ## No breaks identified or sample size too short
    list_of_returns_broken_off.append(returns_to_consider)
    return list_of_returns_broken_off

def how_many_years_approx(returns: pd.Series):
    return int(np.floor(len(returns)/BUS_DAYS_IN_YEAR))

def split_sample_after_n_years(all_returns: pd.Series, n_years: int):
    idx = n_years*BUS_DAYS_IN_YEAR
    return all_returns[:idx], all_returns[idx:]

def test_a_break(first_sample: pd.Series, second_sample: pd.Series, CV: float):
    ## Normalise by standard deviation before considering means
    norm_first_sample =first_sample/first_sample.std()
    norm_second_sample=second_sample/second_sample.std()
    return ttest_ind(norm_first_sample, norm_second_sample).pvalue<CV

On Corn this produces the following:

The break occurs on the 26th August 1982. So to calculate that particular SR estimate we'd only use data from that date onwards.

Is that what you would have guessed? Personally, I would probably have gone for a later break point if identifying it by eye. As humans we are drawn to the sharp upward move in 1989 and would probably have gone for a break just after that. It's possible a search for the likeliest break would have found that point, but remember we are looking for the first break that exceeds the threshold; and once that break happens no further breaks are identified.

A summary of results

Here is a summary of the results for each instrument/forecast pairing with the default 1% critical value:

You can see that breaks are quite rare with only 13% or so of instrument/rules having at least one break. This also suggests that the Sharpe Ratios for trading rule performance are actually quite stable over time; or at least stable enough that they won't fail any statistical tests at a 1% critical value. 

Multiple breaks are even rarer. Just 1.8% have two breaks; 0.4% or 39 instruments have three breaks, ten have four breaks and only two have five breaks. They are:

skewrv365 forecasting EURIBOR-ICE    (yes, there is still a EURIBOR future!)

normmom2 forecasting FTSE250         

Here is Euribor, relative value skew with a 365 day window:

Although five is pushing it, there are certainly three regimes there (pre 2000, 2000 - 2010, and 2010 onwards), and using post 2010 data seems to make some kind of sense.

And FTSE 250 momentum8 (this is pre-cost):

There are certainly at least two regimes there and I wouldn't argue with the automated decision to use only data after 2006 or so, when it looks like; in the words of Pulp in one of my favourite songs, "Something changed".

Of course we will get a different picture with a slacker test. Here is the picture with a 10% critical value:

Now just over half the pairings have at least one break in them.

The decision as to use 0% (equivalent to no breaks at all), 1%, 5% or 10% CV is one we will now address.

An optimisation test

Now the big question is does this actually improve performance? On a pure out of sample test? And is this changed much by using a different critical value?

I follow the same procedure roughly as in previous posts:

• Select 10,20,30 or 40 years of in sample data (I need at least 10 years because with a minimum of five years required for estimation I certainly won't find any breaks, or I will risk finding a break and not having five years of data leftover)
• Select 1 or 5 years of out of sample data
• Pick a random instrument, ensuring there is enough history available (between 11 and 45 years). We will only choose from instruments with sufficient history for the time required.
• Randomly pick N=9 forecasting rules from those available (the same number as in posts #2 and #3)

Then for each of those sumsamples:

• Cycle through using no breaks (0% CV), 1% CV, 5% CV and 10% CV
• Estimate SR on the insample data using eithier all the data (0% CV), or the data after the last break given some critical value. 
• Estimate correlation using all the in sample data
• Use fixed shrinkage levels (estimated here): SR shrinkage 0.5, correlation 0.75 (since we'll always have at least five years of in sample data we don't need to worry about the higher levels of shrinkage required when we have insufficent data). The results won't be much different with any vaguely similar shrinkage.
• Run in sample optimisation and out of sample optimisation on all the options above

Finally once we have all our subsamples:

• Get the median SR from the distribution of subsamples
• Find the optimal CV with the highest SR
• Test to see if that median is significantly higher than the others

10 years in sample, one year out of sample

We only have four options to consider so no need for the huge tables and fancy heatmaps of previous posts:

         SR  pvalue all  pvalue distinct
0.00 -0.021       0.247            0.247
0.01 -0.018       0.295            0.295
0.05 -0.019       0.204            0.204
0.10 -0.014         NaN              NaN

Each row is a different critical value used for breakpoint finding. Zero means the entire in sample period was used. The next column is the out of sample Sharpe Ratio for each option. In the second column is the p-value for a test of the optimal option against the relevant option. NaN is the optimal option, and lower values (say below 0.05) mean the optimal option is significantly better than the alternatives. In the final column I've rerun the staistical t-test but this time I have excluded instances where no breaks were found (so the test is only done comparing the out of sample SR when breaks were found, versus when they were not). This shouldn't affect the p-values, but it's nice to check it doesn't

You can see here that it it looks like a very loose breakpoint policy is the best, but it's not significantly better.

10 years in sample, five years out of sample

         SR  pvalue all  pvalue distinct
0.00  0.166       0.168            0.168
0.01  0.160       0.003            0.003
0.05  0.167       0.007            0.007
0.10  0.170         NaN              NaN

Again the loosest breakpoint works best; but it's hardly logical since it's indistinguishable from no breakpoints at all.

20 years in sample, one year out of sample

         SR  pvalue all  pvalue distinct
0.00 -0.160       0.542            0.542
0.01 -0.160       0.323            0.323
0.05 -0.160       0.050            0.050
0.10 -0.155         NaN              NaN

Loose is better.

20 years in sample, five years out of sample

         SR  pvalue all  pvalue distinct
0.00  0.123         NaN              NaN
0.01  0.116         0.0              0.0
0.05  0.105         0.0              0.0
0.10  0.098         0.0              0.0

No breakpoints are the best. There isn't much consistency here.

30 years in sample, one year out of sample

         SR  pvalue all  pvalue distinct
0.00  0.092         NaN              NaN
0.01 -0.019         0.0              0.0
0.05 -0.078         0.0              0.0
0.10 -0.066         0.0              0.0

Another massive win for not using breaks.

30 years in sample, five years out of sample

        SR  pvalue all  pvalue distinct
0.00 -0.005       0.001            0.001
0.01 -0.011       0.000            0.000
0.05 -0.008       0.014            0.014
0.10 -0.003         NaN              NaN

Or perhaps we should go for the loosest break....

40 years in sample, one year out of sample

         SR  pvalue all  pvalue distinct
0.00 -0.142         NaN              NaN
0.01 -0.249       0.000            0.000
0.05 -0.189       0.000            0.000
0.10 -0.178       0.002            0.002

or no breaks...

40 years in sample, five years out of sample

         SR  pvalue all  pvalue distinct
0.00 -0.060       0.177            0.177
0.01 -0.051         NaN              NaN
0.05 -0.055       0.000            0.000
0.10 -0.055       0.000            0.000

A clear vote for strict breaks, but no breaks at all are also good.

Conclusion

Well that was as clear as a bowl full of mud that has been made even more unclear by painting the bowl a very dark colour and then adding some darker mud. Not very clear at all, in other words.

You think up a nice neat simple way of finding structural breaks, and then it doesn't actually work when used for optimisation. In seven out of the cases we examined not using breaks is eithier optimal, or statistically insignificant from the optimal. Only in one case was it inferior. The loosest possible break (CV 10%) was optimal in four cases. In most cases apart from 30 years/1 year the difference in performance was small between CV alternatives. This partly reflects the fact that breaks aren't that common, especially with a 1% CV.

Of course findings like this are very context dependent. I'm using rules that should probably work over long time periods; indeed they have been in sample selected for such a purpose. For the 30 year and 40 year periods there just aren't that many instruments with that much history; so even repeated random sampling is likely to turn up the same suspects repeatedly.

One issue here might be that we are considering the breaks at a forecast/instrument level. We might get different results if we pool estimates for trading rule performance across instruments. And indeed that is the subject of the next post. So I will return to this topic when I've looked at pooling.

Forecasting statistical estimates when data gets real

 This is my third post in a series about optimisation and fitting. In my previous post I used random data to calibrate and evaluate many portfolio optimisation techniques. It's worth quoting in full from that post:

Random data is not real data: Well duh. But why is this important? Because random data is drawn from a fixed and well behaved distribution. This means the optimiser only has to discover / estimate the parameters of that distribution as more data is revealed to it. But real data doesn't have a fixed and known distribution. It doesn't actually have any distribution at all. We just model it hoping it does.

To summarise then, random data from a fixed distribution differs from real data in three important ways:

• There is no distribution! We just assume there is one.
• The distribution (which doesn't exist) is not known, and thus it's likely the distribution we assume is the wrong one. This is especially true for modelling underlying financial price returns with joint Gaussian models.
• The distribution (which again, doesn't exist) isn't fixed, but can change over time.

And it has one thing in common:

• The unknown parameters of the distribution are unknown and have to be learned over time.

In this post I'm going to explore this learning process for two key statistical estimates: correlations and Sharpe Ratios. What I am interested in is how much wider of the mark our estimates for these two things are likely to be for real data vs random data. This obviously has important implications for optimisation.

Let's look at a plot.

This is for random data generated by a process with a true SR of 1. It shows the evolution of the SR and it's statistical distribution as it is re-estimated each year. There is a burn in year which is missing, and then in the first year we can see our estimate of the SR using all available data so far (in orange), and the SR for the current year (in blue). You can see that the orange line is lagged by a year as it is purely out of sample and always a year behind. I've then used the orange line to estimate the theoretical sampling distribution of the Sharpe Ratio for a one year period, and constructed a 1.96x confidence interval (so about 95%) around the orange line which are the green and red lines. 

Note: The theoretical standard deviation of the sampling distribution of the Sharpe Ratio, assuming i.i.d. returns, is sqrt[(1+0.5SR^2)/N] where N is the number of periods.

Broadly speaking if our estimates are correct then we'd hope to see around 1/20 of the blue points outside the red and green lines, and around 19/20 on the inside. There are 40 years of data here and we go outside the range twice, which is roughly what we'd expect.

Another way of measuring this is to look at our error term, normalised by our standard deviation. This will be equal to:

[(SR estimate this year N) - (SR estimate years 0...N-1)]/(SR sampling std dev error 0... N-1)

If I take the square of this, average of all years and then square root I get the normalised root mean squared error. This comes out at 0.998 for all the data above.

The blue line in this plot shows the absolute value of the error term for each year. The orange line shows the RMSE. You can see this gradually declining over time and settling in at around 0.85

Here are the same two plots for a correlation pair estimate:

Again the RMSE tends to end up around 0.86

Incidentally, we can also do these plots for longer periods. Here is the RMSE evolution for a SR estimate looking ahead over the next 5 years:

The RMSE here is a little higher - around 1.0

Now, let's look at some real data. I'm going to use the p&l from trading the US10 year bond with a 16,64 day EWMAC. Let's begin by trying to forecast the SR one year ahead:

Even without calculating the error we can see that there are more boundary breakages than before with random data. Here is the error:

Notice that it is higher than before (around 1.25; or about sqrt(2) times bigger than the random data RMSE) and doesn't slowly converge as it did with random data, instead it stays roughly constant (ignoring the initial period of luck at the start). 

We get a similar picture for 5 years:

What about correlations? Let's look at the correlation between this slow momentum on 10 year US bonds, and the carry rule on the same instrument:

Wow, that's noisy. The RMSE will be off the charts. What about over 5 years?

Ouch. If we look at the correlation between two variations of the same trading rule, EWMAC64,256 and EWMAC32,128 - which are naturally highly correlated - then it's not much better:

Again the RMSE would be in double digits.

Those might be flukes, so let's look at lots of random results. I'm going to pick an instrument and trading rule randomly, and measure it's final RMSE number. I will then generate some random returns of the same length from the same SR distribution (by measuring the full sample SR for the relevant instrument/rule pairing); and measure that's RMSE. I will then select another rule from the same instrument, get the correlation of the two p&l streams, and generate some more random returns with the given expected correlation. Next and finally I will measure the correlation RMSE for the two sets of real returns, and the two sets of random returns.

If I consider the ratio [RMSE real data / RMSE random data] (both for next one year); then the median of this over a few thousand randomly selected trading strategy components is 1.06 for Sharpe Ratios, and for correlations around 5.6. 

In simple terms, we are a little bit worse than forecasting Sharpe Ratios in real data one year ahead than we would be with random data, but a LOT worse with correlations. 

Partly this is because we are pretty terrible at forecasting SR one year ahead anyway even with a stable underlying distribution; we don't do much worse with real data. However it does seem that correlations are far more unstable in reality than in randomly generated data. Note that these are correlations for trading strategy component returns. In some cases they are mathematically related (eg EWMAC of different speeds) and could be derived with some assumptions, a pencil, and a napkin. They are certainly more stable than the returns of the underlying instruments themselves (think about the changing correlation of stocks and bonds in different inflation environments). 

(Note: These numbers are about the same for five years ahead and also ten years ahead)

If we recall from the prior post that the optimal shrinkage is zero on correlations with random data; we can now see why with actual data we'd probably want to opt for some correlation shrinkage; purely because the sampling error is much larger in practice. That is the empirical finding of the EPO paper. It does feel a bit weird since up to now my gut feeling has been that we have to shrink means a lot because they are much harder to forecast and because they have an outsized effect on portfolio weights compared to differences in correlation. Whilst the latter is still true it seems the former is not.

Food for though. Anyway the next step is to repeat the 'Ultimate Fitting Championships' battle, but this time with real data.

 

The crossword puzzle of fitting - why across and then down?

This will be the first in a series of posts about portfolio optimisation. Main reason being I'm planning to write a book about backtesting, and that will include a big chunk of material on optimisation. Yes, I know, my latest book isn't out yet (it's out in December - in time for Christmas). But this backtesting book is going to be quite deep (and probably long!) so I need to start researching now if it's going to be written any time soon. Today's post is not that deep, and is quite short. It has literally been written whilst waiting for the rather extensive testing of the second post to finish. Anyone, let's begin. 

One of the issues when fitting is to decide how to structure and order the process. In very abstract terms, a component of a trading strategy will consist of a forecast to predict the price of an instrument. A forecast might be something like momentum16,64 - that's the exponentially weighted moving average crossover with spans of 16 and 64 days to you my good sir or madam. An instrument is something like the US 10 year bond future. We can represent all these options in a grid like so:

             momentum16,64                momentum4,16                 carry10

US10             X                            X                           X

SP500            X                            X                           X       

US5              X                            X                           X      

.... where each 'X' is a place on the grid. If those were white squares, and if you can imagine that if there were some forecasts missing from certain instruments which were black squares, then we'd have a crossword grid. Yes that's all I've got. Quite a weak link. Apologies.You think it's easy coming up with catchy blog titles?

And note that this is a tiny subset of the full grid. Instead of these 9 possibilities my full trading system currently has 10,373 options. That's 40 trading rules across 260 different instruments. Some of those instruments are duplicated (eg SP500 mini and micro), some are no longer traded; but that still leaves 204 instruments and over 8,100 options.

Anyone it should be obvious that in doing our fitting we have a few options:

• A joint fit where we fit everything in one go. 8,100 options. In one go. Let that sink in.
• A natural clustering where we cluster together things that are correlated.
• A down and then across structured clustering where we first fit within rules - so for example working out what the best blend of US10, SP500 and US5 is within the carry10 rule - and then across rules - so estimating the best blend of carry10, momentum16,64 and momentum4,16.
• An across and then down  structured clustering where we first fit within instruments - so for example working out what the best blend of carry10, momentum16,64 and momentum4,16 is within US10 - and then across instruments - so estimating the best blend of US10, SP500 and US5.

Now I've generally done the final one of these four options: across and down. And it's quite a natural way of doing things; because we seperate out the idea of predicting the price of a given instrument, and then put together a portfolio of trading substrategies one per instrument. But I've never actually tested the assumption that this is the right thing to do. In particular, if we were to do a natural clustering, would it come out quite like across and down; or would it be something weird? Or would, for example, all the momentum rules be more correlated with other irrespective of instrument, in which case down and across would make more sense?

Programming note: I've done this type of clustering before; here for underlying instrument returns, and here for forecasts. 

Incidentally, I've got three different approaches in the post; as I iteratively updated trying different things each time. TLDR: one of these approaches was inconclusive, the other two were firmly in favour of sticking to 'across then down'.

FIRST ATTEMPT: ONE GIANT CLUSTER

(This was the original blogpost)

Before I begin, I did decide to limit the analysis to the last 10 years. It's kind of slow just calculating a correlation matrix from 52 years of data and many instruments dont't have data except for the last 10 years anyway. I also resampled the returns to a weekly frequency. This is what I do when optimising anyway. Unless your returns are very quick, this won't affect correlations much. That leaves us with 520 weeks (I know that isn't exactly 10 years. Let me check to see if I give a toss. No, I don't) or rows in a dataframe, with over 8,100 columns remember. That's about the limit as to what my laptop can calculate a correlation for; and it's quite a painful process to cluster these bad boys as well.

Anyway let's begin. I've come up with quite a fun way to visualise these clusters which you can see here for the first two cluster plot:

OK as you can see each cluster has a subplot. Each splot has two stacked bars. The lefthand side shows the composition by asset class. The righthand side shows the composition by trading rule. You can just about make out from the legend what the various colours mean. Note these aren't portfolio weights, and just reflect the number of instrument/rule combos in each category. 

If your eyes are very good you will see the tops of the left hand bars don't quite reach 1. This is because I've removed anything with less than 2% of the total from the plots for clarity. Mainly this is a few odd instruments in a few very small asset classes (like volatility). It doesn't affect trading rules, since even the mrinasset rule has 1/40 = 2.5% of the total count.

The important thing here is we aren't yet seeing any evidence of clustering eithier by instrument or by trading rule. If we were, there would be a preponderance of colours on one side or the other. For example, if correlations were higher amoungst trading rules even from different instruments, then one cluster would have a lot of blue and purple in the right hand bar of one cluster (colours I assign to more divergent rules like momentum), and yellow and orange in the other right hand bar (colours that are reserved for convergent rules).

At this stage then there is no evidence that rows or columns makes more sense. 

Here is the three cluster plot. All three clusters look very similar for both instruments and rules, again suggesting there isn't much going on here yet on eithier axis.

I'm going to skip through the next few plots, since none of them show anything interesting. Let's check in at N=16:

Still not much going on here! We aren't seeing clumps of colours developing for eithier bar.

N=36:

49:

Never have so many coloured pixels mean wasted for so little result!

Let's close out with N=64, a nice power of 2 to finish on, as we're reaching the point where the plots are getting so small they're impossible to see:

There you have it folks. No clear evidence for eithier across-> down, or for down-> across. 

Now, there are a few alternative conclusions we could draw here. One is that there is some weird deep correlation pattern that the simple analysis by asset class and trading rule doesn't pick up. I don't buy that. If for example we look at the final cluster from N=64, it looks like this:

[assettrend4 forecasting AEX, breakout80 forecasting AEX, momentum32 forecasting ALUMINIUM, breakout160 forecasting AUD_micro, assettrend16 forecasting BITCOIN, normmom64 forecasting BITCOIN, accel64 forecasting BONO, carry125 forecasting BOVESPA, momentum64 forecasting BOVESPA, breakout80 forecasting BRE, normmom4 forecasting BRENT_W, accel32 forecasting BTP, momentum8 forecasting BTP3, accel16 forecasting BUTTER, assettrend16 forecasting BUTTER, carry125 forecasting CAD, assettrend32 forecasting CAD10, relcarry forecasting CAD2, breakout20 forecasting COAL, breakout40 forecasting COPPER_LME, relmomentum40 forecasting COPPER_LME, relmomentum20 forecasting COTTON, breakout320 forecasting DOW, assettrend8 forecasting EU-AUTO, relmomentum40 forecasting EU-CHEM, assettrend2 forecasting EU-CHEM, normmom2 forecasting EU-CHEM, relmomentum80 forecasting EU-DIV30, relmomentum10 forecasting EU-DJ-TELECOM, relmomentum20 forecasting EU-HOUSE, relmomentum20 forecasting EU-INSURE, carry10 forecasting EU-MEDIA, accel32 forecasting EU-OIL, assettrend2 forecasting EU-TECH, carry10 forecasting EU-TECH, breakout20 forecasting EURAUD, momentum8 forecasting EURCAD, carry30 forecasting EURCHF, carry60 forecasting EURCHF, normmom8 forecasting EURIBOR-ICE, mrinasset1000 forecasting EUROSTX, momentum4 forecasting EUROSTX-SMALL, normmom16 forecasting EUR_micro, assettrend32 forecasting FANG, mrinasset1000 forecasting FED, assettrend4 forecasting FED, normmom32 forecasting FEEDCOW, carry60 forecasting FEEDCOW, accel16 forecasting FTSECHINAH, breakout20 forecasting FTSEINDO, relmomentum20 forecasting FTSEINDO, breakout10 forecasting FTSETAIWAN, normmom8 forecasting FTSEVIET, breakout80 forecasting FTSEVIET, breakout40 forecasting GASOIL, relmomentum10 forecasting GAS_US_mini, momentum8 forecasting GAS_US_mini, mrinasset1000 forecasting GICS, accel32 forecasting HANGENT_mini, skewabs365 forecasting HANGENT_mini, carry30 forecasting HOUSE-US, momentum8 forecasting HOUSE-US, breakout320 forecasting HOUSE-US, assettrend2 forecasting IBEX_mini, relmomentum20 forecasting INR, normmom2 forecasting INR, normmom64 forecasting IRON, accel64 forecasting JGB, carry10 forecasting JGB-SGX-mini, normmom16 forecasting JGB-SGX-mini, assettrend4 forecasting KOSPI_mini, mrinasset1000 forecasting KR10, normmom32 forecasting KR3, momentum4 forecasting KR3, relmomentum40 forecasting KRWUSD_mini, normmom64 forecasting LEAD_LME, carry125 forecasting LEAD_LME, breakout10 forecasting LEANHOG, relmomentum40 forecasting LIVECOW, relcarry forecasting LIVECOW, breakout160 forecasting MILKWET, assettrend2 forecasting MSCIEAFA, skewabs180 forecasting MSCIEMASIA, relmomentum20 forecasting MSCIEMASIA, skewrv365 forecasting MSCIEMASIA, breakout160 forecasting MUMMY, normmom16 forecasting OAT, breakout80 forecasting OJ, skewrv180 forecasting PALLAD, normmom2 forecasting PALLAD, normmom64 forecasting PLAT, normmom16 forecasting RUSSELL, normmom2 forecasting SEK, skewrv365 forecasting SHATZ, normmom64 forecasting SHATZ, accel16 forecasting SHATZ, carry30 forecasting SHATZ, breakout10 forecasting SILVER, assettrend8 forecasting SMI, breakout40 forecasting SMI-MID, momentum16 forecasting SMI-MID, momentum64 forecasting SOFR, carry30 forecasting SOYBEAN_mini, normmom2 forecasting SOYBEAN_mini, skewabs365 forecasting SOYBEAN_mini, skewrv365 forecasting SP500_micro, skewabs365 forecasting STEEL, skewrv365 forecasting SUGAR16, relmomentum80 forecasting TIN_LME, carry10 forecasting TIN_LME, relmomentum40 forecasting TWD, normmom8 forecasting US-ENERGY, carry30 forecasting US-MATERIAL, mrinasset1000 forecasting US10U, carry10 forecasting US2, momentum32 forecasting US30, breakout40 forecasting US30, relcarry forecasting US30, skewrv180 forecasting US5, momentum64 forecasting V2X, carry125 forecasting VIX_mini, breakout10 forecasting VIX_mini, skewabs180 forecasting VNKI, normmom4 forecasting WHEAT, relmomentum10 forecasting YENEUR, assettrend4 forecasting ZAR, accel64 forecasting ZAR]

I look forward to anyone who can give me a coherent story as to why those things are lumped together. 

Personally I'm taking the absence of any contradictory evidence as evidence that I should continue to do what I've done before: fit across and down. Doing some kind of all group clustering or all in one fit, or doing down and then across; none of these seem to offer clear advantages. So why not stick with a simple thing that works?

Note - there is no reason why in theory you could not do an 'all in one' fit or weird clustering, and still use the procedure of generating a combined forecast for an instrument, then a subsystem position for an instrument, and then forming a portfolio. You just take the weights for each rule/instrument pairing from your all in one or weird cluster fit, and then take them across each instrument for forecast weights, and then add up the weights for a given instrument to find instrument weights.

Arguably this has been a waste of time, but the good news is I can recycle this code to visualise forecast weights across a strategy so that's something....

SECOND ATTEMPT: USE AVERAGE CORRELATIONS

As I was posting this up I thought of a much simpler test: if I measure the average correlation of forecasts within instruments (rule/instrument pairings for the same instrument) then it is 0.29. However the average correlation of forecasts within rules (eg rule/instrument pairings with the same rule) is much lower: 0.05. That reinforces that across and down is indeed the way to go.

THIRD ATTEMPT: TRY WITH SMALLER UNIVERSE

Arguably we might not see the pattern we required until we had hundreds of clusters; equal to the number of instruments in the universe. 

So I decided to do a 'small sample' approach. If I were to pick say 5 arbitrary instruments, and 5 arbitrary trading rules, what is the likelihood that these 25 components would cluster into instrument groups rather than rule groups; or nothing at all?

To make this a proper test I need to repeat the random choice of instruments and trading rules many times over. That means I can't just eyeball the charts each time, looking for a preponderance of colours on one side or the other. This is both timeconsuming and subjective.

Let's come up with a systematic rule. Seems on brand for this blog, doesn' it? Given 5 clusters, we consider them to have formed into instrument groups if  50% or more of the weight in a cluster is for a single instrument, and this has happened in 3 or more of the clusters. Alternatively, we consider them to be rule groups if 50% or more of the weight in a  cluster is for a single rule, and this has happened in 3 or more of the groups. Otherwise we consider them to be ungrouped (and that would be the case for all the charts above). Also any cluster with only one or two components is excluded from the calculation.

I do want to show you an example, since I don't want to waste my nice pictures. Here is what might be a particularly extreme example since it includes only US bonds (that are somewhat correlated): US2, US5, US10, US20, US30; and a smattering of trading rules (less likely to be correlated):

Cluster#1: [breakout40 forecasting US2, breakout40 forecasting US20, breakout40 forecasting US30, breakout40 forecasting US10, breakout40 forecasting US5] 

100% in single rule: meets rule grouping criteria

Cluster#2: [momentum64 forecasting US30, momentum64 forecasting US5, momentum64 forecasting US20, momentum64 forecasting US10, momentum64 forecasting US2], 

100% in single rule: meets rule grouping criteria

Cluster#3: [skewrv365 forecasting US2, skewrv365 forecasting US5, skewrv365 forecasting US20, skewrv365 forecasting US10, skewrv365 forecasting US30]

100% in single rule: meets rule grouping criteria

Cluster#4: [carry10 forecasting US10, carry10 forecasting US20, relcarry forecasting US20]

66.6% in single rule: meets rule grouping criteria; 66.6% in single instrument: meets instrument grouping criteria

Cluster#5: [carry10 forecasting US2, relcarry forecasting US2, relcarry forecasting US5, relcarry forecasting US30, relcarry forecasting US10, carry10 forecasting US30, carry10 forecasting US5]

57% in single rule: meets rule grouping criteria

Since 5/5 meet the rule group criteria (more than half), and only 1/5 meets the instrument group criteria, this is a rule group clustering.

Here is a more random selection:

['HEATOIL', 'SMI-MID', 'RUSSELL', 'BOBL', 'EUA']

['skewrv365', 'momentum64', 'carry10', 'relcarry', 'breakout40']

Which clusters as follows:

Cluster#1 [carry10 forecasting RUSSELL, relcarry forecasting RUSSELL, carry10 forecasting SMI-MID, breakout40 forecasting SMI-MID, relcarry forecasting SMI-MID, momentum64 forecasting SMI-MID, breakout40 forecasting RUSSELL, momentum64 forecasting RUSSELL] - this is actually two instrument clusters with exactly 50% in each

Cluster#2 [skewrv365 forecasting SMI-MID, carry10 forecasting EUA, relcarry forecasting EUA, skewrv365 forecasting EUA] - meets both criteria

Cluster#3 [carry10 forecasting HEATOIL, breakout40 forecasting HEATOIL, relcarry forecasting HEATOIL, momentum64 forecasting HEATOIL] - Heating oil cluster

Cluster#4 [breakout40 forecasting EUA, momentum64 forecasting EUA] - ignored, only 2 components.

Cluster#5 [carry10 forecasting BOBL, breakout40 forecasting BOBL, momentum64 forecasting BOBL, skewrv365 forecasting BOBL, relcarry forecasting BOBL, skewrv365 forecasting HEATOIL, skewrv365 forecasting RUSSELL] mostly BOBL

Since 3/4 valid clusters meet the 50% instrument threshold, and only one meets the 50% rule threshold, this would be a case where we would cluster by instruments most logically.

Anyway I repeated this exercise a few thousand times, and here are the results as a proportion of the total:

Meets neithier criteria: 3.8%

Meets both criteria: 0.81%

Meets rule grouping criteria: 0.11%

Meets instrument grouping criteria: 95.3%

That seems pretty conclusive

The State Of Vol

 I'm sometimes asked where I get my ideas for new trading strategies from. The boring truth is I rarely test new trading strategies, and I mostly steal ideas when I feel in the mood. Today for example I saw this tweet post on twitter X:

Think high market volatility leads to high returns? No. Periods with high market volatility are followed by high volatility, but not necessarily better returns. A volatility effect through time. Check out the graph below, based on the US equity market from 1927 to 2023. The… pic.twitter.com/23rrTwIf9O

— Pim van Vliet (@paradoxinvestor) October 2, 2023

The original paper is here  (requires subscription or academic institution membership)

Now I've written in the past about how volatility levels affect the profits of trading strategies, in particular momentum where there is a pretty striking effect (see my new book Advanced Futures Trading Strategies [AFTS] for more info), and I've also written about how the level of past vol affects future vol (and indeed, this incredible predictability of vol is a cornerstone of the inverse volatility sizing formula that lies at the heart of my trading strategies), but I don't think I've ever written about the level of volatility's effect on future price movements. 

There is quite a simple story here which if you can see the paper is in figure 1: absolute returns are pretty flat across different vol regimes, and because vol is pretty autocorrelated from month to month, we'd expect return/vol to be higher if recent vol was lower. To put it another way, there is a link here with my previous post about CAPM; this story is sort of about the time series version of CAPM in which higher vol should be rewarded with higher returns (and hence Sharpe Ratios should not show any pattern conditional on relatively volatility levels), and the opposite happens in single stocks in an analogy to the fact that CAPM doesn't work in cross section for single stocks, in fact we should 'bet against beta' and buy low beta stocks.

But the paper is written for single stocks; a risky asset, and one with some idiosyncratic distributional features. Will the results hold for the wider universe of futures markets? 

Data and definitions

I am not interested in reproducing the results in the paper since I'm more concerned with whether this is a profitable trading strategy sitting within my usual framework, so I will do things a bit differently.

I use my usual set of futures markets, after removing duplicates (eg micro and mini versions, or cross listings) it comes in at 211 instruments. Data goes back to 1970, and I use my standard vol estimate calculated in % terms, which I divide by an exponential moving average of the same with a 10 year vol halflife to get a relative vol measure. If this is 1, then the vol we're seeing is going to be typical of the level over the last couple of decades or so, if it's higher than 1 then the vol is higher, and so on. Again readers of AFTS will recognise this measure. The paper doesn't quite do this; it just looks at the level of vol versus the in sample historic distribution. This means we can't use that forecast as a trading signal.

For the y-axis of response, I will use the returns in the following month normalised by the volatility estimate (in price differences this time) at the beginning of the month for reasons I have discussed before. As I'm trading futures, this is also the Sharpe Ratio. The original paper uses a straight one month moving window of returns for vol estimation, and the pure ex-post Sharpe Ratio using the realised vol in the following month, so some small difference there. However because of the autocorrelation of vol, it shouldn't affect things too much; if anything my results probably should be a bit better because I'm dividing an unknown future estimate of mean by a known current estimate of standard deviation to get Sharpe Ratio, rather than having unknown mean/unknown sigma.

The original papers uses single stocks, I use a massive variety of futures allowing me to see if this effect persists at the stock index level and for other asset classes. 

Some comparable graphs

Let's start with producing the same kind of graphs as in the paper. So each month I look at the current level of vol versus it's historic average, and then rank it on an in sample basis (yes I know, but for the time being...), and then bucket the ranking into quintiles. I measure the risk adjusted return in the following month and take a median of those Sharpe Ratios; medians being somewhat more robust than mean.

Here is the result for the S&P 500:

I use the same ordering; 1 on the left is the lowest quantile of vol, 5 on the right is the highest. That picture is not quite as compelling as the original paper; but ignoring the 2nd bar it has the right pattern: higher recent vol means lower risk adjusted returns in the following month. Nevertheless we are only using 492 data points compared to the thousands we can get with single stocks; plus I am suspicous of anyone that uses a single instrument to prove anything, so let's expand out to all equity futures (with months stacked, which will give a higher weight to instruments with more history):

That is .... not what we'd hoped for.

The name is Bond:

Not great. What about Vol (VSTOXX, VIX)?

OK only two markets, but there is an apparent reverse effect here. You don't want to be long vol on average, but perhaps the worst time to be is when vol of vol is very low.

FX:

At best noise, at worst the reverse of what we expect.

Metals

That is the strongest, wrong way round, effect we have seen. Agricultural:

Just noise really. And finally energies:

The wrong way round, and inconsistent.

It's probably futile, but what happens if we pool our results across all markets?

There could be a story there- higher vol means higher risk adjusted returns, unless vol is very high; or we could be looking at the aggregated result of jamming together instruments that behave very differently*, plus a bunch of instruments that are just noise. 

[* it looks like markets fall into two camps, one where the effect is the 'right way' round (high vol, lower returns; perhaps the S&P 500 is here), others where it is the wrong way (most strikingly, vol), and the third of the two camps doesn't really have a strong effect]

Just to give a point of comparison, here's the bucket plot if I replace the ratio of vol with the level of the 32/128 day momentum forecast level (something we know is very successful as a forecast at a portfolio level, although it is weak in say equities):

Clearly this vol effect is not as strong as momentum.

A trading rule

Although the results aren't what we expect, this could be profitable as a 'wrong way round' rule, which buys when vol is higher, and sells when vol is lower. This won't work for very high vol (since quintile 5 shows a worse return for quintiles 2,3 and 4, but then we would hit forecast capping up there anyway. That isn't much different to what happens with faster momentum rules anyway; if you plot response versus forecast level it's not as good for extreme forecasts (see AFTS for more).

In my first book, Systematic Trading, I said that implementing a rule as a wrong way rule after discovering it is a form of data mining so bear in mind that the backtest results here will effectively be overstated [in reality, you couldn't have implemented this rule until enough historical evidence that the effect was the 'wrong' way round existed; until that moment of statistically significant realisation we would eithier be trading the money losing 'right way round' rule, or a combination of the two, eithier of which is inferior to being all in on 'wrong way round' from the start of the backtest].

I'd add a couple of tweaks as well; firstly rather than quantile buckets which are too coarse and in sample; I will use the percentile of the ratio versus historic levels as the forecast (the same approach I use for the volatility overlay rule in AFTS) , and also chuck in my standard ewm(span=10) smooth to reduce noise on what should be a monthly holding forecast (this reduces turnover from 25 times a year to 10 times without affecting profitability).

Before continuing, it might be worth thinking about how this interacts with the volatility overlay on a trend following rule (which cuts exposure long or short when vol is high).

• If vol is low, we'd be SHORT from a vol rule
• ... and we'd have a stronger trend following signal
• if markets are trending up we'd be net flat
• If markets are trending down we'd have a strong sell
• If vol sits in the middle, we'd be FLAT from a vol rule
• .... and we've have an unadulterated trend signal
• If markets are trending up we'd be long
• If markets are trending down we'd be short
• If vol is high, we'd be LONG from a vol rule
• .... and have a weaker trend following signal
• If markets are trending up we'd be modestly long
• If markets are trending down we'd be flat

Clearly this will work differently for risk on/risk off markets; risky markets like equities will most probably go up with low vol and down with high vol, thus occupying the two lines shown with italics. In these situations we'd have no position on (assuming these are the only two trading rules we're using). Since trend following doesn't work super well in equity indices, this might improve our lives somewhat.

Risk off markets, the most extreme of which is VIX/VSTOXX, will occupy the lines in bold as they have high vol when they are going up, and low vol vol the way down. Notice that this will introduce a short bias (modest long on the way up, strong short on the way down); since these markets tend to lose money on the long side again we might expect a benefit here.

Other markets - which form the majority of futures instruments - will be less clear, so let's see how it goes.

Here's the code for the purely backward looking vol quantile calculation, and the actual trading rule:

def get_vol_quantile_points(self, instrument_code):
    self.log.debug("Calculating vol quantile for %s" % instrument_code)
    daily_vol = self.parent.rawdata.get_daily_percentage_volatility(instrument_code)
    ten_year_vol = daily_vol.rolling(2500, min_periods=10).mean()
    normalised_vol = daily_vol / ten_year_vol

    normalised_vol_q = quantile_of_points_in_data_series(normalised_vol)

    return normalised_vol_q

def quantile_of_points_in_data_series(data_series: pd.Series) -> pd.Series:
    ## With thanks to https://github.com/PurpleHazeIan for this implementation
    numpy_series = np.array(data_series)
    results = []

    for irow in range(len(data_series)):
        current_value = numpy_series[irow]
        count_less_than = (numpy_series < current_value)[:irow].sum()
        results.append(count_less_than / (irow + 1))

    results_series = pd.Series(results, index=data_series.index)
    return results_series

def vol_rule(vol_quantile_points: pd.Series, smooth: int = 10):
    # vol quantile points sits in space 0 to 1.0
    raw_forecast = (vol_quantile_points - 0.5) * 40  ## sits in space -20 to +20
    smoothed_forecast = raw_forecast.ewm(span=smooth).mean()
    return smoothed_forecast

System test

I'm testing this using my current trading system with 147 instruments and relevant instrument weights. I'll be using the 'static' rather than 'dynamically optimised' flavour of the system, to get a feel for pure performance before the noise added by optimisation. This also means I'm going to put in an unrealistically large slug of capital, and remove a few markets that are too expensive to trade, bringing me down to 138 markets. I estimate instrument weights and the IDM (which peaks at 2.15) using my usual optimisation defaults.

To begin with, let's look at the performance of the vol strategy by itself. You're all dying to see it, so here is the money shot with the full account curve before and after costs:

Well it's not terrible but it's not amazing eithier. The Sharpe is a mere 0.10, which is not exactly knocking it out of the park... at best we've tapped the ball and it's dribbled a few feet away. 

Can this thing add value when combined with momentum (particularly given the discussion above)? Let's keep it simple and just use a single ewmac rule, 16/64. Correlation between the rules is actually a little negative, so let's do some god-awful in sample fitting and allocate 10% of our portfolio to the new vol rule.

It's not really worth plotting as these two systems will both look pretty similar, but this relatively small allocation to our new putative signal does indeed push up the Sharpe Ratio from 1.10 to 1.15, with the Sortino rising by a similar amount. Costs are slightly reduced, skew falls slightly (from 0.55 to 0.48), but the more robust lower tail ratio is unchanged (see AFTS for a definition). 

Summary

It does seem like the predictive effect of vol in single equities isn't replicated across the futures universe; if anything the effect is reversed although it is not as strong as in the original paper. Rather than buying instruments with lower vol to get higher risk adjusted returns, we should do the opposite. Rather than 'time series CAPM' failing, and a 'bet against time series standard deviation levels', we in fact see an even stronger 'time series CAPM' where risk adjusted returns aren't constant with relative vol levels, but actually improve when volatility is relatively high.

If we use this idea to construct a simple trading rule the result is not the world's greatest standalone signal. But there does seem to be some promise in adding it to a trend following strategy due to it's negative correlation.