Breaking Badly: finding the structural breaks in parameter estimates

 Here's a nice picture from a lovely book written by a top bloke:

It shows the cumulative p&l from different speeds of momentum over time (for portfolios containing 102 instruments) over 50 years of data. Notice how the two fastest speeds (2&4) get worse in the second half of the sample. I've called the line #2 here the 'second most famous hockey stick graph in history'. It certainly looks like something changed in 1990. 

This is important. If we're optimising portfolios of such things we only want to consider data that is relevant, but we also want as much data as possible for statistical significance. Now if I were a simpleton I'd do this by looking at graphs like that and going 'aha i only need to use data after 1990'. As a simpleton I don't use capital letters. But I am a big fan of not doing in sample fitting, even of meta parameters like this; and I am an even bigger fan of doing things automatically which means not wading through thousands of graphs like that (since there are thousands of SR estimates in my forecast p&l space, plus a good chunk of correlations).

So we need an automatic way of identifying such breaks. Fortunately this is not a new problem as you will know if, like me, you did undergraduate econometrics. Finding structural breaks is an entire industry. We need two things: a test for how likely it is that a break has occured between two sub-samples A and B. And an algorithim for going through all the options of A and B

And in case you haven't realised this is the seventh post in my summer 2026 series on portfolio optimisation.

What parameters

The first question to think about is what parameters we're going to apply this process to. I do two kinds of optimisation:

• Forecast weights
• Instrument weights

And in both cases I have estimates of SR (one per asset) and correlations ([N^2-N]/2 for N assets). I haven't really looked at instrument weight optimisation yet in this series, and there are some wrinkles there so I'm going to park that for now. That just leaves the SR for a forecast (which remember is a pairing of a trading rule and an instrument), and the correlation of such forecasts within an instrument.

Now I am going to ignore correlations in this post. As I discussed in an earlier post, although correlations are relatively unstable in the short term, they are unlikely to have secular trends like SR. And it's quite easy to deal with this by just using a relatively long lookback to estimate them, probably with an ewma on the correlation estimate.

What test

This is quite an easy one, compared to the world of econometrics and linear regression where we have to do such nonsense as a Chow test. Given two sub-samples A and B, to find out if they have different sample means we just do an independent t-test: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html

An important characteristic of such tests is they are only likely to be significant if there is sufficient data. If sample A or sample B is too small in size, it's unlikely we'll find a signficant difference in the means.

Typical values for critical values in t-tests are 1% and 5%; I will also check out 10% later.

Since all the p&l streams in a risk targeted trading system will have the same expected standard deviation in the long run, I can legitmately treat this as a test to see if the SR are different if I adjust the samples so they have an identical standard deviation. 

(If any ex-students of mine are reading this, you should remember this from week 3 of the course!)

What algo / procedure

OK let's make this concrete. Suppose we have 50 years of data as in the original graph above, and we're currently wondering if there were one or more structural breaks. We don't want to have less than five years data to do our estimation. Note these numbers are appropriate for my context, but not in a domain with faster trading, higher SR, and faster alpha decay. We proceed as follows:

• We compare year 1 with years 2 to 50. Since year 1 is very small it's unlikely we'll find a break; but if we do then we do a split (see below).
• If no split has occured, we compare years 1-2 with years 3-50. Again if we find a break, we split.
• If no split has occured, we compare years 1-3 with years 4-50...
• ...
• If no split has occured, we compare years 1-45 with years 46-50. If we still don't find a break, then we use the entire period for estimation (since the period 47-50 will only have four years, we terminate here).

Now what if a split occurs at some point? Then we restart the process, but this time without the pre-split data. Suppose for example a split had occured in year 20 (which is 1992 in the original graph). Then:

• We compare year 21 with years 22 to 50. If we find a break, then we split again.
• If no split has occured, we compare years 21-22 with years 23-50. Again if we find a break, we split.
• If no split has occured, we compare years 21-23 with years 24-50...
• ...
• If no split has occured, we compare years 21-45 with years 46-50. If we still don't find a break, then we use the entire period from years 21-50 for estimation.

You get the idea. This procedure is quite quick and easy to run; and notice we're identifying any break that exceeds a certain threshold rather than finding the most likely break as we would do with a QLR type test. 

The main downside of it is that it won't identify breaks that reverse. For example if the world is in regime A, then regime B, then regime A again then ideally we'd estimate our parameters using both regime A's. But the above test will eithier use only the final regime A, or the last two regimes, or possible all three regimes depending on whether there is a significant difference at the appropriate point(s). There are fancy things we could do to deal with this, but I feel these are corner cases and life is too short.

An example

Let's use a concrete example. This is the performance of the momentum4 rule on CORN. I've chosen it because we already know momentum4 has a structural break, and CORN has plenty of history. Just for fun, before reading on, see if you can identify where the algo finds the break here (there is exactly one break - this isn't a trick question).

Some code

This is probably the first post in this series where it's been practical to actually include the code, since there isn't much of it, nor are there are any dependencies apart from getting the returns:

from copy import copy
from typing import List, Callable

import numpy as np
import pandas as pd
from scipy.stats import  ttest_ind

BUS_DAYS_IN_YEAR = 256
import matplotlib.pyplot as plt

MIN_NUMBER_OF_YEARS = 5

def identify_and_plot_breaks(all_returns: pd.Series, CV: float =0.01):
    breaks_as_dict = identify_all_breaks(all_returns)
    breaks_as_df = pd.DataFrame(breaks_as_dict)
    breaks_as_df = breaks_as_df.bfill(axis=1)
    breaks_as_df.cumsum().plot()
    plt.show(block=True)


def identify_all_breaks(all_returns: pd.Series, CV: float):
    ## returns a dict, turn into a dataframe and you can plot
    returns_to_consider= copy(all_returns)
    returns_to_consider = returns_to_consider.dropna()
    broken_list = identify_all_breaks_recursively(returns_to_consider=returns_to_consider, list_of_returns_broken_off=[],
                                                   CV=CV)
    broken_list.reverse()
    broken_dict = dict([
        (idx, value) for idx, value in enumerate(broken_list)
    ])

    return broken_dict

def identify_all_breaks_recursively(returns_to_consider: pd.Series, list_of_returns_broken_off: List, CV: float) -> List:
    years_in_returns = how_many_years_approx(returns_to_consider)

    for i in range(years_in_returns):
        year_idx=i+1
        first_sample, second_sample = split_sample_after_n_years(returns_to_consider, year_idx)
        if len(second_sample)<(MIN_NUMBER_OF_YEARS*BUS_DAYS_IN_YEAR):
            break
        is_broken_here = test_a_break(first_sample, second_sample, CV=CV)
        if is_broken_here:
            list_of_returns_broken_off.append(first_sample)
            return identify_all_breaks_recursively(
                second_sample, list_of_returns_broken_off=list_of_returns_broken_off,
                CV=CV
            )
        else:
            continue

    ## No breaks identified or sample size too short
    list_of_returns_broken_off.append(returns_to_consider)
    return list_of_returns_broken_off

def how_many_years_approx(returns: pd.Series):
    return int(np.floor(len(returns)/BUS_DAYS_IN_YEAR))

def split_sample_after_n_years(all_returns: pd.Series, n_years: int):
    idx = n_years*BUS_DAYS_IN_YEAR
    return all_returns[:idx], all_returns[idx:]

def test_a_break(first_sample: pd.Series, second_sample: pd.Series, CV: float):
    ## Normalise by standard deviation before considering means
    norm_first_sample =first_sample/first_sample.std()
    norm_second_sample=second_sample/second_sample.std()
    return ttest_ind(norm_first_sample, norm_second_sample).pvalue<CV

On Corn this produces the following:

The break occurs on the 26th August 1982. So to calculate that particular SR estimate we'd only use data from that date onwards.

Is that what you would have guessed? Personally, I would probably have gone for a later break point if identifying it by eye. As humans we are drawn to the sharp upward move in 1989 and would probably have gone for a break just after that. It's possible a search for the likeliest break would have found that point, but remember we are looking for the first break that exceeds the threshold; and once that break happens no further breaks are identified.

A summary of results

Here is a summary of the results for each instrument/forecast pairing with the default 1% critical value:

You can see that breaks are quite rare with only 13% or so of instrument/rules having at least one break. This also suggests that the Sharpe Ratios for trading rule performance are actually quite stable over time; or at least stable enough that they won't fail any statistical tests at a 1% critical value. 

Multiple breaks are even rarer. Just 1.8% have two breaks; 0.4% or 39 instruments have three breaks, ten have four breaks and only two have five breaks. They are:

skewrv365 forecasting EURIBOR-ICE    (yes, there is still a EURIBOR future!)

normmom2 forecasting FTSE250         

Here is Euribor, relative value skew with a 365 day window:

Although five is pushing it, there are certainly three regimes there (pre 2000, 2000 - 2010, and 2010 onwards), and using post 2010 data seems to make some kind of sense.

And FTSE 250 momentum8 (this is pre-cost):

There are certainly at least two regimes there and I wouldn't argue with the automated decision to use only data after 2006 or so, when it looks like; in the words of Pulp in one of my favourite songs, "Something changed".

Of course we will get a different picture with a slacker test. Here is the picture with a 10% critical value:

Now just over half the pairings have at least one break in them.

The decision as to use 0% (equivalent to no breaks at all), 1%, 5% or 10% CV is one we will now address.

An optimisation test

Now the big question is does this actually improve performance? On a pure out of sample test? And is this changed much by using a different critical value?

I follow the same procedure roughly as in previous posts:

• Select 10,20,30 or 40 years of in sample data (I need at least 10 years because with a minimum of five years required for estimation I certainly won't find any breaks, or I will risk finding a break and not having five years of data leftover)
• Select 1 or 5 years of out of sample data
• Pick a random instrument, ensuring there is enough history available (between 11 and 45 years). We will only choose from instruments with sufficient history for the time required.
• Randomly pick N=9 forecasting rules from those available (the same number as in posts #2 and #3)

Then for each of those sumsamples:

• Cycle through using no breaks (0% CV), 1% CV, 5% CV and 10% CV
• Estimate SR on the insample data using eithier all the data (0% CV), or the data after the last break given some critical value. 
• Estimate correlation using all the in sample data
• Use fixed shrinkage levels (estimated here): SR shrinkage 0.5, correlation 0.75 (since we'll always have at least five years of in sample data we don't need to worry about the higher levels of shrinkage required when we have insufficent data). The results won't be much different with any vaguely similar shrinkage.
• Run in sample optimisation and out of sample optimisation on all the options above

Finally once we have all our subsamples:

• Get the median SR from the distribution of subsamples
• Find the optimal CV with the highest SR
• Test to see if that median is significantly higher than the others

10 years in sample, one year out of sample

We only have four options to consider so no need for the huge tables and fancy heatmaps of previous posts:

         SR  pvalue all  pvalue distinct
0.00 -0.021       0.247            0.247
0.01 -0.018       0.295            0.295
0.05 -0.019       0.204            0.204
0.10 -0.014         NaN              NaN

Each row is a different critical value used for breakpoint finding. Zero means the entire in sample period was used. The next column is the out of sample Sharpe Ratio for each option. In the second column is the p-value for a test of the optimal option against the relevant option. NaN is the optimal option, and lower values (say below 0.05) mean the optimal option is significantly better than the alternatives. In the final column I've rerun the staistical t-test but this time I have excluded instances where no breaks were found (so the test is only done comparing the out of sample SR when breaks were found, versus when they were not). This shouldn't affect the p-values, but it's nice to check it doesn't

You can see here that it it looks like a very loose breakpoint policy is the best, but it's not significantly better.

10 years in sample, five years out of sample

         SR  pvalue all  pvalue distinct
0.00  0.166       0.168            0.168
0.01  0.160       0.003            0.003
0.05  0.167       0.007            0.007
0.10  0.170         NaN              NaN

Again the loosest breakpoint works best; but it's hardly logical since it's indistinguishable from no breakpoints at all.

20 years in sample, one year out of sample

         SR  pvalue all  pvalue distinct
0.00 -0.160       0.542            0.542
0.01 -0.160       0.323            0.323
0.05 -0.160       0.050            0.050
0.10 -0.155         NaN              NaN

Loose is better.

20 years in sample, five years out of sample

         SR  pvalue all  pvalue distinct
0.00  0.123         NaN              NaN
0.01  0.116         0.0              0.0
0.05  0.105         0.0              0.0
0.10  0.098         0.0              0.0

No breakpoints are the best. There isn't much consistency here.

30 years in sample, one year out of sample

         SR  pvalue all  pvalue distinct
0.00  0.092         NaN              NaN
0.01 -0.019         0.0              0.0
0.05 -0.078         0.0              0.0
0.10 -0.066         0.0              0.0

Another massive win for not using breaks.

30 years in sample, five years out of sample

        SR  pvalue all  pvalue distinct
0.00 -0.005       0.001            0.001
0.01 -0.011       0.000            0.000
0.05 -0.008       0.014            0.014
0.10 -0.003         NaN              NaN

Or perhaps we should go for the loosest break....

40 years in sample, one year out of sample

         SR  pvalue all  pvalue distinct
0.00 -0.142         NaN              NaN
0.01 -0.249       0.000            0.000
0.05 -0.189       0.000            0.000
0.10 -0.178       0.002            0.002

or no breaks...

40 years in sample, five years out of sample

         SR  pvalue all  pvalue distinct
0.00 -0.060       0.177            0.177
0.01 -0.051         NaN              NaN
0.05 -0.055       0.000            0.000
0.10 -0.055       0.000            0.000

A clear vote for strict breaks, but no breaks at all are also good.

Conclusion

Well that was as clear as a bowl full of mud that has been made even more unclear by painting the bowl a very dark colour and then adding some darker mud. Not very clear at all, in other words.

You think up a nice neat simple way of finding structural breaks, and then it doesn't actually work when used for optimisation. In seven out of the cases we examined not using breaks is eithier optimal, or statistically insignificant from the optimal. Only in one case was it inferior. The loosest possible break (CV 10%) was optimal in four cases. In most cases apart from 30 years/1 year the difference in performance was small between CV alternatives. This partly reflects the fact that breaks aren't that common, especially with a 1% CV.

Of course findings like this are very context dependent. I'm using rules that should probably work over long time periods; indeed they have been in sample selected for such a purpose. For the 30 year and 40 year periods there just aren't that many instruments with that much history; so even repeated random sampling is likely to turn up the same suspects repeatedly.

One issue here might be that we are considering the breaks at a forecast/instrument level. We might get different results if we pool estimates for trading rule performance across instruments. And indeed that is the subject of the next post. So I will return to this topic when I've looked at pooling.

Forecasting statistical estimates when data gets real

 This is my third post in a series about optimisation and fitting. In my previous post I used random data to calibrate and evaluate many portfolio optimisation techniques. It's worth quoting in full from that post:

Random data is not real data: Well duh. But why is this important? Because random data is drawn from a fixed and well behaved distribution. This means the optimiser only has to discover / estimate the parameters of that distribution as more data is revealed to it. But real data doesn't have a fixed and known distribution. It doesn't actually have any distribution at all. We just model it hoping it does.

To summarise then, random data from a fixed distribution differs from real data in three important ways:

• There is no distribution! We just assume there is one.
• The distribution (which doesn't exist) is not known, and thus it's likely the distribution we assume is the wrong one. This is especially true for modelling underlying financial price returns with joint Gaussian models.
• The distribution (which again, doesn't exist) isn't fixed, but can change over time.

And it has one thing in common:

• The unknown parameters of the distribution are unknown and have to be learned over time.

In this post I'm going to explore this learning process for two key statistical estimates: correlations and Sharpe Ratios. What I am interested in is how much wider of the mark our estimates for these two things are likely to be for real data vs random data. This obviously has important implications for optimisation.

Let's look at a plot.

This is for random data generated by a process with a true SR of 1. It shows the evolution of the SR and it's statistical distribution as it is re-estimated each year. There is a burn in year which is missing, and then in the first year we can see our estimate of the SR using all available data so far (in orange), and the SR for the current year (in blue). You can see that the orange line is lagged by a year as it is purely out of sample and always a year behind. I've then used the orange line to estimate the theoretical sampling distribution of the Sharpe Ratio for a one year period, and constructed a 1.96x confidence interval (so about 95%) around the orange line which are the green and red lines. 

Note: The theoretical standard deviation of the sampling distribution of the Sharpe Ratio, assuming i.i.d. returns, is sqrt[(1+0.5SR^2)/N] where N is the number of periods.

Broadly speaking if our estimates are correct then we'd hope to see around 1/20 of the blue points outside the red and green lines, and around 19/20 on the inside. There are 40 years of data here and we go outside the range twice, which is roughly what we'd expect.

Another way of measuring this is to look at our error term, normalised by our standard deviation. This will be equal to:

[(SR estimate this year N) - (SR estimate years 0...N-1)]/(SR sampling std dev error 0... N-1)

If I take the square of this, average of all years and then square root I get the normalised root mean squared error. This comes out at 0.998 for all the data above.

The blue line in this plot shows the absolute value of the error term for each year. The orange line shows the RMSE. You can see this gradually declining over time and settling in at around 0.85

Here are the same two plots for a correlation pair estimate:

Again the RMSE tends to end up around 0.86

Incidentally, we can also do these plots for longer periods. Here is the RMSE evolution for a SR estimate looking ahead over the next 5 years:

The RMSE here is a little higher - around 1.0

Now, let's look at some real data. I'm going to use the p&l from trading the US10 year bond with a 16,64 day EWMAC. Let's begin by trying to forecast the SR one year ahead:

Even without calculating the error we can see that there are more boundary breakages than before with random data. Here is the error:

Notice that it is higher than before (around 1.25; or about sqrt(2) times bigger than the random data RMSE) and doesn't slowly converge as it did with random data, instead it stays roughly constant (ignoring the initial period of luck at the start). 

We get a similar picture for 5 years:

What about correlations? Let's look at the correlation between this slow momentum on 10 year US bonds, and the carry rule on the same instrument:

Wow, that's noisy. The RMSE will be off the charts. What about over 5 years?

Ouch. If we look at the correlation between two variations of the same trading rule, EWMAC64,256 and EWMAC32,128 - which are naturally highly correlated - then it's not much better:

Again the RMSE would be in double digits.

Those might be flukes, so let's look at lots of random results. I'm going to pick an instrument and trading rule randomly, and measure it's final RMSE number. I will then generate some random returns of the same length from the same SR distribution (by measuring the full sample SR for the relevant instrument/rule pairing); and measure that's RMSE. I will then select another rule from the same instrument, get the correlation of the two p&l streams, and generate some more random returns with the given expected correlation. Next and finally I will measure the correlation RMSE for the two sets of real returns, and the two sets of random returns.

If I consider the ratio [RMSE real data / RMSE random data] (both for next one year); then the median of this over a few thousand randomly selected trading strategy components is 1.06 for Sharpe Ratios, and for correlations around 5.6. 

In simple terms, we are a little bit worse than forecasting Sharpe Ratios in real data one year ahead than we would be with random data, but a LOT worse with correlations. 

Partly this is because we are pretty terrible at forecasting SR one year ahead anyway even with a stable underlying distribution; we don't do much worse with real data. However it does seem that correlations are far more unstable in reality than in randomly generated data. Note that these are correlations for trading strategy component returns. In some cases they are mathematically related (eg EWMAC of different speeds) and could be derived with some assumptions, a pencil, and a napkin. They are certainly more stable than the returns of the underlying instruments themselves (think about the changing correlation of stocks and bonds in different inflation environments). 

(Note: These numbers are about the same for five years ahead and also ten years ahead)

If we recall from the prior post that the optimal shrinkage is zero on correlations with random data; we can now see why with actual data we'd probably want to opt for some correlation shrinkage; purely because the sampling error is much larger in practice. That is the empirical finding of the EPO paper. It does feel a bit weird since up to now my gut feeling has been that we have to shrink means a lot because they are much harder to forecast and because they have an outsized effect on portfolio weights compared to differences in correlation. Whilst the latter is still true it seems the former is not.

Food for though. Anyway the next step is to repeat the 'Ultimate Fitting Championships' battle, but this time with real data.

 

CTA index replication and the curse of dimensionality

Programming note: 

So, first I should apologise for the LONG.... break between blogposts. This started when I decided not to do my usual annual review of performance - it is a lot of work, and I decided that the effort wasn't worth the value I was getting from it (in the interests of transparency, you can still find my regularly updated futures trading performance here). Since then I have been busy with other projects, but I now find myself with more free time and a big stack of things I want to research and write blog posts on.

Actual content begins here:

To the point then - if you have heard me talking on the TTU podcast you will know that one of my pet subjects for discussion is the thorny idea of replicating - specifically, replicating the performance of a CTA index using a relatively modest basket of futures which is then presented inside something like an ETF or other fund wrapper as an alternative to investing in the CTA index itself (or to be more precise, investing in the constituents because you can't actually invest in an index).

Reasons why this might be a good thing are: 

• that you don't have to pay fat fees to a bunch of CTA managers, just slightly thinner ones to the person providing you with the ETF. 
• potentially lower transaction costs outside of the fee charged
• Much lower minimum investment ticket size
• Less chance of idiosyncratic manager exposure if you were to deal with the ticket size issue by investing in just a subset of managers rather than the full index

How is this black magic achieved? In an abstract way there are three ways we can replicate something using a subset of the instruments that the underyling managers are trading:

• If we know the positions - by finding the subset of positions which most closely matches the joint positions held by the funds in the index. This is how my own dynamic optimisation works, but it's not really practical or possible in this context.
• Using the returns of individual instruments: doing a top down replication where we try and find the basket of  current positions that does the best job of producing those returns.
• If we know the underlying strategies - by doing a bottom up replication where we try and find the basket of strategies that does the best job of producing those returns.

In this post I discuss in more detail some more of my thoughts on replication, and why I think bottom up is superior to top down (with evidence!).

I'd like to acknowledge a couple of key papers which inspired this post, and from which I've liberally stolen:

In Pursuit of Trend-Following Beta: The Promise and Pitfalls of Replication (Newfound research)

Peering Around Corners: How to Replicate Trend Following Managed Futures (Resolve asset management)

Why are we replicating?

You may think I have already answered this; replication allows us to get close to the returns of an index more cheaply and with lower minimum ticket size than if we invested in the underlying managers. But we need to take a step back: why do we want the returns of the <insert name of CTA index> index?

For many institutional allocators of capital the goal is indeed closely matching and yet beating the returns of a (relatively) arbitrary benchmark. In which case replication is probably a good thing.

If on the other hand you want to get exposure to some latent trend following (and carry, and ...) return factors that you believe are profitable and/or diversifying then other options are equally valid, including investing in a selected number of managers, or doing DIY trend following (and carry, and ...). In both cases you will end up with a lower correlation to the index than with replication, but frankly you probably don't care.

And of course for retail investors where direct manager investment (in a single manager, let alone multiple managers) and DIY trend following aren't possible (both requiring $100k or more) then a half decent and chearp ETF that gives you that exposure is the only option. Note such a fund wouldn't neccessarily need to do any replication - it could just consist of a set of simple CTA type strategies run on a limited universe of futures and that's probably just fine. 

(There is another debate about how wide that universe of futures should be, which I have also discussed in recent TTU episodes and for which this article is an interesting viewpoint). 

For now let's assume we care deeply, deeply, about getting the returns of the index and that replication is hence the way to go.

What exactly are we replicating?

In a very abstract way, we think of there being C_0....C_N CTA managers in an index. For example in the SG CTA index there are 20 managers, whilst in the BTOP50 index there are... you can probably guess. No, not 50, it's currently 20. The 50 refers to the fact it's trying to capture at least 50% of the investable universe. 

In theory the managers could be weighted in various ways (AUM, vol, number of Phds in the front office...) but both of these major indices are equally weighted. It doesn't actually matter what the weighting is for our purposes today.

Each manager trades in X underlying assets with returns R_0.....R_X. At any given time they will have positions in each of these assets, P_c_x (so for manager 0, P_0_0.... P_0_X, for manager 1, P_1_0...P_1_X and in total there will be X*N positions at each time interval). Not every manager has to trade every asset, so many of these positions could be persistently zero.

If we sum positions up across managers for each underlying asset, then there will be a 'index level' position in each underlying asset P_0.... P_X. If we knew that position and were able to know instantly when it was changing, we could perfectly track the index ignoring fees and costs. In practice, we're going to do a bit better than the index in terms of performance as we will get some execution cost netting effects (where managers trade against each other we can net those off), and we're not paying fees. 

Note that not paying performance fees on each manager (the 20 part of '2&20') will obviously improve our returns, but it will also lower our correlation with the index. Management fee savings however will just go straight to our bottom line without reducing correlation. There will be additional noise from things like how we invest our spare margin in different currencies, but this should be tiny. All this means that even in the world of perfectly observable positions we will never quite get to a correlation of 1 with the index.

But we do not know those positions! Instead, we can only observe the returns that the index level positions produce. We have to infer what the positions are from the returns. 

The curse of dimensionality and non stationarity, top down version

How can we do this inference? Well we're finance people, so the first thing we would probably reach for is a regression (it doesn't have to be a regression, and no doubt younger people reading this blog would prefer something a bit more modern, but the advantage of a regression is it's very easy to understand it's flaws and problems unlike some black box ML technique and thus illustrate what's going wrong here).

On the left hand side of the regression is the single y variable we are trying to predict - the returns of the index. On the right hand side we have the returns of all the possible instruments we know our managers are trading. This will probably run into the hundreds, but the maximum used for top down replication is typically 50 which should capture the lions share of the positions held. The regressed 'beta' coefficients on each of these returns will be the positions that we're going to hold in each instrument in our replicating portfolio: P_0... P_X. 

Is this regression even possible? Well, as a rule you want to have lots more data points than you do coefficients to estimate. Let's call the ratio between these the Data Ratio. It isn't called that! But it's as good a name as any. There is a rule of thumb that you should have at least 10x the number of variables in data points. I've been unable to find a source for who invented this rule, so let's call it The Rule Of Thumb.

There are over 3800 data points available for the BTOP50 - 14 years of daily returns, so having say 50 coefficients to estimate gives us a ratio of over 70. So we are all good.

Note - We don't estimate an intercept as we want to do this replication without help or hindrance from a systematic return bias.

In fact we are not good at all- we have a very big problem, which is that the correct betas will change every day as the positions held change every day. In theory then that means we will have to estimate 200 variables with just one piece of data - todays daily return. That's a ratio of 0.005x; well below 10!

Note - we may also have the returns for each individual manager in the index, but a moments thought 

will tell you that this is not actually helpful as it just means we will have twenty regressions to do, each with exactly the same dimensionality problem.

We can get round this. One good thing is that these CTAs aren't trading that quickly, so the position weights we should use today are probably pretty similar to yesterdays. So we can use more than one day of returns to estimate the correct current weights. The general approach in top down replication is to use rolling windows in the 20 to 40 day range. 

We now have a ratio of 40 datapoints: 50 coefficients - which is still less than ten.

To solve this problem we must reduce the number of betas we're trying to estimate by reducing the number of instruments in our replacing portfolio. This can be done by picking a set of reasonably liquid and uncorrelated instruments (say 10 or 15) to the point where we can actually estimate enough position weights to somewhat replicate the portfolio. 

However with 40 days of observations we need to have just four instruments to meet our rule of thumb. It would be hard to find a fixed group of four instruments that suffice to do a good job of replicating a trend index that actually has hundreds of instruments underlying it.

To deal withs problem, we can use some fancy econometrics. With regularisation techniques like LASSO or ridge regression; or stepwise regressions, we can reduce the effective number of coefficients we have to estimate. We would effectively be estimating a small number of coefficients, but they would be the coefficients of four different instruments over time (yes this is a hand waving sentence) which give us the best current fit.

Note that there is a clear trade off here between the choice of lookback window, and the number of coefficients estimated (eithier as an explicit fixed market choice, dynamically through stepwise regression, or in an implicit way through regularisation):

• Very short windows will worsen the curse of dimensionality. Longer windows won't be reactive enough to position changes.
• A smaller set of markets means a better fit, and means we can be more reactive to changes in positions held by the underlying markets, but it also means we're going to do a poorer job of replicating the index.

Introducing strategies and return factors

At this point if we were top down replicators, we would get our dataset and start running regressions. But instead we're going to pause and think a bit more deeply. We actually have additional information about our CTA managers - we know they are CTA managers! And we know that they are likely to do stuff like trend following, as well as other things like carry and no doubt lots of other exotic things. 

That information can be used to improve the top down regression. For example, we know that CTA managers probably do vol scaling of positions. Therefore, we can regress against the vol scaled returns of the underlying markets rather than the raw returns. That will have the benefit of making the betas more stable over time, as well as making the Betas comparable and thus more intuitive when interpreting the results.

But we can also use this information to tip the top down idea on it's head. Recall:

Each manager trades in X underlying assets with returns R_0.....R_X. At any given time they will have positions in each of these assets, P_c_x (so for manager 0, P_0_0.... P_0_X, for manager 1, P_1_0...P_1_X so there will be X*N positions at each time interval). 

Now instead we consider the following:

Each manager trades in Y underlying strategies with returns r_0.....r_Y. At any given time they will have weights in each of these strategies, w_c_y (so for manager 0, w_0_0.... w_0_Y, for manager 1, w_1_0...w_1_Y so there will be Y*N positions at each time interval). 

Why is this good? Well because strategy weights, unlike positions, are likely to be much more stable. I barely change my strategy weights. Most CTAs probably do regular refits, but even if they do then the weights they are using now will be very similar to those used a year ago. Instead of a 40 day window, it wouldn't be unreasonable to use a window length that could be measured in years: thousands of days. This considerably improves the curse of dimensionality problem.

Some simple tables

For a given number of X instruments, and a given number of Y strategies, Z for each instrument:

                                Top down              Bottom up

Approx optimal window size      40 days               2000 days

Number of coefficients            X                     X*Z

Data ratio                  40 / X                   2000 / X*Z

Therefore as long as Z is less than 50 the data ratio of the bottom up strategy will be superior. For example, with some real numbers - 20 markets and 5 strategies per market:

                  

                                Top down              Bottom up

Approx optimal window size        40 days               2000 days

Number of coefficients            20                    100

Data ratio                        2                      20

Alternatively, we could calculate the effective number of coefficients we could estimate to get a data ratio of 10 (eithier as a fixed group, or implicit via regularisation):

                                Top down              Bottom up

Approx optimal window size        40 days               2000 days

Data ratio                        10                    10

Number of coefficients            4                     20

It's clear that with bottom up replication we should get a better match as we can smuggle in many more coefficients, regardless of how fancy our replication is.

A very small number of caveats

There are some "but..."'s, and some "hang on a moment's" though. We potentially have a much larger number of strategies than instruments, given that we probably use more than one strategy on each instrument. Two trend following speeds plus one carry strategy is probably a minimum; tripling the number of coefficients we have to estimate. It could be many more times that.

There are ways round this - the same ways we would use to get round the 'too many instruments' problem we had before. And ultimately the benefit from allowing a much longer window length is significantly greater than the increase in potential coefficients from multiple strategies per instrument. Even if we ended up with thousands of potential coefficients, we'd still end up selecting more of them than we would with top down replication.

A perhaps unanswerable 'but...' is that we don't know for sure which strategies are being used by the various managers, whereas we almost certainly know all the possible underlying instruments they are trading. For basic trend following that's not a problem; it doesn't really matter how you do trend following you end up with much the same return stream. But it's problematic for managers doing other things.

A sidebar on latent factors

Now one thing I have noticed in my research is that asset class trends seem to explain most of instrument trend following returns (see my latest book for details). To put it another way, if you trend follow a global equity index you capture much of the p&l from trend following the individual constituents. In a handwaving way, this is an example of a latent return factor. Latent factors are the reason why both top down and bottom up replication work as well as they do so it's worth understanding them.

The idea is that there are these big and unobservable latent factors that drive returns (and risk), and individual market returns are just manifestations of those. So there is the equity return factor for example, and also a bond one. A standard way of working out what these factors are is to do a decomposition of the covariance matrix and find out what the principal components are. The first few PC will often explain most of the returns. The factor loadings are relatively static and slow moving; the S&P 500 is usually going to have a big weight in the equity return factor.

Taking this idea a step further, there could also be 'alternative' return factors; like the trend following factor or carry factor (or back in equity land, value and quality). These have dynamic loadings versus the underyling instruments; sometimes the trend following factor will be long S&P 500 and sometimes short. This dynamic loading is what makes top down replication difficult.

Bottom up regression reverses this process and begins with some known factors; eg the returns from trend following the S&P 500 at some speed with a given moving average crossover, and then tries to work out the loading on those factors for a given asset - in this case the CTA index. 

Note that this also suggests some interesting research ideas such as using factor decomposition to reduce the number of instruments or strategies required to do top down or bottom up replication, but that is for another day. 

If factors didn't exist and all returns were idiosyncratic both types of replication would be harder; the fact they do seem to exist makes replication a lot easier as it reduces the number of coefficients required to do a good job.

Setup of an empirical battle royale

Let's do a face off then of the two methodologies. The key thing here isn't to reproduce the excellent work done by others (see the referenced papers for examples), or neccessarily to find the best possible way of doing eithier kind of replication, but to understand better how the curse of dimensionality affects each of them. 

My choice of index is the BTOP50, purely because daily returns are still available for free download. My set of instruments will be the 102 I used in my recent book 'AFTS' (actually 103, but Eurodollar is no longer trading) which represent a good spread of liquid futures instruments across all the major asset classes. 

I am slightly concerned about using daily returns, because the index snapshot time is likely to be different from the closing futures price times I am using. This could lead to lookahead bias, although that is easily dealt with by introducing a conservative two day lag in betas as others have done. However it could also make the results worse since a systematic mismatch will lower the correlation between the index returns and underyling instrument returns (and thus also the strategy returns in a bottom up replication). To avoid this I also tested a version using two day returns but it did not affect the results.

For the top down replication I will use six different window sizes from 8 business days up to 256 (about a year) with all the powers of 2 in between. These window sizes exceed the range typically used in this application, deliberately because I want to illustrate the tradeoffs involved. For bottom up replication I will use eight window sizes from 32 business days up to 4096 (about sixteen years, although in practice we only have 14 years of data for the BTP50 so this means using all the available data). 

We will do our regressions every day, and then use an exponential smooth on the resulting coefficients with a span equal to twice the window size. For better intuition, a 16 day exponential span such as we would use with an 8 day window size has a halife of around 5.5 days. The maximum smooth I use is a span of 256 days.

For bottom up replication, I will use seven strategies: three trend following EWMA4,16, EWMAC16,64, EWMAC64,256 and a carry strategy (carry60); plus some additional strategies: acceleration32, mrinasset1000, and skewabs180. For details of what these involve, please see AFTS or various blogposts; suffice to say they can be qualitiatively described as fast, medium and slow trend following, carry, acceleration (change in momentum), mean reversion, fast momentum and skew respectively. Note that in the Resolve paper they use 13 strategies for each instrument, but these are all trend following over different speeds and are likely to be highly correlated (which is bad for regression, and also not helpful for replication).

I will use a limited set of 15 instruments, the same as those used in the Newfound paper, which gives me 15*7 = 105 coefficients to estimate - roughly the same as in the top down replication.

I'm going to use my standard continous forecasting method just because that is the code I have to hand; the Resolve paper does various kinds of sensitivity analysis and concludes that both binary and continous produce similar results (with a large enough universe of instruments, it doesn't matter so much exactly how you do the CTA thing). 

Note - it could make sense to force the coefficients on bottom up replication to be positive, however we don't know for sure if a majority of CTAs are using some of these strategies in reverse, in particular the divergent non trend following strategies.

Approx data ratios with different window sizes if all ~100 coefficients estimated:

                               

8 days                            0.08

16 days                           0.16

32 days                           0.32

64 days                           0.64

128 days                          1.28

256 days                          2.56

512 days                          5.12

1024 days                         10.2    

2048 days                         20.5

4096 days                         41.0

In both cases I need a way to reduce the number of regressors on the right hand side from somewhere just over 100 to something more reasonable. This will clearly be very important with an 8 day window!

Various fancy techniques are commonly used for this including LASSO and ridge regression. There is a nice summary of the pros and cons of these in an appendix of the Resolve paper; one implication being that the right technique will depend on whether we are doing bottom up or top down replication. They also talk about elastic net, a technique that combines both of these techniques. For simplicity I use LASSO, as there is only one hyperparameter to fit (penalty size).

Results

Here are the correlation figures for the two methods with different lookback windows:

As you can see, the best lookback for the top down method needs to be quite short to capture changing positions. Since strategy weights are more stable, we can use a longer lookback for the bottom up method. For any reasonable length of lookback the correlation produced by the top down method is pretty stable, and significantly better than the bottom up method.

Footnote: Why not do both?

One of the major contributions of the Resolve paper is the idea of combining both top down and bottom up methods. We can see why this make sense. Although bottom up is superior as it causes less dimensionality issues, it does suffer because there might be some extra 'secret sauce' that our bottom up models don't capture. By including the top down element as well we can possibly fill this gap.

Footnote on 'Creating a CTA from scratch'

You may have seen some bottom up 'replication' articles that don't use any regression, such as this one. They just put together a set of simple strategies with some sensible weights and then do an ex-post cursory check on correlation with the index. The result, without trying, is a daily correlation of 0.6 with the SG CTA index, in line with the best bottom up results above without any of the work or the risks involved with doing potentially unstable regressions on small amounts of data. Indeed, my own trading strategies (monthly) correlation with the SG CTA index was 0.8 last time I checked. I have certainly done no regressions to get that that!

As I mentioned above, if you are a retail investor or an institutional investor who is not obsessed with benchmarking, then this might be the way to go. There is then no limit on the number of markets and strategies you can include.

Conclusion

I guess my conclusion comes back to why... why are we doing this.

If we really want to replicate the index then we should be agnostic about methodology and go with what is best. This will involve mostly bottom up with a longish window for the reasons discussed above, although it can probably be improved by including an averaging with top down.

But if we are trying to get 'exposure to some trend following factors' without caring about the index then I would probably start with the bottom up components of simple strategies on a diversified set of instruments with sensible but dumb 'no-information' weights that probably use some correlation information but not much else (see all the many posts I have done on portfolio optimisation). Basically the 'CTA from scratch' idea.

And then it might make sense to move in the direction of trying to do a bottom up replication of the index if you did decide to reduce your tracking error, though I'd probably use a robust regression to avoid pulling the strategy weights too far from the dumb weights.

Optimising my way out of a small fund problem - part one

This is part one of a series of posts about using optimisation to get the best possible portfolio given a relatively small amount of capital.

In this short post I present the idea, and discuss some issues that I need to resolve. It's a bit of a stream of conciousness! It's less of a blog post, and more my random jottings on the subject converted from scribbles to electronic prose. It's a precursor to further posts where I will start designing and testing the method.

I am sorry for my size

There is a little known book about the City of London in the 80's (The buck stops here), in which there is quite an amusing anecote. The stockbroker - who has recently been fired - goes for a meal / drink with a Japanese client:

"His enzymes had let him down again, and he was a bit drunk, in a benign sort of way 'I am sorry, Mr Parton for my size' he kept on muttering. I caught the stares of a few passers-by and wanted to say to them, this man does not mean what you think he means."

Of course the Japanese fund manager is referring to the size of his fund which is relatively modest (and this is why the broker has been canned in the first place. As a specialist in selling European equities to Japanese investors who prefer to invest domestically, or at a push in the US, he is doomed).

My fund, or rather my trading account, is also relatively modest. It's larger than the average retail account, but by no means the multi billion dollars I used to jockey back in the days when I had a proper job. 

This is.... unfortunate. Why does it matter? Obviously it means fewer bragging rights in Soho wine bars, but that doesn't bother me (especially as at the time of writing, Soho wine bars are outside table service with NHS track and trace enabled only). No, what bothers me is this:

The graph shows the increase in expected Sharpe Ratio as you add instruments to a simple trading strategy consisting of a single moving average crossover (And like a good boy, I've put error bars around the Sharpe Ratio estimates). So with one (randomly chosen) instrument the average SR is around 0.24; but if I add another (randomly chosen) instrument it goes up to around 0.3. And with a few wiggles, the increase continues pretty monotonically. And as those error bars show, the improvement is statistically significant.

(I could do even better especially for the first few assets if I deliberately added instruments that diversified the existing pool at each stage, rather than just randomly choosing).

This graph is striking, especially if I compared it to another graph where I added trading rules but kept the number of instruments constant. There the increase is slower, and also begins to show reduced marginal gains. Here we're still getting fairly steady improvements in performance at the 33 instrument mark. If there is an optimal number of instruments (at which point the marginal improvement became non existent) one could trade it's clearly much more than 33, or even the 37 or so I've traded with (give or take) since 2014.

To make a famous quote more accurate:

Diversification across instruments is the only free lunch in finance.

However it isn't actually a free lunch. Every extra instrument you trade will use up capital (this isn't true for trading rules, at least not the way my system is implemented). This problem is most pressing for futures traders, since you can't trade fractions of a futures contract, and most contracts are very large in dollar risk compared to the average persons trading account.

This means that with less capital you can't trade the 400+ or so instruments traded by AHL and other large CTAs.  Even if we put aside the OTC instruments and cash equities that these funds trade, and just stick to futures, there are something like 70 additional futures markets I don't already trade which are liquid enough, not massive in size, have cheap data, and don't cost too much. But there is no way I could trade over 100 markets with my capital.

And this is a serious problem for retail traders, which is why I wrote a whole book about how to make the best use of scarce capital (the subject is also discussed at length in my first and second books). Diversification across instruments is the main competitive advantage that large funds have.

So I'm stuck with around 37 instruments, and I can only manage that many because of an ugly hack that I wrote about at some length here. 

That ugly hack is worth a brief discussion (though you are welcome to read the post). It relies on the fact that, with some exceptions, a larger forecast (my scaled measure of expected risk adjusted return) implies a larger ex-post risk adjusted return. This is something I analysed in more detail in this more recent post. 

So in the ugly hack I ignore forecasts that are too small, and then scale up forecasts beyond some threshold more aggresively scale up trades (to ensure that the scaling properties of the forecast are unchanged). I have to do this in markets where my modest capital is most pressing: those with relatively large contract sizes. 

The important point here is that larger forecasts are better - hold on to that point.

Optimisation to the rescue

Now any financial quant worth their salt would read what I've just written and say 'Pff! That's just an optimisation problem'.

'Pff?' I'd reply.

'Mais Oui*. All you need to do is take the expected returns and covariance matrix, limit the optimisation weights to discrete values, and press F9**'  

* Thanks to their excellent Grand Ecole system, most quants are French

** Surprisingly large amounts of the financial system, especially on the sell side, run in Excel

'But where do I get the expected returns from?'

'Boff! You already have the, how do you say, forecasts? A higher forecast means a higher expected return, does it not?'

'Yes, but there is no obvious mapping... Also aren't optimisations somewhat.... well not robust?'

'Only if handled by an inexperienced Rosbif like yourself. For a suitable fee I can of course help you out....'

Now I can't afford to pay this imaginary Quant a fee, and of course she is imaginary, so we'll have to come up with a better solution using a methodology that I understand (no doubt much simpler than is taught in the hallowed lecture theatres of the Ecole Polytechnique). And the building block we're going to use is Black-Litterman.

A brief idiots guide to Black-Litterman

 

Well Black and Litterman are of course the legendary (and sadly missed) Fischer Black of BSM and BDT; and GSAM legend Bob Litterman. And their model deals with the problem I highlighted above 'But where do I get the expected returns from?'

And the answer is you get them from an inverse portfolio optimisation. You start with a portfolio of weights (let's put aside for the moment the question of where they come from). Then you estimate a covariance matrix. Then you run the classical Markowitz optimisation (find the optimal weights given a vector of expected returns and a covariance matrix, and some risk tolerance or utility function) in reverse so it becomes find the expected returns given a vector of weights and a covariance matrix.

BL (as I will say henceforth) used the market portfolio for their starting weights, and hence the resulting implied returns are the 'equilibrium returns'; the returns that are expected given that the 'average' (in a cumulative sense) investor must hold the market portfolio by construction.

Once you have your expected returns you can combine them with some forecasted returns. Perhaps you want to include the discretionary opinion of your chief economist. Or maybe you've got some kind of systematic model for forecasting returns. In any case you take a weighted average of the original equilibrium returns and your forecasts (so this is Bayesian in character as we shrink our forecasts towards the equilibrium returns). Now with your new vector of expected returns you run the normal optimisation forward; using the same covariance matrix you derive a new set of optimal weights.

(The full paper is here)

BL portfolios have some nice properties. If you make no changes at all to the expected returns then you'll recover the original weights (this is a good way to check your code is working!). If you replace them completely, you'll basically have the portfolio implied by your forecasts (which will usually be not very robust at all, with the usual extreme weights problem highlighted). But a blend of the two sets of expected returns, if weighted mostly towards the equlibrium returns, will produce robust portfolios that are tilted away from the market cap weights to reflect our forecasts.

I'm a fan of BL because it accounts, to an extent, for the hierarchy of inputs to a portfolio optimisation. Expected returns are the hardest to forecast, and small changes have a big effect on the output. Standard deviations are relatively easy to forecast, and small changes have a small effect on the output. Correlations fall somewhere in the middle. BL effectively assumes we can predict standard deviations and correlations perfectly, but doesn't make the same assumption about expected returns.

But I don't actually use BL for optimisation, mainly because in the kind of problem I'm usually dealing with (eg deciding how to linearly weight a variety of trading rules and instruments) as it isn't obvious what the 'market cap portfolio' should be. And I'm not going to use it for it's intended purpose here eithier.

The brilliant idea

We can use the BL methodology to do something rather cool and interesting, and fun (and completely different from the original intent). We can run the backward optimisation, and then the forward, without making any changes to the expected returns. Instead we make some other change to the optimisation. Most commonly this would be the introduction of constraints; like a limit on Emerging market exposure, or a position size limit, or ... and this is relevant.... a discrete position size constraint.

So the plan looks something like this:

• Run my standard position generation function, which will produce a vector of desired contract positions across instruments, all of which will be non integer. Let's call this the 'original' portfolio weights. The main inputs into this calculation are the forecast, instrument weight (as a proportion of risk capital allocated), current volatility of the instrument, long run target volatility and the instrument diversification multiplier (see here, and search for 'why does expected risk vary')
• Estimate a covariance matrix Σ and a risk aversion coefficient λ
• Using a reverse Markowitz, BL style, calculate the implied expected returns for each instrument, µ. There is a closed form for the reverse optimisation, since this doesn't have constraints: λΣw
• Run the optimisation forward using µ, Σ, λ, with a constraint that only integer contract positions can be taken.

Intuitively the sort of thing this process would do is to trade more of instruments with smaller contract size, if they are posiitvely correlated with instruments that are too big to trade. So it's going to be superior to something that just gives you the rounded version of the optimal portfolio (like for example minimising Euclidian distance); which if you have enough instruments and insufficient capital is going to be a vector of zero weights.

The brilliant idea is harder than it first sounds: some small problems

Now there are a lot of unanswered questions here. I've spent a long time thinking about this idea (over 18 months); and it's actually much more complicated than it might first seem.

For the discrete optimisation we're probably going to want to use some kind of grid search. That's going to be slow, especially if I end up with my 'dream' portfolio of 100+ instruments.

In fact it's worse than that, because a great feature of this approach is we can calculate forecasts for instruments we have no intention of trading (because they aren't sufficiently liquid, or are too expensive) as well as instruments that we'd like to trade but the contract size is inordinately large so we can't. And then we can use their forecasts to inform us what our overall portfolio should look like once we apply the discrete constraints; for example the (way too large) Ethereum contract could give me useful information about how to trade the micro Bitcoin future. 

In fact my full wish list currently stands at a total of 228 instruments. Anything we can do to reduce the area that has to be searched would be good! For example, I'd be reluctant to put more than 10% of my risk capital in a single instrument. That sets an upper and lower limit on position size.

I'd also be unhappy changing the sign of a position as a result of an optimisation. I don't want to end up with weird spreading behaviour, just because two instruments are negatively correlated doesn't mean I want to go long/short if say both forecasts are positive. So the lower limit would be zero for a long optimal position, and the upper limit would be zero for a short optimal.

It would probably make a lot of sense to do some kind of coarse to fine search, but I'll discuss specific options for that later. 

It's possible that contracts will move in and out of the 'tradeable / not tradeable' region over time, and rather than adding/removing them manually it would be better to allow an optimisation to do this. There would need to be a list of instruments in a state of 'reduce only', for which the maximum would be the current position (if long, the minimum if short). This list would be updated automatically for instruments that fell below or suddenly qualified for my required criteria for volume and costs.  There would no need to eliminate instruments that were 'too big to trade'; this would happen naturally if 10% of risk capital wasn't sufficient to take even a single contract of position.

It's plausible that there could also be instruments that we couldn't trade at all - eithier permanently or temporarily. For these the maximum and minimum would be equal to the current position. For example, it might be that I get end of day data from one source, for a market I can't afford to get live L1 data for to trade with.

Notice that for these last few points the optimisation would need to have knowledge of the current positions held by the system. In production this means it would make most sense in the pysystemtrade layer that generates 'instrument orders', which sits between the strategy optimal position generation and the execution layer.

In my current trading system this layer currently implements a buffering algo to reduce turnover. It would make sense to replace this with an optimisation that considered explicit costs in it's calculation. It's trivial to calculate the expected cost per contract to do a given trade, assuming you have expected slippage and commision data (which I have). An open question is wether those costs should be amortised over the required holding period rather than assume we're optimising until the next optimisation (in 24 hours presumably), or whether a multiplier should be applied to reflect that costs are more certain than returns (for example, I apply a multiplier of 2 in my normal optimisation of forecast and instrument weights).

Something I have skated over is the fact that my initial strategy will produce desired weights in contract units, and the final optimisation also needs to know about discrete contract units, but 'w' is expressed as a notional position size as a proportion of capital (costs per contract would be in £,$,... units, but one can easily convert that to be a proportion of capital). So I'd need to work out what a single contract was in units of w when determining what the possible discrete step sizes were for each instrument.

Finally one can imagine extending this further; for example by introducing margin requirements into the optimisation.

And some big problems

All of the above problems are mostly just <vaguely waves hand> engineering. I know what needs to be done, it's just a matter of coding it up.

A more difficult question lies around the coefficient of risk aversion, λ. I'm not used to thinking in terms of that at all. However in theory it won't actually matter what λ is set to, as long as we use the same λ in both the reverse and forward optimisation (trivially, so that it is consistent with the closed form of the initial reverse optimisation the form of the forward optimisation must be to maximise max w'µ − λ w'Σw/ 2 rather than the more modern version where we specify a maximum risk and solve for highest return). That should naturally result in a portfolio which has about the same amount of risk as the original. Which is important, because there is information in the amount of risk that the original strategy positions want to take. 

[Note that I could still impose a maximum risk constraint (at say twice my expected annualised target risk of 25% a year); this would replace part of my exogenous risk overlay which effectively fulfills the same function.]

I've left the hardest problem until last, and this is 'What covariance matrix should we use'? Remember a covariance is just the offspring of a correlation matrix and a standard deviation that love each other very much. 

Well the easy part is the standard deviations; I'll just use estimates of percentage annualised risk for each instrument (since we're dealing in w units as a proportion of capital, % risk is the most appropriate). And this seems as good a time as any to introduce a blended estimate of volatility (as discussed here, which will also make another part of my exogenous risk overlay redundant, since it includes a mean reverting component). But what about the correlation?

Should we use the correlation of instrument returns, or should we use the correlation of trading subsystem returns, which after all is what was used (although not directly) to calculate the instrument weights? And which of these should we use in our initial ('reverse') and second ('forward') optimisations?

Let's look at an example. Suppose that we have a 50% instrument weight in SP500, 25% each in US2 and US5 (because the trading subsystems for the two bonds are highly correlated, and historically they've been relatively uncorrelated with SP500), and also suppose those weights are a result of doing a naive markowitz optimisation with some specific correlation matrix of trading subsystem returns (not true in practice, but we'll come to that).

And suppose also that we have equal positive forecasts in all three assets (we we expect the same risk adjusted return). We'll have long positions, but with a larger long position in SP500 than in the other two assets (ignoring the effect of risk; in practice we'd have apply risk scaling to these positions).

What will the implied expected returns look like for these assets after we do the initial reverse optimisation? 

Well if we use the correlation of trading subsystem returns, then in theory we'd end up with expected returns that were equal (actually risk adjusted returns that were equal, but we're ignoring risk and focusing on correlation for now). Which is all fine and correct - since the forecasts are equal.

Let's also suppose however that right now the current correlation of the instrument returns of US2, US5 and SP500 are all equal and positive (so the world has changed, and stocks and bonds are now highly correlated). Then if were to use this correlation matrix in the initial forward optimisation then our implied expected returns would be higher for SP500 than it is for US2 and US10 year (ignoring risk again). This doesn't seem right.

Now what happens if we run the forward optimisation with each of the two matrices. The better option, for me, is to use the current correlation of instrument returns. This deals with the problem I highligted here. If we were to use the long run matrix of subsystem returns we wouldn't be taking into account the change in risk conditions (stocks becoming more correlated with bonds), which is arguably a major flaw of the type of trading system I like to use (forecasts developed independently, and expected risk does not take changes into correlation into account). 

We have four cases:

Reverse / Forward optimisation: which correlation matrix used

A: Subsystem correlation / Subsystem correlation

Implied expected returns will be correct (see above). Final positions will take no account of the fact that stocks are now more correlated with bonds. Using identical matrices will result in consistency and more intuitive results. 

B: Current instrument correlation / current instrument correlation

Implied expected returns will be wrong (see above). Final positions will take account of changes in stock and bond correlations. Using identical matrices will result in consistency and more intuitive results. 

C: Current instrument correlation / Subsystem correlation

Implied expected returns will be wrong (see above).  Final positions will take no account of the fact that stocks are now more correlated with bonds.Using different matrices will result in less intuitive results, may not result in robust portfolios, and could result in unhelpful effects around expected risk targeting.

D: Subsystem correlation / Current instrument correlation

Implied expected returns will be correct (see above). Final positions will take account of changes in stock and bond correlations. Using different matrices will result in less intuitive results, may not result in robust portfolios, and could result in unhelpful effects around expected risk targeting.

We can discount option C right away; it really is the worst of all worlds. 

I don't like option B, since it will result in the 'wrong' expected returns, but perhaps that doesn't actually matter as much as I think it should in practice. As it's using current correlations, it will be more adaptive to different risk conditions. And as it's the same matrix in both optimisations, the BL machinery will work as expected.

Option A will also work, but it won't give us the nice property of giving us a more holistic and dynamic adaptation to portfolio risk. It will be much more like the existing system in character.  

I'm intrigued by option D. In some ways it's the best of both worlds. If it works, then in the example it would have the correct identical expected returns, but then allocate away from SP500 due to it's (temporarily) higher correlation with the other two assets. It gives us a nice holistic and dynamic adaptation. However I'm worried that using two different correlation matrices will make the thing rather weird. It strikes me as likely that the character of the resulting portfolio could be very different from the original, even if we prevent the signs of positions changing.

Also, will it produce the same amount of required risk if I use the same coefficient of risk aversion? Or do I need to calculate the required target risk (for example by scaling the long run risk target I use, 25%, by the aggregate strength of forecasts) and then run the forward optimisation using a maximum standard deviation rather than a coefficient of risk aversion?

There is a technical issue with both options A and D as the correct correlation matrix of subsystem returns (that will result in the expected returns being implied as 'correct' i.e. proportional to forecasts) won't be the same as one you just estimate, because the instrument weights aren't just naively derived from a given correlation matrix; they are robustly optimised. Perhaps that doesn't matter so much for option A since it's the same correlation matrix in both cases (to an extent the correlation matrix is arbitrary). For option D however all the benefit of recovering the correct expected returns will be lost if we don't have the 'right' matrix on the initial optimisation.

I think I have to dismiss option D on the grounds of complexity.

It comes down then to a fight between using the long run correlation of subsystem returns for both forward and reverse optimisation (option A), and using the current correlation of instrument returns for both (option B).  

I'll need to test both of these options to get a feel for how well they work, and whether they have the properties I expect and want.

Some thoughts on testing

I'd be surprised if I was able to run a full backtest with 228 instruments doing a daily optimisation for 40 odd years of data without my laptop committing digital suicide. Instead I'm going to work with a smaller universe of instruments to test what is going on. As well as checking the optimisation does things that make sense, I'm interested in the tracking error between the p&l that would be possible with a large amount of capital versus what the system can actually produce through the optimisation, and whether the expected risk is broadly similar for the original and optimised portfolio.

What next

In the next post I go ahead and test this crazy idea out.