I got more than 99 instruments in my portfolio but butter ain't one of them - another take on instrument diversification

As those of you who follow me on the Elon Musk Daily News App will know, I received physical copies of my new book last week (exciting!). Global supply chains being what they are, you lot will have to wait until April to get your copies. Sorry.

Anyway one of the themes I touch on in the book is the truely amazing diversification that trend following strategies offer when run across multiple instruments. 

I have touched on this before (although never in detail with a complete blogpost on the subect), but I thought I would present some updated results using the data set I use in the book (102 instruments), presented with some different plots to help better grasp the intuition of what's going on. This isn't a sneak preview of stuff in the book, you'll still have to buy it :-), but goes into more detail to help understand just how amazing this effect. 

Incidentally I don't trade butter because it isn't liquid enough; less than 20 contracts a day. Maybe if I left it out of the fridge in hot weather....

(Code is here. These results are not exactly the same as in my latest book, as I'm using a different methodology)

Measuring diversification

The most intuitive way to measure diversification is a simple ratio: expected return for many things divided by the expected return for one thing.

But there is another way of measuring diversification through 'independent bets', which I was reminded of when I reread this 2018 article by the always excellent Cory Hoffstein, and also this article by Resolve (Adam Butler et al). 

Cory's article talks about the three axis of diversification: what, how and when. That article talks a great deal about 'how': diversification across trading rules in my parlance. This blogpost and the Resolve post consider 'what': diversification across instruments. You can also consider 'when': broadly speaking, diversifying across different forecast horizons or speeds of trading rules.

The maths behind the independent bet idea is simple; if you have a portfolio of N return streams (eg trading strategies running on different instruments), scaled to the same volatility (normal for trading strategies) and they are uncorrelated, then your standard deviation will reduce by sqrt(N) versus the risk of one strategy. Assuming they have the same expected mean, your Sharpe Ratio will therefore increase by sqrt(N). If you can use leverage (like a futures trader), then you can purchase higher returns with that rise in Sharpe Ratio.

Most portfolios aren't uncorrelated, so we can bring in the idea of effective independent bets. That's just the value of N with uncorrelated assets that would give you the same reduction in standard deviation that you actually get in practice. For example, if your correlation is 0.5 with two assets equally weighted, then your SR will increase by 1.15. That is the same improvement that we would get from having 1.15^2 = 1.33 independent bets. 

Similarly, if we have three assets with an average correlation of 0.5, then the SR will improve by 1.225 which is the same as having 1.5 independent bets.

A couple of wrinkles: The average correlation is a nice guide, but in the actual calculation you should use the entire correlation matrix. Also, the simple independent bets calculation assumes that all your assets are equally weighting in the zero correlation version. But in calculating the standard deviation for the uncorrelated portfolio you should use the actual weights. 

For example, with identical off diagonal correlations of 0.3 and equal weights, there are 1.88 independent bets. But consider this piece of pysystemtrade code:

from sysquant.estimators.diversification_multipliers import *
corr = correlationEstimate(np.array([[1,0.5, .3], [.5,1, .1], [.3,.1,1]]), columns=['a', 'b','c'])

     a    b    c
a  1.0  0.5  0.3
b  0.5  1.0  0.1
c  0.3  0.1  1.0

corr.average_corr()
0.3

weights = portfolioWeights(dict(a=.45, b=.45, c=.1))

## independent bets
diversification_mult_single_period(corrmatrix=corr, weights=weights)**3
1.53

By allocating less to the more diversifying asset we have reduced the available diversification.

However in this post I will always optimise instrument weights to minimise the portfolio standard deviation using only the correlation matrix as an input, hence maximising the available diversification.

Now, there two ways to calculate diversification, eithier as a ratio of standard deviations (Sharpe Ratios) or as a number of independent bets. The first is to use the correlation matrix of returns, and weights, as I've already done. We can think of this as expected diversification benefit. In fact I do this already in my code - it's called the instrument diversification multiplier (IDM). So if take IDM^2 then that will be equal to the expected benefit, measured in units of independent bets.

The second way of measuring diversification is by looking at the realised Sharpe Ratio of the portfolio, and comparing that to the Sharpe Ratio for a single instrument. We can think of that as the realised diversification benefit. And if I square that ratio, I will again get the realised benefit measured in independent bets.

There is a third way, which is to measure the realised standard deviation of the portfolio having not applied any diversification multiplier. But assuming we are pretty good at predicting risk - and we are - this will be pretty similar to the IDM figure. 

So in summary we can measure:

• Expected diversification ratio (equivalent to IDM)
• Expected number of independent bets (IDM^2)
• Realised diversification ratio (SR ratio)
• Realised diversification as a number of independent bets

Which markets?

There is a big problem here, which is that we're comparing the diversification of one instrument with say two instruments. But which 'one instrument'? And which two instruments? We'll get different answers if we compare a portfolio of Corn and US 10 year bonds versus a portfolio with just Eurodollar, or a portfolio of Dry milk and Crude oil, versus just US 2 year.

That isn't so much of a problem with correlations, but for Sharpe Ratio which exhibits a lot of variation across instruments, it's a big problem.

The answer, is to Monte Carlo this. For each portfolio size N that we want to test we run a bunch of different random runs. Each run involves picking a random group of N markets, allocate instrument weights, and measure the resulting portfolio IDM (expected diversification) and Sharpe Ratio. Then we take the average (median) to get the values for a given N.

Incidentally I calculated instrument weights in two ways to check the robustness of my results:

• Optimised using using correlation as an input into the optimisation, with flooring at zero and some shrinkage for stability.
• Equal weights. I have some heterogeneity in my instruments, eg a lot of equities, so this may not be appropriate.

Long only

Let's first consider a long only portfolio. This will be identical to the trend following portfolio except it uses a constant forecast (the 'asset allocating investor in my first book').

Interestingly, the average correlation of instrument returns across the entire portfolio of 102 instruments is 0.224 (effective number of independent bets, 4.3), but the correlation of subsystem returns (basically trading each instrument, sizing positions for current volatility) is just 0.09 (number of independent bets 6.3). 

First of all let's look at the expected diversification, measured by IDM, with optimised weights. Just to give you a flavour of the monte carlo, the following plot shows the individual data points for each monte carlo, plus a median line. Notice the noise falls as we get more instruments; that's because the portfolios get more similar. At the limit when we're choosing 100 out of 103 instruments, the results are going to be fairly similar.

At the top end the IDM is around 4.8, which means we've got 4.8^2 = 23 independent bets.

Note: I am optimising weights in such a way that will maximise the IDM, hence the IDM's here are higher than in the book for both long only and trend following strategies.

This sounds pretty amazing, but what about the Sharpe ratio, realised diversification? To get something with a comparable y-axis we need to divide the Sharpe Ratios by the median SR for a single instrument. Here is what we get:

I've truncated this plot to make things clearer - there is a lot of noise for smaller N as you'd expect.

But the bottom line is that is rather disappointing. We only get a doubling of SR, equivalent to 4 independent bets. That's a lot less than the IDM would suggest, but then our portfolio weights have effectively been chosen to maximise the IDM (in sample!). But it's also a little short of the 6.3 independent bets that the average correlation would suggest. In fact, it's actually closer to what we'd expect from the average correlation of the raw instrument returns (0.22) before vol scaling. Vol scaling doesn't really add diversification, it just makes the relationship between returns more non linear, and hence reduces correlation.

What difference will equal rather than optimised weights make? Check out the following plot for IDM first, with just the median lines:

The blue line we've already seen, and is for optimised weights. The orange line is with equal weights. This orange is more like the classic 'diversification' graph that you see everywhere, encouraging you to believe that there is no point owning more than X stocks, where here X is perhaps 40. With equal weights the IDM tops out at ~3, or 9 independent bets.

Here are the same two lines, but this time for SR (relative to the median SR of a single instrument portfolio):

Equal weights actually comes out slightly better here, but we're still looking at only around 4.3 independent bets.

The bottom line is that the long only portfolio over promises (high IDM, between 9 and 24 independent bets depending on the weighting scheme), and under delivers (twofold SR improvement, for around 4 to 4.3 independent bets).

Trend following

Now let's consider the same setup, but for a trend following strategy. For simplicity I just used a single trading rule, the EWMAC16,64 rule AKA the starter system from Leveraged Trading. But the results will be similar for a portfolio of such rules. First of all let's plot the IDM, for both optimised (blue) and equal weights (orange).

Pretty similar to before; if anything the diversification benefit available is a little lower than for long only strategies, despite similar average correlation of around 0.08. Using the more optimistic blue line with optimised weights it looks like our expected diversification is of the order of 4.2, equivalent to 18 independent bets.

Now for SR, again blue is with optimised weights and orange is equal weights:

For those crying 'in sample fitting' with respect to the optimised weights on the blue line, note that (a) my optimisation was purely on the correlation matrix, not the SR of the portfolio; (b) correlations are relatively stable over time for trading subsystems, and hence (c) I have got these sorts of improvements in SR for 'proper' backtests using rolling out of sample instrument weights. So I am pretty confident that the blue line is achievable. 

Looking at the blue line then, it's pretty darn good. Rather than the twofold SR improvement seen earlier, we have a relative improvement of over 4.5 times. This is actually a little higher than the improvement promised by the IDM, and hence over 20 independent bets. Even the subpar equal weight version with the orange line manages a threefold improvement, for a score of 9 independent bets.

Another important point is that it looks like the blue line here is still going, suggesting adding even more instruments could potentially boost SR further. In contrast the long only portfolio asymptotes at around 50 instruments, with further diversification not adding anything meaningful.

Summary

Whenever I have plotted these graphs before it has always been to make the point that instrument diversification is the best kind. Adding different kinds of trading rule, or trading at different speeds, or both, will boost your performance - but not by very much. You will be lucky to get two independent bets worth of diversification from this source. In contrast, we can get realised performance equivalent to over twenty independent bets from diversifying across a trend following portfolio of 100 instruments; versus just over four in a long only setup (and it would be even lower without volatility targeting!).

Unfortunately adding instruments requires more capital or doing something smart like my own dynamic optimisation technique. Diversification is a free lunch, but you still need to get to the buffet... 

Equities, Bonds and maximising CAGR

Lots of things have changed in the last year. Many unthinkable things are now thinkable. A war in Europe. The UK coming 2nd in the Eurovision song contest rather than the usual dismal 'null points'. And of course, the correlation of stocks and bonds has recently gone more positive than it has been for over 20 years:

Rolling 12 month correlation of weekly returns for S&P 500 equity and US 10 year bond futures

I thought it would be 'fun' to see how the optimal stock/bond portfolio is affected by correlation and expected return assumptions.

In my second book, Smart Portfolios, I noted that a 100% equity portfolio made no sense under the Kelly criteria (AKA maximising CAGR), and that pretty much everyone should have some exposure to bonds regardless of their risk tolerance, even though they will have a lower expected arithmetic return due to their lower risk. For a while my own strategic risk weighting has been 10% in bonds, equating to cash weights of around 80/20.

I am currently reviewing my long only portfolio and it seems as good a time as any to check that 80/20 still makes sense.

Simple python code is liberally scattered throughout.

Assumptions and base case

I'm assuming a two asset, fully invested portfolio with two assets: a global stock, and a global bond (including both government and corporates). Both assets are Gaussian normal and have a linear relationship, so I can use an approximation for geometric return.

I assume that the standard deviation of the stocks is around 20% and the bonds around 10% (it's gone up recently, can't think why). Furthermore, I assume that my central case for the expected return in stocks is around 8%, and 5% in bonds. That corresponds to a simple SR (without risk free rate) of 0.4 and 0.5 respectively; eg a SR advantage for bonds versus the average SR of 0.05.

(Real return expectations are taken from AQR plus an assumed 3% inflation)

My utility function is to maximise real CAGR, which in itself implies I will be fully invested. Note that means I will be at 'full Kelly' - something that isn't usually advised. However we're determining allocations here, not leverage, so it's probably not as dangerous as you might think.

import numpy as np
import pandas as pd


def calculate_cagr(
    correlation, equity_weight, mean_eq, mean_bo, stdev_eq=0.2, stdev_bo=0.1
):
    bond_weight = 1 - equity_weight
    mean_return = (equity_weight * mean_eq) + (bond_weight * mean_bo)
    variance = (
        ((equity_weight**2) * (stdev_eq**2))
        + ((bond_weight**2) * (stdev_bo**2))
        + 2 * bond_weight * equity_weight * stdev_bo * stdev_eq * correlation
    )

    approx_cagr = mean_return - 0.5 * variance

    return approx_cagr

Effect of correlation varying with base case assumptions

I'm going to vary the correlation between stocks and bonds, between -0.8 and +0.8

list_of_weight_indices = list(np.arange(0, 1, 0.001))

def iterate_cagr(correlation, mean_eq, mean_bo):
    cagr_list = [
        calculate_cagr(
            correlation=correlation,
            equity_weight=equity_weight,
            mean_bo=mean_bo,
            mean_eq=mean_eq,
        )
        for equity_weight in list_of_weight_indices
    ]
    return cagr_list


corr_list = list(np.arange(-0.8, 0.8, 0.1))
corr_list = [round(x, 1) for x in corr_list]
## plot correlation varying
results = dict(
    [(correlation, iterate_cagr(correlation, 0.08, 0.05)) for correlation in corr_list]
)

results = pd.DataFrame(results)
results.columns = corr_list

results.index = list_of_weight_indices

results.plot()

So each line on this plot is a different correlation level. The x-axis is the cash weight on equities, and the y-axis is the geometric return / CAGR. You can see that as correlations get less negative and then positive, we get less diversification from bonds, and a higher weight to equities.

Let's look at the maximum CAGR in each case;

def weight_with_max_cagr(correlation, mean_eq, mean_bo):
    cagr_list = iterate_cagr(correlation, mean_eq, mean_bo)
    max_cagr = np.max(cagr_list)
    index_of_max = cagr_list.index(max_cagr)
    wt_of_max = list_of_weight_indices[index_of_max]

    return wt_of_max


results = pd.Series(
    [weight_with_max_cagr(correlation, 0.08, 0.05) for correlation in corr_list],
    index=corr_list,
)

On the x-axis is the correlation, and on the y-axis is the weight to equities which maximises the CAGR. Remember, these are cash weights. You can see that with zero correlations my original cash weight of 80% in equities is about right. But if correlations go above around 0.4 there is no point owning any bonds at all.

Effect of SR varying 

Now let's see what happens when we tweak the relative SR. I'm going to vary the relative simple Sharpe Ratio (return/standard deviation) between -0.5 and +0.5, keeping the average at 0.45 (positive numbers mean that equities are better). Note that the base case abve is equivalent to a differential of -0.05, in favour of bonds. To begin with, let's keep the correlation fixed at zero. 

def means_from_sr_diff(sr_diff, avg_sr=0.45, stdev_eq=0.2, stdev_bo=0.1):
    ## higher sr_diff is better for equities
    sr_eq = avg_sr + sr_diff
    sr_bo = avg_sr - sr_diff

    mean_eq = sr_eq * stdev_eq
    mean_bo = sr_bo * stdev_bo

    return mean_eq, mean_bo


def weight_with_max_cagr_given_sr_diff(correlation, sr_diff):
    mean_eq, mean_bo = means_from_sr_diff(sr_diff)
    return weight_with_max_cagr(correlation, mean_eq, mean_bo)


# fix corr at zero
sr_diff_list = list(np.arange(-0.5, 0.5, 0.01))
results = pd.Series(
    [weight_with_max_cagr_given_sr_diff(0, sr_diff) for sr_diff in sr_diff_list],
    index=sr_diff_list,
)

Just because CAGR isn't the mean return of standard mean variance optimisation, doesn't mean it won't suffer from the same problem of massive sensitivity to small differences in means (and Sharpe Ratios)! We wouldn't allocate *anything* to equities if the SR difference went below -0.2 (and which point the mean returns are 5% in equities and 6.5% in bonds), or anything to bonds if it's above -0.02 (8.7% in equities and 4.7% in bonds).  

Effect of SR and correlations varying 

sr_diff_list = list(np.arange(-0.25, 0.0501, 0.05))
sr_diff_list = [sr_diff.round(2) for sr_diff in sr_diff_list]
results = pd.DataFrame(
    dict(
        [
            (
                correlation,
                [
                    weight_with_max_cagr_given_sr_diff(correlation, sr_diff)
                    for sr_diff in sr_diff_list
                ],
            )
            for correlation in corr_list
        ]
    )
)

results.index = sr_diff_list
results.columns = corr_list

results = results.transpose()

results.plot()

Here again the x-axis is correlation, and the y-axis shows the weight to equities that maximises CAGR. 

Each of the lines on this plot is a different SR difference. The blue line has a SR advantage of 0.25 to bonds (label -0.25), and the light purple line (lillac?) is a small 0.05 SR advantage to equities. The brown line has no advantage to eithier asset class (misleadingly labelled -0.00 SR). The purple line is a -0.05SR advantage to bonds, which is equal to the base case I was using above - hence you can see the purple line matches the earlier plot of optimal weight versus correlation.

Notice that a SR advantage to bonds, SR difference = -0.1 (red line) results in 50% weights, irrespective of correlation. The lines above it, with a weaker advantage to bonds, put more in equities as correlations become more positive. The lillac line, SR difference 0.05, is 100% in equities, irrespective of correlations. The lines below the red line put less in equities as correlations become more positive.

Sensitivity

My usual base case is that expected SR differences between asset classes are zero, which implies I am somewhere on the brown line. Unless correlations are going to be somewhat negative, this implies 100% in equities. But the AQR base case figures for SR differences allow much more headroom for correlations. Even with correlations at the average 2022 level of around 0.30, one should still have a 10% cash weight in bonds.

How sensitive are my likely CAGR for the following 3 portfolios: 80% in equities, 90% and 100% in equities; over different assumptions of correlation and SR differential?

results = []
for correlation in [-0.4, 0, 0.4]:
    for sr_diff in [-0.25, 0, 0.25]:
        cagr80 = cagr_with_sr_diff(0.8, correlation, sr_diff)
        cagr90 = cagr_with_sr_diff(0.9, correlation, sr_diff)
        cagr100 = cagr_with_sr_diff(1, correlation, sr_diff)
        loss80 = cagr100 - cagr80
        loss90 = cagr100 - cagr90
        results.append(
            dict(
                correlation=correlation,
                sr_diff=sr_diff,
                cagr100=round(cagr100 * 100, 1),
                cagr90=round(cagr90 * 100, 1),
                cagr80=round(cagr80 * 100, 1),
                loss80=round(loss80 * 100, 2),
                loss90=round(loss90 * 100, 2),
            )
        )

print(pd.DataFrame(results))

  correlation  sr_diff  cagr100  cagr90  cagr80  loss80  loss90
0         -0.4    -0.25      2.0     2.7     3.4   -1.43   -0.75
1         -0.4     0.00      7.0     7.0     6.9    0.07    0.00
2         -0.4     0.25     12.0    11.2    10.4    1.57    0.75
3          0.0    -0.25      2.0     2.7     3.3   -1.30   -0.68
4          0.0     0.00      7.0     6.9     6.8    0.20    0.07
5          0.0     0.25     12.0    11.2    10.3    1.70    0.82
6          0.4    -0.25      2.0     2.6     3.2   -1.17   -0.60
7          0.4     0.00      7.0     6.9     6.7    0.33    0.15
8          0.4     0.25     12.0    11.1    10.2    1.83    0.90

For different scenarios of correlation and SR differential (positive better for equities remember), we can see the expected CAGR for portfolios with 100%, 90% and 80% in equities. 10.0 means a CAGR of 10% a year. The final two columns show the loss for an 80% vs 100%, and 90% vs 100% portfolio. Positive numbers mean the lower % of equities is worse, negative means they have outperformed all equities. Eg 1.00 means that the relevant portfolio has a 1% lower CAGR than 100% equities would.

Conclusion

100% in equities is never going to wash for me. Even if correlations have risen, and even with equal SR differentials, I'd be uncomfortable running that. But my current strategic cash weighting of 80% in equities also feels a little low given that correlations are elevated; even if I buy the AQR differential in favour of bonds.

On balance I've decided to increase my strategic cash allocation to ~87% in equities, which corresponds to a risk weighting of 93%.

ESG - Extremely Serious Goalpost moving: Rob goes green

Just a quick one:

I'm moving the goalposts on my example long only portfolio using UK listed ETFs (original blog post is here), done in the spirit of my second book Smart Portfolios: 

I've replaced all the ETFs in that portfolio with ESG funds

This is something I've wanted to do for a while, but the availability of ESG funds has really exploded recently and now the coverage is good enough that I think it's realistic to run an entire portfolio with just ESG.

The criteria is similar to before: low TER, reasonable AUM and ideally a distributing fund; although I've had to be a bit more flexible as the choice obviously still isn't as good. 

My ESG criteria was simple: I used the ESG criteria checkbox in justetf.com; I am not going to get into an argument about good versus bad ESG as I'm not an expert on that subject. My logic is that any ESG fund is probably better for the environment than an average non ESG fund, even if on balance there are going to be varying degrees of ESG fund. The portfolio reflects what an average investor can achieve without doing vast amounts of research, or investing directly in the underlying stocks (which again, will require significant research).

The only fund I excluded on ESG grounds was LGGG, which is very ESG-lite and only seems to exclude companies that actively murder people (one of it's largest holdings is Exxon!). Inevitably there are a few categories where I just couldn't find a suitable ESG fund. I also added a new bond category, bond issues by multilateral institutions, which matches the fund MDBU.

Rather pleasingly, and much to my surprise, the like for like matched simple average of TER across all the various ETF is virtually unchanged from before: 0.23%, just 1bp higher. This may just reflect fee pressure in the industry generally - I haven't updated the fees of the original portfolio for a few years and they have probably come down a bit, or cheaper funds may be available that weren't around originally.

I've created a new spreadsheet with the new tickers and portfolio in; the old spreadsheet link is still around but won't be updated. 

The plan is also to move my own investments into these funds, although it might take a few years as I don't want to incur a massive capital gains tax bill in the process.