A little demonstration of portfolio optimisation

I've had a request for the code used to do the optimisations in chapter 4 of my book "Systematic Trading" (the 'one-period' and 'bootstrapping' methods; there isn't much point in including code to the 'handcrafted' method as it's supposed to avoid programming).

Although this post will make more sense if you've read the book, it can also be read independently as I'll be dropping brief explanations in as we go. Hopefully it will whet your appetite!

You can get the code you need from here:

https://github.com/robcarver17/systematictradingexamples/blob/master/optimisation.py

The code also includes a function for generating "expanding window", "rolling window" and "in sample" back test time periods which could be useful for general fitting.

The problem

The problem we are trying to solve here is "What portfolio weights should we have held in the past (between 2000 and mid 2015) given the returns of 3 assets: S&P 500 equity index, NASDAQ equity index and a US 20 year bond*?"

* You can think of this as a synthetic constant maturity bond, or what you'd get if you held the 20 year US bond future and also earned interest on the cash you saved from getting exposure via a derivative.

Some of the issues I will explore in this post are:

• This is a backtest that we're running here - a historical simulation. So how do we deal with the fact that 10 years ago we wouldn't have had data from 2005 to 2015? How much data should we use to fit?
• These assets have quite different volatility. How can we express our portfolio weights in a way which accounts for this?
• Standard portfolio optimisation techniques produce very unstable and extreme weights. Should we use them, or another method like bootstrapping which takes account of the noise in the data?
• Most of the instability in weights comes from having slightly different estimates of the mean return. Should we just assume all assets have the same mean return?

In sample

Let's begin by doing some simple in sample testing. Here we cheat, and assume we have all the data at the start.

I'm going to do the most 'vanilla' optimisation possible:

opt_and_plot(data, "in_sample", "one_shot", equalisemeans=False, equalisevols=False)

This is a very boring plot, but it shows that we would have put 78% of our portfolio into US 20 year bonds and 22% into S&P500, with nothing in NASDAQ. Because we're cheating we have the same information throughout the backtest so the weights don't change. We haven't accounted for the uncertainty in our data; nor done anything with our estimated means - this is just vanilla 'one period' optimisation - so the weights are pretty extreme.

Let's deal with the first problem - different volatility. In my book I use the technique of volatility normalisation to make sure that the assets we are optimising weights for have the same expected risk. That isn't the case here. Bonds are much less volatile than stocks. To compensate for this they have a much bigger weight.

We can change the optimisation function so it does a type of normalisation; measure the standard deviation of returns in the dataset and change all the returns so they have some arbitrary annualised risk (20% by default). This has the effect of turning the covariance matrix into a correlation matrix.


opt_and_plot(data, "in_sample", "one_shot", equalisemeans=False, equalisevols=True)

Now things are looking slightly more reasonable. The weights we are seeing here are 'risk allocations'; they are conditional on the assets having the same volatility. Even if we aren't lucky enough to have assets like that it's more intuitive to look at weights in this vol adjusted space.

However it's still a pretty extreme portfolio. Poor NASDAQ doesn't get a look in. A very simple way of dealing with this is to throw away the information we have about expected mean returns, and assume all assets have the same mean return (notice that as we have equalised volatility this is the same as assume the same Sharpe Ratio for all assets; and indeed this is actually what the code does).


opt_and_plot(data, "in_sample", "one_period", equalisemeans=True, equalisevols=True)

Now we have something I could actually live with. The only information we're using here is correlations; clearly bonds are uncorrelated with equities and get almost half the weight (which is what they'd get with handcrafting - the simple, no computer required, method I discuss in my own). S&P 500 is, for some reason, slightly less diversifying than NASDAQ in this dataset, and gets a slightly higher weight.

However what if our assets do have different expected returns, and in a statistically significant way? A better way of doing the optimisation is not to throw away the means, but to use bootstrapping. With bootstrapping we pull returns out of our data at random (500 times in this example); do an optimisation on each sample of returns, and then take an average of the weights from each sample.

opt_and_plot(data, "in_sample", "bootstrap", equalisemeans=False, equalisevols=True, monte_carlo=500)

Notice the weights are 'wiggling' around slightly. This is because although the code is using the same data (as we're optimising in sample), it's doing a new set of 500 optimisations each year, and each will be slightly different due to the randomness of each sample. If I'd used a smaller value for monte_carlo then there would be even more noise. I quite like this 'wiggliness' - it exposes the underlying uncertainty in the data.

Looking at the actual weights they are similar to the previous example with no means, although NASDAQ (which did really badly in this sample) is slightly downweighted. In this case using the distribution of average returns (and correlations, for what it's worth) hasn't changed our minds very much. There isn't a statistically significant difference in the returns of these three assets over this period.

Rolling window

Let's stop cheating and run our optimisation in such a way that we only know the returns of the past. A common method to do this is 'walk forward testing', or what I call 'a rolling window'. In each year that we're testing for we use the returns of the last N years to do our optimisation.

To begin with let's use 'one period' optimisation with a lookback of a single year.

opt_and_plot(data, "rolling", "one_period", rollyears=1, equalisemeans=False, equalisevols=True)

As I explain at length in my book one year is wholly inadequate to give you significant information about returns. Notice how unstable and extreme these weights are. What about 5 years?


opt_and_plot(data, "rolling", "one_period", rollyears=5, equalisemeans=False, equalisevols=True)

These are a little more stable, but still very extreme. In practice you usually need a lot more than 5 years of data to do any kind of optimisation, and chapter 3 of my book expands on this point.

I won't show the results for bootstrapping with a rolling window; this is left as an exercise for the reader.

Expanding window

It's my preference to use an expanding window (sometimes called anchored fitting). Here we use all the data that we have available as we step through each year. So our window gets bigger over time.

opt_and_plot(data, "expanding", "one_period", equalisemeans=False, equalisevols=True)

These weights are more stable as we get more data; by the end of the period we're only adding 7% more information so it doesn't affect the results that much. However the weights are still extreme. Adding more data to a one-shot optimisation is only helpful up to a point.

Let's go back to the boostrapped method. This is my own personal favourite optimisation method:

opt_and_plot(data, "expanding", "bootstrap", equalisemeans=False, equalisevols=True)

Things are a bit hairy in the first year* but the weights quickly settle down to non extreme values, gradually adjusting as we get more data as the window expands.

* I'm using 250 days - about a year - of data in each bootstrap sample (you can change this with the monte_length parameter). With the underlying sample also only a year long this is pushing things to their limit - I normally suggest you use a window size around 10% of the total data. If you must optimise with only a year of data then you should probably use samples of around 25 business days. However my simple code doesn't support a varying window size; though it would be easy to use the 10% guideline eg by adding monte_length=int(0.1*len(returns_to_bs.index)) to the start of the function bootstrap_portfolio.

Just to reinforce the point that these are 'risk weightings' here is the same optimisation done with the actual 'cash' weights and no normalisation of volatility:

opt_and_plot(data, "expanding", "bootstrap", equalisemeans=False, equalisevols=False)

Conclusion

I hope this has been useful both to those who have bought my book, and those who haven't yet bought it (I'm feeling optimistic!). If there is any python code that I've used to write the book you would like to see, let me know.