Thursday, 28 February 2019

Skew and Trend following

In this post I discuss a well known stylised fact of the investment industry: "Trend following is a positively skewed strategy".

Spoiler alert: yes it is (sort of), but it's much more complicated (and interesting!) than you might think.

A quick primer on positive skew

So what actually is positive skew? Essentially it's an asset, or trading strategy, whose returns have the following profile:
  • A high proportion of relatively poor returns
  • The losing returns are smaller in magnitude than the winning returns
Or... if you prefer a pretty picture:

Or, if you prefer maths, then the skew is the third moment of the statistical distribution, and positive skew means there is more skew relative to a Gaussian normal distribution which has zero skew.

It's generally felt that positive skew is a good thing, and people are generally willing to pay a premium for owning assets with positive skew (and vice versa for negative skew) [where by 'a premium', I mean the assets have a higher risk adjusted return than you would expect when risk is measured purely by the second moment - standard deviation]. 

A coherent explanation of this comes from behavioural finance, and specifically prospect theory. A cognitive bias results in people overweighting the chances of low probability outcomes. They get fixated on the small chance of a large gain that positive skew offers. Equally with negative skew people get scared of the small chance of large losses which are threatened by negative skew.

(It might also be worth reading this paper, written by a bunch of people I used to work with, and some other people I haven't worked with).

In fact, we can check to see if we get paid for skew. If I look at the skew over the last 3 months of daily returns, and see how well that predicts the next 3 months annualised Sharpe Ratio, then I find that with negative skew the average SR is +0.33. With positive skew it's -0.016. The difference is statistically significant; if I do a regression the p-value is 0.01.

(At this point you might be thinking 'ah-ha! I can use skew as a predictor in a trading strategy. I will be rich!' .This is not an original idea! See for example, this paper.)

Intuitively, why should trend following produce positively skewed returns?

Trend following is effectively like buying a synthetic straddle* (a combination of long put and call options). This is a well known and fairly old result (see the seminal Fung and Hsieh 2001). Intuitively this makes sense, since both strategies will do well if volatility rises, and do badly if prices remain pinned. It's equally well known that any long volatility strategy, like buying straddles, should produce positively skewed returns: a lot of small negative returns when prices don't move and we hand over our premium, plus a smaller number of large positive returns when prices move enough for one leg of our straddle to be in the money.

* actually it's a look back straddle, but the distinction isn't important here.

So, positive skew is definitely one of the reasons why people like to allocate to trend following strategies, the others being:
  • Linear diversification; low correlation with traditional asset classes
  • Non linear diversification; good performance in tail events like 2008 (if you're from a fixed income background like yours truly, you can also think of this as 'positive convexity')
  • They sometimes even make money! 

However trend following also has it's problems. People don't like the long drawdowns that trend following type strategies produce, but these are an inevitable consequence of positive skew (for a given risk adjusted return the size of the average drawdown will be higher than for positively skewed assets). For example, suppose you're trading a strategy with a Sharpe Ratio of 0.5 and an annual risk target of 25%. With zero skew a bad drawdown (one that is achieved 10% of the time) will be 9.3% in magnitude. With positive skew that would rise to 11%, and would be just 3.7% with negative skew (skews of +1 and -2 respectively).

If trend following generates positive returns (and there is no clear evidence it has stopped doing so) then people must be more scared of those ugly drawdowns than they are of the advantages I've listed above. But (spoiler alert!) there might be something else going on.

The evidence

Economists and quant finance 'professionals' often pretend to be scientists (many of them have actual Phds in actual scientific subjects). So, let's pretend to be scientists and actually check to see if the evidence supports our expectations.

I'm going to use three types of trend following trading rule: a 2,8 day EWMAC; all the way up to a 64,256 day EWMAC (Exponentially weighted moving average crossover). Finally the results will be calculated over the 40ish futures contracts in my dataset. The whole thing is being done under the auspices of pysystemtrade, and you can find the usual ugly code here.

* actually 2,8 is actually a bit expensive to trade, but costs don't affect the calculation of skew since they just shift the distribution of returns to the left a bit.

For reasons that will become obvious I'm going to measure skew over different time periods: daily, weekly, monthly, and annual returns.

Let's start with the daily returns

Skew by trading rule, daily returns

Let me explain these plots. The y-axis is the measurement of skew, and the x-axis is the fast parameter value in the moving average pair (2,4,8,... 64). Each dot represents the skew measurement for a single instrument, and for a single 5 year period. This gives an indication of the uncertainty in our skew estimate (yes...I can't stop banging on about uncertainty).

Here are the median values for each rule:

2_8 = -0.04, 4_16 = -0.07, 8_32 = -0.51, 16_64 = -0.73, 32_128 = -0.94, 64_256 = -0.82

So... WTF?! Negative skew across the board, with significantly negative values for the slower crossovers. Something weird going on here.

Let's check the other time periods out:

Skew by trading rule, weekly returns
Skew by trading rule, monthly returns
Skew by trading rule, annual returns
Interesting. It looks like for bigger time periods the estimate of skew does indeed become positive. We can see this if we plot the median values for each rule, by time period:

Skew of a trend following rules profits, measured at different time horizons, from left to right: daily, weekly, monthly, annual
The results run from (on the left) daily, to (on the right) annual. Generally, skew gets more positive the slower the time period we use. The exception to this are the very fastest trading rules, which have a 'sweet spot' for skew at the monthly time period.

The puzzle

Does it make sense that positive skew only appears at certain frequencies of measurement, with a more infrequent measurement required for slower trading strategies? Yes, it does. Think about a fairly slow trend following rule. Maybe it changes it's positions every few months. When it is not changing it's positions, then it's skew of daily returns will be dictated by the skew of the underlying assets. 

So if it's trend following say equities (negative skew), then half the time you'd expect to see negative skew of (when it's long), and half the time (when it's short) you'd see positive skew. Overall your skew will be zero (and this result should hold for positive skew assets as well).

However if you start looking at annual returns, you're more likely to see the characteristically positive skew of trend following. The point at which the skew becomes significantly positive will depend on the speed of the trend following rule. With the faster rules we see positive skew with weekly and monthly returns; with the slower rules it isn't until we get to annual returns that the positive skew reveals itself.

(This is not an original finding. See this, written by someone else I used to work with)

But... that doesn't explain one thing. Why is the skew strongly negative at the shorter time frames? It should be zero, or close to it.

The only explanation is that trend following strategies like to be long negatively skewed assets, and short positively skewed assets

This is kind of interesting (well I think it is!). Perhaps the positive returns of trend following (a 'positively skewed' trading strategy) aren't that surprising at all, if it actually loads on to negatively skewed assets. Perhaps trend following is just a way of collecting the negative skew premium.

And... thinking some more... it sort of makes sense. If negative skew assets earn a premium in the market, then on average they will go up more often than they go down. And assets which go up more often than they go down, will tend to exhibit more bullish trends. And assets which exhibit more bullish trends, well they will be bought by trend following strategies.

This is all assuming that negative skew assets are negative before we buy them, and remain so. I will check this in a second.

What is the conditional relationship between skew and trend following

Let's do the following exercise. We'll find out the median skew, conditional on a trading rule being long or short, for a given trading rule.  I'm going to measure the skew over a period of a month, using daily returns.

First, let's look at the skew of a given instrument in the month after a trading rule has taken it's position. Remember its this skew that matters in determining what the skew of the returns of a trading rule will be (at least for the slower rules, which will 'inherit' the skew of the underlying asset).

Skew in the month after a trading rule takes it's position; conditioned on the trading rule being short (left hand side) or long (right hand side)

If a trading rule is long, then in the month following the forecast being made the skew is negative. If the rule is short, then the skew is closer to zero, or even positive. The effect is more noticeable for slower rules (faster rules will have changed their position during the following month anyway, perhaps multiple times).
This is a confirmation of our earlier intuition that slower trend following rules are likely to have negative skewed returns, because when they are long the underlying asset is negatively skewed; and when they are short the underlying asset is positively skewed (giving the strategy the opposite: more negatively skewed returns).

Now we look at the skew in the month before the trading rule decides what position it is taking:

Skew in the month before a trading rule takes it's position; conditioned on the trading rule being short (left hand side) or long (right hand side)
Now this is interesting. The slowest moving average does what we'd expect; it tends to be short when skew has been positive (or at least less negative), and goes long when skew has recently been negative. This confirms the theory that we end up loading up on negative skew as trend followers because negatively skewed assets are more likely to have positive drift (as a reward for that awful skew).

But for all the other trading rules we get the opposite effect*! For them the story is very weird: if skew has recently been negative they go short. But (from the previous graph), they then end up being short assets which subsequently have positive skew (which gives the trading  strategy negatively skewed returns). The skew flips sign. 

* in truth the penultimately slow trading rule is sort of flat.

If skew has been positive, the rule goes long, and then the underlying asset has negative skew (which again gives the trading strategy negatively skewed returns).

What is going on here? One possible explanation is this; for risky assets strongly negative skew usually appears after a sharp sell off. After such a sell off most trading rules will go short. But skew, like volatility at the right time horizon, is a mean reverting parameter. The trading rule starts with the skew the 'right' way round for generating positive skew (it goes short recent negative skew, and long recent positive skew) but then the sign of skew flips, and it ends up with exactly the wrong position!

The slowest moving average isn't affected by this; instead it's more likely to pick up the secular positive drift from negatively skewed assets.


Trend following rules do indeed have the positive skew you'd expect... but only at the right time horizon. For slower trend following rules you don't see them appear until you are using annual returns. At shorter time horizons they have persistently negative skew.

An asset which is negatively skewed at one time horizon, and positively skewed at another is... weird. Should we want to own it? I guess it depends on our own 'investment horizon'. If you only look at annual returns, you're going to love trend following! If you look at more frequent returns... you'll be less impressed. Given the long drawdowns of trend following strategies, you would be best off looking at your portfolio every 20 years or so :-)

For the slowest trend following rule I use it looks like this occurs because negatively skewed assets have a return premium, which leads to positive drift. So slow trend following rules will have a secular long bias to negatively skewed assets.

For other trend following rules this explanation is wrong. Instead, they tend to short assets whose skew has recently gone negative, and vice versa. It seems likely this is due to sharp selloffs in risky assets creating both negative skew and bearish recent trends. However skew is mean reverting; so the other rules end up being short assets which subsequently have positive skew, and vice versa.

This also means that if you're planning to use negative skew as a trading signal in combination with trend following, it will be a great diversifier! Except for the slowest moving average crossover, the momentum rule will usually do the opposite to a skew trading rule: it will short negative skewed assets, and go long positively skewed assets.

Saturday, 9 February 2019

Portfolio construction through handcrafting: Empirical tests

This post is all about handcrafting; a method for doing portfolio construction which human beings can do without computing power, or at least with a spreadsheet. The method aims to achieve the following goals:
  • Humans can trust it: intuitive and transparent method which produces robust weights
  • Can be easily implemented by a human in a spreadsheet
  • Can be back tested
  • Grounded in solid theoretical foundations
  • Takes account of uncertainty in data estimates
  • Decent out of sample performance
  • Addresses the problem of allocating capital to assets on a long only basis, or to trading strategies. 
This is the final post in a series on the handcrafting method.
  1. The first post can be found here, and it motivates the need for a method like this.
  2. In the second post I build up the various components of the method, and discuss why they are needed. 
  3. In the third post, I explained how you'd actually apply the method step by step, with code. 
  4. This post will test the method with real data, addressing the question of robust weights and out of sample performance
The testing will be done using psystemtrade. If you want to follow along, get the latest version.

PS apologies for the weird formatting in this post. It's out of my hands...

The Test Data

The test data is the 37 futures instruments in my usual data set, with the following trading rules:
  • Carry
  • Exponentially weighted moving average crossover (EWMAC) 2 day versus 8 day
  • EWMAC 4,16
  • EWMAC 8,32
  • EWMAC 16,64
  • EWMAC 32,128
  • EWMAC 64,256

I'll be using handcrafting to calculate both the forecast and instrument weights. By the way, this isn't a very stern test of the volatility scaling, since everything is assumed to have the same volatility in a trading system. Feel free to test it with your own data.
The handcrafting code lives here (you've mostly seen this before in a previous post, just some slight changes to deal with assets that don't have enough data) with a calling function added here in my existing optimisation code (which is littered with #FIXME NEEDS REFACTORING comments, but this isn't the time or the place...).

The Competition

I will be comparing the handcrafted method to the methods already coded up in pysystemtrade, namely:
  • Naive Markowitz
  • Bootstrapping
  • Shrinkage
  • Equal weights
All the configuration options for each optimiser will be the default for pysystemtrade (you might want to read this). All optimisation will be done on an 'expanding window' out of sample basis.
from systems.provided.futures_chapter15.estimatedsystem import *
system = futures_system()
system.config.forecast_weight_estimate['method']='handcraft' # change as appropriate
system.config.instrument_weight_estimate['method']='handcraft'  # change as appropriate


Evaluating the weights

Deciding which optimisation to use isn't just about checking profitability (although we will check that in a second). We also want to see robust weights; stable, without too many zeros.
Let's focus on the forecast weights for the S&P 500 (not quite arbitrary example, this is a cheap instrument so can allocate to most of the trading rules. Looking at say instrument weights would result in a massive messy plot).

Forecast weights with handcrafting
Pretty sensible weights here, with ~35% in carry and the rest split between the other moving averages. There are some variations when the correlations shift instruments slightly between groups.

# this will give us the final Portfolio object used for optimisation (change index -1 for others)
# See previous post in this series (


# eg to see the sub portfolio tree

[' Contains 3 sub portfolios',
 ["[0] Contains ['ewmac16_64', 'ewmac32_128', 'ewmac64_256']"], # slow momentum
 ["[1] Contains ['ewmac2_8', 'ewmac4_16', 'ewmac8_32']"],  # fast momentum
 ["[2] Contains ['carry']"]]  # carry
Makes a lot of sense to me...
Forecast weights with naive Markowitz
The usual car crash you’d expect from Naive Markowitz, with lots of variation, and very unrobust weights (at the end it’s basically half and half between carry and the slowest momentum).

Forecast weights with shrinkage

Smooth and pretty sensible. This method downweights the faster moving averages a little more than the others; they are more expensive and also don't perform so well in equities.

Forecast weights with bootstrapping
A lot noisier than shrinkage due to the randomness involved, but pretty sensible.

I haven't shown equal weights, as you can probably guess what those are.

Although I’m not graphing them, I thought it would be instructive to look at the final instrument weights for handcrafting:

AEX        0.016341
AUD        0.024343
BOBL       0.050443
BTP        0.013316
BUND       0.013448
CAC        0.014476
COPPER     0.024385
CORN       0.031373
CRUDE_W    0.029685
EDOLLAR    0.007732
EUR        0.010737
EUROSTX    0.012372
GAS_US     0.031425
GBP        0.010737
GOLD       0.012900
JPY        0.012578
KOSPI      0.031301
KR10       0.051694
KR3        0.051694
LEANHOG    0.048684
LIVECOW    0.031426
MXP        0.028957
NASDAQ     0.034130
NZD        0.024343
OAT        0.014660
PALLAD     0.013194
PLAT       0.009977
SHATZ      0.057006
SMI        0.040494
SOYBEAN    0.029706
SP500      0.033992
US10       0.005511
US2        0.031459
US20       0.022260
US5        0.007168
V2X        0.042326
VIX        0.042355
WHEAT      0.031373

Let's summarise these:

Ags 17.2%
Bonds 31.8%
Energy 6.1%
Equities 18.3%
FX 11.1%
Metals 6.0%
STIR 0.77%
Vol 8.4%

[' Contains 3 sub portfolios', # bonds, equities, other ['[0] Contains 3 sub portfolios', # bonds ["[0][0] Contains ['BOBL', 'SHATZ']"], # german short bonds ["[0][1] Contains ['KR10', 'KR3']"], # korean bonds ['[0][2] Contains 3 sub portfolios', # other bonds ["[0][2][0] Contains ['BUND', 'OAT']"], # european 10 year bonds ex BTP ['[0][2][1] Contains 2 sub portfolios', # US medium and long bonds ["[0][2][1][0] Contains ['EDOLLAR', 'US10', 'US5']"], # us medium bonds ["[0][2][1][1] Contains ['US20']"]], # us long bond ["[0][2][2] Contains ['US2']"]]], # us short bonds ['[1] Contains 3 sub portfolios', # equities and vol ['[1][0] Contains 2 sub portfolios', # European equities ["[1][0][0] Contains ['AEX', 'CAC', 'EUROSTX']"], # EU equities ["[1][0][1] Contains ['SMI']"]], # Swiss equities ["[1][1] Contains ['NASDAQ', 'SP500']"], # US equities ["[1][2] Contains ['V2X', 'VIX']"]], # US vol ['[2] Contains 3 sub portfolios', # other ['[2][0] Contains 3 sub portfolios', # FX and metals ['[2][0][0] Contains 2 sub portfolios', # FX, mostly ["[2][0][0][0] Contains ['EUR', 'GBP']"], ["[2][0][0][1] Contains ['BTP', 'JPY']"]], ["[2][0][1] Contains ['AUD', 'NZD']"], ['[2][0][2] Contains 2 sub portfolios', # Metals ["[2][0][2][0] Contains ['GOLD', 'PALLAD', 'PLAT']"], ["[2][0][2][1] Contains ['COPPER']"]]], ['[2][1] Contains 2 sub portfolios', # letfovers ["[2][1][0] Contains ['KOSPI', 'MXP']"], ["[2][1][1] Contains ['GAS_US', 'LIVECOW']"]], ['[2][2] Contains 3 sub portfolios', # ags and crude ["[2][2][0] Contains ['CORN', 'WHEAT']"], ["[2][2][1] Contains ['CRUDE_W', 'SOYBEAN']"], ["[2][2][2] Contains ['LEANHOG']"]]]]
Some very interesting groupings there, mostly logical but a few unexpected (eg BTP, KOSPI). Also instructive to look at the smallest weights:
US10, US5, EDOLLAR, PLAT, EUR, GBP, EUROSTX (used to hedge), JPY
Those are markets I could potentially think about removing if I wanted to. 

Evaluating the profits

As the figure shows the ranking of performance is as follows:

- Naive Markowitz, Sharpe Ratio (SR) 0.82
- Shrinkage, SR 0.96
- Bootstrap, SR 0.97
- Handcrafted SR 1.01
- Equal weighting SR 1.02
So naive is definitely sub optimal, but the others are pretty similar, with perhaps handcrafting and equal weights a fraction ahead of the rest. This is borne out by the T-statistics from doing pairwise comparisons between the various curves. 

boot = system.accounts.portfolio() ## populate the other values in the dict below appropriately
results = dict(naive=oneperiodacc, hc=handcraft_acc, equal=equal_acc, shrink=shrink, boot=boot)

from syscore.accounting import account_test

for type1 in types:
    for type2 in types:
        if type1==type2:
            continue        print("%s vs %s" % (type1, type2))
        print(account_test(results[type1], results[type2]))

A T-statistic of around 1.9 would hit the 5% critical value, and 2.3 is a 2% critical value):

         Naive Shrink Boot   HC   Equal
Shrink   2.01
Boot     1.56   0.03
HC       2.31   0.81  0.58
Equal    2.33   0.97  0.93  0.19

Apart from bootstrapping, all the other methods handily beat naive with 5% significance. However the rest of the t-statistics are pretty indifferent.

Partly this is because I’ve constrained all the optimisations in similar ways so they don’t do anything too stupid; for example ignoring Sharpe Ratio when optimising over instrument weights. Changing this would mostly penalise the naive optimisation further, but probably wouldn't change things much elsewhere.

It’s always slightly depressing when equal weights beats more complicated methods, but this is partly a function of the data set. Everything is vol scaled, so there is no need to take volatility into account. The correlation structure is reasonably friendly: 

  • for instrument weights we have a pretty even set of instruments across different asset classes, so equal weighted and handcrafted aren’t going to be radically different, 
  • for forecast weights, handcrafting (and all the other methods) produce carry weights of between 30% and 40% for S&P 500, whilst equal weighting would give carry just 14%. However this difference won’t be as stark for other instruments which can only afford to trade 2 or 3 EWMAC crossovers.

Still there are many contexts in which equal weight wouldn't make sense

Incidentally the code for handcrafting runs pretty fast; only a few seconds slower than equal weights which of course is the fastest (not that pysystemtrade is especially quick... speeding it up is on my [long] to do list). Naive and bootstraps run a bit slower (as they are doing a single optimisation per time period), whilst bootstrap is slowest of all (as it’s doing 100 optimisations per time period).


Handcrafting produces sensible and reasonably stable weights, and it's out of sample performance is about as good as more complicated methods. The test for handcrafting was not to produce superior out of sample performance. All we needed was performance that was indistinguishable from more complex methods. I feel that it has passed this test with flying colours, albeit on just this one data set.

So if I review the original motivation for producing this method:

  • - Humans can trust it; intuitive and transparent method which produces robust weights (yes, confirmed in this post)
  • - Can be easily implemented by a human in a spreadsheet (yes, see post 3)
  • - Can be back tested (yes, confirmed in this post)
  • - Grounded in solid theoretical foundations (yes, see post 2)
  • - Takes account of uncertainty (yes, see post 2)
  • - Decent out of sample performance (yes, confirmed in this post)

We can see that there is a clear tick in each category. I’m pretty happy with how this test has turned out, and I will be switching the default method for optimisation in pysystemtrade to use handcrafting.

Friday, 14 December 2018

Portfolio construction through handcrafting: implementation

This post is all about handcrafting; a method for doing portfolio construction which human beings can do without computing power, or at least with a spreadsheet. The method aims to achieve the following goals:
  • Humans can trust it: intuitive and transparent method which produces robust weights
  • Can be easily implemented by a human in a spreadsheet
  • Can be back tested
  • Grounded in solid theoretical foundations
  • Takes account of uncertainty in data estimates
  • Decent out of sample performance
  • Addresses the problem of allocating capital to assets on a long only basis, or to trading strategies. It won't be suitable for a long /short portfolio.

This is the third in a series of posts on the handcrafting method.
  1. The first post can be found here, and it motivates the need for a method like this.
  2. In the second post I build up the various components of the method, and discuss why they are needed. 
  3. In this, the third post, I'll explain how you'd actually apply the method step by step, with code. 
  4. Post four will test the method with real data

This will be a 'twin track' post; in which I'll outline two implementations:
  • a spreadsheet based method suitable for small numbers of assets where you need to do a one-off portfolio for live trading rather than repeated backtest. It's also great for understanding the intution of the method - a big plus point of this technique.
  • a python code based method. This uses (almost) exactly the same method, but can be backtested (the difference is that the grouping of assets is done manually in the spreadsheet based method, but automatically here based on the correlation matrix). The code can be found here; although this will live within the pysystemtrade ecosystem I've deliberately tried to make it as self contained as possible so you could easily drop this out into your own framework.

The demonstration

To demonstrate the implementation I'm going to need some data. This won't be the full blown real data that I'll be using to test the method properly, but we do need *something*. It needs to be an interesting data set; with the following characteristics:
  • different levels of volatility (so not a bunch of trading systems)
  • heirarcy of 3 levels (more would be too complex for the human implementaiton, less wouldn't be a stern enough test)
  • not too many assets such that the human implementation is too complex

I'm going to use long only weekly returns from the following instruments: BOBL, BUND, CORN, CRUDE_W, EURODOLLAR, GAS_US, KR10, KR3, US10, US20; from 2014 to the present (since for some of these instruments I only have data for the last 5 years).

Because this isn't a proper test I won't be doing any fancy rolling out of sample optimisation, just a single portfolio.

The descriptive statistics can be found here. The python code which gets the data (using pysystemtrade), is here.

(I've written the handcrafting functions to be standalone; when I come to testing them with real data I'll show you how to hook these into pysystemtrade]

Overview of the method

Here are the stages involved in the handcrafting method. Note there are a few options involved:
  1. (Optional if using a risk target, and automated): partition the assets into high and low volatility
  2. Group the assets hierarchically (if step 1 is followed, this will form the top level grouping). This will done either by (i) an automated clustering algorithm or (ii) human common sense.
  3. Calculate volatility weights within each group at the lowest level, proceeding upwards. These weights will either be equal, or use the candidate matching technique described in the previous post.
  4. (Optionally) Calculate Sharpe Ratio adjustments. Apply these to the weights from step 3.
  5. Calculate diversification multipliers for each group. Apply these to the weights from step 4.
  6. Calculate cash weights using the volatility of each asset.
  7. (Optionally) if a risk target was used with a manual method, partition the top level groups into high and low volatility.
  8. (Optionally) if a risk target was supplied; use the technique outlined in my previous post to ensure the target is hit.

Spreadsheet: Group the assets hierarchically

A suggested grouping is here. Hopefully it's fairly self explanatory. There could be some debate about whether Eurodollar and bonds should be glued together, but part of doing it this way was to see if the diversification multiplier fixes this potential mistake.

Spreadsheet: Calculate volatility weights

The calculations are shown here.

Notice that for most groups there are only one or two assets, so things are relatively trivial. Then at the top level (level 1) we have three assets, so things are a bit more fun. I use a simple average of correlations to construct a correlation matrix for the top level groups. Then I use a weighted average of two candidate matrices to work out the required weights for the top level groups.

The weights come out as follows:
  • Developed market bonds, which we have a lot of, 3.6% each for a total of 14.4%
  • Emerging market bonds (just Korea), with 7.2% each for a total of 14.4%
  • Energies get 10.7% each, for a total of 21.4%
  • Corn gets 21.4%
  • Eurodollar gets 28.6%

Spreadsheet: Calculate Sharpe Ratio adjustments (optionally)

Adjustments for Sharpe Ratios are shown in this spreadsheet. You should follow the calculations down the page, as they are done in a bottom up fashion. I haven't bothered with interpolating the heuristic adjustments, instead I've just used VLOOKUP to match the closest adjustment row. 

Spreadsheet: Calculate diversification multipliers (DM)

DM calculations are shown in this sheet. DMs are quite low in bonds (where the assets in each country are highly correlated), but much higher in commodities. The final set of changes in particular striking; note the reallocation from the single instrument rates group (initial weight 30.7%, falls to 24.2%) to commodities (initial weight 29%, rises to 36.5%).

Spreadsheet: Calculate cash weights

(Almost) finally we calculate our cash weights, in this spreadsheet. Notice the huge weight to low volatility Eurodollar. 

Spreadsheet: Partition into high and low volatility 

(optional: if risk target used with manual method)

If we're using a risk target we'll need to partition our top level groups (this is done automatically with python, but spreadsheet people are allowed to choose their own groupings). Let's choose an arbitrary risk target: 10%. This should be achievable since the average risk of our assets is 10.6%

This is the average volatility of each group (calculated here):

Bonds: 1.83%
Commodities: 14.6%
Rates: 0.89%

So we have:

High vol: commodities
Low vol: Rates and bonds

(Not a massive surprise!!)

Spreadsheet: Check risk target is hit, adjust weights if required

(optional: with risk target)

The natural risk of the portfolio comes out at 1.09% (calculated here). Let's explore the possible scenarios:
  • Risk target lower than 1.09%, eg 1%: We'd need to add cash to the portfolio. Using the spreadsheet with a 1% risk target you'd need to put 8.45% of your portfolio into cash; with the rest going into the constructed portfolio.
  • Risk target higher than 1.09% with leverage allowed: You'd need to apply a leverage factor; with a risk target of 10% you'd need a leverage factor of 9.16
  • Risk target higher than 1.09% without leverage: You'd need to constrain the proportion of the portfolio that allocated to low risk assets (bonds and rates). The spreadsheet shows that this comes out at 31.4% cash weight, with the rest in commodities. I've also recalculated the weights with this constraint to show how it comes out.
And here are those final weights (to hit 10% risk with no leverage):

BOBL 2.17%
BUND 0.78%
US10 0.44%
US20 0.23%
KR3 7.25%
KR10 1.86%
EDOLLAR 18.67%
CORN 36.67%
CRUDE_W 19.47%
GAS_US 12.45%

Python code

The handcrafting code is here. Although this file will ultimately be dumped into pysystemtrade, it's designed to be entirely self contained so you can use it in your own applications.

The code expects weekly returns, and for all assets to be present. It doesn't do rolling optimisation, or averages over multiple assets. I need to write code to hook it into pysystemtrade, and to achieve these various objectives.

The only input required is a pandas data frame returns with named columns containing weekly returns. The main object you'll be interacting with is called Portfolio

Simplest use case, to go from returns to cash weights without risk targeting:


I won't document the API or methodology fully here, but hopefully you will get the idea.

Python: Partition the assets into high and low volatility

(If using a risk target, and automated)

Let's try with a risk target of 10%:

p=Portfolio(returns, risk_target=.1)

Out[575]: [Portfolio with 7 instruments, Portfolio with 3 instruments]

Out[576]: Portfolio with 7 instruments
Out[577]: ['BOBL', 'BUND', 'EDOLLAR', 'KR10', 'KR3', 'US10', 'US20']

Out[578]: ['CORN', 'CRUDE_W', 'GAS_US']

So all the bonds get put into one group, the other assets into another. Seems plausible.

Using an excessively high risk target is a bad idea:

p=Portfolio(returns, risk_target=.3)
Not many instruments have risk higher than target; portfolio will be concentrated to hit risk target
Out[584]: [Portfolio with 9 instruments, Portfolio with 1 instruments]

This is an even worse idea:

p=Portfolio(returns, risk_target=.4)
Exception: Risk target greater than vol of any instrument: will be impossible to hit risk target

The forced partitioning into two top level groups will not happen if leverage is allowed, or no risk target is supplied:

p=Portfolio(returns) # no risk target
Natural top level grouping used
[Portfolio with 7 instruments,
 Portfolio with 2 instruments,
 Portfolio with 1 instruments]
p=Portfolio(returns, risk_target=.3, allow_leverage=True)
Natural top level grouping used
[Portfolio with 7 instruments,
 Portfolio with 2 instruments,
 Portfolio with 1 instruments]

Python: Group the assets hierarchically

Here's an example when we're allowing the grouping to happen naturally:
Natural top level grouping used Out[48]: [' Contains 3 sub portfolios', ['... Contains 3 sub portfolios', ["...... Contains ['KR10', 'KR3']"], ["...... Contains ['EDOLLAR', 'US10', 'US20']"], ["...... Contains ['BOBL', 'BUND']"]], ["... Contains ['CRUDE_W', 'GAS_US']"], ["... Contains ['CORN']"]]
We have three top level groups: interest rates, energies, and Ags. The interest rate group is further divided into second level groupings by country: Korea, US and Germany. Here's an example when we're doing a partition by risk

p=Portfolio(returns, risk_target=.1)
Applying partition to hit risk target
Partioning into two groups to hit risk target of 0.100000

[' Contains 2 sub portfolios',
 ['... Contains 3 sub portfolios',
  ["...... Contains ['KR10', 'KR3']"],
  ["...... Contains ['EDOLLAR', 'US10', 'US20']"],
  ["...... Contains ['BOBL', 'BUND']"]],
 ["... Contains ['CORN', 'CRUDE_W', 'GAS_US']"]]

There are now two top level groups as we saw above.

If you're a machine learning enthusiast who wishes to play around with the clustering algorithm, then the heavy lifting of the clustering algo is all done in this method of the portfolio object:

def _cluster_breakdown(self):

    X = self.corr_matrix.values
    d = sch.distance.pdist(X)
    L = sch.linkage(d, method='complete')

    # play with this line at your peril!!!
    ind = sch.fcluster(L, MAX_CLUSTER_SIZE, criterion='maxclust')

    return list(ind)

However I've found the results to be very similar regardless of the method used.

Python: Calculate volatility weights

p=Portfolio(returns, use_SR_estimates=False)  # turn off SR estimates for now
Natural top level grouping used
[' Contains 3 sub portfolios',
 ['... Contains 3 sub portfolios',
  ["...... Contains ['KR10', 'KR3']"],
  ["...... Contains ['EDOLLAR', 'US10', 'US20']"],
  ["...... Contains ['BOBL', 'BUND']"]],
 ["... Contains ['CRUDE_W', 'GAS_US']"],
 ["... Contains ['CORN']"]]

Let's look at a few parts of the portfolio. Firstly the very simple single asset Corn portfolio:

# Just Corn, single asset
Out[54]: [1.0]

The Energy portfolio is slightly more interesting with two assets; but this will default to equal volatility weights:

# Just two assets, so goes to equal vol weights
Out[55]: [0.5, 0.5]

Only the US bonds (and STIR) portfolio has 3 assets, and so will use the candidate matching algorithm:

# The US bond group is the only interesting one
          EDOLLAR      US10      US20
EDOLLAR  1.000000  0.974097  0.872359
US10     0.974097  1.000000  0.924023
US20     0.872359  0.924023  1.000000
# Pretty close to equal weighting
Out[57]: [0.28812193544790643, 0.36572016685796049, 0.34615789769413313]

Python: Calculate Sharpe Ratio adjustments (optionally)

p=Portfolio(returns) # by default Sharpe Ratio adjustments are on unless we turn them off

Let's examine a simple two asset portfolio to see how these work:

# Let's look at the energies portfolio
Out[61]: Portfolio with 2 instruments
# first asset is awful, second worse
Out[63]: array([-0.55334564, -0.8375069 ])

# Would be equal weights, now tilted towards first asset
Out[62]: [0.5399245657079913, 0.46007543429200887]

# Can also see this information in one place
                      CRUDE_W    GAS_US
Raw vol (no SR adj)  0.500000  0.500000
Vol (with SR adj)    0.539925  0.460075
Sharpe Ratio        -0.553346 -0.837507 
Portfolio containing ['CRUDE_W', 'GAS_US'] instruments  

Python: Calculate diversification multipliers

Natural top level grouping used

# not much diversification for bonds /rates within each country
Out[67]: 1.0389170782708381  #korea
Out[68]: 1.0261371453175774  #US bonds and STIR
Out[69]: 1.0226377699075955  # german bonds
# Quite decent when you put them together though p.sub_portfolios[0].div_mult
Out[64]: 1.2529917422729928

# Energies group only two assets but quite uncorrelated
Out[65]: 1.2787613327950775

# only one asset in corn group
Out[66]: 1.0
# Not used in the code but good to know
Out[71]: 2.0832290180687183

Python: Aggregate up sub-portfolios

The portfolio in the python code is built up in a bottom up fashion. Let's see how this happens, by focusing on the 10 year US bond.

Natural top level grouping used

First the code calculates the vol weight for US bonds and rates, including a SR adjustment:

                      EDOLLAR      US10      US20
Raw vol (no SR adj)  0.288122  0.365720  0.346158
Vol (with SR adj)    0.292898  0.361774  0.345328
Sharpe Ratio         0.218935  0.164957  0.185952 
 Portfolio containing ['EDOLLAR', 'US10', 'US20'] instruments  

This portfolio then joins the wider bond portfolio (here in column '1' - there are no meaningful names for parts of the wider portfolio - the code doesn't know this is US bonds):

                                  0         1         2
Raw vol (no SR adj or DM)  0.392114  0.261486  0.346399
Vol (with SR adj no DM)    0.423425  0.162705  0.413870
SR                         0.985267  0.192553  1.185336
Div mult                   1.038917  1.026137  1.022638 
 Portfolio containing 3 sub portfolios aggregate 

The Sharpe Ratios, raw vol, and vol weights shown here are for the groups that we're aggregating together here. So the raw vol weight on US bonds is 0.26. To see why look at the correlation matrix:
          0         1         2
0  1.000000  0.493248  0.382147
1  0.493248  1.000000  0.715947
2  0.382147  0.715947  1.000000
You can see that US bonds are more highly correlated with asset 0 and asset 2, than they are with each other. So it gets a lower raw weight. It also has a far worse Sharpe Ratio, so get's further downweighted relative to the other countries.

We can now work out what the weight of US 10 year bonds is amongst bonds as a whole:


                       BOBL      BUND   EDOLLAR      KR10       KR3      US10  \

Vol wt in group    0.519235  0.480765  0.292898  0.477368  0.522632  0.361774   

Vol wt. of group   0.413870  0.413870  0.162705  0.423425  0.423425  0.162705   
Div mult of group  1.022638  1.022638  1.026137  1.038917  1.038917  1.026137   
Vol wt.            0.213339  0.197533  0.047473  0.203860  0.223189  0.058636   
Vol wt in group    0.345328  
Vol wt. of group   0.162705  
Div mult of group  1.026137  
Vol wt.            0.055971   
 Portfolio containing 3 sub portfolios 

The first row is the vol weight of the asset within it's group; we've already seen this calculated. The next row is the vol weight of the group as a whole; again we've already seen the figures for US bonds calculated above. After that is the diversification multiplier for the US bond group. Finally we can see the volatility weight of US 10 year bonds in the bond group as a whole; equal to the vol weight within the group, multiplied by the vol weight of the group, multiplied by the diversification multiplier of the group; and then renormalised to add up to 1.

Finally we're ready to construct the top level group, in which the bonds as a whole is asset '0'. First the correlation matrix:

notUsedYet = p.volatility_weights
          0         1         2
0  1.000000 -0.157908 -0.168607
1 -0.157908  1.000000  0.016346
2 -0.168607  0.016346  1.000000

All these assets, bonds [0], energies [1], and corn [2] are pretty uncorrelated, though bonds might just have the edge:

                                  0         1         2
Raw vol (no SR adj or DM)  0.377518  0.282948  0.339534
Vol (with SR adj no DM)    0.557443  0.201163  0.241394
SR                         1.142585 -0.871979 -0.801852
Div mult                   1.252992  1.278761  1.000000 
 Portfolio containing 3 sub portfolios aggregate 

Now to calculate the final weights:



                       BOBL      BUND      CORN   CRUDE_W   EDOLLAR    GAS_US  \

Vol wt in group    0.213339  0.197533  1.000000  0.539925  0.047473  0.460075   
Vol wt. of group   0.557443  0.557443  0.241394  0.201163  0.557443  0.201163   
Div mult of group  1.252992  1.252992  1.000000  1.278761  1.252992  1.278761   
Vol wt.            0.124476  0.115254  0.201648  0.116022  0.027699  0.098863   
                       KR10       KR3      US10      US20  
Vol wt in group    0.203860  0.223189  0.058636  0.055971  
Vol wt. of group   0.557443  0.557443  0.557443  0.557443  
Div mult of group  1.252992  1.252992  1.252992  1.252992  
Vol wt.            0.118945  0.130224  0.034212  0.032657   
 Portfolio containing 3 sub portfolios 

We've now got the final volatility weights. Here's another way of viewing them:

# First remind ourselves of the volatility weights
dict([(instr,wt) for instr,wt in zip(p.instruments, p.volatility_weights)])
{'BOBL': 0.12447636469041611,
 'BUND': 0.11525384132670763,
 'CORN': 0.20164774158721335,
 'CRUDE_W': 0.11602155610023207,
 'EDOLLAR': 0.027698823230085486,
 'GAS_US': 0.09886319534295436,
 'KR10': 0.11894543449866347,
 'KR3': 0.13022374999090081,
 'US10': 0.034212303586599956,
 'US20': 0.032656989646226771}
The most striking difference to the spreadsheet is that by lumping Eurodollar in with the other US bonds it has a much smaller vol weight. German and Korean bonds have gained as a result; the energies and Corn are pretty similar.

Python: Calculate cash weights

dict([(instr,wt) for instr,wt in zip(p.instruments, p.cash_weights)])
Natural top level grouping used
{'BOBL': 0.21885945926487166,
 'BUND': 0.079116240615862948,
 'CORN': 0.036453365347104472,
 'CRUDE_W': 0.015005426640542012,
 'EDOLLAR': 0.10335586678017628,
 'GAS_US': 0.009421184504702888,
 'KR10': 0.10142345423259323,
 'KR3': 0.39929206844323878,
 'US10': 0.025088747004851766,
 'US20': 0.011984187166055982}

Obviously the less risky assets like 3 year Korean bonds and Eurodollar get a larger cash weight. It's also possible to see how these were calculated from the final volatility weights:
                  BOBL      BUND      CORN   CRUDE_W   EDOLLAR    GAS_US  \
Vol weights   0.124476  0.115254  0.201648  0.116022  0.027699  0.098863   
Std.          0.018965  0.048575  0.184449  0.257816  0.008936  0.349904   
Cash weights  0.218859  0.079116  0.036453  0.015005  0.103356  0.009421   
                  KR10       KR3      US10      US20  
Vol weights   0.118945  0.130224  0.034212  0.032657  
Std.          0.039105  0.010875  0.045470  0.090863  
Cash weights  0.101423  0.399292  0.025089  0.011984   
 Portfolio containing 10 instruments (cash calculations) 

Python: Check risk target is hit, adjust weights if required

(optional: with risk target)

The natural risk of the unconstrained portfolio is quite low: 1.59% (a bit higher than the spreadsheet version, since we haven't allocated as much to Eurodollar)

Natural top level grouping used
Out[82]: 0.015948015324395711

Let's explore the possible scenarios:
  • Risk target lower than 1.59%, eg 1%: We'd need to add cash to the portfolio. 
p=Portfolio(returns, risk_target=.01)

# if cash weights add up to less than 1, must be including cash in the portfolio


Calculating weights to hit a risk target of 0.010000

Natural top level grouping used
Too much risk 0.372963 of the portfolio will be cash
Out[84]: 0.62703727056889502

# check risk target hit
Out[85]: 0.01

With a 1% risk target you'd need to put 37.3% of your portfolio into cash; with the rest going into the constructed portfolio.
  • Risk target higher than 1.59% with leverage allowed, eg 10%
p=Portfolio(returns, risk_target=.1, allow_leverage=True)

# If sum of cash weights>1 we must be using leverage
Calculating weights to hit a risk target of 0.100000
Natural top level grouping used
Not enough risk leverage factor of 6.270373 applied
Out[87]: 6.2703727056889518

# check target hit
Out[88]: 0.10000000000000001

You'd need to apply a leverage factor; with a risk target of 10% you'd need a leverage factor of 6.27
  • Risk target higher than 1.59% without leverage: 
p=Portfolio(returns, risk_target=.1)
Calculating weights to hit a risk target of 0.100000
Not enough risk, no leverage allowed, using partition method
Applying partition to hit risk target
Partitioning into two groups to hit risk target of 0.100000
Need to limit low cash group to 0.005336 (vol) 0.323992 (cash) of portfolio to hit risk target of 0.100000
Applying partition to hit risk target
Partitioning into two groups to hit risk target of 0.100000

# look at cash weights
dict([(instr,wt) for instr,wt in zip(p.instruments, p.cash_weights)])
{'BOBL': 0.07548008030352539,
 'BUND': 0.027285547606928903,
 'CORN': 0.3285778602871447,
 'CRUDE_W': 0.19743348662518673,
 'EDOLLAR': 0.035645291049388697,
 'GAS_US': 0.15010566887898191,
 'KR10': 0.034978842111056153,
 'KR3': 0.13770753839879318,
 'US10': 0.0086525875783564771,
 'US20': 0.0041330971606378854}

# check risk target hit
Out[91]: 0.10001663416516968

In this case the portfolio to constrain the proportion of the portfolio that allocated to low risk assets (bonds and rates). 

What's next

In the next post I'll test the method (in it's back testable python format - otherwise (a) the results could arguably be forward looking, and (b) I have now seen more than enough spreadsheets for 2018 thank you very much) against some alternatives. It could take me a few weeks to post this, as I will be somewhat busy with Christmas, university, and book writing commitments!