## Thursday 28 February 2019

### Skew and Trend following

In this post I discuss a well known stylised fact of the investment industry: "Trend following is a positively skewed strategy".

Spoiler alert: yes it is (sort of), but it's much more complicated (and interesting!) than you might think.

## Economists and quant finance 'professionals' often pretend to be scientists (many of them have actual Phds in actual scientific subjects). So, let's pretend to be scientists and actually check to see if the evidence supports our expectations.I'm going to use three types of trend following trading rule: a 2,8 day EWMAC; all the way up to a 64,256 day EWMAC (Exponentially weighted moving average crossover). Finally the results will be calculated over the 40ish futures contracts in my dataset. The whole thing is being done under the auspices of pysystemtrade, and you can find the usual ugly code here.* actually 2,8 is actually a bit expensive to trade, but costs don't affect the calculation of skew since they just shift the distribution of returns to the left a bit.For reasons that will become obvious I'm going to measure skew over different time periods: daily, weekly, monthly, and annual returns.Let's start with the daily returns

 Skew by trading rule, daily returns

## So... WTF?! Negative skew across the board, with significantly negative values for the slower crossovers. Something weird going on here.

Let's check the other time periods out:

 Skew by trading rule, weekly returns
 Skew by trading rule, monthly returns
 Skew by trading rule, annual returns
Interesting. It looks like for bigger time periods the estimate of skew does indeed become positive. We can see this if we plot the median values for each rule, by time period:

 Skew of a trend following rules profits, measured at different time horizons, from left to right: daily, weekly, monthly, annual
The results run from (on the left) daily, to (on the right) annual. Generally, skew gets more positive the slower the time period we use. The exception to this are the very fastest trading rules, which have a 'sweet spot' for skew at the monthly time period.

## Does it make sense that positive skew only appears at certain frequencies of measurement, with a more infrequent measurement required for slower trading strategies? Yes, it does. Think about a fairly slow trend following rule. Maybe it changes it's positions every few months. When it is not changing it's positions, then it's skew of daily returns will be dictated by the skew of the underlying assets.

So if it's trend following say equities (negative skew), then half the time you'd expect to see negative skew of (when it's long), and half the time (when it's short) you'd see positive skew. Overall your skew will be zero (and this result should hold for positive skew assets as well).

However if you start looking at annual returns, you're more likely to see the characteristically positive skew of trend following. The point at which the skew becomes significantly positive will depend on the speed of the trend following rule. With the faster rules we see positive skew with weekly and monthly returns; with the slower rules it isn't until we get to annual returns that the positive skew reveals itself.

(This is not an original finding. See this, written by someone else I used to work with)

But... that doesn't explain one thing. Why is the skew strongly negative at the shorter time frames? It should be zero, or close to it.

The only explanation is that trend following strategies like to be long negatively skewed assets, and short positively skewed assets

This is kind of interesting (well I think it is!). Perhaps the positive returns of trend following (a 'positively skewed' trading strategy) aren't that surprising at all, if it actually loads on to negatively skewed assets. Perhaps trend following is just a way of collecting the negative skew premium.

And... thinking some more... it sort of makes sense. If negative skew assets earn a premium in the market, then on average they will go up more often than they go down. And assets which go up more often than they go down, will tend to exhibit more bullish trends. And assets which exhibit more bullish trends, well they will be bought by trend following strategies.

This is all assuming that negative skew assets are negative before we buy them, and remain so. I will check this in a second.

## What is the conditional relationship between skew and trend following

Let's do the following exercise. We'll find out the median skew, conditional on a trading rule being long or short, for a given trading rule.  I'm going to measure the skew over a period of a month, using daily returns.

First, let's look at the skew of a given instrument in the month after a trading rule has taken it's position. Remember its this skew that matters in determining what the skew of the returns of a trading rule will be (at least for the slower rules, which will 'inherit' the skew of the underlying asset).

 Skew in the month after a trading rule takes it's position; conditioned on the trading rule being short (left hand side) or long (right hand side)

If a trading rule is long, then in the month following the forecast being made the skew is negative. If the rule is short, then the skew is closer to zero, or even positive. The effect is more noticeable for slower rules (faster rules will have changed their position during the following month anyway, perhaps multiple times).
This is a confirmation of our earlier intuition that slower trend following rules are likely to have negative skewed returns, because when they are long the underlying asset is negatively skewed; and when they are short the underlying asset is positively skewed (giving the strategy the opposite: more negatively skewed returns).

## Summary

Trend following rules do indeed have the positive skew you'd expect... but only at the right time horizon. For slower trend following rules you don't see them appear until you are using annual returns. At shorter time horizons they have persistently negative skew.

An asset which is negatively skewed at one time horizon, and positively skewed at another is... weird. Should we want to own it? I guess it depends on our own 'investment horizon'. If you only look at annual returns, you're going to love trend following! If you look at more frequent returns... you'll be less impressed. Given the long drawdowns of trend following strategies, you would be best off looking at your portfolio every 20 years or so :-)

For the slowest trend following rule I use it looks like this occurs because negatively skewed assets have a return premium, which leads to positive drift. So slow trend following rules will have a secular long bias to negatively skewed assets.

For other trend following rules this explanation is wrong. Instead, they tend to short assets whose skew has recently gone negative, and vice versa. It seems likely this is due to sharp selloffs in risky assets creating both negative skew and bearish recent trends. However skew is mean reverting; so the other rules end up being short assets which subsequently have positive skew, and vice versa.

This also means that if you're planning to use negative skew as a trading signal in combination with trend following, it will be a great diversifier! Except for the slowest moving average crossover, the momentum rule will usually do the opposite to a skew trading rule: it will short negative skewed assets, and go long positively skewed assets.

## Saturday 9 February 2019

### Portfolio construction through handcrafting: Empirical tests

This post is all about handcrafting; a method for doing portfolio construction which human beings can do without computing power, or at least with a spreadsheet. The method aims to achieve the following goals:
• Humans can trust it: intuitive and transparent method which produces robust weights
• Can be easily implemented by a human in a spreadsheet
• Can be back tested
• Grounded in solid theoretical foundations
• Takes account of uncertainty in data estimates
• Decent out of sample performance
• Addresses the problem of allocating capital to assets on a long only basis, or to trading strategies.
This is the final post (sort of) in a series on the handcrafting method.
1. The first post can be found here, and it motivates the need for a method like this.
2. In the second post I build up the various components of the method, and discuss why they are needed.
3. In the third post, I explained how you'd actually apply the method step by step, with code.
4. This post will test the method with real data, addressing the question of robust weights and out of sample performance
5. Bonus: post five explains how to account for uncertainty in asset Sharpe Ratios (effectively an appendix to post two)
6. Bonus: post six explains how I come up with subportfolio weights that account for correlation uncertainty (effectively an appendix to post two)
The testing will be done using psystemtrade. If you want to follow along, get the latest version.

PS apologies for the weird formatting in this post. It's out of my hands...

## The Test Data

The test data is the 37 futures instruments in my usual data set, with the following trading rules:
• Carry
• Exponentially weighted moving average crossover (EWMAC) 2 day versus 8 day
• EWMAC 4,16
• EWMAC 8,32
• EWMAC 16,64
• EWMAC 32,128
• EWMAC 64,256

I'll be using handcrafting to calculate both the forecast and instrument weights. By the way, this isn't a very stern test of the volatility scaling, since everything is assumed to have the same volatility in a trading system. Feel free to test it with your own data.
The handcrafting code lives here (you've mostly seen this before in a previous post, just some slight changes to deal with assets that don't have enough data) with a calling function added here in my existing optimisation code (which is littered with #FIXME NEEDS REFACTORING comments, but this isn't the time or the place...).

```
```

## The Competition

I will be comparing the handcrafted method to the methods already coded up in pysystemtrade, namely:
• Naive Markowitz
• Bootstrapping
• Shrinkage
• Equal weights
All the configuration options for each optimiser will be the default for pysystemtrade (you might want to read this). All optimisation will be done on an 'expanding window' out of sample basis.
```from systems.provided.futures_chapter15.estimatedsystem import *
system = futures_system()
system.config.forecast_weight_estimate['method']='handcraft' # change as appropriate```
```system.config.instrument_weight_estimate['method']='handcraft'  # change as appropriate
del(system.config.instruments)
del(system.config.rule_variations)

system.set_logging_level("on")```

## Evaluating the weights

Deciding which optimisation to use isn't just about checking profitability (although we will check that in a second). We also want to see robust weights; stable, without too many zeros.
Let's focus on the forecast weights for the S&P 500 (not quite arbitrary example, this is a cheap instrument so can allocate to most of the trading rules. Looking at say instrument weights would result in a massive messy plot).
`system.combForecast.get_forecast_weights("SP500")`

 Forecast weights with handcrafting
Pretty sensible weights here, with ~35% in carry and the rest split between the other moving averages. There are some variations when the correlations shift instruments slightly between groups.

```# this will give us the final Portfolio object used for optimisation (change index -1 for others)
# See previous post in this series (https://qoppac.blogspot.com/2018/12/portfolio-construction-through_14.html)

portfolio=system.combForecast.calculation_of_raw_estimated_forecast_weights("SP500").results[-1].diag['hc_portfolio']

# eg to see the sub portfolio tree

portfolio.show_subportfolio_tree()
[' Contains 3 sub portfolios',
["[0] Contains ['ewmac16_64', 'ewmac32_128', 'ewmac64_256']"], # slow momentum
["[1] Contains ['ewmac2_8', 'ewmac4_16', 'ewmac8_32']"],  # fast momentum
["[2] Contains ['carry']"]]  # carry

```
Makes a lot of sense to me...
 Forecast weights with naive Markowitz
 The usual car crash you’d expect from Naive Markowitz, with lots of variation, and very unrobust weights (at the end it’s basically half and half between carry and the slowest momentum). @page { margin: 2cm } p { margin-bottom: 0.25cm; line-height: 120% } Forecast weights with shrinkage

Smooth and pretty sensible. This method downweights the faster moving averages a little more than the others; they are more expensive and also don't perform so well in equities.

 Forecast weights with bootstrapping
A lot noisier than shrinkage due to the randomness involved, but pretty sensible.

I haven't shown equal weights, as you can probably guess what those are.

Although I’m not graphing them, I thought it would be instructive to look at the final instrument weights for handcrafting:

`system.portfolio.get_instrument_weights().tail(1).transpose()`
```AEX        0.016341
AUD        0.024343
BOBL       0.050443
BTP        0.013316
BUND       0.013448
CAC        0.014476
COPPER     0.024385
CORN       0.031373
CRUDE_W    0.029685
EDOLLAR    0.007732
EUR        0.010737
EUROSTX    0.012372
GAS_US     0.031425
GBP        0.010737
GOLD       0.012900
JPY        0.012578
KOSPI      0.031301
KR10       0.051694
KR3        0.051694
LEANHOG    0.048684
LIVECOW    0.031426
MXP        0.028957
NASDAQ     0.034130
NZD        0.024343
OAT        0.014660
PLAT       0.009977
SHATZ      0.057006
SMI        0.040494
SOYBEAN    0.029706
SP500      0.033992
US10       0.005511
US2        0.031459
US20       0.022260
US5        0.007168
V2X        0.042326
VIX        0.042355
WHEAT      0.031373
```

`Let's summarise these:`
```
```
`Ags 17.2%`
`Bonds 31.8%`
`Energy 6.1%`
`Equities 18.3%`
`FX 11.1%`
`Metals 6.0%`
`STIR 0.77%`
`Vol 8.4%`

portfolio=system.portfolio.calculation_of_raw_instrument_weights().results[-1].diag['hc_portfolio']
```portfolio.show_subportfolio_tree()

[' Contains 3 sub portfolios', # bonds, equities, other
['[0] Contains 3 sub portfolios',     # bonds
["[0][0] Contains ['BOBL', 'SHATZ']"],  # german short bonds
["[0][1] Contains ['KR10', 'KR3']"], # korean bonds
['[0][2] Contains 3 sub portfolios', # other bonds
["[0][2][0] Contains ['BUND', 'OAT']"], # european 10 year bonds ex BTP
['[0][2][1] Contains 2 sub portfolios', # US medium and long bonds
["[0][2][1][0] Contains ['EDOLLAR', 'US10', 'US5']"], # us medium bonds
["[0][2][1][1] Contains ['US20']"]], # us long bond
["[0][2][2] Contains ['US2']"]]],  # us short bonds
['[1] Contains 3 sub portfolios', # equities and vol
['[1][0] Contains 2 sub portfolios', # European equities
["[1][0][0] Contains ['AEX', 'CAC', 'EUROSTX']"], # EU equities
["[1][0][1] Contains ['SMI']"]], # Swiss equities
["[1][1] Contains ['NASDAQ', 'SP500']"], # US equities
["[1][2] Contains ['V2X', 'VIX']"]], # US vol
['[2] Contains 3 sub portfolios',  # other
['[2][0] Contains 3 sub portfolios', # FX and metals
['[2][0][0] Contains 2 sub portfolios', # FX, mostly
["[2][0][0][0] Contains ['EUR', 'GBP']"],
["[2][0][0][1] Contains ['BTP', 'JPY']"]],
["[2][0][1] Contains ['AUD', 'NZD']"],
['[2][0][2] Contains 2 sub portfolios', # Metals
["[2][0][2][0] Contains ['GOLD', 'PALLAD', 'PLAT']"],
["[2][0][2][1] Contains ['COPPER']"]]],
['[2][1] Contains 2 sub portfolios', # letfovers
["[2][1][0] Contains ['KOSPI', 'MXP']"],
["[2][1][1] Contains ['GAS_US', 'LIVECOW']"]],
['[2][2] Contains 3 sub portfolios', # ags and crude
["[2][2][0] Contains ['CORN', 'WHEAT']"],
["[2][2][1] Contains ['CRUDE_W', 'SOYBEAN']"],
["[2][2][2] Contains ['LEANHOG']"]]]]

```
Some very interesting groupings there, mostly logical but a few unexpected (eg BTP, KOSPI). Also instructive to look at the smallest weights:
US10, US5, EDOLLAR, PLAT, EUR, GBP, EUROSTX (used to hedge), JPY
Those are markets I could potentially think about removing if I wanted to.

Evaluating the profits

As the figure shows the ranking of performance is as follows:

- Naive Markowitz, Sharpe Ratio (SR) 0.82
- Shrinkage, SR 0.96
- Bootstrap, SR 0.97
- Handcrafted SR 1.01
- Equal weighting SR 1.02
So naive is definitely sub optimal, but the others are pretty similar, with perhaps handcrafting and equal weights a fraction ahead of the rest. This is borne out by the T-statistics from doing pairwise comparisons between the various curves.

```boot = system.accounts.portfolio() ## populate the other values in the dict below appropriately
```
```results = dict(naive=oneperiodacc, hc=handcraft_acc, equal=equal_acc, shrink=shrink, boot=boot)

from syscore.accounting import account_test

types=results.keys()
for type1 in types:
for type2 in types:
if type1==type2:
continue        print("%s vs %s" % (type1, type2))
print(account_test(results[type1], results[type2]))```

A T-statistic of around 1.9 would hit the 5% critical value, and 2.3 is a 2% critical value):

Naive Shrink Boot   HC   Equal
Naive
Shrink   2.01
Boot     1.56   0.03
HC       2.31   0.81  0.58
Equal    2.33   0.97  0.93  0.19

Apart from bootstrapping, all the other methods handily beat naive with 5% significance. However the rest of the t-statistics are pretty indifferent.

Partly this is because I’ve constrained all the optimisations in similar ways so they don’t do anything too stupid; for example ignoring Sharpe Ratio when optimising over instrument weights. Changing this would mostly penalise the naive optimisation further, but probably wouldn't change things much elsewhere.

It’s always slightly depressing when equal weights beats more complicated methods, but this is partly a function of the data set. Everything is vol scaled, so there is no need to take volatility into account. The correlation structure is reasonably friendly:

• for instrument weights we have a pretty even set of instruments across different asset classes, so equal weighted and handcrafted aren’t going to be radically different,
• for forecast weights, handcrafting (and all the other methods) produce carry weights of between 30% and 40% for S&P 500, whilst equal weighting would give carry just 14%. However this difference won’t be as stark for other instruments which can only afford to trade 2 or 3 EWMAC crossovers.

Still there are many contexts in which equal weight wouldn't make sense

Incidentally the code for handcrafting runs pretty fast; only a few seconds slower than equal weights which of course is the fastest (not that pysystemtrade is especially quick... speeding it up is on my [long] to do list). Naive and bootstraps run a bit slower (as they are doing a single optimisation per time period), whilst bootstrap is slowest of all (as it’s doing 100 optimisations per time period).

## Conclusion

Handcrafting produces sensible and reasonably stable weights, and it's out of sample performance is about as good as more complicated methods. The test for handcrafting was not to produce superior out of sample performance. All we needed was performance that was indistinguishable from more complex methods. I feel that it has passed this test with flying colours, albeit on just this one data set.

So if I review the original motivation for producing this method:

• - Humans can trust it; intuitive and transparent method which produces robust weights (yes, confirmed in this post)
• - Can be easily implemented by a human in a spreadsheet (yes, see post 3)
• - Can be back tested (yes, confirmed in this post)
• - Grounded in solid theoretical foundations (yes, see post 2)
• - Takes account of uncertainty (yes, see post 2)
• - Decent out of sample performance (yes, confirmed in this post)

We can see that there is a clear tick in each category. I’m pretty happy with how this test has turned out, and I will be switching the default method for optimisation in pysystemtrade to use handcrafting.