Wednesday, 17 April 2019

Trading and investing performance - year five

Hard to believe, but it has been five and a half years since I had to go to an office to manage other peoples money, and exactly five years since I began systematically trading my own. Time then for another annual review. Perhaps it is confusing for overseas readers, but these reviews follow the UK tax year which runs from 6th April to 5th April, rather than any logical period like a calendar year (or even 1st to April to 31st March would make more sense, frankly).

Previous updates can be found hereherehere and here.

Overview of my world

My investments fall into the following categories:

  • In my investment accounts:
    • 1 UK stocks
    • 2 Various ETFs, covering stocks, bonds, gold and property
    • 3 Usually some uninvested cash
  • In my trading account:
    • 4 Various ETFs, covering stocks and bonds
    • 5 A futures contract hedge against those long only ETFs in 2.1, so that the net Beta is around zero
    • 6 Futures contracts traded by my fully automated trading system
    • 7 Cash needed for futures margin, and to cover potential trading losses (there is also some cash in my investment accounts, but it's pretty much a rounding error)

Excluded from this analysis is:

  • Net property equity in my PPR (primary private residence, i.e. the bricks and mortar I am currently writing this in)
  • My 'cash float', roughly 6 months of household expenditure that is kept separately from my investment and trading accounts. This is used to smooth out lumpy income arriving from multiple sources and means I can sleep easily at night.

For the purposes of benchmarking it makes most sense to lump my investments in the following way:

  • A: UK single stocks
    • Benchmarked against ISF, a cheap FTSE 100 ETF (FTSE 350 is probably a better benchmark but these ETFs tend to be more expensive).
  • B: Long only investments: All ETFs (in both investment and trading accounts) and UK stocks
    • Benchmarked against a cheap 60:40 fund. This is the type of top down asset allocation portfolio I deal with in my second book.
  • C: Equity neutral: The ETFs in my trading account, plus the equity hedge. 
    • Benchmark is zero.
  • D: Futures trading: Return from the futures contracts traded by my fully automated system. This is the type of portfolio I deal with in chapter 15 of my first book. The denominator of performance here is the notional capital at risk in my account (usually close to, but not exactly the same as the account value).
    • Benchmarks are a similar fund run by my ex employers, or the SG CTA index, adjusted for volatility.
  • E: Trading account value: This is essentially everything in my trading account, and consists of equity neutral + futures trading. 
    • No relevant benchmark.
  • F: Everything: Long only investments, plus futures hedge, plus futures trading. I include the value of any cash included in my trading or investment accounts, since if I wasn't trading I could invest this. 
    • For the benchmark here again I use a cheap 60:40 fund.

If you prefer maths, then the relationship to the first set of categories is:

A = 1
B = 1 + 2 + 4
C = 4 + 5
D = 6 + 7
E = 4 + 5 + 6 + 7 = C + D
F = 1 + 2 + 3 + 4 + 5 + 6 + 7 = B + 3 + 5 + D

Performance contribution

The figures shown are the contribution of each category to my total investment performance (on an XIRR basis):

A, or 1) UK equities -0.45%
2) ETFs +4.35%
B) Long only investments +3.9%
C) Equity hedge -0.2%
D) Systematic futures trading +1.0%
E) Trading account: +0.8%
F) Total 4.4% (won't add up exactly because of XIRR effects)

UK Equities

This is effectively a 'trading' portfolio, using a mechanical system described here. Since I published that post I have added a new twist in that I enforce sector diversification.

Mostly it's held inside SIPP and ISAs, to avoid paying CGT. There are also a couple of legacy stocks with larger positions, which are held outside tax shelters. I have been gradually reducing their position tactically over time.

Start of the year:

ICP 18.8% (legacy)
STOB 17.1% (legacy)
BKG 10.4%
VSVS 9.5%
RMG 8.9%
LGEN 7.6%
GOG 7.5%
HSBA 7.4%
IBST 6.9%
BP 6.0%

I sold some STOB to crystallise a CGT loss, and all the other trades were mechanical:

Bought and sold Babcock, for a 26% loss.
Sold RMG for a 31% loss
Sold IBST for a 20% loss
Bought CEY
Bought PTEC

So now the portfolio looks like this:

ICP 20.98%
VSVS 10.79%
BKG 10.59%
CEY 8.91%
STOB 8.76%
GOOG 8.69%
PTEC 8.66%
LGEN 8.47%
HSBA 7.21%
BP 6.94%

Although the yield was 4.4% the total return was negative; an XIRR of -2.3% to be precise. This compares badly with the FTSE 100 tracker I use as my benchmark coming in at 7.6%. Looking back over the years since I started doing being this anal about my performance, this is the first year I've underperformed. Apart from the losers I sold Stobart also underperformed (though it's still up 100% from my original purchase price), and both CEY and PTEC have lost money since I bought them. On the upside BP and GOOG (which is Go-ahead, not Google in case you didn't know) both earned around 20% over the year.

I have been a net seller of UK equities for many years now, but my allocation is now about right. I reinvested the dividends earned from all UK stocks back into the UK, but they are still a smaller proportion of my portfolio due to the relatively poor performance. I'll return to questions of portfolio allocation later.

ETFs and funds

All my non UK and non equity exposure is in ETFs, with a smattering of investment trusts. As usual trading was done for tax optimisation, to generate funds for SIPP and ISA Investment, and to get the right risk exposure (discussed later). I don't look at the performance of my ETF portfolio seperately, only in conjuction with UK shares.

Long only

Performance here was better, but still not fantastic. Dividends on the whole piece were identical to the UK share, with a yield on starting value of 4.4%. The aggregate XIRR was 4% exactly, hence a small capital loss was made. This under performed a benchmark Vanguard 60:40 fund, which came in at 7.2%.

Systematic futures trading and equity hedge

The systematic futures trading system I run is effectively what you can in find in "Systematic Trading", and code-wise is an older version of the implementation in pysystemtrade.

For my trading account as a whole the breakdown looks like this (all numbers are as a % of the notional maximum capital at risk, which happens to be a bit less than the trading account value):

Hedging futures: -1.1%
Hedged stocks, total return: +2.0%
Net equity neutral: +1.0%

Futures trading-
MTM: 5.7%
Interest: 0.08%
Fees: -0.04%
Commissions: -0.6%
FX gain/loss: +0.02%
Net futures trading: +5.2%

Grand total: +6.1%

Some more statistics (futures trades only):

Profit factor: 1.0
Win/loss ratio: 1.31
Profit factor: 1.0
Hit rate: 43.2%
Avg holding period, winners: 39 days
Avg holding period, losers: 29 days

So is this good or bad? Let's look at benchmarks.

 'Bench1' is this AHL fund, using monthly returns from April to March in each year, and a new benchmark 'Bench2' is the SG CTA index, with matching daily returns. Both have returns scaled up to match my volatility. Remember the benchmark should only be compared against futures trading, not the equity neutral component of the portfolio.

Year:    14/15   15/16   16/17   17/18  18/19

Total:   57.2%   39.6%    0.3%    0.4%   6.1%
Hedge:   -1.1%   16.3%   14.4%    4.1%   1.0%  
Futures: 58.2%   23.2%  -14.0%   -3.7%   5.2%

Bench1: 106.9%  -10.6%   -6.2%   16.4%  10.3%
Bench2:          -6.7%* -21.9%   -3.8%   0.7%

* From 13th April 2015

A very good performance from 'Bench1' here, but against the entire industry (bench2) my performance is more respectable.

You will be used to seeing pretty pictures and more detailed analysis here. Unfortunately I couldn't run my normal annual performance attribution in the increasingly creaky old legacy code that my system runs on (after all it's five years old now). I am gradually building up to replacing this with pysystemtrade, however this is a slow process (interrupted by the new book I am finishing off, plus various random side projects like writing this blog post).

Total investment return

My total return on all my investments, including cash held for futures margin, came in at an XIRR of 4.4%. Once again Vanguard 60:40 seems an appropriate benchmark (since if I wasn't trading futures I could throw all my cash into that fund), at 7.2%. All in all then a slightly disappointing year, although unlike the last two years my futures trading added to rather than detracted from my performance.


My investment portfolio asset allocation at the end of the year in cash terms stood at 22.7% in bonds, 65.1% in equities, 9.5% in cash and 2.7% in other (Gold and commercial property).

This is more cash than I normally have. About 2.6% (of the 9.5%) was transferred to ISAs and will be invested shortly. I am toying with the idea of starting another little portfolio which will invest in investment trusts using some simple filters around discount and yield. I also have a little more than usual in my current account, and in my trading account. In practical terms this just means I have more of a cushion against the rather lumpy arrival of income in the form of dividends and royalties.

My allocation in my preferred risk weighted terms looks like this:

|Asset    |Strategic|Start of year|Current|
|Bonds    |   22%   |   13.1%     | 12.0% |
|Equity   |   50%   |   59.4%     | 60.7% |
|Futures  |   25%   |   24.5%     | 24.7% |
|Other    |    3%   |    2.9%     |  2.5% |

As usual the lower allocation to bonds reflects a mechanical trading rule that uses the last 12 months of relative risk adjusted performance; bonds are down around -0.5% and equities up 5.5%. See the relevant chapters of "Smart Portfolios" for more detail.

Regional exposures, should you care (each row adds up to 100%):

|      | Asia | EM   | Euro |  UK   |  US  |
|Bonds | 0%   | 22%  | 19% |  18%  |  41% |
|Equity| 26%  | 27%  | 17% |  27%  |   3% |

The 0% weight in Asia reflects a lack of decent bond ETFs, whilst the 3% in US equities is because they are frightfully expensive (again there are mechanical rules lurking behind these numbers, this time based on relative valuation metrics: dividend yield and PE ratios).

Thoughts and plans

There are enough numbers in these posts that there will always be good news (relative futures trading, equity hedge) and bad news (everything else). Naturally none of it is statistically significant. So there are no conclusions to draw in the shape of "Oh my gawd, this is awful, I need to change everything right now".

But gradual improvement in a sensible way is fine. I haven't yet got round to implementing any serious changes to my futures trading (waiting for my code migration to finish first), but in the mean time I've started a very small portfolio which buys investment trusts trading at a large discount and with large yield, with some minimum liquidity requirements. So far this constitutes PGIT and VSL.

Wednesday, 27 March 2019

Risk targeting and dynamic asset allocation: absolute or relative momentum?

Quite a few of my recent blog pieces have been picked up by the lovely folk at allocate smartly. So I thought I'd write an asset allocation piece, as the readers of my second book "Smart Portfolios" probably feel neglected with the lack of articles on investment rather than trading.

Absolute or relative momentum?

The motivation for this comes from a table in my second book, which exposes an interesting problem. Here is the table (actually a slightly modified version of it, so you won't recognise the precise numbers), and I'll explain what it means and what the problem is:

                   Arithmetic mean  Geometric mean   Std. Deviation   Sharpe Ratio

Fixed weight            8.37%         8.04%             8.14%             1.03
Relative momentum       9.26%         8.89%             8.62%             1.07
Absolute momentum       8.93%         8.61%             7.96%             1.12

Fixed weight: This is a portfolio with 75:25 risk weightings in US equities and US bonds (using the last 12 months of monthly returns to calculate the appropriate volatility for risk weighting; this works out to roughly 60:40 cash weightings on average).

Relative momentum: This portfolio tactically rebalances the strategic fixed weights using the relative 12 month total risk adjusted return of equities and bonds. The rebalancing is a 'tilt' to account for forecasting uncertainty; the maximum tilt is to 148% of the original portfolio weight, and the minimum is 60% of the original. The relative momentum portfolio is always fully invested.

Absolute momentum: This portfolio tactically rebalances the strategic fixed weights according to the absolute 12 month total risk adjusted return of equities and bonds, again using a 'tilt'. The absolute momentum portfolio may not be fully invested if momentum is relatively weak in one or both assets. The minimum investment is 60% (which is not unusual), and the average is 93%.

All portfolios are rebalanced monthly, using data from January 1954 to March 2016 (I could update this, but I wanted to use the same data as in the book, and it wouldn't affect the results much). Returns shown are excess returns, net of the risk free rate.

Let's do some basic analysis of these results. Using absolute momentum results in a slightly higher risk than for fixed weights, because equities have spent more time going up in a risk adjusted sense. I call this the historical volatility boost. That more than compensates for the fact we aren't always fully invested, which drags down risk. The average cash weight to equities is 67% versus the 61% under fixed weights. But the extra risk is well rewarded with a higher arithmetic and geometric return, and a higher Sharpe Ratio.

Absolute momentum is a super popular asset allocation methodology, because people like the 'downside protection' of being partly in cash when markets are selling off.

Relative momentum has even higher risk; again it has a systematic bias towards equities and a historical volatility boost, but because we are always fully invested that all hits the 'bottom line' in the form of higher risk. The average cash weight to equities is 70%. The extra risk is rewarded with a higher arithmetic and geometric mean return, but the Sharpe Ratio is actually lower than for absolute momentum (though still better than fixed weights).

Relative momentum is less popular amongst the general public, as it seems hard to justify a big allocation to bonds just because they aren't falling quite as fast as equities. It also has a worse Sharpe Ratio, so 'theoretically' it's inferior (if you're an investor who can use leverage).

In my book I rather blandly concluded that relative momentum was better due to the higher geometric mean.

However, we're not comparing like with like. Strictly speaking we should probably compare relative momentum with an absolute version that has a higher strategic allocation to equities, so that their risk levels are comparable. To put it another way: is it better* to use relative momentum, or to use absolute momentum and crank up your strategic risk target to compensate for the reduction in risk?

Already we can see that this is a variation of the classic dilemma that investors without access to leverage and high risk tolerance have: should I opt for the highest Sharpe Ratio, or for something with higher risk (and return) but a lower Sharpe Ratio? However this story is more complicated, because we have two moving parts: the original risk weights, and the choice of rebalancing strategy (fixed weights, absolute, or relative). The interaction of these will produce portfolios with different return and risk profiles.

* The dilemma would be the same** for any type of forecast, but momentum is a popular and well understood rule to establish conditional returns.

** Strictly speaking the idea of an 'absolute' forecast requires some kind of equilibrium value at which we have a zero position. So dividend yield as a forecast wouldn't be helpful for absolute weighting, but something like (divided yield - interest rates)*** would make sense. 

*** the 'Fed model'

The experiment

The general question we want to answer is: For a given risk tolerance, what is the best choice of strategic risk weights and rebalancing strategy?

My criteria will be to judge a particular outcome by looking at the geometric mean (my reasons for choosing that are documented here), and the standard deviation of returns.

The range of strategic risk weights I will consider are from 10% equities 90% bonds, up to 90% equities 10% bonds. All strategic risk weight portfolios will be fully invested. Note that people with really low risk appetites will be best served by the maximum Sharpe Ratio portfolio plus a cash allocation; however I won't consider that option here. After all the problem we are exploring is most acute for investors with higher risk appetites.

To make the results starker, I'm going to allow the two tactical portfolios to 'tilt' all the way from 10% to 200% of the original strategic weight. Obviously this won't affect the fixed weights. For less aggressive tilts the relative results will be the same, but the numbers will be closer together.

First let's look at the Sharpe Ratios:

The black line is what you'd expect; the maximum SR portfolio is roughly 50:50 in equities and bonds. Absolute momentum is mostly inferior to the other options except for relatively high allocations to equities. Relative momentum shows declining performance as we increase the risk weight.

However these differences in SR might not be significant (I'll discuss this later in the post), but more importantly 'we can't eat Sharpe Ratios' if we're not leveraged investors, so let's instead focus on the geometric means and standard deviations.

Each line shows a classic 'efficient frontier', with one line for fixed weights, one for relative weights, and one for absolute weights. Each cross is a different strategic allocation, in 10% steps. So the first black cross on the bottom end of the fixed weights line is 10% risk weight in equities, the next cross is 20% in equities, and so on up to 90% on the top right end of the line.

We can safely ignore all the portfolios with lower risk than 30% equities; for these we'd be better off mixing the maximum Sharpe Ratio portfolio with cash.

It's clear from this graph that the out performance of relative momentum is pretty consistent. For a given risk target relative momentum is better than fixed weights or absolute momentum. It also looks like there is no benefit from using a risk target of greater than 80% in equities.

Which strategic portfolio weights should we use?

There is an important question that is not easily answered by the graphs above: how much should I adjust my strategic weights to compensate for the effect of applying a relative or absolute momentum tactical weight?

So, for example, if you want a standard deviation of 8% you could use:

  • a fixed risk weight of ~75% to equities
  • tactical absolute weighting with a strategic risk weight of ~68% to equities 
  • tactical relative weighting with a strategic risk weight of ~40% to equities

That is some substantial difference!

How robust are these results?

First let's consider the differences in geometric means. I'm extremely confident that 12 month momentum is a robust effect that has existed in the past, though we can argue about whether it will continue in the future. So I'd expect both types of momentum to beat fixed weights.

What about the out performance of relative momentum? Cross sectional momentum across asset classes is a less popular idea (though super popular within asset classes e.g. across stocks), but it would be surprising if there was a substantial difference between the two types of forecast.

However in a long only portfolio absolute momentum is operating with one hand tied behind it's back, as it cannot go short. This might explain the relatively poor performance of absolute momentum. Even when it has a slightly higher Sharpe Ratio (for relatively high equity weightings), the reduction in volatility means that absolute momentum can't compete on a geometric mean basis.

What about the differences in standard deviations? By construction the standard deviation for absolute momentum will always be lower than that for relative momentum.

The reasons for the increase in standard deviation when using relative momentum is less robust (risk also rises for absolute momentum, except for very low or very high equity allocations). In theory, if both equities and bonds had the same average forecast going forward, then the standard deviation would be the same for relative momentum as it is for fixed weights.

Radically reducing your strategic weight to equities to compensate for the expected 'volatility boost' from your tactical overlay might not be wise. The existence of a risk boost is probably the least robust finding here - I wouldn't be 100% sure it will exist in the future.


I'm reasonably happy that my superficial analysis in "Smart Portfolios" was correct when put through a more thorough test: relative momentum gives a higher geometric mean than absolute momentum, except for investors with low tolerance to risk. Therefore for most investors it's preferable.

In terms of more specific advice, the graphs above suggest the optimal portfolios are:

  • If you can use leverage, the highest Sharpe Ratio comes from using relative momentum tactical weighting with a risk weight to equities of somewhere between 30% (15:85 equity/bonds in cash weights based on current vols) and 50% (30:70 equity/bonds in cash weights). Within that range I'd err towards a higher weight in equities in case the 'risk boosting' that occurred in the past is absent. Recommend: Strategic risk weights 50% equity 50% bond, cash weights 30% equity 70% bonds, relative momentum tactical weighting.

  • If you can't use leverage and have a high risk tolerance, the highest geometric mean comes from using  relative momentum tactical weighting with a risk weight to equities of somewhere between 60% (40:60 in cash weights based on current vols) and 90% (80:20 in cash weights).  Within that range I'd err towards a higher weight in equities in case the 'risk boosting' that occurred in the past is absent. Recommend: Strategic risk weights 90% equity 10% bond, cash weights 80% equity 20% bonds, relative momentum tactical weighting.

  • If you can't use leverage and have a modest risk tolerance (but higher than 8% standard deviation a year): I'd use relative momentum but with a lower risk weight. If you don't buy the 'risk boosting' story then you will need between 60% and 90% risk weighting in equities; if you do buy the story and believe history will repeat itself, between 30% and 60% risk in equities. Recommend: Strategic risk weights 60% equity 40% bond, cash weights 40% equity 60% bonds, relative momentum tactical weighting.

  • If you can't use leverage and have a low risk tolerance (lower than 8% standard deviation a year): I'd invest in the maximum Sharpe Ratio portfolio (see above), and blend it with cash.

Thursday, 28 February 2019

Skew and Trend following

In this post I discuss a well known stylised fact of the investment industry: "Trend following is a positively skewed strategy".

Spoiler alert: yes it is (sort of), but it's much more complicated (and interesting!) than you might think.

A quick primer on positive skew

So what actually is positive skew? Essentially it's an asset, or trading strategy, whose returns have the following profile:
  • A high proportion of relatively poor returns
  • The losing returns are smaller in magnitude than the winning returns
Or... if you prefer a pretty picture:

Or, if you prefer maths, then the skew is the third moment of the statistical distribution, and positive skew means there is more skew relative to a Gaussian normal distribution which has zero skew.

It's generally felt that positive skew is a good thing, and people are generally willing to pay a premium for owning assets with positive skew (and vice versa for negative skew) [where by 'a premium', I mean the assets have a higher risk adjusted return than you would expect when risk is measured purely by the second moment - standard deviation]. 

A coherent explanation of this comes from behavioural finance, and specifically prospect theory. A cognitive bias results in people overweighting the chances of low probability outcomes. They get fixated on the small chance of a large gain that positive skew offers. Equally with negative skew people get scared of the small chance of large losses which are threatened by negative skew.

(It might also be worth reading this paper, written by a bunch of people I used to work with, and some other people I haven't worked with).

In fact, we can check to see if we get paid for skew. If I look at the skew over the last 3 months of daily returns, and see how well that predicts the next 3 months annualised Sharpe Ratio, then I find that with negative skew the average SR is +0.33. With positive skew it's -0.016. The difference is statistically significant; if I do a regression the p-value is 0.01.

(At this point you might be thinking 'ah-ha! I can use skew as a predictor in a trading strategy. I will be rich!' .This is not an original idea! See for example, this paper.)

Intuitively, why should trend following produce positively skewed returns?

Trend following is effectively like buying a synthetic straddle* (a combination of long put and call options). This is a well known and fairly old result (see the seminal Fung and Hsieh 2001). Intuitively this makes sense, since both strategies will do well if volatility rises, and do badly if prices remain pinned. It's equally well known that any long volatility strategy, like buying straddles, should produce positively skewed returns: a lot of small negative returns when prices don't move and we hand over our premium, plus a smaller number of large positive returns when prices move enough for one leg of our straddle to be in the money.

* actually it's a look back straddle, but the distinction isn't important here.

So, positive skew is definitely one of the reasons why people like to allocate to trend following strategies, the others being:
  • Linear diversification; low correlation with traditional asset classes
  • Non linear diversification; good performance in tail events like 2008 (if you're from a fixed income background like yours truly, you can also think of this as 'positive convexity')
  • They sometimes even make money! 

However trend following also has it's problems. People don't like the long drawdowns that trend following type strategies produce, but these are an inevitable consequence of positive skew (for a given risk adjusted return the size of the average drawdown will be higher than for positively skewed assets). For example, suppose you're trading a strategy with a Sharpe Ratio of 0.5 and an annual risk target of 25%. With zero skew a bad drawdown (one that is achieved 10% of the time) will be 9.3% in magnitude. With positive skew that would rise to 11%, and would be just 3.7% with negative skew (skews of +1 and -2 respectively).

If trend following generates positive returns (and there is no clear evidence it has stopped doing so) then people must be more scared of those ugly drawdowns than they are of the advantages I've listed above. But (spoiler alert!) there might be something else going on.

The evidence

Economists and quant finance 'professionals' often pretend to be scientists (many of them have actual Phds in actual scientific subjects). So, let's pretend to be scientists and actually check to see if the evidence supports our expectations.

I'm going to use three types of trend following trading rule: a 2,8 day EWMAC; all the way up to a 64,256 day EWMAC (Exponentially weighted moving average crossover). Finally the results will be calculated over the 40ish futures contracts in my dataset. The whole thing is being done under the auspices of pysystemtrade, and you can find the usual ugly code here.

* actually 2,8 is actually a bit expensive to trade, but costs don't affect the calculation of skew since they just shift the distribution of returns to the left a bit.

For reasons that will become obvious I'm going to measure skew over different time periods: daily, weekly, monthly, and annual returns.

Let's start with the daily returns

Skew by trading rule, daily returns

Let me explain these plots. The y-axis is the measurement of skew, and the x-axis is the fast parameter value in the moving average pair (2,4,8,... 64). Each dot represents the skew measurement for a single instrument, and for a single 5 year period. This gives an indication of the uncertainty in our skew estimate (yes...I can't stop banging on about uncertainty).

Here are the median values for each rule:

2_8 = -0.04, 4_16 = -0.07, 8_32 = -0.51, 16_64 = -0.73, 32_128 = -0.94, 64_256 = -0.82

So... WTF?! Negative skew across the board, with significantly negative values for the slower crossovers. Something weird going on here.

Let's check the other time periods out:

Skew by trading rule, weekly returns
Skew by trading rule, monthly returns
Skew by trading rule, annual returns
Interesting. It looks like for bigger time periods the estimate of skew does indeed become positive. We can see this if we plot the median values for each rule, by time period:

Skew of a trend following rules profits, measured at different time horizons, from left to right: daily, weekly, monthly, annual
The results run from (on the left) daily, to (on the right) annual. Generally, skew gets more positive the slower the time period we use. The exception to this are the very fastest trading rules, which have a 'sweet spot' for skew at the monthly time period.

The puzzle

Does it make sense that positive skew only appears at certain frequencies of measurement, with a more infrequent measurement required for slower trading strategies? Yes, it does. Think about a fairly slow trend following rule. Maybe it changes it's positions every few months. When it is not changing it's positions, then it's skew of daily returns will be dictated by the skew of the underlying assets. 

So if it's trend following say equities (negative skew), then half the time you'd expect to see negative skew of (when it's long), and half the time (when it's short) you'd see positive skew. Overall your skew will be zero (and this result should hold for positive skew assets as well).

However if you start looking at annual returns, you're more likely to see the characteristically positive skew of trend following. The point at which the skew becomes significantly positive will depend on the speed of the trend following rule. With the faster rules we see positive skew with weekly and monthly returns; with the slower rules it isn't until we get to annual returns that the positive skew reveals itself.

(This is not an original finding. See this, written by someone else I used to work with)

But... that doesn't explain one thing. Why is the skew strongly negative at the shorter time frames? It should be zero, or close to it.

The only explanation is that trend following strategies like to be long negatively skewed assets, and short positively skewed assets

This is kind of interesting (well I think it is!). Perhaps the positive returns of trend following (a 'positively skewed' trading strategy) aren't that surprising at all, if it actually loads on to negatively skewed assets. Perhaps trend following is just a way of collecting the negative skew premium.

And... thinking some more... it sort of makes sense. If negative skew assets earn a premium in the market, then on average they will go up more often than they go down. And assets which go up more often than they go down, will tend to exhibit more bullish trends. And assets which exhibit more bullish trends, well they will be bought by trend following strategies.

This is all assuming that negative skew assets are negative before we buy them, and remain so. I will check this in a second.

What is the conditional relationship between skew and trend following

Let's do the following exercise. We'll find out the median skew, conditional on a trading rule being long or short, for a given trading rule.  I'm going to measure the skew over a period of a month, using daily returns.

First, let's look at the skew of a given instrument in the month after a trading rule has taken it's position. Remember its this skew that matters in determining what the skew of the returns of a trading rule will be (at least for the slower rules, which will 'inherit' the skew of the underlying asset).

Skew in the month after a trading rule takes it's position; conditioned on the trading rule being short (left hand side) or long (right hand side)

If a trading rule is long, then in the month following the forecast being made the skew is negative. If the rule is short, then the skew is closer to zero, or even positive. The effect is more noticeable for slower rules (faster rules will have changed their position during the following month anyway, perhaps multiple times).
This is a confirmation of our earlier intuition that slower trend following rules are likely to have negative skewed returns, because when they are long the underlying asset is negatively skewed; and when they are short the underlying asset is positively skewed (giving the strategy the opposite: more negatively skewed returns).

Now we look at the skew in the month before the trading rule decides what position it is taking:

Skew in the month before a trading rule takes it's position; conditioned on the trading rule being short (left hand side) or long (right hand side)
Now this is interesting. The slowest moving average does what we'd expect; it tends to be short when skew has been positive (or at least less negative), and goes long when skew has recently been negative. This confirms the theory that we end up loading up on negative skew as trend followers because negatively skewed assets are more likely to have positive drift (as a reward for that awful skew).

But for all the other trading rules we get the opposite effect*! For them the story is very weird: if skew has recently been negative they go short. But (from the previous graph), they then end up being short assets which subsequently have positive skew (which gives the trading  strategy negatively skewed returns). The skew flips sign. 

* in truth the penultimately slow trading rule is sort of flat.

If skew has been positive, the rule goes long, and then the underlying asset has negative skew (which again gives the trading strategy negatively skewed returns).

What is going on here? One possible explanation is this; for risky assets strongly negative skew usually appears after a sharp sell off. After such a sell off most trading rules will go short. But skew, like volatility at the right time horizon, is a mean reverting parameter. The trading rule starts with the skew the 'right' way round for generating positive skew (it goes short recent negative skew, and long recent positive skew) but then the sign of skew flips, and it ends up with exactly the wrong position!

The slowest moving average isn't affected by this; instead it's more likely to pick up the secular positive drift from negatively skewed assets.


Trend following rules do indeed have the positive skew you'd expect... but only at the right time horizon. For slower trend following rules you don't see them appear until you are using annual returns. At shorter time horizons they have persistently negative skew.

An asset which is negatively skewed at one time horizon, and positively skewed at another is... weird. Should we want to own it? I guess it depends on our own 'investment horizon'. If you only look at annual returns, you're going to love trend following! If you look at more frequent returns... you'll be less impressed. Given the long drawdowns of trend following strategies, you would be best off looking at your portfolio every 20 years or so :-)

For the slowest trend following rule I use it looks like this occurs because negatively skewed assets have a return premium, which leads to positive drift. So slow trend following rules will have a secular long bias to negatively skewed assets.

For other trend following rules this explanation is wrong. Instead, they tend to short assets whose skew has recently gone negative, and vice versa. It seems likely this is due to sharp selloffs in risky assets creating both negative skew and bearish recent trends. However skew is mean reverting; so the other rules end up being short assets which subsequently have positive skew, and vice versa.

This also means that if you're planning to use negative skew as a trading signal in combination with trend following, it will be a great diversifier! Except for the slowest moving average crossover, the momentum rule will usually do the opposite to a skew trading rule: it will short negative skewed assets, and go long positively skewed assets.

Saturday, 9 February 2019

Portfolio construction through handcrafting: Empirical tests

This post is all about handcrafting; a method for doing portfolio construction which human beings can do without computing power, or at least with a spreadsheet. The method aims to achieve the following goals:
  • Humans can trust it: intuitive and transparent method which produces robust weights
  • Can be easily implemented by a human in a spreadsheet
  • Can be back tested
  • Grounded in solid theoretical foundations
  • Takes account of uncertainty in data estimates
  • Decent out of sample performance
  • Addresses the problem of allocating capital to assets on a long only basis, or to trading strategies. 
This is the final post in a series on the handcrafting method.
  1. The first post can be found here, and it motivates the need for a method like this.
  2. In the second post I build up the various components of the method, and discuss why they are needed. 
  3. In the third post, I explained how you'd actually apply the method step by step, with code. 
  4. This post will test the method with real data, addressing the question of robust weights and out of sample performance
The testing will be done using psystemtrade. If you want to follow along, get the latest version.

PS apologies for the weird formatting in this post. It's out of my hands...

The Test Data

The test data is the 37 futures instruments in my usual data set, with the following trading rules:
  • Carry
  • Exponentially weighted moving average crossover (EWMAC) 2 day versus 8 day
  • EWMAC 4,16
  • EWMAC 8,32
  • EWMAC 16,64
  • EWMAC 32,128
  • EWMAC 64,256

I'll be using handcrafting to calculate both the forecast and instrument weights. By the way, this isn't a very stern test of the volatility scaling, since everything is assumed to have the same volatility in a trading system. Feel free to test it with your own data.
The handcrafting code lives here (you've mostly seen this before in a previous post, just some slight changes to deal with assets that don't have enough data) with a calling function added here in my existing optimisation code (which is littered with #FIXME NEEDS REFACTORING comments, but this isn't the time or the place...).

The Competition

I will be comparing the handcrafted method to the methods already coded up in pysystemtrade, namely:
  • Naive Markowitz
  • Bootstrapping
  • Shrinkage
  • Equal weights
All the configuration options for each optimiser will be the default for pysystemtrade (you might want to read this). All optimisation will be done on an 'expanding window' out of sample basis.
from systems.provided.futures_chapter15.estimatedsystem import *
system = futures_system()
system.config.forecast_weight_estimate['method']='handcraft' # change as appropriate
system.config.instrument_weight_estimate['method']='handcraft'  # change as appropriate


Evaluating the weights

Deciding which optimisation to use isn't just about checking profitability (although we will check that in a second). We also want to see robust weights; stable, without too many zeros.
Let's focus on the forecast weights for the S&P 500 (not quite arbitrary example, this is a cheap instrument so can allocate to most of the trading rules. Looking at say instrument weights would result in a massive messy plot).

Forecast weights with handcrafting
Pretty sensible weights here, with ~35% in carry and the rest split between the other moving averages. There are some variations when the correlations shift instruments slightly between groups.

# this will give us the final Portfolio object used for optimisation (change index -1 for others)
# See previous post in this series (


# eg to see the sub portfolio tree

[' Contains 3 sub portfolios',
 ["[0] Contains ['ewmac16_64', 'ewmac32_128', 'ewmac64_256']"], # slow momentum
 ["[1] Contains ['ewmac2_8', 'ewmac4_16', 'ewmac8_32']"],  # fast momentum
 ["[2] Contains ['carry']"]]  # carry
Makes a lot of sense to me...
Forecast weights with naive Markowitz
The usual car crash you’d expect from Naive Markowitz, with lots of variation, and very unrobust weights (at the end it’s basically half and half between carry and the slowest momentum).

Forecast weights with shrinkage

Smooth and pretty sensible. This method downweights the faster moving averages a little more than the others; they are more expensive and also don't perform so well in equities.

Forecast weights with bootstrapping
A lot noisier than shrinkage due to the randomness involved, but pretty sensible.

I haven't shown equal weights, as you can probably guess what those are.

Although I’m not graphing them, I thought it would be instructive to look at the final instrument weights for handcrafting:

AEX        0.016341
AUD        0.024343
BOBL       0.050443
BTP        0.013316
BUND       0.013448
CAC        0.014476
COPPER     0.024385
CORN       0.031373
CRUDE_W    0.029685
EDOLLAR    0.007732
EUR        0.010737
EUROSTX    0.012372
GAS_US     0.031425
GBP        0.010737
GOLD       0.012900
JPY        0.012578
KOSPI      0.031301
KR10       0.051694
KR3        0.051694
LEANHOG    0.048684
LIVECOW    0.031426
MXP        0.028957
NASDAQ     0.034130
NZD        0.024343
OAT        0.014660
PALLAD     0.013194
PLAT       0.009977
SHATZ      0.057006
SMI        0.040494
SOYBEAN    0.029706
SP500      0.033992
US10       0.005511
US2        0.031459
US20       0.022260
US5        0.007168
V2X        0.042326
VIX        0.042355
WHEAT      0.031373

Let's summarise these:

Ags 17.2%
Bonds 31.8%
Energy 6.1%
Equities 18.3%
FX 11.1%
Metals 6.0%
STIR 0.77%
Vol 8.4%

[' Contains 3 sub portfolios', # bonds, equities, other ['[0] Contains 3 sub portfolios', # bonds ["[0][0] Contains ['BOBL', 'SHATZ']"], # german short bonds ["[0][1] Contains ['KR10', 'KR3']"], # korean bonds ['[0][2] Contains 3 sub portfolios', # other bonds ["[0][2][0] Contains ['BUND', 'OAT']"], # european 10 year bonds ex BTP ['[0][2][1] Contains 2 sub portfolios', # US medium and long bonds ["[0][2][1][0] Contains ['EDOLLAR', 'US10', 'US5']"], # us medium bonds ["[0][2][1][1] Contains ['US20']"]], # us long bond ["[0][2][2] Contains ['US2']"]]], # us short bonds ['[1] Contains 3 sub portfolios', # equities and vol ['[1][0] Contains 2 sub portfolios', # European equities ["[1][0][0] Contains ['AEX', 'CAC', 'EUROSTX']"], # EU equities ["[1][0][1] Contains ['SMI']"]], # Swiss equities ["[1][1] Contains ['NASDAQ', 'SP500']"], # US equities ["[1][2] Contains ['V2X', 'VIX']"]], # US vol ['[2] Contains 3 sub portfolios', # other ['[2][0] Contains 3 sub portfolios', # FX and metals ['[2][0][0] Contains 2 sub portfolios', # FX, mostly ["[2][0][0][0] Contains ['EUR', 'GBP']"], ["[2][0][0][1] Contains ['BTP', 'JPY']"]], ["[2][0][1] Contains ['AUD', 'NZD']"], ['[2][0][2] Contains 2 sub portfolios', # Metals ["[2][0][2][0] Contains ['GOLD', 'PALLAD', 'PLAT']"], ["[2][0][2][1] Contains ['COPPER']"]]], ['[2][1] Contains 2 sub portfolios', # letfovers ["[2][1][0] Contains ['KOSPI', 'MXP']"], ["[2][1][1] Contains ['GAS_US', 'LIVECOW']"]], ['[2][2] Contains 3 sub portfolios', # ags and crude ["[2][2][0] Contains ['CORN', 'WHEAT']"], ["[2][2][1] Contains ['CRUDE_W', 'SOYBEAN']"], ["[2][2][2] Contains ['LEANHOG']"]]]]
Some very interesting groupings there, mostly logical but a few unexpected (eg BTP, KOSPI). Also instructive to look at the smallest weights:
US10, US5, EDOLLAR, PLAT, EUR, GBP, EUROSTX (used to hedge), JPY
Those are markets I could potentially think about removing if I wanted to. 

Evaluating the profits

As the figure shows the ranking of performance is as follows:

- Naive Markowitz, Sharpe Ratio (SR) 0.82
- Shrinkage, SR 0.96
- Bootstrap, SR 0.97
- Handcrafted SR 1.01
- Equal weighting SR 1.02
So naive is definitely sub optimal, but the others are pretty similar, with perhaps handcrafting and equal weights a fraction ahead of the rest. This is borne out by the T-statistics from doing pairwise comparisons between the various curves. 

boot = system.accounts.portfolio() ## populate the other values in the dict below appropriately
results = dict(naive=oneperiodacc, hc=handcraft_acc, equal=equal_acc, shrink=shrink, boot=boot)

from syscore.accounting import account_test

for type1 in types:
    for type2 in types:
        if type1==type2:
            continue        print("%s vs %s" % (type1, type2))
        print(account_test(results[type1], results[type2]))

A T-statistic of around 1.9 would hit the 5% critical value, and 2.3 is a 2% critical value):

         Naive Shrink Boot   HC   Equal
Shrink   2.01
Boot     1.56   0.03
HC       2.31   0.81  0.58
Equal    2.33   0.97  0.93  0.19

Apart from bootstrapping, all the other methods handily beat naive with 5% significance. However the rest of the t-statistics are pretty indifferent.

Partly this is because I’ve constrained all the optimisations in similar ways so they don’t do anything too stupid; for example ignoring Sharpe Ratio when optimising over instrument weights. Changing this would mostly penalise the naive optimisation further, but probably wouldn't change things much elsewhere.

It’s always slightly depressing when equal weights beats more complicated methods, but this is partly a function of the data set. Everything is vol scaled, so there is no need to take volatility into account. The correlation structure is reasonably friendly: 

  • for instrument weights we have a pretty even set of instruments across different asset classes, so equal weighted and handcrafted aren’t going to be radically different, 
  • for forecast weights, handcrafting (and all the other methods) produce carry weights of between 30% and 40% for S&P 500, whilst equal weighting would give carry just 14%. However this difference won’t be as stark for other instruments which can only afford to trade 2 or 3 EWMAC crossovers.

Still there are many contexts in which equal weight wouldn't make sense

Incidentally the code for handcrafting runs pretty fast; only a few seconds slower than equal weights which of course is the fastest (not that pysystemtrade is especially quick... speeding it up is on my [long] to do list). Naive and bootstraps run a bit slower (as they are doing a single optimisation per time period), whilst bootstrap is slowest of all (as it’s doing 100 optimisations per time period).


Handcrafting produces sensible and reasonably stable weights, and it's out of sample performance is about as good as more complicated methods. The test for handcrafting was not to produce superior out of sample performance. All we needed was performance that was indistinguishable from more complex methods. I feel that it has passed this test with flying colours, albeit on just this one data set.

So if I review the original motivation for producing this method:

  • - Humans can trust it; intuitive and transparent method which produces robust weights (yes, confirmed in this post)
  • - Can be easily implemented by a human in a spreadsheet (yes, see post 3)
  • - Can be back tested (yes, confirmed in this post)
  • - Grounded in solid theoretical foundations (yes, see post 2)
  • - Takes account of uncertainty (yes, see post 2)
  • - Decent out of sample performance (yes, confirmed in this post)

We can see that there is a clear tick in each category. I’m pretty happy with how this test has turned out, and I will be switching the default method for optimisation in pysystemtrade to use handcrafting.