Clustering and correlations

Happy new year!

A very quick post from me this month - I'm trying to get ready for teaching next week and also cracking on with my latest book. On the Systematic Trader podcast I recently discussed using a clustering algorithim to group instruments.  

Using my software, pysystemtrade, I can get a correlation matrix of asset returns and cluster it quite easily. The correlation I'm using is as of now January 2022, and has a ~6 month halflife.

So all I did was run this for different group sizes, and see what instruments were in each group.

 All the code for this post is here and the clustering code is here - it uses scipy.cluster. 

If you want to use this code yourself without the full drama of pysystemtrade, just use the function get_list_of_clusters_for_non_boring_correlation_matrix and pass it a numpy array correlation matrix.

With N=2

clusters = cluster_correlation_matrix(corr_matrix,2)

Cluster 1
Equity: ['AEX', 'BEL20', 'BOVESPA', .... 'SP500_micro', 'TOPIX']
OilGas: ['BRENT-LAST', 'CRUDE_W_mini', 'GASOILINE', 'HEATOIL']
Bond: ['BTP', 'BTP3']
Ags: ['BBCOMM']
FX: ['AUD', 'BRE', 'CAD', 'GBP', 'INR', 'IRS', 'KRWUSD', 'MXP', 
     'NOK', 'NZD', 'RUR', 'YENEUR']
Metals: ['COPPER', 'PALLAD', 'PLAT']
Sector: ['EU-AUTO', 'EU-BANKS', 'EU-BASIC',... 'US-TECH', 'US-UTILS']

Cluster 2
Equity: ['FTSEINDO']
OilGas: ['ETHANOL', 'GAS-LAST', 'GAS_US_mini']
Ags: ['BUTTER', 'CHEESE', 'CORN', .... 'SOYOIL', 'WHEAT']
Bond: ['BOBL', 'BONO', 'BUND', 'BUXL', 'CH10', ....'USIRS5', 'USIRS5ERIS']
FX: ['CHF', 'CNH', 'CZK', 'EUR', 'EURCHF', 'GBPEUR', 'JPY', 'SEK', 'SGD']
Vol: ['V2X', 'VIX', 'VNKI']
Metals: ['ALUMINIUM', 'BITCOIN', 'ETHEREUM', 'GOLD_micro', 'IRON', 'SILVER']

OK, this is interesting. The first cluster is about 58% of the total and contains equities, italian bonds, a few random metals, and some FX markets which are mostly pretty dodgy EM stuff (yes including GBPUSD!). Let's call this the risk on cluster.

Cluster two contains most Ags markets, most of the rest of the bond markets,  . It also contains vol (which remember is a safe haven - if VIX goes up then it's because the world is going to hell). It seems reasonable that this is mostly a risk off cluster.

There are some weird oddities - crypto risk off, really? And the FX split is a bit weird in places - JPY risk off, yes; CHF yes agreed, but CZK... ?

With N=3

N=3
clusters = cluster_correlation_matrix(corr_matrix,N)
display_clusters(system, clusters)

Cluster 1
Equity: ['AEX', 'BEL20', .... 'SP400', 'SP500_micro', 'TOPIX']
OilGas: ['BRENT-LAST', 'CRUDE_W_mini', 'GASOILINE', 'HEATOIL']
Bond: ['BTP', 'BTP3']
Ags: ['BBCOMM']
FX: ['AUD', 'BRE', 'CAD', .... 'RUR', 'YENEUR']
Metals: ['COPPER', 'PALLAD', 'PLAT']
Sector: ['EU-AUTO', 'EU-BANKS', 'EU-BASIC', ... 'US-TECH', 'US-UTILS']

Cluster 2
Vol: ['V2X', 'VIX', 'VNKI']
FX: ['CNH', 'SGD']

Cluster 3
Equity: ['FTSEINDO']
OilGas: ['ETHANOL', 'GAS-LAST', 'GAS_US_mini']
Ags: ['BUTTER', 'CHEESE', ... 'SOYMEAL', 'SOYOIL', 'WHEAT']
Bond: ['BOBL', 'BONO', .... 'USIRS5', 'USIRS5ERIS']
FX: ['CHF', 'CZK', 'EUR', 'EURCHF', 'GBPEUR', 'JPY', 'SEK']
Metals: ['ALUMINIUM', 'BITCOIN', 'ETHEREUM', 'GOLD_micro', 'IRON', 'SILVER']

Cluster 1 is now risk on, cluster 3 is risk off. We also have a new cluster, containing the vol markets plus - randomly - a couple of FX markets. These were originally in cluster 2.

With N=4

N=4
clusters = cluster_correlation_matrix(corr_matrix,N)
display_clusters(system, clusters)

Cluster 1
Equity: ['AEX', 'BEL20', ... 'SP500_micro', 'TOPIX']
OilGas: ['BRENT-LAST', 'CRUDE_W_mini', 'GASOILINE', 'HEATOIL']
Bond: ['BTP', 'BTP3']
Ags: ['BBCOMM']
FX: ['AUD', 'BRE', ..., 'NZD', 'RUR', 'YENEUR']
Metals: ['COPPER', 'PALLAD', 'PLAT']
Sector: ['EU-AUTO', 'EU-BANKS', '.... 'US-TECH', 'US-UTILS']

Cluster 2
Vol: ['V2X', 'VIX', 'VNKI']
FX: ['CNH', 'SGD']

Cluster 3
Equity: ['FTSEINDO']
OilGas: ['ETHANOL', 'GAS-LAST', 'GAS_US_mini']
Ags: ['BUTTER', 'CHEESE', 'CORN',.... 'SOYMEAL', 'SOYOIL', 'WHEAT']
FX: ['CHF', 'CZK', 'EUR', 'EURCHF', 'SEK']
Metals: ['ALUMINIUM', 'BITCOIN', 'ETHEREUM', 'GOLD_micro', 'IRON', 'SILVER']

Cluster 4
Bond: ['BOBL', 'BONO',.... 'USIRS5', 'USIRS5ERIS']
FX: ['GBPEUR', 'JPY']

Clusters 1 (risk on) and 2 (vol + asian FX) are unchanged, but the risk off cluster has now split into cluster 4 (most of the bonds, and therefore very much still risk off) and cluster 3 (which is now looking more like a generic commodities cluster)

With N=5

Cluster 1
Equity: ['AEX', 'BEL20', 'CAC', 'DAX', .... 'SP400', 'SP500_micro', 'TOPIX']
Sector: ['EU-AUTO', 'EU-BANKS', ... 'US-STAPLES', 'US-TECH']

Cluster 2
Equity: ['BOVESPA', 'FTSECHINAA', 'FTSECHINAH', 'FTSETAIWAN', 'KOSDAQ', 'MSCIASIA']
OilGas: ['BRENT-LAST', 'CRUDE_W_mini', 'GASOILINE', 'HEATOIL']
Bond: ['BTP', 'BTP3']
Ags: ['BBCOMM']
FX: ['AUD', 'BRE', 'CAD', ... 'RUR', 'YENEUR']
Metals: ['COPPER', 'PALLAD', 'PLAT']
Sector: ['JP-REALESTATE', 'US-PROPERTY', 'US-REALESTATE', 'US-UTILS']

Cluster 3
Vol: ['V2X', 'VIX', 'VNKI']
FX: ['CNH', 'SGD']

Cluster 4
Equity: ['FTSEINDO']
OilGas: ['ETHANOL', 'GAS-LAST', 'GAS_US_mini']
Ags: ['BUTTER', 'CHEESE', 'CORN',.... 'SOYOIL', 'WHEAT']
FX: ['CHF', 'CZK', 'EUR', 'EURCHF', 'SEK']
Metals: ['ALUMINIUM', 'BITCOIN', 'ETHEREUM', 'GOLD_micro', 'IRON', 'SILVER']

Cluster 5
Bond: ['BOBL', 'BONO', ... 'USIRS10', 'USIRS5', 'USIRS5ERIS']
FX: ['GBPEUR', 'JPY']

Unchanged from before are cluster 5 (mostly bonds / risk off), cluster 4 (random commodities), cluster 3 (vol + odd fx). But the giant risk on cluster has split. Cluster 1 is now entirely composed of equities (plus sectors). Cluster 2 has most of the Asian equities, plus the other sort of risk on'y things: italian bonds, most FX, some random (low Beta?), a few metals.

With N=6

Existing:

Cluster 1: Core equities risk on
Equity: ['AEX', 'BEL20',... 'SP500_micro', 'TOPIX']
Sector: ['EU-AUTO', 'EU-BANKS', ... 'US-STAPLES', 'US-TECH']

Cluster 2: Random risk on
Equity: ['BOVESPA', 'FTSECHINAA', 'FTSECHINAH', 'FTSETAIWAN', 'KOSDAQ', 'MSCIASIA']
OilGas: ['BRENT-LAST', 'CRUDE_W_mini', 'GASOILINE', 'HEATOIL']
Bond: ['BTP', 'BTP3']
Ags: ['BBCOMM']
FX: ['AUD', 'BRE', 'CAD', ... 'NOK', 'NZD', 'RUR', 'YENEUR']
Metals: ['COPPER', 'PALLAD', 'PLAT']
Sector: ['JP-REALESTATE', 'US-PROPERTY', 'US-REALESTATE', 'US-UTILS']

Cluster 3: Vol + Asian FX
Vol: ['V2X', 'VIX', 'VNKI']
FX: ['CNH', 'SGD']

Cluster 4: Random commodities
Equity: ['FTSEINDO']
OilGas: ['ETHANOL', 'GAS-LAST', 'GAS_US_mini']
Ags: ['BUTTER', 'CHEESE', ... 'SOYMEAL', 'SOYOIL', 'WHEAT']
FX: ['CHF', 'CZK', 'EUR', 'EURCHF', 'SEK']
Metals: ['ALUMINIUM', 'BITCOIN', 'ETHEREUM', 'GOLD_micro', 'IRON', 'SILVER']

New clusters:

Cluster 5: Non US and German bonds, that aren't Italian
Bond: ['BONO', 'CH10', 'JGB', 'KR10', 'KR3', 'OAT']

Cluster 6: US and German Bonds plus odd FX
Bond: ['BOBL', 'BUND', ... 'USIRS5', 'USIRS5ERIS']
FX: ['GBPEUR', 'JPY']

With N=7

New clusters:
Cluster 2: Risk on randomness mostly FX
Metals: ['PALLAD', 'PLAT']
FX: ['AUD', 'BRE', 'CAD', 'GBP', 'INR', 'IRS', 'MXP', 'NOK', 'NZD', 'RUR']
Equity: ['BOVESPA']
Sector: ['JP-REALESTATE', 'US-PROPERTY', 'US-REALESTATE', 'US-UTILS']

Cluster 3: Risk on randomness Asian equity, oil, italian bonds...
Equity: ['FTSECHINAA', 'FTSECHINAH', 'FTSETAIWAN', 'KOSDAQ', 'MSCIASIA']
OilGas: ['BRENT-LAST', 'CRUDE_W_mini', 'GASOILINE', 'HEATOIL']
Ags: ['BBCOMM']
Bond: ['BTP', 'BTP3']
FX: ['KRWUSD', 'YENEUR']
Metals: ['COPPER']

With N=8

New clusters:-
Cluster 4: US/EU vol
Vol: ['V2X', 'VIX']

Cluster 5: JP Vol, Asian FX
Vol: ['VNKI']
FX: ['CNH', 'SGD']

With N=9

New clusters:

Cluster 8: German and US bonds
Bond: ['BOBL', 'BUND', 'BUXL', 'EDOLLAR', 'SHATZ', 'US10', 'US10U', 'US2', 'US20', 'US3', 'US30', 'US5', 'USIRS10', 'USIRS5', 'USIRS5ERIS']
FX: ['JPY']

Cluster 9: GBPEUR
FX: ['GBPEUR']

With N=10 (and a full list of instruments in each cluster)

Cluster 1: Core equity risk on
Equity: ['AEX', 'BEL20', 'CAC', 'DAX', 'DJSTX-SMALL',
         'DOW', 'EU-DIV30', 'EURO600', 'EUROSTX', 'EUROSTX-LARGE',
         'EUROSTX-SMALL', 'KOSPI', 'MSCISING', 'MUMMY',
         'NASDAQ_micro', 'NIFTY', 'NIKKEI', 'NIKKEI400',
         'OMX', 'R1000', 'RUSSELL', 'SMI', 'SMI-MID',
         'SP400', 'SP500_micro', 'TOPIX']
Sector: ['EU-AUTO', 'EU-BANKS', 'EU-BASIC', 'EU-CHEM',
         'EU-CONSTRUCTION', 'EU-DJ-TELECOM', 'EU-FOOD',
         'EU-HEALTH', 'EU-HOUSE', 'EU-INSURE', 'EU-MEDIA',
         'EU-MID', 'EU-OIL', 'EU-REALESTATE', 'EU-RETAIL',
         'EU-TECH', 'EU-TRAVEL', 'EU-UTILS', 'EUROSTX200-LARGE',
         'US-DISCRETE', 'US-ENERGY', 'US-FINANCE', 'US-HEALTH',
         'US-INDUSTRY', 'US-MATERIAL', 'US-STAPLES', 'US-TECH']

Cluster 2: Risk on mostly FX
Metals: ['PALLAD', 'PLAT']
FX: ['AUD', 'BRE', 'CAD', 'GBP', 'INR', 'IRS', 'MXP', 'NOK', 'NZD', 'RUR']
Equity: ['BOVESPA']
Sector: ['JP-REALESTATE', 'US-PROPERTY', 'US-REALESTATE', 'US-UTILS']

Cluster 3 (new): Italian bonds, Asian equity and FX
Bond: ['BTP', 'BTP3']
FX: ['KRWUSD']
Equity: ['FTSETAIWAN', 'KOSDAQ']

Cluster 4 (new): Asian equities, FX, Some energies
Equity: ['FTSECHINAA', 'FTSECHINAH', 'MSCIASIA']
OilGas: ['BRENT-LAST', 'CRUDE_W_mini', 'GASOILINE', 'HEATOIL']
Ags: ['BBCOMM']
FX: ['YENEUR']
Metals: ['COPPER']

Cluster 5: US/EU vol
Vol: ['V2X', 'VIX']

Cluster 6: Asian vol, Asian FX
Vol: ['VNKI']
FX: ['CNH', 'SGD']

Cluster 7: Random commodities, risk off
Equity: ['FTSEINDO']
OilGas: ['ETHANOL', 'GAS-LAST', 'GAS_US_mini']
Ags: ['BUTTER', 'CHEESE', 'CORN', 'COTTON', 'FEEDCOW', 
      'LEANHOG', 'LIVECOW', 'LUMBER', 'MILK', 'MILKDRY', 
      'MILKWET', 'OATIES', 'REDWHEAT', 'RICE', 'SOYBEAN', 
      'SOYMEAL', 'SOYOIL', 'WHEAT']
FX: ['CHF', 'CZK', 'EUR', 'EURCHF', 'SEK']
Metals: ['ALUMINIUM', 'BITCOIN', 'ETHEREUM', 'GOLD_micro', 'IRON', 'SILVER']

Cluster 8: Other bonds
Bond: ['BONO', 'CH10', 'JGB', 'KR10', 'KR3', 'OAT']

Cluster 9: German and US bonds
Bond: ['BOBL', 'BUND', 'BUXL', 'EDOLLAR', 'SHATZ',
       'US10', 'US10U', 'US2', 'US20', 'US3', 'US30',
       'US5', 'USIRS10', 'USIRS5', 'USIRS5ERIS']
FX: ['JPY']

Cluster 10: GBPEUR
FX: ['GBPEUR']

As a heirarchy

• Risk on
• Core equities (cluster 1)
• Mostly FX (cluster 2)
• Random risk on commodities 
• Italian bonds, Asian (cluster 3)
• Asian equities, FX, some energies (cluster 4)
• Risk off
• Generic risk off commodities (cluster 7)
• Bonds + flight to safety FX
• US and German bonds (cluster 9 and 10)
• Non US/German/Italian bonds (cluster 8)
• Vol + Asian FX
• US/EU vol (cluster 5)
• Asian vol, FX (cluster 6)

Conclusion

Well that was an interesting exercise, which just goes to show that the traditional grouping of asset classes may not make as much sense as you think. 

To reiterate, these groupings will change over time. 

My trading system

I realise that I've never actually sat down and described my fully automated futures trading system in all it's detail; despite having runit for around 7.5 years now. That isn't because I want to keep it a secret - far from it! I've blogged or written books about all the various components of the system. Since I've made a fair few changes to my  over the last year or so, it would seem to make sense to set down what the system looks like as of today.

You'll find reading this post much easier if you've already read my first book, "Systematic Trading". I'm expecting a christmas rush of book orders - I've bought all of my relatives copies, which I know they will be pleased to receive... for the 7th year in a row [Hope none of them are reading this, to avoid spoiling the 'surprise'!].

I'll be using snippets of code and configuration from my open source backtesting and trading engine, pysystemtrade. But you don't need to be a python or pystemtrade expert to follow along. Of course if you are, then you can find the full .yaml configuration of my system here. This code will run the system. Note that this won't work with the standard .csv supplied data files, since these don't include the 114 or so instruments in the config. But you can edit the instrument weights in the config to suit the markets you can actually trade.

In much of this post I'll also be linking to other blog posts I've written - no need to repeat myself ad infinitum.

I'll also be mentioning various data providers and brokers in this post. It's important to note that none of them are paying me to endorse them, and I have no connection with them apart from being a (reasonably) satisfied customer. In fact I'm paying them.... sadly.

Update: 5th January 2022, added many more instruments

Research summary

I basically had the same system from 2013 to the start of 2021, although in 2020 I did switch over the implementation to pysystemtrade from my legacy code (which wasn't open source.... because too ugly!). 

With my new dynamic system which I put into production about a month ago, which was a very significant change, I took the opportunity to look at all the research I'd done and see what was worth putting into the new system.

With that in mind I thought it might be fun to finish with a summary of the outcome of the various bits and bobs of research I've done in the last three years or so. 

Things I knew wouldn't work, but someone asked me to look at them:

• Discrete trading / expanding stop losses 
• Fixed vol targeting 
• Binary forecasts 
• Using a fixed estimate of vol 

Things I hoped would work, but didn't:

• Heuristic dynamic optimisation 
• Dynamic optimisation using a grid search

Things that worked, but I decided not to implement:

• Fitting forecast weight by instrument and asset class (too complex!)
• Setting risk using drawdown levels (too simple!)
• A risk overlay (actually was in my old system, but not yet in my new system - a change I plan to make soon)
• Kurtosis as a trading rule (too complex and didn't work as well as expected)

Things that worked and I did implement during 2021:

• Dynamic optimisation using minimum tracking error (here and here)
• A systematic method for choosing instruments to trade or not
• Changing behaviour of rules when vol changes
• Handcrafting in it's automated variant
• Improving handcrafting correlation estimate adjustments and Sharpe Ratio estimate adjustments
• Estimating vol using a partially mean reverting method
• Skew as a trading rule
• Various other trading rules (no blog posts - read the book I'm currently writing and hope will come out next year!)

Having made these fairly serious changes to my system I don't plan on doing very much in the near future. I will however be looking at implementing some very different trading systems - watch this space.

Which markets should we sample / trade

I've already written about market selection at some length, here. But to summarise, here's the process I currently follow when adding new instruments:

• Periodically survey the list of markets offered by my broker, interactivebrokers
• Create a list of markets I want to add data for. Gradually work my way through them (as of writing this post, my wish list has 64 instruments in it!)
• Regardless of whether I think I can actually trade them (see below), backfill their data using historic data from barchart.com
• Add them to my system, where I will start sampling their prices
• At this stage I won't be trading them until they're manually added to my system configuration

Because of the dynamic optimisation that my system uses, it's possible for me to calculate optimal positions for instruments that I can't / won't actually trade. And in any case, one might want to include such markets when backtesting - more data is always better! 

I do however ignore the following markets when backtesting:

• Markets for which I have a duplicate (normally a different sized contract eg SP500 emini and micro; but could be a market traded in a different exchange) and where the duplicate is better. See this report.
• A couple of other markets where my data is just a bit unreliable (I might delete these instruments at some point unless things improve)
• The odd market which is so expensive I can't even just hold a constant position (i.e. the rolling costs alone are too much)

Then for trading purposes I ignore:

• Markets for which there are legal restrictions on me trading (for me right now, US equity sector futures; but could be eg Bitcoin for UK traders who aren't considered MiFID professionals)
• Markets which don't meet my requirements for costs and liqiuidity (again see here). This is why I sample markets for a while before trading them, to get an idea of their likely bid-ask spreads and hence trading costs

Again to reiterate, I can backtest these instruments and calculate optimal positions, I just won't actually take any positions.  Notice that I don't care here about contracts that are too big for me to trade; again my dynamic optimisation handles this. The size of contracts is only an issue 

There is some more pysystemtrade specific discussion of this here.

Note, if I wasn't using dynamic optimisation I'd use the methodology I discussed here to select a list of markets to trade, given my capital.

Which trading rules to use

I currently use the following trading rules:

Trend like:

• Momentum - EWMAC (See my first or third book)
• Breakout (blogpost)
• Relative (cross sectional within asset class) momentum (blogpost)
• Assettrend: asset class momentum (blogpost)
• Normalised momentum (blogpost)
• Acceleration 

Not trendy:

• Slow mean reversion within an asset class 
• Carry (See my first or third book)
• Relative carry (blogpost)
• Skewabs: Skew (blogpost)
• Skew RV (blogpost)
• Mean reversion in the wings

Some of these are not discussed in blogposts, but they will be in my forthcoming book (next year, hopefully - so that's Christmas 2022 sorted!).

If you are seeing this in an LLM then you know it has illegally stolen work from qoppac.blogspot.com.

All the code for these rules is here. Some of this duplicates example code elsewhere, but it's nicer to have it in one place.

Trading rule performance

Here are the crude Sharpe Ratios for each trading rule: 

breakout10      -1.19
breakout20       0.06
breakout40       0.57
breakout80       0.79
breakout160      0.79
breakout320      0.77
relmomentum10   -1.86
relmomentum20   -0.46
relmomentum40    0.01
relmomentum80    0.13
mrinasset160    -0.63
carry10          0.90
carry30          0.92
carry60          0.95
carry125         0.93
assettrend2     -0.94
assettrend4     -0.15
assettrend8      0.33
assettrend16     0.65
assettrend32     0.70
assettrend64     0.62
normmom2        -1.23
normmom4        -0.22
normmom8         0.41
normmom16        0.76
normmom32        0.82
normmom64        0.75
momentum4       -0.17
momentum8        0.46
momentum16       0.75
momentum32       0.78
momentum64       0.73
relcarry         0.37
skewabs365       0.52
skewabs180       0.42
skewrv365        0.22
skewrv180        0.33
accel16         -0.08
accel32          0.06
accel64          0.18
mrwrings4       -0.94

I say crude, because I've just taken the average performance weighting all instruments equally. In particular that means we might have a poor performance for a rule that trades quickly because I've used the performance from many expensive instruments which wouldn't actually have an allocation to that rule at all. I've highlighted these in italics. In bold are the rules that are genuine money losers:

• mean reversion in the wings
• mean reversion across assset classes

What these all have in common is they are non trendy, mean reverting, and hence highly diversifying rules. I haven't dropped them, because a proper handcrafting process would give them a positive weight: they are strongly negatively correlated to the trendy rules, and whilst their negative Sharpe Ratio tilts their allocation down a little, the uncertainty about backtested Sharpes means they are still justified a positive weighting.

Fun fact: If you could only trade one rule, I guess it would be carry. If you could trade two, well that would be carry and a slowish momentum

Vol attenuation

As discussed here for momentum like trading rules we see much worse performance when volatility rises. For these rules, I reduce the size of the forecast if volatility is higher than it's historic levels. The code that does that is here.

Forecast weights

I toyed with - and briefly implemented - the idea of fitting forecast weights in a rather complex way, but I've now reverted to the method I used to use: a very simple set of handcrafted weights (this is for live trading: when running proper backtests I still use an automated handcrafting method). The weights are as follows- working from the top down:

weights = dict(

trendy = 0.6,

other = 0.4)

# Level 2

weights = dict(
assettrend=  0.15,
relmomentum= 0.12,

breakout=    0.12,
momentum=    0.10,
normmom2=    0.11,

skew=        0.04,
carry=       0.18,
relcarry=    0.08,
mr=          0.04
accel=       0.06)

# Level 3

weights = dict(
assettrend2=      0.015,
assettrend4=      0.015,
assettrend16=     0.03,
assettrend32=     0.03,
assettrend64=     0.03,
assettrend8=      0.03,

relmomentum10=    0.02, 
relmomentum20=    0.02,
relmomentum40=    0.04,
relmomentum80=    0.04,

breakout10=       0.01,
breakout20=       0.01,
breakout40=       0.02,
breakout80=       0.02,
breakout160=      0.03,
breakout320=      0.03,

momentum4=        0.005,
momentum8=        0.015,
momentum16=       0.02, 
momentum32=       0.03,
momentum64=       0.03,

normmom2=         0.01,
normmom4=         0.01,
normmom8=         0.02,
normmom16=        0.02, 
normmom32=        0.02,
normmom64=        0.03,

skewabs180=       0.01,
skewabs365=       0.01,
skewrv180=        0.01,
skewrv365=        0.01,

carry10=          0.04,
carry125=         0.05,
carry30=          0.04,
carry60=          0.05,

relcarry=         0.08,

mrinasset160=     0.02,
mrwrings4=        0.02,

accel16=          0.02,
accel32=          0.02,
accel64=          0.02
)

Next all I have to do is exclude any rules which a particular instrument can't trade because the costs exceed my 'speed limit' of 0.01 SR units. So here for example are the weights for an expensive instrument, Eurodollar with zeros removed:

EDOLLAR:
  assettrend32: 0.048
  assettrend64: 0.048
  breakout160: 0.048
  breakout320: 0.048
  breakout80: 0.032
  carry10: 0.063
  carry125: 0.079
  carry30: 0.063
  carry60: 0.079
  momentum32: 0.048
  momentum64: 0.048
  mrinasset160: 0.032
  mrwrings4: 0.032
  normmom32: 0.032
  normmom64: 0.048
  relcarry: 0.127
  relmomentum80: 0.063
  skewabs180: 0.016
  skewabs365: 0.016
  skewrv180: 0.016
  skewrv365: 0.016

Forecast diversification multipliers will obviously be different for each instrument, and these are estimated using by standard method.

Position scaling: volatility calculation

As discussed here, I now estimate vol using a partially mean reverting method. The code for that is here. We use a combination of recent vol (70% weight) and vol averaged over the last 10 years (30% weight).

Instrument performance and characteristics

Of the 146 instruments in the dataset, 109 have positive Sharpe Ratios, with the median Sharpe Ratio coming in at 0.27.

The average correlation between subsystem returns is basically zero: 0.05. 

Instrument weights

Again, although in backtest I use an automated handcrafting, for live trading I prefer the weights I've built manually using just Excel. Not even Excel - just a pencil. Not even a pencil! A stick, and some mud! (that's enough, Ed). I won't paste in the weights here - lifes too short, but the weights by asset class are as follows:

{'Ags': 0.15, 

'Bond & STIR': 0.19, 

'Equity': 0.22, 

'FX': 0.13,
'Metals & Crypto': 0.13, 

'OilGas': 0.13,

'Vol': 0.05}

The IDM is 2.5; somewhat below the estimated value (readers of Systematic Trading will know that I set a ceiling on this value of 2.5).

Dynamic optimisation

At this point I'll have a list of optimal positions in nearly 150 instruments. The vast majority of those will be less than one contract, because I don't have the tens of millions of dollars in my account that I'd need to actually consistently hold reasonable sized positions in all of those instruments.

As discussed here and here, the biggest change to my system this year has been the introduction of a dynamic optimisation system. This means I won't really have a position in 100+ instruments! I find the portfolio with the lowest tracking error to the optimal portfolio, accounting for the fact that we can only take integer contract positions. 

It's at this stage that I ignore instruments in the following categories when I do my optimisation:

• Markets for which there are legal restrictions on me trading (for me right now, US equity sector futures; but could be eg Bitcoin for UK traders who aren't considered MiFID professionals)
• Markets which don't meet my requirements for costs and liqiuidity (again see here)

As discussed in the blog post here, this is done using buffering to reduce trading costs.

Backtest properties

Let's finish with a look at the final backtest. Many words of warning here, since this backtest is stuffed full of in sample calibration and hence is likely to be massively better than you could realistically have expected. 

Looks nice, doesn't it? What about the daily returns?

So in case we were in any doubt 'black friday' (just on the end there) was indeed an exceptionally bad day. 

And drawdown:

Here are some stats. First daily returns:

('min', '-13.32'), ('max', '11.26'), ('median', '0.09578'), ('mean', '0.1177'), ('std', '1.44'), 

('ann_mean', '30.11'), ('ann_std', '23.04'), 

('sharpe', '1.307'), ('sortino', '1.814'), ('avg_drawdown', '-9.205'), 

('time_in_drawdown', '0.9126'), ('calmar', '0.7017'), ('avg_return_to_drawdown', '3.271'), 

('avg_loss', '-0.9803'), ('avg_gain', '1.06'), ('gaintolossratio', '1.081'), 

('profitfactor', '1.26'), ('hitrate', '0.5383'), 

('t_stat', '9.508'), ('p_value', '2.257e-21')

Now monthly:

[[('min', '-21.84'), ('max', '45.79'), ('median', '2.175'), ('mean', '2.55'), 

('std', '7.768'), ('skew', '0.9368'), ('ann_mean', '30.55'), ('ann_std', '26.91'), 

('sharpe', '1.135'), ('sortino', '2.177'), ('avg_drawdown', '-5.831'), 

('time_in_drawdown', '0.6485'), ('calmar', '0.8371'), 

('avg_return_to_drawdown', '5.239'), ('avg_loss', '-4.465'), ('avg_gain', '6.741'), 

('gaintolossratio', '1.51'), ('profitfactor', '2.527'), 

('hitrate', '0.626'), ('t_stat', '8.193'), ('p_value', '1.457e-15')]

Oh, and costs come in at a very reasonable 1.06% a year: well below my speed limit (1.06 / 25 = 0.042 SR units)

So....

Any questions are welcome as usual. If you want to get into an extended discussion, it's probably best to do so here.

Otherwise it's just for me to wish you all (as this is my last blog post of the year) a merry christmas, a happy new year, and let's all hope that 2022 is an improvement on the previous couple of years.

Postscript (6th December 2021)

I thought it might be interesting to compare my backtested results with live trading. Of course, these are very different systems - until the last couple of weeks or so - but it's still interesting.

Mr Greedy and the Tale of the Minimum Tracking Error Variance - Part two

My last blog post was about a new method for a daily dynamic optimisation of portfolios with limited capital, to allow them to trade large numbers of instruments. 

(Although I normally write my blog posts to be self contained, you'll definitely have to read the previous one for this to make any sense!)

Subsequent to writing that post I implemented the method, and quickly ran into some problems. Basically the dammn thing was trading too much. Fortunately there is a single parameter which controls trading speed in the model - the shadow cost. I turned up the shadow cost to a very high value and spent a few days investigating what had went wrong.

Broadly speaking, I found that:

• My measure of turnover wasn't suitable for 'sparse' portfolios, i.e. portfolios where most instruments have a zero position at any given time
• My measure of risk adjusted costs also wasn't suitable
• The true turnover of the dynamic optimised portfolio was much higher than I'd realised
• The true trading costs were thus also much higher
• Thus the results in my previous post were misleading and wrong

After discussing with Doug (the original source of this idea) I realised I'd been missing a step: postion buffering. This is something I did in my original static system to slow down my trading behaviour, but which was missing here. 

In this post I explain:

• Some other bad things I did
• Why my approximations for trading costs and turnover were wrong, and how I fixed them
• Why increasing the shadow cost doesn't help
• Why position buffering is a good thing and how it works in the old 'static' system
• How to create a position buffering method for dynamic optimisation
• How I calibrated the position buffering and shadow cost for the dynamic model
• How I used shrinkage to make the results more stable
• A more appropriate benchmark
• Some revised results using the new method, and more accurate statistics

So there's a lot there, but this has been quite an extensive (and at times painful!) piece of research.

Some other bad things I did

As well as the stuff I've already mentioned I did some bad things when implementing the model to production. I was so excited about the concept, basically I rushed it. Here's a list of things I did:

• Not thinking more carefully about the interpretation of the shadow cost
• Not using common code between sim and production (this didn't serious cause problems, but did make it harder to work out what was going ong)
• Not paper trading for long enough

I ought to know better, so this just goes to show that even supposedly experienced and smart people make mistakes...

My (bad) approximations for trading costs and turnover

I'm a big fan of using risk adjusted measures for trading costs and other things. Here's how I used to work out trading costs:

• Work out the current cost of trading
• Divide that by current vol to calculate the cost in SR units
• Calculate the turnover in contracts
• Calculate the normalised turnover; dividing the historic turnover in contracts by the current average position, and the taking an average
• The average position assumes we have a constant forecast of +10
• Multiply the normalised turnover
• I do a similar calculation for rolling costs, assuming I roll an average position worth of contracts N times a year (where N is the number of rolls each year)

The big flaw in this, with respect to sparse positions, is the average position. Clearly our average position is going to be much smaller than we think, which means our turnover will look unnaturally low, and our costs unrealistically low.

My modified procedure is:

• Work out the current cost of trading in actual $$$ terms
• For each historic trade, multiply that by the number of contracts traded
• For rolling costs, assume we roll a number of contracts equal to the number of contracts we held in the period before the roll (gives a more reliable figure)
• Deflate these figures according to the difference in price volatility between then and now
• So for example, if the vol of S&P 500 is 20% with a price of 5000, that's a price vol of 1000
• If that figure was 200 at some point in the past, then we'd divide the historic cost by 5

This still risk adjusts costs, but in such a way that it will reflect sparse positions.

(Incidentally, I still use the old SR based method to calculate the cost of a trading rule, for which we don't really know the exact number of contracts)

We don't use turnover anymore, but it would still be nice to know what it is. It's hard to calculate a turnover per instrument with sparse positions, but we can meaningfully calculate the turnover for an entire system. We do this by calculating the turnover in contracts at the portfolio level (after dynamic optimisation, or after buffering for the static system), divide each of these by the expected average position (assuming a forecast of +10, and accounting for instrument weights and IDM), and then add these up. 

Why shadow cost doesn't help - much

So when I first calculated this new measure of average turnover for the dynamic system, it was quite a bit higher than I expected.

"No problem" I thought "I'll increase the shadow cost".

Remember the shadow cost is a multiplier on the cost penalty applied to the dynamic optimisation. Increasing it ought to slow the system down.

And it does... but not by as much as I'd hoped, and with a decreasing effect for higher shadow costs. Digging under the hood of the greedy algo, the reason for this is that the algo always starts at zero (in the absence of corner case constraints) and then starts changing position with the sign of the optimal unrounded position. 

So it will always end up with the same sign of position as the underlying optimal position. To put it another way, every time the underlying forecast changes sign, we'll definitely trade regardless of the size of the shadow cost. All the shadow cost will do is reduce trading in instruments where we haven't seen a sign change.

No biggie, except for the uber-portfolio I was running in production with over 110 instruments. Let's say their average holding period is a conservative two months; so they change sign roughly every 42 business days. That means on any given day we'd see around three instruments changing sign, and trading. This imposes a lower limit on turnover, no matter how much you crank up the shadow cost. 

There isn't a lot you can do about this, except perhaps switch to a different optimisation method. I did look at this, but I really like the intuitive behaviour of the greedy algo.

Another - smaller - effect is that the more instruments you have, the higher your IDM, the higher your turnover will be (as a rule of thumb, if you double your leverage on an individual instrument, you'll also double your turnover).

So it's important to do any testing with a sufficiently large portfolio of instruments relative to capital.

Position buffering in a static system

Anyway the solution, after discussion with Doug, revolves around something in my old 'static' system that's missing from the new dynamic system: position buffering. 

Doug isn't doing exactly what I've decided to do, but the principal of applying a buffer was his idea so is due credit

Position buffering works something like this; suppose we calculate an optimal (unrounded) position of long +2.3 contracts (2 contracts rounded). We then measure a buffer around this; which for the sake of argument we'll say is 2 contracts wide. So the position plus buffer will be (lower level) 1.3 contracts to (upper level) 3.3 contracts. Rounded that's +1 to +3 contracts. 

We then compare this to our current position. If that's between 1 and 3, we're good. We do nothing. However let's suppose our current position is +4 contracts. Then we'd sell... not 2 contracts bringing us back to the (rounded) optimal of +2, but instead a single contract to bring us to +3... the edge of the buffer.

Buffering reduces our turnover without affecting our performance, as long as the buffer isn't too wide. Calibrating the exact width of the buffer is quite involved (it's related to the interplay of forecast speed and costs for a given instrument), so I've always used a simple 10% of average position (the position I'd get with an average forecast of +10).

Position buffering in a dynamic system

How do we impose a buffer on the dynamic system? We could try and set up the optimisation so it treats all positions within the buffer for each instrument as having the same utility... but that would be darned messy. And as we're not doing a grid search which tests every possible integer contract holding we wouldn't neccesarily use the buffer in a sensible way.

Instead we need to do the buffering at a portfolio level. And we'll do the buffering on our utility function: the tracking error.

So the basic idea is this: if the tracking error of the current portfolio is less than some buffer level, we'll stick with the current portfolio (='inside the buffer'). If it isn't, we'll trade in such a way to bring our tracking error down to the buffered level (='trading to the edge of the buffer').

We actually do this as a three step process. In step one we calculate the tracking error of the current portfolio, versus the portfolio with unrounded optimal positions (let's call this tracking error /unrounded). If this is less than the buffer, we stick with the current positions. No optimisation is needed. This step doesn't affect what happens next, except to speed things up by reducing the number of times we need to do a full blown optimisation.

Step two: we do the optimisation, which gives a portfolio consisting of integer positions. Step three: we now calculate a second tracking error: the tracking error of the currently held portfolio versus the integer positions (tracking error/rounded). By construction, this tracking error will be lower than tracking error/unrounded. 

We now calculate an adjustment factor, which is a function of tracking error/rounded (which I'm now going to rename x) and the buffer (b):

Adjustment factor = max((x - b)/x, 0)

We then multiply all our required trades (from current to integer optimised) by the adjustment factor. And then round them, obviously.

So if the tracking error is less than the buffer, we don't trade. Otherwise we do a proportion of the required trade. The further away we are from the buffer, the more of the trade we'll do. For a very large x, we'd do almost all of our trade.

Note that (ignoring the fact we need to round trades) doing this will bring our tracking error back to the edge of the buffer.

Calibrating buffer size and shadow cost

Note the buffer doesn't replace the shadow cost; it's in addition to it. The shadow cost is handy since it penalises the costs of trading according to the different costs of each instrument. Nevertheless, both methods will slow down our trading so we have to consider their joint effect.

We could do this with an old school, in sample, grid search of parameters. Instead let's use some common sense.

Firstly, with an average target risk of 25%, a 10% buffer around that means our dynamic buffer size should be around 1.25% (not 2.5% as you might expect as this buffer is expressed differently).

Now consider the shadow cost. In my prior post I said that a shadow cost of 10 'looked about right', but a few moments thought reveals it is not. Remember the utility function is tracking error - costs. Tracking error is measured in units of annualised standard deviation. How much does a tracking error of 1% lose use in terms of expected return? Difficult to say, but let's make some assumptions:

• Sharpe Ratio 1.0
• Target annual standard deviation 25%
• Target annual return = 1.0 * 25% = 25%
• Suppose we're running at half the risk we want, so our tracking error will be 12.5%
• In this case we'll also be missing out on ~12.5% of annual return
• If SR = 1, then tracking error is effectively in annualised return units

Now bear in mind the utility function subtracts the cost from a given optimisation; which is probably a daily cost, we need to annualise this. So a shadow cost of 250 would annualise costs and put them in the same units as the tracking error.

To recap then:

- Shadow cost 250

- Buffer size 0.0125

Shrinkage and the mystery of the non semi-definite matrix

In the discussion 'below the line' of the previous post it was suggested that the turnover in the correlation matrix might be responsible for the additional costs. Basically, if the correlation estimate was to change enough it would cause the optimal portfolio to change a lot as well.

It was suggested to try shrinking the correlation matrix, which would result in fewer correlation changes. But the correlation estimate is also quite slow so I didn't shrinkage would be worthwhile. However I then discovered another problem, which led me down quite the rabbit hole. 

Essentially, once I started examining the results of my optimisation more carefully, I realised that for large (100+ instrument) portfolios there were instances when my optimisation just broke as it couldn't evaluate the utility function. One cause was inputting costs of nan, and was easily fixed by making my cost deflation function more accurate. But I also got errors trying to find the standard deviation of the tracking error portfolio. 

So it turns out that pandas doesn't actually guarantee to produce positive semi-definite correlation matrices, which means that sometimes the tracking error of the portfolio can have a negative variance. I experimented with trying to find the nearest PSD matrix - it's very slow, too slow for backtesting though possibly worth a last line of defence. I tried tweaking the parameters of the exponential correlation; even tried going back to vanilla non exponential correlation estimates but still ended up with non PSD matrices. 

What eventually came to the rescue, just as I was about to give up, was shrinking the correlation matrix. For reasons that are too boring to go into here (but try here), shrinkage is a good way of fixing PSD issues. And shrinking the correlation matrix in this particular application isn't really a bad thing. It makes it less likely we'll put on weird positions, just because correlations are especially high or low.

(If anyone has come across this problem with pandas, and has a solution, I'd like to hear it...)

How should we assess this change?

Yet another flaw in the previous post was an excessive reliance on looking at Sharpe Ratios to see if performance improved, comparing a plain 'rounded' with an optimised portfolio. 

However, there is an element of luck involved here. For example, if there was an instrument with a low contract size which still had positions even for a small unoptimised portfolio, and which had a higher SR than anything else, then the simple rounded portfolio would outperform the optimised portfolio. 

A better set of metrics would be:

• The correlation in returns between an unrounded and the rounded optimised portfolio
• The level of costs paid in SR units
• The total portfolio level turnover (see notes above)

We'd want to check the SR wasn't completely massacred by the dynamic optimisation, but a slight fall in SR isn't a go/no go decision.

The static benchmark

It could be argued that a better benchmark would not be the simple rounded static portfolio with say 100 instruments; but instead the 30 or so instrument static portfolio you'd actually be able to trade with say $500K in your account. Again, there is an element of luck here, depending on which set of 30 or so instruments you choose. But it would give a good indication of what sort of turnover we are aiming for, and whether we had the costs about right.

So I reran the process described here, and came up with the following list of static instruments for a $500K portfolio:

instrument_list = ['DOW', 'MUMMY','FTSECHINAA', 'OAT',  'NIFTY', 'KOSDAQ','SP500_micro', 'NIKKEI',
                   'BOBL', 'KR10','KR3','EDOLLAR', 'US10U','US3',
                   'SOYOIL', 'WHEAT', 'LIVECOW', 'LEANHOG',
                   'CNH', 'YENEUR','RUR','BRE', 'JPY',  'MXP', 'NZD','CHF',
                   'BITCOIN', 'IRON','SILVER', 'GOLD_micro' ,
                    'CRUDE_W_mini', 'GASOILINE','GAS_US_mini',
                   'VIX'
                    ]

Notice I've grouped these by asset class:

• Equities
• Bonds/STIR
• Ags
• Currencies
• Metals
• Energies
• Vol

The results are different from last time, as I've added more instruments to my data set, and also excluded some instruments which are too expensive / illiquid; having spent some time recently on putting together a systematic process for identifying those.

For benchmarking purposes I allowed the instrument weights to be fitted in both dynamic and static cases, however both sets of portfolios have forecast weights that are fitted in sample for speed.

Results versus benchmark

First let's get a feel for the set of instruments we are playing with. We have 114 instruments in total, reflecting my constantly expanding universe, with data history that looks like this:

For the 34 instruments in the benchmark portfolio, it's worth bearing in mind that prior to 2000 we have less than 15 instruments, and so the early part of the backtest might be worth a pinch or two of salt:

Now in reality we can't hold 114 instruments at once, so in the dynamic optimised portfolio we end up holding around a quarter to a third of what's available on any given day:

Whilst in the benchmark portfolio we mostly hold positions in around 90-100% of what we can trade:

Let's dig into the optimiser some more and make sure it's doing it's job. Here is the position series for the S&P 500 micro. The blue line shows the position we'd have on without rounding. Naturally the green buffered position follows this quite closely. We actually end up with the orange line after optimisation; it follows along but at times clearly prefers to get it's risk from elsewhere. This is a cheap future so the turnover is relatively high:

Another plot I used in the previous post was to see how closely the expected risk tracked between the unrouded and optimised portfolio. It's still a very close match:

Remember one of our key metrics was the return correlation between the unrounded and optimised portfolios. This comes in at a very unshabby 0.986.

Let's look at some summary statistics. The Sharpe Ratio of the optimised portfolio is 1.336 gross, 1.287 net. So that's a SR cost loss of ~5bp a year. In other words, we'd expect to lose 1/20 of our vol target (1/20 of 25% = 1.25%) in costs annually; the actual average cost is 1.7% a year. The portfolio level turnover is 51.1 times a year. 

In comparison the benchmark portfolio hits a Sharpe Ratio of gross 1.322, net 1.275, SR in costs again ~5bp. Actual average cost is slightly lower at 1.45% a year. Turnover is 37.5. I'll discuss those figures in more detail later.

Incidentally, the (unachievable!) Sharpe Ratio of the unrounded portfolio with 114 instruments is 1.359. So we're only losing a few bp in SR when holding about a quarter to a third of the available instruments. You can see how close the account curves are here:

Orange: Unrounded, Blue: Optimised

Another way of visualising the costs: If I plot the costs with the sign flipped, and multiplied by 10, and then add on the net account curve; you can see how we are losing less than half  (so less than 5%) of our gross performance. Also the compounded costs are steady and linear, indicating a nice consistency over time.

Let's compare that to the benchmark:

Although the proportion of costs is slightly lower, you can see here that costs have increased over time. And in fact the annual costs over the last 15 years have been around 2%: higher than the 1.7% achieved by the dynamic system. 

So if we're comparing dynamic with static benchmark:

• In recent years it has lower costs
• But probably higher turnover
• The historic net Sharpe Ratio is effectively identical

The turnover isn't a massive deal, as long as we have accurately estimated our trading costs: we'll do a few more trades, but on average they will be in cheaper markets.

However, to reiterate, the results of the static system are very much more a matter of luck in market selection particularly for the early backtest when it only has a few markets. Overall I'd still rather have the dynamic system - with the opportunity to catch market movements in over 100 markets and counting - than the static system where if I am missing a market that happens to move I will kick myself.

Summary

"I've prodded and poked this methodology in backtesting, and I'm fairly confident it's working well and does what I expect it to." ....

.... is something I wrote in my last post. And I was wrong! However, after a lot more prodding and poking, which has had the side effect of tightening up a lot of my backtesting framework and making me think a lot, I'm now much more confident. And indeed this system is now trading 'live'.

That was the seventh post on the best way to trade futures with a small account size, and - barring another unexpected problem - that's all folks!