Tuesday, 18 January 2022

Clustering and correlations

Happy new year!

A very quick post from me this month - I'm trying to get ready for teaching next week and also cracking on with my latest book. On the Systematic Trader podcast I recently discussed using a clustering algorithim to group instruments.  

Using my software, pysystemtrade, I can get a correlation matrix of asset returns and cluster it quite easily. The correlation I'm using is as of now January 2022, and has a ~6 month halflife.

So all I did was run this for different group sizes, and see what instruments were in each group.

 All the code for this post is here and the clustering code is here - it uses scipy.cluster. 
If you want to use this code yourself without the full drama of pysystemtrade, just use the function get_list_of_clusters_for_non_boring_correlation_matrix and pass it a numpy array correlation matrix.


With N=2

clusters = cluster_correlation_matrix(corr_matrix,2)
Cluster 1
Equity: ['AEX', 'BEL20', 'BOVESPA', .... 'SP500_micro', 'TOPIX']
OilGas: ['BRENT-LAST', 'CRUDE_W_mini', 'GASOILINE', 'HEATOIL']
Bond: ['BTP', 'BTP3']
Ags: ['BBCOMM']
FX: ['AUD', 'BRE', 'CAD', 'GBP', 'INR', 'IRS', 'KRWUSD', 'MXP', 
     'NOK', 'NZD', 'RUR', 'YENEUR']
Metals: ['COPPER', 'PALLAD', 'PLAT']
Sector: ['EU-AUTO', 'EU-BANKS', 'EU-BASIC',... 'US-TECH', 'US-UTILS']

Cluster 2
Equity: ['FTSEINDO']
OilGas: ['ETHANOL', 'GAS-LAST', 'GAS_US_mini']
Ags: ['BUTTER', 'CHEESE', 'CORN', .... 'SOYOIL', 'WHEAT']
Bond: ['BOBL', 'BONO', 'BUND', 'BUXL', 'CH10', ....'USIRS5', 'USIRS5ERIS']
FX: ['CHF', 'CNH', 'CZK', 'EUR', 'EURCHF', 'GBPEUR', 'JPY', 'SEK', 'SGD']
Vol: ['V2X', 'VIX', 'VNKI']
Metals: ['ALUMINIUM', 'BITCOIN', 'ETHEREUM', 'GOLD_micro', 'IRON', 'SILVER']

OK, this is interesting. The first cluster is about 58% of the total and contains equities, italian bonds, a few random metals, and some FX markets which are mostly pretty dodgy EM stuff (yes including GBPUSD!). Let's call this the risk on cluster.

Cluster two contains most Ags markets, most of the rest of the bond markets,  . It also contains vol (which remember is a safe haven - if VIX goes up then it's because the world is going to hell). It seems reasonable that this is mostly a risk off cluster.

There are some weird oddities - crypto risk off, really? And the FX split is a bit weird in places - JPY risk off, yes; CHF yes agreed, but CZK... ?


With N=3


N=3
clusters = cluster_correlation_matrix(corr_matrix,N)
display_clusters(system, clusters)
Cluster 1
Equity: ['AEX', 'BEL20', .... 'SP400', 'SP500_micro', 'TOPIX']
OilGas: ['BRENT-LAST', 'CRUDE_W_mini', 'GASOILINE', 'HEATOIL']
Bond: ['BTP', 'BTP3']
Ags: ['BBCOMM']
FX: ['AUD', 'BRE', 'CAD', .... 'RUR', 'YENEUR']
Metals: ['COPPER', 'PALLAD', 'PLAT']
Sector: ['EU-AUTO', 'EU-BANKS', 'EU-BASIC', ... 'US-TECH', 'US-UTILS']

Cluster 2
Vol: ['V2X', 'VIX', 'VNKI']
FX: ['CNH', 'SGD']

Cluster 3
Equity: ['FTSEINDO']
OilGas: ['ETHANOL', 'GAS-LAST', 'GAS_US_mini']
Ags: ['BUTTER', 'CHEESE', ... 'SOYMEAL', 'SOYOIL', 'WHEAT']
Bond: ['BOBL', 'BONO', .... 'USIRS5', 'USIRS5ERIS']
FX: ['CHF', 'CZK', 'EUR', 'EURCHF', 'GBPEUR', 'JPY', 'SEK']
Metals: ['ALUMINIUM', 'BITCOIN', 'ETHEREUM', 'GOLD_micro', 'IRON', 'SILVER']

Cluster 1 is now risk on, cluster 3 is risk off. We also have a new cluster, containing the vol markets plus - randomly - a couple of FX markets. These were originally in cluster 2.

With N=4

N=4
clusters = cluster_correlation_matrix(corr_matrix,N)
display_clusters(system, clusters)

Cluster 1
Equity: ['AEX', 'BEL20', ... 'SP500_micro', 'TOPIX']
OilGas: ['BRENT-LAST', 'CRUDE_W_mini', 'GASOILINE', 'HEATOIL']
Bond: ['BTP', 'BTP3']
Ags: ['BBCOMM']
FX: ['AUD', 'BRE', ..., 'NZD', 'RUR', 'YENEUR']
Metals: ['COPPER', 'PALLAD', 'PLAT']
Sector: ['EU-AUTO', 'EU-BANKS', '.... 'US-TECH', 'US-UTILS']

Cluster 2
Vol: ['V2X', 'VIX', 'VNKI']
FX: ['CNH', 'SGD']

Cluster 3
Equity: ['FTSEINDO']
OilGas: ['ETHANOL', 'GAS-LAST', 'GAS_US_mini']
Ags: ['BUTTER', 'CHEESE', 'CORN',.... 'SOYMEAL', 'SOYOIL', 'WHEAT']
FX: ['CHF', 'CZK', 'EUR', 'EURCHF', 'SEK']
Metals: ['ALUMINIUM', 'BITCOIN', 'ETHEREUM', 'GOLD_micro', 'IRON', 'SILVER']

Cluster 4
Bond: ['BOBL', 'BONO',.... 'USIRS5', 'USIRS5ERIS']
FX: ['GBPEUR', 'JPY']
Clusters 1 (risk on) and 2 (vol + asian FX) are unchanged, but the risk off cluster has now split into cluster 4 (most of the bonds, and therefore very much still risk off) and cluster 3 (which is now looking more like a generic commodities cluster)


With N=5

Cluster 1
Equity: ['AEX', 'BEL20', 'CAC', 'DAX', .... 'SP400', 'SP500_micro', 'TOPIX']
Sector: ['EU-AUTO', 'EU-BANKS', ... 'US-STAPLES', 'US-TECH']

Cluster 2
Equity: ['BOVESPA', 'FTSECHINAA', 'FTSECHINAH', 'FTSETAIWAN', 'KOSDAQ', 'MSCIASIA']
OilGas: ['BRENT-LAST', 'CRUDE_W_mini', 'GASOILINE', 'HEATOIL']
Bond: ['BTP', 'BTP3']
Ags: ['BBCOMM']
FX: ['AUD', 'BRE', 'CAD', ... 'RUR', 'YENEUR']
Metals: ['COPPER', 'PALLAD', 'PLAT']
Sector: ['JP-REALESTATE', 'US-PROPERTY', 'US-REALESTATE', 'US-UTILS']

Cluster 3
Vol: ['V2X', 'VIX', 'VNKI']
FX: ['CNH', 'SGD']

Cluster 4
Equity: ['FTSEINDO']
OilGas: ['ETHANOL', 'GAS-LAST', 'GAS_US_mini']
Ags: ['BUTTER', 'CHEESE', 'CORN',.... 'SOYOIL', 'WHEAT']
FX: ['CHF', 'CZK', 'EUR', 'EURCHF', 'SEK']
Metals: ['ALUMINIUM', 'BITCOIN', 'ETHEREUM', 'GOLD_micro', 'IRON', 'SILVER']

Cluster 5
Bond: ['BOBL', 'BONO', ... 'USIRS10', 'USIRS5', 'USIRS5ERIS']
FX: ['GBPEUR', 'JPY']


Unchanged from before are cluster 5 (mostly bonds / risk off), cluster 4 (random commodities), cluster 3 (vol + odd fx). But the giant risk on cluster has split. Cluster 1 is now entirely composed of equities (plus sectors). Cluster 2 has most of the Asian equities, plus the other sort of risk on'y things: italian bonds, most FX, some random (low Beta?), a few metals.


With N=6

Existing:
Cluster 1: Core equities risk on
Equity: ['AEX', 'BEL20',... 'SP500_micro', 'TOPIX']
Sector: ['EU-AUTO', 'EU-BANKS', ... 'US-STAPLES', 'US-TECH']

Cluster 2: Random risk on
Equity: ['BOVESPA', 'FTSECHINAA', 'FTSECHINAH', 'FTSETAIWAN', 'KOSDAQ', 'MSCIASIA']
OilGas: ['BRENT-LAST', 'CRUDE_W_mini', 'GASOILINE', 'HEATOIL']
Bond: ['BTP', 'BTP3']
Ags: ['BBCOMM']
FX: ['AUD', 'BRE', 'CAD', ... 'NOK', 'NZD', 'RUR', 'YENEUR']
Metals: ['COPPER', 'PALLAD', 'PLAT']
Sector: ['JP-REALESTATE', 'US-PROPERTY', 'US-REALESTATE', 'US-UTILS']

Cluster 3: Vol + Asian FX
Vol: ['V2X', 'VIX', 'VNKI']
FX: ['CNH', 'SGD']

Cluster 4: Random commodities
Equity: ['FTSEINDO']
OilGas: ['ETHANOL', 'GAS-LAST', 'GAS_US_mini']
Ags: ['BUTTER', 'CHEESE', ... 'SOYMEAL', 'SOYOIL', 'WHEAT']
FX: ['CHF', 'CZK', 'EUR', 'EURCHF', 'SEK']
Metals: ['ALUMINIUM', 'BITCOIN', 'ETHEREUM', 'GOLD_micro', 'IRON', 'SILVER']

New clusters:
Cluster 5: Non US and German bonds, that aren't Italian
Bond: ['BONO', 'CH10', 'JGB', 'KR10', 'KR3', 'OAT']

Cluster 6: US and German Bonds plus odd FX
Bond: ['BOBL', 'BUND', ... 'USIRS5', 'USIRS5ERIS']
FX: ['GBPEUR', 'JPY']

With N=7
New clusters:
Cluster 2: Risk on randomness mostly FX
Metals: ['PALLAD', 'PLAT']
FX: ['AUD', 'BRE', 'CAD', 'GBP', 'INR', 'IRS', 'MXP', 'NOK', 'NZD', 'RUR']
Equity: ['BOVESPA']
Sector: ['JP-REALESTATE', 'US-PROPERTY', 'US-REALESTATE', 'US-UTILS']

Cluster 3: Risk on randomness Asian equity, oil, italian bonds...
Equity: ['FTSECHINAA', 'FTSECHINAH', 'FTSETAIWAN', 'KOSDAQ', 'MSCIASIA']
OilGas: ['BRENT-LAST', 'CRUDE_W_mini', 'GASOILINE', 'HEATOIL']
Ags: ['BBCOMM']
Bond: ['BTP', 'BTP3']
FX: ['KRWUSD', 'YENEUR']
Metals: ['COPPER']


With N=8

New clusters:-
Cluster 4: US/EU vol
Vol: ['V2X', 'VIX']

Cluster 5: JP Vol, Asian FX
Vol: ['VNKI']
FX: ['CNH', 'SGD']

With N=9
New clusters:

Cluster 8: German and US bonds
Bond: ['BOBL', 'BUND', 'BUXL', 'EDOLLAR', 'SHATZ', 'US10', 'US10U', 'US2', 'US20', 'US3', 'US30', 'US5', 'USIRS10', 'USIRS5', 'USIRS5ERIS']
FX: ['JPY']

Cluster 9: GBPEUR
FX: ['GBPEUR']

With N=10 (and a full list of instruments in each cluster)
Cluster 1: Core equity risk on
Equity: ['AEX', 'BEL20', 'CAC', 'DAX', 'DJSTX-SMALL',
         'DOW', 'EU-DIV30', 'EURO600', 'EUROSTX', 'EUROSTX-LARGE',
         'EUROSTX-SMALL', 'KOSPI', 'MSCISING', 'MUMMY',
         'NASDAQ_micro', 'NIFTY', 'NIKKEI', 'NIKKEI400',
         'OMX', 'R1000', 'RUSSELL', 'SMI', 'SMI-MID',
         'SP400', 'SP500_micro', 'TOPIX']
Sector: ['EU-AUTO', 'EU-BANKS', 'EU-BASIC', 'EU-CHEM',
         'EU-CONSTRUCTION', 'EU-DJ-TELECOM', 'EU-FOOD',
         'EU-HEALTH', 'EU-HOUSE', 'EU-INSURE', 'EU-MEDIA',
         'EU-MID', 'EU-OIL', 'EU-REALESTATE', 'EU-RETAIL',
         'EU-TECH', 'EU-TRAVEL', 'EU-UTILS', 'EUROSTX200-LARGE',
         'US-DISCRETE', 'US-ENERGY', 'US-FINANCE', 'US-HEALTH',
         'US-INDUSTRY', 'US-MATERIAL', 'US-STAPLES', 'US-TECH']

Cluster 2: Risk on mostly FX
Metals: ['PALLAD', 'PLAT']
FX: ['AUD', 'BRE', 'CAD', 'GBP', 'INR', 'IRS', 'MXP', 'NOK', 'NZD', 'RUR']
Equity: ['BOVESPA']
Sector: ['JP-REALESTATE', 'US-PROPERTY', 'US-REALESTATE', 'US-UTILS']

Cluster 3 (new): Italian bonds, Asian equity and FX
Bond: ['BTP', 'BTP3']
FX: ['KRWUSD']
Equity: ['FTSETAIWAN', 'KOSDAQ']

Cluster 4 (new): Asian equities, FX, Some energies
Equity: ['FTSECHINAA', 'FTSECHINAH', 'MSCIASIA']
OilGas: ['BRENT-LAST', 'CRUDE_W_mini', 'GASOILINE', 'HEATOIL']
Ags: ['BBCOMM']
FX: ['YENEUR']
Metals: ['COPPER']

Cluster 5: US/EU vol
Vol: ['V2X', 'VIX']

Cluster 6: Asian vol, Asian FX
Vol: ['VNKI']
FX: ['CNH', 'SGD']

Cluster 7: Random commodities, risk off
Equity: ['FTSEINDO']
OilGas: ['ETHANOL', 'GAS-LAST', 'GAS_US_mini']
Ags: ['BUTTER', 'CHEESE', 'CORN', 'COTTON', 'FEEDCOW', 
      'LEANHOG', 'LIVECOW', 'LUMBER', 'MILK', 'MILKDRY', 
      'MILKWET', 'OATIES', 'REDWHEAT', 'RICE', 'SOYBEAN', 
      'SOYMEAL', 'SOYOIL', 'WHEAT']
FX: ['CHF', 'CZK', 'EUR', 'EURCHF', 'SEK']
Metals: ['ALUMINIUM', 'BITCOIN', 'ETHEREUM', 'GOLD_micro', 'IRON', 'SILVER']

Cluster 8: Other bonds
Bond: ['BONO', 'CH10', 'JGB', 'KR10', 'KR3', 'OAT']

Cluster 9: German and US bonds
Bond: ['BOBL', 'BUND', 'BUXL', 'EDOLLAR', 'SHATZ',
       'US10', 'US10U', 'US2', 'US20', 'US3', 'US30',
       'US5', 'USIRS10', 'USIRS5', 'USIRS5ERIS']
FX: ['JPY']

Cluster 10: GBPEUR
FX: ['GBPEUR']

As a heirarchy
  • Risk on
    • Core equities (cluster 1)
    • Mostly FX (cluster 2)
    • Random risk on commodities 
      • Italian bonds, Asian (cluster 3)
      • Asian equities, FX, some energies (cluster 4)
  • Risk off
    • Generic risk off commodities (cluster 7)
    • Bonds + flight to safety FX
      • US and German bonds (cluster 9 and 10)
      • Non US/German/Italian bonds (cluster 8)
    • Vol + Asian FX
      • US/EU vol (cluster 5)
      • Asian vol, FX (cluster 6)


Conclusion
Well that was an interesting exercise, which just goes to show that the traditional grouping of asset classes may not make as much sense as you think. 

To reiterate, these groupings will change over time. 










Posted by



at





5 comments:
  


































        

      









Subscribe to:
Posts (Atom)