Static optimisation of the best set of instruments to hold in a futures trading system

 In a couple of recent posts (here and here) I explored the idea of using dynamic optimisation to deal with the following problem: diversification across markets is good, but requires more capital.

That didn't work out so well!

I can also appreciate that this is *way* beyond most peoples idea of a simple trend following system. And it flies in the face of much I've said in terms of keeping things as simple as possible. Many people would prefer to trade a fixed subset of markets, which gives them the best expected outcome for their capital.

I've explored this somewhat in the past, in particular in this post and also in my third book Leveraged Trading, but it's fair to say I have never before presented and tested systematic and automated methodology for market selection given capital requirements.

Until now...

Don't want to read the post, but just want the results? This page has the current list of instruments I would trade with a given level of capital.

How should we choose which instrument(s) to trade?

This should be a fairly well worn list of criteria if you've read anything of mine before, but here goes:

• Instruments should be not too expensive. In this post I talked about a maximum cost of 1 SR unit per trade.
• Instruments should be not too large. In particular (as discussed here) if we can't hold at least three contracts with a maximum forecast of 20 (twice the average forecast), then we'll suffer a lower SR through not being able to target our required expected risk as closely. 
• Instruments should meet basic liquidity criteria; this post suggests at least $1.25 million per day in risk units, and 100 contracts per day.
• Instruments should be as diversifying as possible. A portfolio consisting of all the US bond futures would be spectacularly pointless.

What is not on this list? Well, we aren't interested in the pre-cost backtested SR of an instrument. These are just too unreliable and have minimal statistical significance.

Thinking a little more deeply about the above criteria, I would say that we can quantify:

• The effect of costs: once we've fitted some forecast weights for a given instrument, we know it's expected annualised turnover, and then we can calculate the expected cost penalty in SR units.
• The effect of size (at least in the handwaving way described here: "There is around a 20% penalty to Sharpe if you can only hold one contract; around 5% with a maximum position of two. This falls to zero if you can hold 3 or 4 contracts" 
• The effect of diversification. Correlations are fairly predictable, and the correlation of instrument trading subsystems are even more so. 

With some assumption of pre-cost SR for each instrument (call it 0.5), and some instrument weights (naturally fitted by handcrafting), we can then calculate the expected post cost SR for a portfolio of any set of instruments. This portfolio will consist of a series of trade offs; an expensive instrument may be included because it is massively diversifying.

Liquidity is harder to think about: instead I'd stick to using the hard limit mentioned above.

To proceed we could do a massive grid search (I'm good at those!) where each node of the grid is a subset of the possible instruments, but I think it makes more sense to proceed iteratively. We begin with one instrument and then succesively add further instruments to that portfolio. At some point it wouldn't be worth adding more instruments (because the benefit of more diversification would be outweighed by the cost of size effects, or by running out of cheap instruments to trade).

The advantage of this is that we can easily do things like add more instruments as our capital grows, or replace instruments if some have to be deleted.  

It is also possible that the goodness of instruments can change over time. In particular, if an instrument becomes riskier then we'll hold smaller positions (can potentially cause size issues), but it would also be cheaper to trade on a risk adjusted basis (improves cost). The reverse would be true if volatility fell. Liquidity and correlations can also change, but typically more slowly (I use very long lookbacks to measure this kind of correlation).

Most likely this substitution of instruments would be an annual process in live trading. 

With the iterative method we can produce what is effectively a 'Top 100' of instruments, ranked in order of preference, allowing some kind of buffering on addition and deletion (like with equity indices; we'd only drop an instrument if it fell more than a certain number of places outside the top section).

We can envisage a situation in which we occasionally swap instruments round by phasing their instrument weight from an old to a new instrument.

Alternatively, we could just keep the set of instruments fixed.

How should we choose the starting instrument

First we remove all instruments that fail the liquidity criteria from our universe (we don't do this in the backtest, but it's something to consider for production). Then we calculate, estimating across all instruments:

• the expected position size (for now assuming a nominal instrument weight of 5% and instrument diversification multiplier of 2.5), and hence the size effect penalty
• the expected turnover of the instrument
• an expected SR for that instrument  equal to Notional SR - (turnover * cost per trade SR) - (size effect penalty)

The size effect penalty is unlikely to have any effect, except for the very largest contracts (like Lumber and full size Bitcoin). It is calculated as follows (assuming notional SR of 0.5):

• Using current risk, what is the current optimal position with a forecast of 20. Call that P
• Remember: "There is around a 20% penalty to Sharpe if you can only hold one contract; around 5% with a maximum position of two. This falls to zero if you can hold 3 or 4 contracts". 
• With a notional SR of 0.5 a 20% penalty is 0.1 SR units and a 5% penalty is 0.025 units. A slightly conservative fit to these points is a penalty of 0.125 / P^2 SR units.
• Something with a P of less than 0.5 is effectively untradeable and should have a penalty of 'infinity' SR units. 

Here's some pysystemtrade code. You can of course hopefully modify this for your own purposes if you're not a user.

def net_SR_for_instrument_in_system(system, instrument_code, instrument_weight_idm=0.25):

    maximum_pos_final = calculate_maximum_position(system, instrument_code, instrument_weight_idm=instrument_weight_idm)
    trading_cost = calculate_trading_cost(system, instrument_code)

    return net_SR_for_instrument(maximum_position=maximum_pos_final,
                                 trading_cost=trading_cost)

# To begin with, we assume that the instrument weight is at least 5% with an IDM of 1.0
# Otherwise we'd end up adding too many large sized contracts initially
# You may need to tweak this for small portfolios

max_instrument_weight = 0.05
notional_starting_IDM = 1.0
minimum_instrument_weight_idm = max_instrument_weight * notional_starting_IDM

from copy import  copy
def calculate_maximum_position(system, instrument_code,
                               instrument_weight_idm = 0.25
                               ):
    if instrument_weight_idm ==0:
        return 0.0

    if instrument_weight_idm>minimum_instrument_weight_idm:
        instrument_weight_idm = copy(minimum_instrument_weight_idm)
    pos_at_average = system.positionSize.get_volatility_scalar(instrument_code)
    pos_at_average_in_system = pos_at_average * instrument_weight_idm
    forecast_multiplier = system.combForecast.get_forecast_cap() / system.positionSize.avg_abs_forecast()

    maximum_pos_final = pos_at_average_in_system.iloc[-1] * forecast_multiplier

    return maximum_pos_final

def calculate_trading_cost(system, instrument_code):
    turnover = system.accounts.subsystem_turnover(instrument_code)
    SR_cost_per_trade = system.accounts.get_SR_cost_per_trade_for_instrument(instrument_code)

    trading_cost = turnover * SR_cost_per_trade

    return trading_cost

def net_SR_for_instrument(maximum_position, trading_cost, notional_SR= 0.5):
    return notional_SR - trading_cost - size_penalty(maximum_position)

def size_penalty(maximum_position):
    if maximum_position<0.5:
           return 9999
    return 0.125 / maximum_position**2



list_of_instruments = system.get_instrument_list()
all_results = []
for instrument_code in list_of_instruments:
    all_results.append((instrument_code,
                        net_SR_for_instrument_in_system(system, instrument_code)))

all_results = sorted(all_results, key = lambda tup: tup[1])

The 'worst' instruments using this metric are Copper, AEX and Palladium which all have less than 0.5 contracts of position.

And here are the very best instruments right now:

[('EU-DIV30', 0.452), ('US10', 0.455), ('EDOLLAR', 0.455), 

('KOSPI_mini', 0.458), ('GAS_US_mini', 0.46), ('US5', 0.463), 

('NASDAQ_micro', 0.466), ('MXP', 0.472), ('SP500_micro', 0.483), 

('GOLD_micro', 0.483)]

So we're going to start trading with the Gold micro future as our first instrument.

best_market = all_results[-1][0]

How should we choose the n+1 instrument

Now what? We need to choose another instrument! And the another, and then another...

• iterate over all instruments not currently in the portfolio
• for a given instrument, construct a portfolio consisting of the old portfolio + the given instrument
• allocate instrument weights using the handcrafting portfolio weighting methodology
• Given the expected SR for each instrument, and the instrument weights, measure the expected portfolio SR
• Choose the instrument with the highest expected portfolio SR. This will be an instrument that provides the best tradeoff between diversification, costs, and size penalty.
• Repeat

list_of_correlations = system.portfolio.get_instrument_correlation_matrix()
corr_matrix = list_of_correlations.corr_list[-1]

from sysquant.optimisation.optimisers.handcraft import *
from sysquant.estimators.estimates import Estimates, meanEstimates, stdevEstimates
from sysquant.optimisation.shared import neg_SR
from syscore.dateutils import WEEKS_IN_YEAR

def portfolio_sizes_and_SR_for_instrument_list(system, corr_matrix, instrument_list):

    estimates = build_estimates(
                                instrument_list=instrument_list,
                                corr_matrix=corr_matrix)

    handcraft_portfolio = handcraftPortfolio(estimates)
    risk_weights = handcraft_portfolio.risk_weights()

    SR = estimate_SR_given_weights(system=system,
                                   risk_weights=risk_weights,
                                   handcraft_portfolio=handcraft_portfolio)

    portfolio_sizes = estimate_portfolio_sizes_given_weights(system,
                                                      risk_weights=risk_weights,
                                                      handcraft_portfolio=handcraft_portfolio)

    return portfolio_sizes, SR


def build_estimates( instrument_list, corr_matrix, notional_years_data=30):
    # we ignore differences in SR for creating instrument weights

    mean_estimates = meanEstimates(dict([
        (instrument_code, 1.0)
        for instrument_code in instrument_list
    ]))

    stdev_estimates = stdevEstimates(dict([
        (instrument_code, 1.0) for instrument_code in instrument_list
    ]))

    estimates = Estimates(correlation=corr_matrix.subset(instrument_list),
                          mean=mean_estimates,
                          stdev=stdev_estimates,
                          frequency="W",
                          data_length=notional_years_data * WEEKS_IN_YEAR)

    return estimates

def estimate_SR_given_weights(system, risk_weights, handcraft_portfolio: handcraftPortfolio):
    instrument_list = list(risk_weights.keys())

    mean_estimates = mean_estimates_from_SR_function_actual_weights(system,
                                                                    risk_weights=risk_weights,
                                                                    handcraft_portfolio=handcraft_portfolio)

    wt=np.array(risk_weights.as_list_given_keys(instrument_list))
    mu=np.array(mean_estimates.list_in_key_order(instrument_list))
    cm=handcraft_portfolio.estimates.correlation_matrix

    SR = -neg_SR(wt, cm, mu)

    return SR

def mean_estimates_from_SR_function_actual_weights(system, risk_weights, handcraft_portfolio: handcraftPortfolio):
    instrument_list = list(risk_weights.keys())
    actual_idm = min(2.5, handcraft_portfolio.div_mult(risk_weights))
    mean_estimates = meanEstimates(dict([
        (instrument_code, net_SR_for_instrument_in_system(system, instrument_code,
                                                          instrument_weight_idm=actual_idm * risk_weights[instrument_code]))
        for instrument_code in instrument_list
    ]))

    return mean_estimates


def estimate_portfolio_sizes_given_weights(system, risk_weights, handcraft_portfolio: handcraftPortfolio):
    instrument_list = list(risk_weights.keys())
    idm = handcraft_portfolio.div_mult(risk_weights)

    portfolio_sizes =dict([
                             (instrument_code,
        round(calculate_maximum_position(system,
                                   instrument_code,
                                   instrument_weight_idm=risk_weights[instrument_code]*idm),1))

    for instrument_code in instrument_list
    ])

    return portfolio_sizes


set_of_instruments_used = [best_market]

unused_list_of_instruments = copy(list_of_instruments)
unused_list_of_instruments.remove(best_market)
max_SR = 0.0
while len(unused_list_of_instruments)>0:
    SR_list= []
    portfolio_sizes_dict = {}
    for instrument_code in unused_list_of_instruments:
        instrument_list= set_of_instruments_used+[instrument_code]

        portfolio_sizes, SR_this_instrument =\
            portfolio_sizes_and_SR_for_instrument_list(system,
                                                       corr_matrix=corr_matrix,
                                                       instrument_list=instrument_list)
        SR_list.append((instrument_code,SR_this_instrument))
        portfolio_sizes_dict[instrument_code] = portfolio_sizes

    SR_list = sorted(SR_list, key=lambda tup: tup[1])
    selected_market = SR_list[-1][0]
    new_SR = SR_list[-1][1]
    if (new_SR)<(max_SR*.9):
        print("PORTFOLIO TOO BIG! SR falling")
        break
    portfolio_size_with_market = portfolio_sizes_dict[selected_market]
    print("Portfolio %s SR %.2f" % (str(set_of_instruments_used), new_SR))
    print(str(portfolio_size_with_market))

    set_of_instruments_used.append(selected_market)
    unused_list_of_instruments.remove(selected_market)
    if new_SR>max_SR:
        max_SR = new_SR

And here is the output, well at least some of it:

Portfolio ['GOLD_micro'] SR 0.69
{'GOLD_micro': 80.0, 'KOSPI_mini': 54.9}
Portfolio ['GOLD_micro', 'KOSPI_mini'] SR 0.87
{'GOLD_micro': 62.2, 'KOSPI_mini': 58.8, 'SHATZ': 334.5}
Portfolio ['GOLD_micro', 'KOSPI_mini', 'SHATZ'] SR 1.09

Expected SR is ramping up as we add our first few instruments: a metal, an equity, and a bond.  Let's skip ahead a bit:

Portfolio ['GOLD_micro',.... 'GBP'] SR 1.67
Portfolio ['GOLD_micro', ..., 'KR3'] SR 1.71
Portfolio ['GOLD_micro', ... 'V2X'] SR 1.73
Portfolio ['GOLD_micro', ... 'NZD'] SR 1.74
Portfolio ['GOLD_micro', .... 'BTP'] SR 1.75
Portfolio ['GOLD_micro', ...'NASDAQ_micro'] SR 1.76
Portfolio ['GOLD_micro', ...., 'EUR'] SR 1.77
Portfolio ['GOLD_micro', ...., 'KR10'] SR 1.79
Portfolio ['GOLD_micro', .... 'LIVECOW'] SR 1.77
Portfolio ['GOLD_micro', ..., 'SMI'] SR 1.76
Portfolio ['GOLD_micro', ...., 'US10'] SR 1.77
Portfolio ['GOLD_micro', ...., 'BITCOIN'] SR 1.77
Portfolio ['GOLD_micro', .... 'EU-DIV30'] SR 1.77
Portfolio ['GOLD_micro', ..., 'BOBL'] SR 1.77
Portfolio ['GOLD_micro', ...., 'EUROSTX'] SR 1.77
Portfolio ['GOLD_micro', ...., 'WHEAT'] SR 1.77
Portfolio ['GOLD_micro', ...., 'OAT'] SR 1.74
Portfolio ['GOLD_micro', ..., 'CORN'] SR 1.75
Portfolio ['GOLD_micro', ...., 'US20'] SR 1.72
Portfolio ['GOLD_micro', ..., 'BUND'] SR 1.70
Portfolio ['GOLD_micro', ...., 'PLAT'] SR 1.71
Portfolio ['GOLD_micro', ...., 'SP500_micro'] SR 1.70
Portfolio ['GOLD_micro', ...     'AUD'] SR 1.68
Portfolio ['GOLD_micro', ... 'FEEDCOW'] SR 1.66
PORTFOLIO TOO BIG! SR falling

You can hopefully see why I allow a 10% tolerance from the maximum achieved Sharpe Ratio before halting. For starters, it's possible for the Sharpe Ratio to go down before rising again to a new level. Next, it's possible to have several portfolios with every similar Sharpe Ratios. On balance we'd want to choose the one with the most instruments I think. Finally, you might be willing to add slightly more instruments than is optimal for a modest theoretical loss in Sharpe Ratio. 

I decided that the portfolio with a SR of 1.75 (ending in Corn) was the one I wanted. After that there is a consistent fall in SR. It has 28 instruments, versus the 18 instruments of the strict maximal SR (ending in KR10 above).

Here it is in full:

Portfolio ['GOLD_micro', 'KOSPI_mini', 'SHATZ', 'US2', 'JPY', 

'LEANHOG', 'MXP', 'GAS_US_mini', 'EDOLLAR', 'CRUDE_W_mini', 

'GBP', 'KR3', 'V2X', 'NZD', 'BTP', 'NASDAQ_micro', 'EUR',

 'KR10', 'LIVECOW', 'SMI', 'US10', 'BITCOIN', 'EU-DIV30', 

'BOBL', 'EUROSTX', 'WHEAT', 'OAT', 'CORN'] SR 1.75

Maximum positions, contracts:

{'GOLD_micro': 9.4, 'KOSPI_mini': 8.1, 'SHATZ': 32.3, 'US2': 45.6,

 'JPY': 4.3, 'LEANHOG': 3.4, 'MXP': 8.9, 'GAS_US_mini': 8.0, 

'EDOLLAR': 11.3, 'CRUDE_W_mini': 3.8, 'GBP': 3.2, 'KR3': 23.2, 

'V2X': 6.1, 'NZD': 2.7, 'BTP': 2.6, 'NASDAQ_micro': 2.7, 'EUR': 2.1,

 'KR10': 3.4, 'LIVECOW': 3.3, 'SMI': 1.3, 'US10': 4.1, 'BITCOIN': 8.5,

 'EU-DIV30': 2.2, 'BOBL': 5.6, 'EUROSTX': 1.2, 'WHEAT': 1.9,

'OAT': 3.1, 'CORN': 1.8, 'US20': 3.2}

Well this is a pretty nice portfolio. It is pretty well diversified, with 28 instruments. About what you'd expect with $500,000 in capital. We have all the sectors represented:

• Metals 2 (including Bitcoin)
• Energies 2
• Equities 5
• Bonds 9
• Ags 4
• Currencies 5
• Vol 1

A few of the instruments do have maximum positions of a couple of contracts or less, but in most of them we're able to have some decent position adjustment. 

We are a little bit overweight bonds perhaps, but that reflects the fact we have quite a few bond markets in the broader universe and many of them are able to take smaller contract sizes. Of course the instrument weights will apportion risk more evenly anyway (I'd do a proper fit rather than the quick and dirty method done here, although the results probably wouldn't be that different).

Different account sizes
Let's run this thing with a few different fund sizes and see what comes out:
A $1 million portfolio
{'GOLD_micro': 8.6, 'NASDAQ_micro': 2.7, 'SHATZ': 81.9, 'US2': 74.4, 
'JPY': 5.0, 'EDOLLAR': 34.2, 'KR3': 47.4, 'CORN': 3.2, 
'CRUDE_W_mini': 4.7, 'LEANHOG': 3.5, 'MXP': 14.7, 'GBP': 3.8, 
'NZD': 5.1, 'BTP': 6.0, 'LIVECOW': 6.7, 'BITCOIN': 10.4, 
'GAS_US_mini': 23.2, 'US10': 12.4, 'WHEAT': 3.4, 'KOSPI_mini': 11.5, 
'SOYBEAN': 2.4, 'OAT': 4.8, 'V2X': 6.1, 'EU-DIV30': 2.5, 'SMI': 1.5,
 'BOBL': 14.2, 'KR10': 9.7, 'COPPER': 1.3, 'FEEDCOW': 4.1,
 'BUND': 2.5, 'SP500_micro': 5.3, 'PLAT': 1.3, 'US20': 1.8, 
'EUR': 4.0, 'EUROSTX': 1.4, 'AUD': 3.2, 'VIX': 0.8}
36 markets. Not perhaps as much of an improvement as you'd have expected - there are diminishing returns to adding markets. 

A $100K portfolio
max_instrument_weight = 0.20
{'GOLD_micro': 2.4, 'KOSPI_mini': 1.4, 'SHATZ': 28.8, 'US2': 23.3, 
'JPY': 0.9, 'LEANHOG': 1.1, 'GAS_US_mini': 4.5, 'EDOLLAR': 8.0, 
'KR3': 5.4, 'NASDAQ_micro': 0.8, 'CRUDE_W_mini': 0.9, 'MXP': 2.8, 
'GBP': 0.9, 'BTP': 1.0, 'NZD': 0.7, 'KR10': 1.2}

A $50K portfolio 
max_instrument_weight = 0.33
{'GOLD_micro': 2.8, 'KOSPI_mini': 1.2, 'SHATZ': 9.8, 
'US2': 3.9, 'V2X': 0.9, 'EDOLLAR': 4.5, 'KR3': 4.3,
 'GAS_US_mini': 3.6, 'LEANHOG': 0.7, 'BITCOIN': 0.7, 
'JPY': 1.0, 'BOBL': 1.7, 'MXP': 0.8, 'EU-DIV30': 0.6}

This page has the current list of instruments I would trade with a given level of capital. It's periodically updated.


Backtesting: I don't think so
I could very easily backtest the above code: reselecting a group of instruments every single year. However I don't see the point. I'm not expecting it to add performance value compared to a benchmark of just using my current set of instruments, or a randomly chosen set of instruments - performance per se isn't one of the things I'm considering. I wouldn't expect it do as well as the hypothetical portfolio where I can take unrounded positions (equivalent to having a much larger account size).

Running in production
To run this concept in production requires a few decisions to be made, and things to be set up:
How often do we want to run this process? 
Costs and volatility will change. Liquidity may also change, and I'm in the process of continously adding potential instruments to my database. New instruments are launched all the time (micro Bitcoin recently, and coming this summer some new yield curve futures, to name just a few). But constant chopping and changing isn't ideal; perhaps once a year?

How will we get the information to make these decisions? 
Liquidity (used as a filter): I do collect volume information, but I would need a process to aggregate this across contracts and combine with risk information.
Trading costs: Commissions. Should hopefully be reasonably stable.
Trading costs: Slippage. For instruments I already trade, I'd need a process to automate the analysis of bid/ask and execution costs. For others, I'd need to set up a process to regularly collect bid/ask price data.

Other calculations can be pulled out of a backtest, once the above are calculated.

What action to take
Once some instruments are ranked in the process described, what changes should we make? Should we always trade the top N instruments, or use some kind of buffer (eg for 30 instruments, if an instrument falls below 35 then always replace it, if it goes above 25 then always include it: similar to how index buffering works). 
Should we have stricter rules for instruments that have failed the liquidity criteria: remove immediately?

How to make changes
How should we transition between the old and new set of instruments? For example, should we use <close only> overrides on instruments that are falling out of favour? Should we smoothly change instrument weights and allow the system to do the rest, with buffering reducing trading costs? Should we add new instruments before removing old ones?

The first transition
For now I have the following current portfolio of instruments (in no particular order):
'AEX', 'AUD', 'BOBL', 'BTP', 'BUND', 'CAC', 'COPPER', 'CORN', 'CRUDE_W_mini', 'EDOLLAR', 'EUR', 'GAS_US_mini', 'GBP', 'GOLD_micro', 'JPY', 'KOSPI_mini', 'KR10', 'KR3', 'LEANHOG', 'LIVECOW', 'MXP', 'NASDAQ_micro', 'NZD', 'OAT', 'BITCOIN', 'SHATZ', 'SMI', 'SOYBEAN', 'SP500_micro', 'US10', 'US2', 'US20', 'US5', 'V2X', 'VIX', 'WHEAT'A
And I want the following reduced set (in order of preference):
['GOLD_micro', 'KOSPI_mini', 'SHATZ', 'US2', 'JPY', 'LEANHOG', 'MXP', 'GAS_US_mini', 'EDOLLAR', 'CRUDE_W_mini', 'GBP', 'KR3', 'V2X', 'NZD', 'BTP', 'NASDAQ_micro', 'EUR', 'KR10', 'LIVECOW', 'SMI', 'US10', 'BITCOIN', 'EU-DIV30', 'BOBL', 'EUROSTX', 'WHEAT', 'OAT', 'CORN']
In theory that would involve dropping the following instruments:
{'AEX', 'BUND', 'SP500_micro', 'AUD', 'VIX', 'COPPER', 'US20', 'SOYBEAN', 'CAC', 'US5'}
And adding these:
{'EUROSTX', 'EU-DIV30'}
And in the process gradually changing/ increasing the instrument weights on other markets.
I'm going to sit on this decision for a little bit longer, whilst I think about the best way to implement this. It may involve a tactical game, waiting for positions to be closed before replacing instruments.

Summary

As a retail trader you are unlikely to have the money to trade 200+ futures markets. You probably only need 15 to 30 for adequate diversification, but which 15 to 30? I've shown how to use a systematic method to select markets based on contract size and costs, but ignoring pre-cost performance - something that isn't sufficiently robust to make these kinds of decisions.
In the next (And final) post on this series I'll consider yet another way of making the best use of small capital - using a dynamic instrument selection method on top of a relatively simple futures trading system.