Monday, 7 June 2021

Optimising my way out of a small fund problem - part one

This is part one of a series of posts about using optimisation to get the best possible portfolio given a relatively small amount of capital.

In this short post I present the idea, and discuss some issues that I need to resolve. It's a bit of a stream of conciousness! It's less of a blog post, and more my random jottings on the subject converted from scribbles to electronic prose. It's a precursor to further posts where I will start designing and testing the method.


I am sorry for my size


There is a little known book about the City of London in the 80's (The buck stops here), in which there is quite an amusing anecote. The stockbroker - who has recently been fired - goes for a meal / drink with a Japanese client:

"His enzymes had let him down again, and he was a bit drunk, in a benign sort of way 'I am sorry, Mr Parton for my size' he kept on muttering. I caught the stares of a few passers-by and wanted to say to them, this man does not mean what you think he means."

Of course the Japanese fund manager is referring to the size of his fund which is relatively modest (and this is why the broker has been canned in the first place. As a specialist in selling European equities to Japanese investors who prefer to invest domestically, or at a push in the US, he is doomed).



My fund, or rather my trading account, is also relatively modest. It's larger than the average retail account, but by no means the multi billion dollars I used to jockey back in the days when I had a proper job. 

This is.... unfortunate. Why does it matter? Obviously it means fewer bragging rights in Soho wine bars, but that doesn't bother me (especially as at the time of writing, Soho wine bars are outside table service with NHS track and trace enabled only). No, what bothers me is this:


The graph shows the increase in expected Sharpe Ratio as you add instruments to a simple trading strategy consisting of a single moving average crossover (And like a good boy, I've put error bars around the Sharpe Ratio estimates). So with one (randomly chosen) instrument the average SR is around 0.24; but if I add another (randomly chosen) instrument it goes up to around 0.3. And with a few wiggles, the increase continues pretty monotonically. And as those error bars show, the improvement is statistically significant.

(I could do even better especially for the first few assets if I deliberately added instruments that diversified the existing pool at each stage, rather than just randomly choosing).

This graph is striking, especially if I compared it to another graph where I added trading rules but kept the number of instruments constant. There the increase is slower, and also begins to show reduced marginal gains. Here we're still getting fairly steady improvements in performance at the 33 instrument mark. If there is an optimal number of instruments (at which point the marginal improvement became non existent) one could trade it's clearly much more than 33, or even the 37 or so I've traded with (give or take) since 2014.

To make a famous quote more accurate:
Diversification across instruments is the only free lunch in finance.

However it isn't actually a free lunch. Every extra instrument you trade will use up capital (this isn't true for trading rules, at least not the way my system is implemented). This problem is most pressing for futures traders, since you can't trade fractions of a futures contract, and most contracts are very large in dollar risk compared to the average persons trading account.

This means that with less capital you can't trade the 400+ or so instruments traded by AHL and other large CTAs.  Even if we put aside the OTC instruments and cash equities that these funds trade, and just stick to futures, there are something like 70 additional futures markets I don't already trade which are liquid enough, not massive in size, have cheap data, and don't cost too much. But there is no way I could trade over 100 markets with my capital.

And this is a serious problem for retail traders, which is why I wrote a whole book about how to make the best use of scarce capital (the subject is also discussed at length in my first and second books). Diversification across instruments is the main competitive advantage that large funds have.

So I'm stuck with around 37 instruments, and I can only manage that many because of an ugly hack that I wrote about at some length here

That ugly hack is worth a brief discussion (though you are welcome to read the post). It relies on the fact that, with some exceptions, a larger forecast (my scaled measure of expected risk adjusted return) implies a larger ex-post risk adjusted return. This is something I analysed in more detail in this more recent post

So in the ugly hack I ignore forecasts that are too small, and then scale up forecasts beyond some threshold more aggresively scale up trades (to ensure that the scaling properties of the forecast are unchanged). I have to do this in markets where my modest capital is most pressing: those with relatively large contract sizes. 

The important point here is that larger forecasts are better - hold on to that point.



Optimisation to the rescue


Now any financial quant worth their salt would read what I've just written and say 'Pff! That's just an optimisation problem'.

'Pff?' I'd reply.

'Mais Oui*. All you need to do is take the expected returns and covariance matrix, limit the optimisation weights to discrete values, and press F9**'  

* Thanks to their excellent Grand Ecole system, most quants are French

** Surprisingly large amounts of the financial system, especially on the sell side, run in Excel

'But where do I get the expected returns from?'

'Boff! You already have the, how do you say, forecasts? A higher forecast means a higher expected return, does it not?'

'Yes, but there is no obvious mapping... Also aren't optimisations somewhat.... well not robust?'

'Only if handled by an inexperienced Rosbif like yourself. For a suitable fee I can of course help you out....'


Now I can't afford to pay this imaginary Quant a fee, and of course she is imaginary, so we'll have to come up with a better solution using a methodology that I understand (no doubt much simpler than is taught in the hallowed lecture theatres of the Ecole Polytechnique). And the building block we're going to use is Black-Litterman.



A brief idiots guide to Black-Litterman

 

Well Black and Litterman are of course the legendary (and sadly missed) Fischer Black of BSM and BDT; and GSAM legend Bob Litterman. And their model deals with the problem I highlighted above 'But where do I get the expected returns from?'

And the answer is you get them from an inverse portfolio optimisation. You start with a portfolio of weights (let's put aside for the moment the question of where they come from). Then you estimate a covariance matrix. Then you run the classical Markowitz optimisation (find the optimal weights given a vector of expected returns and a covariance matrix, and some risk tolerance or utility function) in reverse so it becomes find the expected returns given a vector of weights and a covariance matrix.

BL (as I will say henceforth) used the market portfolio for their starting weights, and hence the resulting implied returns are the 'equilibrium returns'; the returns that are expected given that the 'average' (in a cumulative sense) investor must hold the market portfolio by construction.

Once you have your expected returns you can combine them with some forecasted returns. Perhaps you want to include the discretionary opinion of your chief economist. Or maybe you've got some kind of systematic model for forecasting returns. In any case you take a weighted average of the original equilibrium returns and your forecasts (so this is Bayesian in character as we shrink our forecasts towards the equilibrium returns). Now with your new vector of expected returns you run the normal optimisation forward; using the same covariance matrix you derive a new set of optimal weights.

(The full paper is here)

BL portfolios have some nice properties. If you make no changes at all to the expected returns then you'll recover the original weights (this is a good way to check your code is working!). If you replace them completely, you'll basically have the portfolio implied by your forecasts (which will usually be not very robust at all, with the usual extreme weights problem highlighted). But a blend of the two sets of expected returns, if weighted mostly towards the equlibrium returns, will produce robust portfolios that are tilted away from the market cap weights to reflect our forecasts.

I'm a fan of BL because it accounts, to an extent, for the hierarchy of inputs to a portfolio optimisation. Expected returns are the hardest to forecast, and small changes have a big effect on the output. Standard deviations are relatively easy to forecast, and small changes have a small effect on the output. Correlations fall somewhere in the middle. BL effectively assumes we can predict standard deviations and correlations perfectly, but doesn't make the same assumption about expected returns.

But I don't actually use BL for optimisation, mainly because in the kind of problem I'm usually dealing with (eg deciding how to linearly weight a variety of trading rules and instruments) as it isn't obvious what the 'market cap portfolio' should be. And I'm not going to use it for it's intended purpose here eithier.



The brilliant idea


We can use the BL methodology to do something rather cool and interesting, and fun (and completely different from the original intent). We can run the backward optimisation, and then the forward, without making any changes to the expected returns. Instead we make some other change to the optimisation. Most commonly this would be the introduction of constraints; like a limit on Emerging market exposure, or a position size limit, or ... and this is relevant.... a discrete position size constraint.

So the plan looks something like this:

  • Run my standard position generation function, which will produce a vector of desired contract positions across instruments, all of which will be non integer. Let's call this the 'original' portfolio weights. The main inputs into this calculation are the forecast, instrument weight (as a proportion of risk capital allocated), current volatility of the instrument, long run target volatility and the instrument diversification multiplier (see here, and search for 'why does expected risk vary')
  • Estimate a covariance matrix Σ and a risk aversion coefficient λ
  • Using a reverse Markowitz, BL style, calculate the implied expected returns for each instrument, µ. There is a closed form for the reverse optimisation, since this doesn't have constraints: λΣw
  • Run the optimisation forward using µ, Σ, λ, with a constraint that only integer contract positions can be taken.
Intuitively the sort of thing this process would do is to trade more of instruments with smaller contract size, if they are posiitvely correlated with instruments that are too big to trade. So it's going to be superior to something that just gives you the rounded version of the optimal portfolio (like for example minimising Euclidian distance); which if you have enough instruments and insufficient capital is going to be a vector of zero weights.


The brilliant idea is harder than it first sounds: some small problems


Now there are a lot of unanswered questions here. I've spent a long time thinking about this idea (over 18 months); and it's actually much more complicated than it might first seem.

For the discrete optimisation we're probably going to want to use some kind of grid search. That's going to be slow, especially if I end up with my 'dream' portfolio of 100+ instruments.

In fact it's worse than that, because a great feature of this approach is we can calculate forecasts for instruments we have no intention of trading (because they aren't sufficiently liquid, or are too expensive) as well as instruments that we'd like to trade but the contract size is inordinately large so we can't. And then we can use their forecasts to inform us what our overall portfolio should look like once we apply the discrete constraints; for example the (way too large) Ethereum contract could give me useful information about how to trade the micro Bitcoin future. 

In fact my full wish list currently stands at a total of 228 instruments. Anything we can do to reduce the area that has to be searched would be good! For example, I'd be reluctant to put more than 10% of my risk capital in a single instrument. That sets an upper and lower limit on position size.

I'd also be unhappy changing the sign of a position as a result of an optimisation. I don't want to end up with weird spreading behaviour, just because two instruments are negatively correlated doesn't mean I want to go long/short if say both forecasts are positive. So the lower limit would be zero for a long optimal position, and the upper limit would be zero for a short optimal.

It would probably make a lot of sense to do some kind of coarse to fine search, but I'll discuss specific options for that later. 

It's possible that contracts will move in and out of the 'tradeable / not tradeable' region over time, and rather than adding/removing them manually it would be better to allow an optimisation to do this. There would need to be a list of instruments in a state of 'reduce only', for which the maximum would be the current position (if long, the minimum if short). This list would be updated automatically for instruments that fell below or suddenly qualified for my required criteria for volume and costs.  There would no need to eliminate instruments that were 'too big to trade'; this would happen naturally if 10% of risk capital wasn't sufficient to take even a single contract of position.

It's plausible that there could also be instruments that we couldn't trade at all - eithier permanently or temporarily. For these the maximum and minimum would be equal to the current position. For example, it might be that I get end of day data from one source, for a market I can't afford to get live L1 data for to trade with.

Notice that for these last few points the optimisation would need to have knowledge of the current positions held by the system. In production this means it would make most sense in the pysystemtrade layer that generates 'instrument orders', which sits between the strategy optimal position generation and the execution layer.

In my current trading system this layer currently implements a buffering algo to reduce turnover. It would make sense to replace this with an optimisation that considered explicit costs in it's calculation. It's trivial to calculate the expected cost per contract to do a given trade, assuming you have expected slippage and commision data (which I have). An open question is wether those costs should be amortised over the required holding period rather than assume we're optimising until the next optimisation (in 24 hours presumably), or whether a multiplier should be applied to reflect that costs are more certain than returns (for example, I apply a multiplier of 2 in my normal optimisation of forecast and instrument weights).

Something I have skated over is the fact that my initial strategy will produce desired weights in contract units, and the final optimisation also needs to know about discrete contract units, but 'w' is expressed as a notional position size as a proportion of capital (costs per contract would be in £,$,... units, but one can easily convert that to be a proportion of capital). So I'd need to work out what a single contract was in units of w when determining what the possible discrete step sizes were for each instrument.

Finally one can imagine extending this further; for example by introducing margin requirements into the optimisation.



And some big problems


All of the above problems are mostly just <vaguely waves hand> engineering. I know what needs to be done, it's just a matter of coding it up.

A more difficult question lies around the coefficient of risk aversion, λ. I'm not used to thinking in terms of that at all. However in theory it won't actually matter what λ is set to, as long as we use the same λ in both the reverse and forward optimisation (trivially, so that it is consistent with the closed form of the initial reverse optimisation the form of the forward optimisation must be to maximise max w'µ − λ w'Σw/ 2 rather than the more modern version where we specify a maximum risk and solve for highest return). That should naturally result in a portfolio which has about the same amount of risk as the original. Which is important, because there is information in the amount of risk that the original strategy positions want to take. 

[Note that I could still impose a maximum risk constraint (at say twice my expected annualised target risk of 25% a year); this would replace part of my exogenous risk overlay which effectively fulfills the same function.]

I've left the hardest problem until last, and this is 'What covariance matrix should we use'? Remember a covariance is just the offspring of a correlation matrix and a standard deviation that love each other very much. 

Well the easy part is the standard deviations; I'll just use estimates of percentage annualised risk for each instrument (since we're dealing in w units as a proportion of capital, % risk is the most appropriate). And this seems as good a time as any to introduce a blended estimate of volatility (as discussed here, which will also make another part of my exogenous risk overlay redundant, since it includes a mean reverting component). But what about the correlation?

Should we use the correlation of instrument returns, or should we use the correlation of trading subsystem returns, which after all is what was used (although not directly) to calculate the instrument weights? And which of these should we use in our initial ('reverse') and second ('forward') optimisations?

Let's look at an example. Suppose that we have a 50% instrument weight in SP500, 25% each in US2 and US5 (because the trading subsystems for the two bonds are highly correlated, and historically they've been relatively uncorrelated with SP500), and also suppose those weights are a result of doing a naive markowitz optimisation with some specific correlation matrix of trading subsystem returns (not true in practice, but we'll come to that).

And suppose also that we have equal positive forecasts in all three assets (we we expect the same risk adjusted return). We'll have long positions, but with a larger long position in SP500 than in the other two assets (ignoring the effect of risk; in practice we'd have apply risk scaling to these positions).

What will the implied expected returns look like for these assets after we do the initial reverse optimisation? 

Well if we use the correlation of trading subsystem returns, then in theory we'd end up with expected returns that were equal (actually risk adjusted returns that were equal, but we're ignoring risk and focusing on correlation for now). Which is all fine and correct - since the forecasts are equal.

Let's also suppose however that right now the current correlation of the instrument returns of US2, US5 and SP500 are all equal and positive (so the world has changed, and stocks and bonds are now highly correlated). Then if were to use this correlation matrix in the initial forward optimisation then our implied expected returns would be higher for SP500 than it is for US2 and US10 year (ignoring risk again). This doesn't seem right.

Now what happens if we run the forward optimisation with each of the two matrices. The better option, for me, is to use the current correlation of instrument returns. This deals with the problem I highligted here. If we were to use the long run matrix of subsystem returns we wouldn't be taking into account the change in risk conditions (stocks becoming more correlated with bonds), which is arguably a major flaw of the type of trading system I like to use (forecasts developed independently, and expected risk does not take changes into correlation into account). 

We have four cases:

Reverse / Forward optimisation: which correlation matrix used

A: Subsystem correlation / Subsystem correlation

Implied expected returns will be correct (see above). Final positions will take no account of the fact that stocks are now more correlated with bonds. Using identical matrices will result in consistency and more intuitive results. 


B: Current instrument correlation / current instrument correlation

Implied expected returns will be wrong (see above). Final positions will take account of changes in stock and bond correlations. Using identical matrices will result in consistency and more intuitive results. 


C: Current instrument correlation / Subsystem correlation

Implied expected returns will be wrong (see above).  Final positions will take no account of the fact that stocks are now more correlated with bonds.Using different matrices will result in less intuitive results, may not result in robust portfolios, and could result in unhelpful effects around expected risk targeting.


D: Subsystem correlation / Current instrument correlation

Implied expected returns will be correct (see above). Final positions will take account of changes in stock and bond correlations. Using different matrices will result in less intuitive results, may not result in robust portfolios, and could result in unhelpful effects around expected risk targeting.




We can discount option C right away; it really is the worst of all worlds. 

I don't like option B, since it will result in the 'wrong' expected returns, but perhaps that doesn't actually matter as much as I think it should in practice. As it's using current correlations, it will be more adaptive to different risk conditions. And as it's the same matrix in both optimisations, the BL machinery will work as expected.

Option A will also work, but it won't give us the nice property of giving us a more holistic and dynamic adaptation to portfolio risk. It will be much more like the existing system in character.  

I'm intrigued by option D. In some ways it's the best of both worlds. If it works, then in the example it would have the correct identical expected returns, but then allocate away from SP500 due to it's (temporarily) higher correlation with the other two assets. It gives us a nice holistic and dynamic adaptation. However I'm worried that using two different correlation matrices will make the thing rather weird. It strikes me as likely that the character of the resulting portfolio could be very different from the original, even if we prevent the signs of positions changing.

Also, will it produce the same amount of required risk if I use the same coefficient of risk aversion? Or do I need to calculate the required target risk (for example by scaling the long run risk target I use, 25%, by the aggregate strength of forecasts) and then run the forward optimisation using a maximum standard deviation rather than a coefficient of risk aversion?

There is a technical issue with both options A and D as the correct correlation matrix of subsystem returns (that will result in the expected returns being implied as 'correct' i.e. proportional to forecasts) won't be the same as one you just estimate, because the instrument weights aren't just naively derived from a given correlation matrix; they are robustly optimised. Perhaps that doesn't matter so much for option A since it's the same correlation matrix in both cases (to an extent the correlation matrix is arbitrary). For option D however all the benefit of recovering the correct expected returns will be lost if we don't have the 'right' matrix on the initial optimisation.

I think I have to dismiss option D on the grounds of complexity.

It comes down then to a fight between using the long run correlation of subsystem returns for both forward and reverse optimisation (option A), and using the current correlation of instrument returns for both (option B).  

I'll need to test both of these options to get a feel for how well they work, and whether they have the properties I expect and want.



Some thoughts on testing


I'd be surprised if I was able to run a full backtest with 228 instruments doing a daily optimisation for 40 odd years of data without my laptop committing digital suicide. Instead I'm going to work with a smaller universe of instruments to test what is going on. As well as checking the optimisation does things that make sense, I'm interested in the tracking error between the p&l that would be possible with a large amount of capital versus what the system can actually produce through the optimisation, and whether the expected risk is broadly similar for the original and optimised portfolio.



What next


I think it makes most sense to solve the hardest problem first: which correlation matrices to use. So in the next post I'll be pulling together a little toy system and some examples to get a feel for that. Then subsequently I can think about the simpler problems that mostly involve adding a cost penalty and some more constraints.


Thursday, 27 May 2021

Fit forecast weights by instrument, by group or fit across all markets? Or all three?

I've long been a critic of the sort of people who think that one should run a different trading system for each instrument that you trade. It is the sort of thing that makes intuitive sense; surely the S&P 500 is a completely different animal to the Corn future? And that's probably true for high frequency traders, but not at the sort of timescales that I tend to trade over (holding periods of a couple of weeks up to a couple of months). There I'm using rules that I expect to work over pretty much any instrument I trade, and to perform consistently over long periods of time.

So I've generally advocated pooling information across markets when fitting. My preferred method is to pool gross returns, then apply the costs for each individual instrument, so more expensive instruments will end up trading slower; otherwise everything will look pretty similar.

But... might instrument specific fitting actually work? Or even if that doesn't work, what about pooling together information for similar instruments? Or.... is there some way of getting the best out of all three worlds here: using a blend of instrument specific, globally pooled, and similarity pooled information?

Let's find out.



What exactly is wrong with fitting by instrument?

Let's think about a simple momentum system, where the combined forecast is a weighted average of N different trend signals, each with different speeds. These could be moving average crossovers with some length, or breakouts with some varying window. The only fitting that can be done in this kind of system is to allocate risk weightings differently to different speeds of momentum. Naturally this is a deliberate design decision to avoid 'free-form' fitting of large numbers of parameters, and reduce the issue to a portfolio optimisation problem (which is relatively well understood) with just N-1 degrees of freedom.

The decision we have to make is this: What forecast weights should a given instrument have?

Important note: my trading systems are carefully designed to abstract away any differences in instruments, mostly by the use of risk scaling or risk normalisation. Thus we don't need to estimate or re-estimate 'magic numbers' for each instrument, or calibrate them seperately to account for differences in volatility. Similarly forecasts from each trading rule are normalised to have the same expected risk, so there are no magic numbers required here eithier. This is done automatically by the use of forecast scalars and risk normalisation. 

In a simple portfolio optimisation where all assets have the same expected volatility what matters in determining the weights: Basically correlation and relative Sharpe Ratio (equivalent to mean, given the identical volatilities). 

But it turns out that when you analyse the different correlation across trading rules for different instruments, you get very similar results. 

(There's chunks of pysystemtrade code scattered throughout this post, but hopefully the general approach will be applicable to your own trading system. You may find it helpful to read my posts on optimising with costs, and my preferred optimisation method, handcrafting)


def corr_from(system, instrument):
y = system.combForecast.calculation_of_raw_estimated_monthly_forecast_weights(instrument)
return y.optimiser_over_time.optimiser.calculate_correlation_matrix_for_period(
y.optimiser_over_time.fit_dates[-1]).as_pd().round(2)
corr_from(system, "CORN")
momentum16 momentum32 momentum4 momentum64 momentum8
momentum16 1.00 0.88 0.65 0.61 0.89
momentum32 0.88 1.00 0.41 0.88 0.64
momentum4 0.65 0.41 1.00 0.21 0.89
momentum64 0.61 0.88 0.21 1.00 0.37
momentum8 0.89 0.64 0.89 0.37 1.00

corr_from(system, "SP500")
momentum16 momentum32 momentum4 momentum64 momentum8
momentum16 1.00 0.92 0.60 0.79 0.90
momentum32 0.92 1.00 0.40 0.94 0.71
momentum4 0.60 0.40 1.00 0.29 0.85
momentum64 0.79 0.94 0.29 1.00 0.57
momentum8 0.90 0.71 0.85 0.57 1.00

We can see that the results are fairly similar: in fact they'd result in very similar weights (all other things being equal). 

This is partly because my handcrafted method is robust to correlation differences that aren't significant, but even a vanilla MVO wouldn't result in radically different weights. In fact I advocate using artifical data to estimate the correlations for momentum rules of different speed, since it will give a robust but accurate result.

(Things are a bit different for carry and other more exotic trading rules, but I'll be bringing those in later)

What about Sharpe Ratio? Well there are indeed some differences....

def SR_from(system, instrument):
y = system.combForecast.calculation_of_raw_estimated_monthly_forecast_weights(instrument)
std = np.mean(list(y.optimiser_over_time.optimiser.calculate_stdev_for_period(y.optimiser_over_time.fit_dates[-1]).values()))
means =y.optimiser_over_time.optimiser.calculate_mean_for_period(y.optimiser_over_time.fit_dates[-1])
SR = dict([
(key, round(mean/std,3)) for key,mean in means.items()
])

return SR
SR_from(system, "CORN")
{'momentum16': 0.39, 'momentum32': 0.296, 'momentum4': -0.25, 'momentum64': 0.102,
'momentum8': 0.206}

SR_from(system, "SP500")
{'momentum16': 0.147, 'momentum32': 0.29, 'momentum4': -0.207, 'momentum64': 0.359,
'momentum8': -0.003}

We can see the well known effect that faster momentum isn't much cop for equity indices, as well as some other differences.

But are they significant differences? Are they significant enough that we should use them in determining what weights to use? Here are the forecast weights with no pooling of gross returns for each instrument:


system.config.forecast_weight_estimates['pool_gross_returns'] = False
system.combForecast.get_forecast_weights("CORN").iloc[-1].round(2)
momentum16 0.39
momentum4 0.00
momentum8 0.13
momentum64 0.16
momentum32 0.32
system.combForecast.get_forecast_weights("SP500").iloc[-1].round(2)
momentum16 0.22
momentum4 0.00
momentum8 0.08
momentum64 0.36
momentum32 0.33


The weights are certainly a bit different, although my use of a robust optimisation process (handcrafting) means they're not that crazy. Or maybe it makes more sense to pool our results:

system.config.forecast_weight_estimate['pool_gross_returns'] = True
system.combForecast.get_forecast_weights("CORN").iloc[-1].round(2)
momentum16 0.21
momentum4 0.00
momentum8 0.11
momentum64 0.30
momentum32 0.38
system.combForecast.get_forecast_weights("SP500").iloc[-1].round(2)
momentum16 0.22
momentum4 0.01
momentum8 0.16
momentum64 0.24
momentum32 0.37

(The small differences here are because we're still using the specific costs for each instrument - it's only gross returns that we pool). 

There is a tension here: We want more data to get robust fitting results (which implies pooling across instruments is the way to go) and yet we want to account for idiosyncratic differences in performance between instruments (which implies not pooling).

At the moment there is just a binary choice: we eithier pool gross returns, or we don't (we could also pool costs, and hence net returns, but to me that doesn't make a lot of sense - I think the costs of an instrument should determine how it is traded).

And the question is more complex again, because what instruments should we pool across?  But maybe it would make more sense to pool across instruments within the same asset class? This effectively is what was done at AHL when I worked there due to the fact that we ran seperate teams for each asset class (I was head of fixed income), and each team fitted their own models (What they do now, I dunno. Probably some fancy machine learning nonsense). Or across everything, regardless of costs?

Really, we have three obvious alternatives:

  • Fit by instrument, reflecting the idiosyncractic nature of each instrument
  • Fit with information pooled across similar instruments (same asset class? Perhaps)
  • Fit with information pooled across all instruments

So the point of this post is to test these alternatives out. But what I also want to try is something else: a method which uses a blend of all three methods. In this post I develop a methodology to do this kind of 'blended weights' (not a catchy name! Suggestions are welcome!).



A brief interlude: The speed limit

In my first book I introduce the idea of a 'speed limit' on costs, measured in annualised risk adjusted terms (so effectively a Sharpe Ratio). The idea is that on a per instrument, per trading rule basis it's unlikely (without overfitting) you will get an average SR before costs of more than about 0.40 on average, and you wouldn't want to spend more than a third of that on costs (about 0.13). Therefore it makes no sense to include any trading rules which breach this limit for a given instrument (which will happen if they trade too quickly, and the instrument concerned is relatively expensive to trade).

Now whilst I do like the idea of the speed limit, one could argue that it is unduly conservative. For starters, expensive rules are going to be penalised anyway since I optimise on after costs returns, and I am taking SR into account when deciding the correct weights to use. In fact they get penalised twice, since I include a scaling factor of 2.0 on all costs when optimising. Secondly, a fast rule might not affect turnover on the entire system once added to a bunch of slower rules, especially if it has some diversifying effects. Thirdly, I apply a buffering on the final position for a given instrument, which reduces turnover and thus costs anyway, so the marginal effect of allocating to a faster rule might be very small. 

It turns out that this question of whether to apply the speed limit is pretty important. It will result in different individually fitted instrument weights, different asset groupings, and different results. For this reason I'll be running the results both with, and without the speed limit. And of course I'll be checking what effect this difference has on the pre-cost and after-costs SR.



The setup


Although just looking at momentum rules alone is an interesting exercise to get a feel for the process, and make sure the code worked (I did find a few bugs!), the fact is the rules involved are far too similar to produce meaningfully different results; especially because the handcrafting method I use for optimisation is designed to produce robust weights. 

Instead I decided to use a more interesting set of rules, which basically constitute an evenly spread sample from the rules I use myself:


Here's the correlation matrix for these guys (pooling all instrument returns together)

               assettrend32  breakout10  breakout160  carry10  kurtS_abs30  momentum4
assettrend32 1.00 0.16 0.75 0.29 -0.04 0.29
breakout10 0.16 1.00 0.18 0.08 -0.01 0.82
breakout160 0.75 0.18 1.00 0.37 -0.04 0.35
carry10 0.29 0.08 0.37 1.00 -0.05 0.12
kurtS_abs30 -0.04 -0.01 -0.04 -0.05 1.00 -0.02
momentum4 0.29 0.82 0.35 0.12 -0.02 1.00
momentum64 0.73 0.15 0.89 0.46 -0.03 0.28
mrinasset160 0.02 -0.05 -0.38 -0.11 0.03 -0.11
normmom32 0.80 0.18 0.89 0.33 -0.04 0.34
relcarry 0.04 0.00 0.19 0.63 -0.02 0.02
relmomentum20 0.02 0.25 0.19 0.05 0.01 0.42
skewabs90 -0.02 0.01 0.02 0.11 -0.06 -0.03

               momentum64  mrinasset160  normmom32  relcarry  relmomentum20  skewabs90
assettrend32 0.73 0.02 0.80 0.04 0.02 -0.02
breakout10 0.15 -0.05 0.18 0.00 0.25 0.01
breakout160 0.89 -0.38 0.89 0.19 0.19 0.02
carry10 0.46 -0.11 0.33 0.63 0.05 0.11
kurtS_abs30 -0.03 0.03 -0.04 -0.02 0.01 -0.06
momentum4 0.28 -0.11 0.34 0.02 0.42 -0.03
momentum64 1.00 -0.41 0.87 0.25 0.16 0.08
mrinasset160 -0.41 1.00 -0.45 -0.19 -0.26 -0.01
normmom32 0.87 -0.45 1.00 0.13 0.22 -0.03
relcarry 0.25 -0.19 0.13 1.00 0.03 0.10
relmomentum20 0.16 -0.26 0.22 0.03 1.00 -0.06
skewabs90 0.08 -0.01 -0.03 0.10 -0.06 1.00

There are some rules with high correlation, mostly momentum of similar speeds defined differently. And the mean reversion rule is obviously negatively correlated with the trend rules; whilst the skew and kurtosis rules are clearly doing something quite different.

Here are the Sharpe Ratios (using data pooled across instruments):

{'momentum4': 0.181, 'momentum64': 0.627, 'carry10': 0.623, 
'breakout10': -0.524, 'breakout160': 0.714, 'mrinasset160': -0.271, 
'relmomentum20': 0.058, 'assettrend32': 0.683, 'normmom32': 0.682, 
'relcarry': 0.062, 'skewabs90': 0.144, 'kurtS_abs30': -0.600}


Not all of these rules are profitable! That's because I didn't cherry pick rules which I know made money; I want the optimiser to decide - otherwise I'm doing implicit fitting.

As this exercise is quite time consuming, I also used a subset of my full list of instruments, randomly picked mainly to see how well the clustering of groups worked (so there is quite a lot of fixed income for example):

'AEX', 'AUD', 'SP500', 'BUND', "SHATZ",'BOBL','US10', 'US2','US5', 'EDOLLAR', 'CRUDE_W', 'GAS_US', 'CORN', 'WHEAT'



Fit weights for individual instrument

Step one is to fit weights for each individual instrument. We'll use these for three different purposes:

  • To test instrument specific fitting
  • To decide what instruments to pool together for 'pool similar' fitting
  • To provide some of the weights to blend together for 'blended' weights


system.config.forecast_weight_estimate['ceiling_cost_SR'] = 9999 # Set to 0.13 to get weights with speed limit
system.config.forecast_weight_estimate['pool_gross_returns'] = False
system.config.forecast_weight_estimate['equalise_SR'] = False
system.config.use_forecast_weight_estimates = True
system.config.instruments = ['AEX', 'AUD', 'SP500', 'BUND', "SHATZ",'BOBL','US10', 'US2','US5', 'EDOLLAR', 'CRUDE_W', 'GAS_US', 'CORN', 'WHEAT']

system = futures_system()

wts_dict = {}
for instrument in system.get_instrument_list():
wts_dict[instrument] = system.combForecast.get_forecast_weights(instrument)


Get instrument groupings


The next stage is to decide which instruments to group together for fitting purposes. Now I could, as I said, do this by asset class. But it seems to make more sense to let the actual forecast weights tell me how they should be clustered, whilst also avoiding any implicit fitting through human selection of what constitutes an asset class. I'll use k-means clustering, which I also used for handcrafting. This takes the wts_dict we produced above as it's argument (remember this is a dict of pandas Data Frames, on per instrument):

import pandas as pd
from sklearn.cluster import KMeans


def get_grouping_pd(wts_dict, n_clusters=4):
all_wts_common_columns_as_dict = create_aligned_dict_of_weights(wts_dict)
## all aligned so can use a single index

all_wts_as_list_common_index = list(all_wts_common_columns_as_dict.values())[0].index
## weights are monthly, let's do this monthly or we'll be here all day
annual_range = range(0, len(all_wts_as_list_common_index), int(len(all_wts_as_list_common_index)/40))
list_of_groupings = [
get_grouping_for_index_date(all_wts_common_columns_as_dict,
index_number, n_clusters=n_clusters)
for index_number in annual_range]

pd_of_groupings = pd.DataFrame(list_of_groupings)
date_index = [all_wts_as_list_common_index[idx] for idx in annual_range]
pd_of_groupings.index = date_index

return pd_of_groupings


def get_grouping_for_index_date(all_wts_common_columns_as_dict: dict,
index_number: int, n_clusters = 4):
print("Grouping for %d" % index_number)
as_pd = get_df_of_weights_for_index_date(all_wts_common_columns_as_dict, index_number)
results_as_dict = get_clusters_for_pd_of_weights(as_pd, n_clusters = n_clusters)

print(results_as_dict)

return results_as_dict

def get_df_of_weights_for_index_date(all_wts_common_columns_as_dict: dict,
index_number: int):

dict_for_index_date = dict()
for instrument in all_wts_common_columns_as_dict.keys():
wts_as_dict = dict(all_wts_common_columns_as_dict[instrument].iloc[index_number])
wts_as_dict = dict([
(str(key), float(value))
for key, value in wts_as_dict.items()
])
dict_for_index_date[instrument] =wts_as_dict
as_pd = pd.DataFrame(dict_for_index_date)
as_pd = as_pd.transpose()

as_pd[as_pd.isna()] = 0.0

return as_pd


def get_clusters_for_pd_of_weights(as_pd, n_clusters = 4):
kmeans = KMeans(n_clusters=n_clusters).fit(as_pd)
klabels = list(kmeans.labels_)
row_names = list(as_pd.index)
results_as_dict = dict([
(instrument, cluster_id) for instrument, cluster_id in
zip(row_names, klabels)
])

return results_as_dict

As an example, here are the groupings for the final month of data (I've done this particular fit with a subset of the trading rules to make the results easier to view):

get_grouping_for_index_date(all_wts_common_columns_as_dict, -1)
{'AEX': 3, 'AUD': 3, 'BOBL': 0, 'BUND': 0, 'CORN': 1, 'CRUDE_W': 3, 'EDOLLAR': 0,
'GAS_US': 3, 'SHATZ': 0, 'SP500': 0, 'US10': 0, 'US2': 2, 'US5': 0, 'WHEAT': 1}

There are four groups (I use 4 clusters throughout, a completely arbitrary decision that seems about right with 14 instruments):

- A bond group containing BOBL, BUND, EDOLLAR, SHATZ, US5 and US10; but curiously also SP500

- An Ags group: Corn and Wheat

- US 2 year

- The rest: Crude & Gas; AEX and AUD

These are close but not quite the same as asset classes (for which you'd have a bond group, an Ags group, Energies, and equities/currency). Let's have a look at the weights to see where these groups came from (remember I'm using a subset here):

get_df_of_weights_for_index_date(all_wts_common_columns_as_dict, -1).round(2)

carry10 momentum16 momentum32 momentum4 momentum64 momentum8
AEX 0.39 0.06 0.08 0.31 0.14 0.02
AUD 0.42 0.16 0.10 0.03 0.11 0.19
CRUDE_W     0.40        0.15        0.15       0.05        0.13       0.12
GAS_US      0.42        0.09        0.08       0.12        0.11       0.18

         carry10  momentum16  momentum32  momentum4  momentum64  momentum8
CORN        0.17        0.28        0.23       0.00        0.12       0.20
WHEAT       0.28        0.18        0.23       0.00        0.24       0.07


         carry10  momentum16  momentum32  momentum4  momentum64  momentum8
US2         1.00        0.00        0.00       0.00        0.00       0.00


         carry10  momentum16  momentum32  momentum4  momentum64  momentum8
BOBL        0.66        0.10        0.11       0.00        0.13       0.00
BUND 0.64 0.06 0.13 0.01 0.13 0.03

EDOLLAR     0.67        0.00        0.19       0.00        0.14       0.00

SHATZ 0.79 0.00 0.00 0.00 0.21 0.00
SP500 0.64 0.08 0.10 0.04 0.11 0.04
US10 0.60 0.12 0.12 0.00 0.11 0.06
US5 0.56 0.12 0.13 0.00 0.13 0.06


It's a pretty convincing grouping I think! They key difference between the groups is the amount of carry that they have: a lot (bonds, S&P), a little (the Ags markets) or some (Energies and markets beginning with the letter A). (Note that US 2 year can only trade carry in this particular run - which is with the speed limit on rules. The other rules are too expensive, due to US2 very low volatility. Shatz is a tiny bit cheaper and can also trade very slow momentum. This is enough to put it in the same groups as the other bonds for now).


Fit the system by group


Now we want to fit the system with data pooled for the groups we've just created. These weights will be used for:

  • To test group fitting
  • To provide weights to blend together 
It would be straightforward (and in-sample cheating!) to use a static set of groups for fitting, but we want to use different groups for different time periods.

So the code here is a bit complicated, but it's here in this gist if you're interested.



Fit the entire system with everything pooled


Now for the final fitting, where we pool the gross returns of every instrument. The key configuration change to the default are these two:

system.config.forecast_weight_estimate['pool_gross_returns'] = True # obviously!
system.config.forecast_weight_estimate['ceiling_cost_SR'] = 9999 # ensures all markets grouped

The removal of the speed limit (sharpe ratio ceiling) is key, otherwise the system will only pool returns for instruments with similar costs. Without the ceiling we'll pool gross returns across every instrument.  I can modify the fitted weights to remove rules that exceed the SR ceiling in post processing, when I want to look at the results with the speed limit included.



Use a blended set of weights

Now for the final system: using a blended set of weights. I don't need to do any optimisation here, just take an average of:

  • The individual weights
  • The group fitted weights
  • Weights from results pooled across the entire system

I did originally think I'd do something funky here; perhaps using weights for the averaging which reflected eg the amount of data an individual instrument had (which would increase over time). But I decided to keep things simple and just take a simple average of all three weighting schemes. In any case the handcrafting method already accounts for the length of data when deciding how much faith to put in the SR estimates used for a given estimate, so an instrument with less data would have weights that were less extreme anyway.


What do the weights look like?

To get a feel for the process, here are the weights for US 2 year (with the speed limit imposed, so only rules cheap enough to trade are included). As already noted this is an expensive instrument, so the use of a speed limit will reduce the number of rules it can actually trade (making the results more tractable).  There are some noticeable effects; in particular slow intra asset mean reversion does very badly for US 2 year, but pretty well within it's group and across the entire set of instruments.

               Ind  Group  Entire  Blend
assettrend32 0.24 0.24 0.23 0.24
breakout160 0.14 0.06 0.06 0.09
carry10 0.27 0.17 0.17 0.22
mrinasset160 0.00 0.24 0.32 0.14
normmom32 0.11 0.12 0.06 0.11
relcarry 0.24 0.16 0.16 0.20
Where we to look at the rule weightings for Shatz (another expensive short duration bond market), we'd find that the individual weights were different (again look at mrinasset160), but the grouped and entire system weights would be very similar (since they are in the same group in this case); except that Shatz has an extra rule that is too expensive for US2, and because the instrument costs are a little different: 

               Ind  Group  Entire  Blend
assettrend32 0.17 0.23 0.21 0.20
breakout160 0.06 0.06 0.06 0.06
carry10 0.33 0.17 0.15 0.24
momentum64 0.06 0.06 0.11 0.06
mrinasset160 0.19 0.21 0.28 0.19
normmom32 0.06 0.12 0.05 0.10
relcarry 0.14 0.15 0.14 0.15

Similarly the rules for S&P 500 would be different again, both individually and for the group, but for the entire system they'd be fairly similar (except again, that SP500 has a few more rules it can trade, and is cheaper).

                Ind  Group  Entire  Blend
assettrend32 0.09 0.16 0.14 0.13
breakout10 0.02 0.04 0.03 0.03
breakout160 0.02 0.04 0.04 0.04
carry10 0.15 0.09 0.10 0.12
kurtS_abs30 0.18 0.17 0.05 0.18
momentum4 0.06 0.05 0.05 0.05
momentum64 0.02 0.09 0.07 0.06
mrinasset160 0.06 0.07 0.09 0.07
normmom32 0.04 0.04 0.04 0.04
relcarry 0.07 0.06 0.06 0.06
relmomentum20 0.10 0.07 0.07 0.08
skewabs90 0.17 0.12 0.27 0.13

In all three cases the 'blended' weights are (roughly) an average of the first three columns.


The results!


Remember we have have eight possible schemes here:

  • Fitting individually
  • Fitting across groups
  • Fitting across everything
  • A blend of the above
... and each of these can be done with, or without a 'speed limit' on costs (Any trading rule with a Sharpe Ratio of costs that is above 0.13 will have it's weight set to zero, regardless of what the fitted weights are). We also need a benchmark. Let's use equal weights; which will be hard to beat with the selection of rules we have (not an unusual correlation structure, or any deliberately bad rules). 

Let's just show raw Sharpe Ratios. 


                 All rules                     Speed limit
Individual          0.602                          0.545
Groups              0.656                          0.546
Everything          0.651                          0.587
Blend               0.656                          0.587
Equal wt.           0.657                          0.726

Now these are very similar Sharpe Ratios. We'd get more dramatic results if we used a crap, non robust, fitting method which didn't account for noise: something like Naive Markowitz for example. In this case we'd expect very poor results from the individual instrument weighting, and probably very good results from the blended method, with the other two methods coming somewhere between.

The first thing we notice is that using all rules is consistently better, after costs, than excluding expensive rules based on my 'speed limit' figure. Remember we'll still be giving those costly rules a lower weight for the relevant instruments because of their higher costs; but a rule that manages to handily outperform even it's cost penalty will get a decent weight.

(The exception is for equal weights; if we just equally weight *all* trading rules, that will include some that are far too expensive to trade. Equally weighting only those that pass the speed limit is a great method, and beats everything else!)

What is going on here? Let's look at the effects of costs on SR:

def net_costs(system):
return system.accounts.portfolio().gross.sharpe() - system.accounts.portfolio().sharpe()


                 All rules                       Speed limit
Individual          0.120                           0.083
Groups              0.109                           0.085
Everything          0.096                           0.079
Blend               0.090                           0.083
Equal wt.           0.178                           0.074

Ignoring equal weights, removing the speed limit does increase costs a little, but only by around 1 to 2 SR basis points; versus an improvement in net performance of between 4 and 10 SR basis points (which means gross performance must have gone up by 5 to 12 basis points). 

(For equal weights we have an extra 10 basis points of costs, but only 3 basis points of gross return improvement; hence a net loss of 7 basis points of net return)

The next thing we notice is that pooling across groups and pooling across everything is better than fitting on an individual instrument (the ranking is slightly different, depending on whether we are using the speed limit or not). Blending weights together does about as well as any other option. Equal weights is as good as or better than that.

It doesn't surprise me that pooling across everything is better than fitting by instrument; that was my original opinion. Pooling across groups is equally good; and in fact with more instruments in the portfolio I'd expect the two to end up pretty similar. What might be surprising is that pooling across groups doesn't help much when we only choose cheap rules. But think about how we formed our groups; we clustered things that had similar weights together; with the speed limit these are things that are likely to have the same level of costs. 

It isn't surprising that blended weights are better than everything, as it's a well known effect that averaging weights generally improves robustness and therefore out of sample peformance. Nor is it surprising that equal weights does so well; although it wouldn't look as good with a more esoteric set of trading rules (including ones I hadn't already pre-selected as profitable). 


Summary

To an extent this kind of post is a bit pointless, as trying to improve your optimisation technique is a time-sink that will not result in any serious improvement in performance - though it might result in more robustness. Still it's an itch I had to scratch, and I got to play with my favourite ML tool - clustering.

Why is it pointless? Well it doesn't matter so much what data you use to find your trading rule portfolio weights you use, if you're already using a robust method like handcrafting for fitting. The robust method will mostly correct for anything you do that is stupid.

Having said that a clearly stupid thing to do is to fit weights for each instrument individually - there just isn't enough meaningful information in a single market. Fitting by grouped instruments will make things more robust and also probably improve performance. By the way in my full portfolio I'd probably use more clusters since I have a lot more instruments.

Fitting across the entire portfolio seems to do okay here; but I can't help thinking there are situations in which different instruments will behave differently; I'm thinking for example of a set of rules that includes both slower momentum and faster mean reversion, where the boundary between one and the other working is fluid and depends on the instrument involved (there is some of that in the rules I've got, but not much).

Using a blend of weights is a cop-out if you can't decide what is best, which has other advantages: any kind of averaging makes things more robust. The bad news is that there is quite a lot of work involved here to get the blended weights. A compromise would be to use an average of individual instrument weights and weights fitted across the entire portfolio; this will speed things up a lot as it is estimating the grouped weights that really slows you down.

A more interesting question - and a more surprising result - is whether I should stick to using my 'speed limit' concept. Given most of the rules I trade are fairly slow anyway, it might be worth increasing it a little. This will be especially true if I change my system to one that directly optimises positions in the presence of costs, rather than just buffer.

Finally if you're dealing with a list of trading rules that you know work fairly well across instruments, and you've filtered out those that are too expensive, and the correlation structure is fairly regular: You'd be mad not to use equal weights. 

Friday, 7 May 2021

Adding new instruments, or how I learned to stopped worrying and love the Bitcoin (future)

For the last seven years since I started trading my own account I've pretty much kept the same set of futures markets: around 40 or so, with very occasional changes. The number is limited, as to trade more markets I'd need more capital. The set of markets I have is a compromise between getting a diversified portfolio, avoiding low volatility, not paying too much in trading costs, not incurring excessive data fees, and being able to trade without running into problems with the minimum size required to trade a particular contract.

However, the times they are a' changing. I've been toying with an idea for a new trading system method that will allow me to trade a very large number of markets; to be more precise it will be an optimisation process with a universe of markets that it could trade, in which I'll only take selective positions that make the best use of my limited capital.

A side effect of this is that I will be able to calculate optimal positions for markets I have no intention of trading at all (because they are too expensive, too large, or not sufficiently liquid) and then use that information to make a better decision about what positions I should hold in another market. 

A pre-requisite for that is to actually add more markets to my price database. So that's what this post is all about: deciding which markets to add (using the python interactive brokers [IB] API 'layer' I use, ib_insync), the order to add them in, and explaning the process I follow to include more markets in my open source system, pysystemtrade

(And yes - one of those markets is Bitcoin!)

Of course you don't have to be a pysystemtrade, ib_insync, or python user; or even an Interactive Brokers customer; as much of what I will say will be relevant to all futures traders (and perhaps even to people who trade other things as well).

It also goes without saying, I think, that this would be useful for someone who is building a list of markets from scratch rather than adding to an existing list.



The initial universe


Sadly my broker, interactive brokers (IB), doesn't seem to have a giant .csv file or spreadsheet list of products - at least not externally. So I went on the IB website and basically copied down the list of all the futures products I could find. I excluded single stock futures, since that would have resulted in an even bigger list and I don't want to go those for now. I also excluded markets that I already trade. The key fields I was after were the IB code and Exchange identifier. For example, Ethanol, has an IB code of 'AC' and the exchange is 'ECBOT'.

Here is the initial list.


EDIT: On my initial pass I missed out quite a few of the Singapore SGX instruments (thanks to @HobbyTrading on ET). These are now included at the end of the file.


 

Resolving duplicates


Now the symbol and exchange aren't sufficient to uniquely identify every instrument, since in some markets there are multiple contracts with different multipliers and currencies. So I ran the following code (which doesn't require pysystemtrade, only ib_insync):

import pandas as pd
from ib_insync import Future, IB
missing_data = object()
def identify_duplicates(symbol, exchange):
print("%s %s" % (symbol, exchange))
future = Future(symbol=symbol, exchange = exchange)
contracts = ib.reqContractDetails(future)
if len(contracts)==0:
print("Missing data for %s/%s" % (symbol, exchange))
return missing_data
list_of_ccy= [contract.contract.currency for contract in contracts]
list_of_multipliers = [contract.contract.multiplier for contract in contracts]
unique_list_of_ccy = list(set(list_of_ccy))
unique_list_of_multipliers= list(set(list_of_multipliers))
if len(unique_list_of_multipliers)>1:
print("%s/%s more than one multiplier %s" %
(symbol, exchange, str(unique_list_of_multipliers)))
multiplier = str(unique_list_of_multipliers)
else:
multiplier = unique_list_of_multipliers[0]
if len(unique_list_of_ccy)>1:
print("%s/%s more than one currency %s" %
(symbol, exchange, str(unique_list_of_multipliers)))
currency = str(unique_list_of_ccy)
else:
currency = unique_list_of_ccy[0]
return [symbol, exchange, currency, multiplier]

ib=IB()
ib.connect('127.0.0.1', 4001, clientId=5)
all_market_data = pd.read_csv('/home/rob/private/projects/new_markets/Initial_market_list.csv')
new_market_data = []
for row_data in all_market_data.iterrows():
new_data = identify_duplicates(symbol=row_data[1].Symbol, exchange=row_data[1].Exchange)
if new_data is missing_data:
continue
new_market_data.append(new_data)
new_market_data_pd = pd.DataFrame(new_market_data)
new_market_data_pd.columns=['Symbol','Exchange','Currency','Multiplier']
new_market_data_pd.to_csv('/home/rob/with_additional_fields.csv')


Perusing the list there don't appear to be any duplicates for currency, only half a dozen or so for multipliers. We don't want to make a decision at this stage about which multiplier to trade, so I manually split those rows out in the spreadsheet (I could do this with python of course, but there aren't enough instances to make it worth writing the code). For example, if I look at 'BRR/CMECRYPTO' I can see the multiplier is both 5 and 0.1; the latter is the new micro contract for Bitcoin, the former figure is for the original massive contract.

BRR CMECRYPTO USD ['5', '0.1']

I split that out so it becomes:

BRR CMECRYPTO USD 0.1

BRR CMECRYPTO USD 5



Market selection


There are a few characteristics about markets that are important when making a selection:

  1. Do they diversify our current set of markets?
  2. Are they cheap or expensive to trade?
  3. How much money do I need to trade them?
  4. What kind of volume is there?
  5. Can I get market data for them?
Remember that with my new methodology I can still add instruments which don't diversify, are too expensive, too large, or have insufficient volume, I just wouldn't trade them. But in terms of priority I will want to add instruments which I can actually trade first. 


Measuring trading volume


There are many ways to measure trading volume, but I'm going to do the following. Get the volume for the last 20 business days for all contracts that are currently traded. Divide by 20 to get a daily average. Assuming we have multiple contracts for a given instrument, select the contract with the higest volume (since we'd normally only be trading a single contract at once). 

We now want to work out what that is risk adjusted terms, so we need the annualised risk per contract:

Risk per contract = 16* (daily standard deviation in price units) * M

For example for Eurodollar (December 2023, which I currently hold) the daily standard deviation in price units is about 0.027 price units, so the risk per contract is around $1100. And now we can work out the volume:

Volume in $ risk terms = FX rate * Avg daily volume * risk per contract

So for Eurodollars if the average daily volume was 100,000 contracts then the volume in $ risk terms would about $110 million per day.

In terms of data we're going to need, we want:
  • The volumes traded in each contract (we get this from IB)
  • The exchange rate (we can get this from IB)
  • The multiplier (which we already have)
  • The daily standard deviation in price units (we can calculate this, given ~20 business days of closing prices from IB)

What is a minimum size for daily volume? Well it depends on your typical trade size, and how much the daily volume you are prepared to use up. 

For example, with an account size of $500,000, let's assume that we are only trading a single instrument (which is conservative, of course we'd actually be trading more) with annual risk of 25%. That equates to $125,000 of annual risk. The fastest trading systems I have a turnover of about 25 times a year, so with about 250 business days a year that equates to about 1/10 of annual risk being turned over each trading day: $12,500. If I was willing to be 1% of the market that equates to a minimum volume of $1.25 million a day in risk units.

For Eurodollar we'd only be $25K / $110m = 0.0227% of the market.

Clearly I have no problem trading Eurodollar (!), but it would rule out quite a few almost moribund instruments that only trade a couple of contracts a day. Of course with my new system I could still add them as price inputs, but my priority is going to be adding markets I can actually trade first.

EDIT: There was an interesting question from Chris (see below) which made me think that I should add an additional filter. It would be possible to meet the $1.25 million limit with only 1 contract in volume per day, if that contract was big enough! To be <1% of the market there must, by construction, be more than 100 contracts traded each day. 

Footnote: For some IB contracts (eg Corn) the stated multiplier is actually larger than the actual one. So we need to divide the multiplier used to access the data by contract.priceMagnifier to get the actual multiplier for our calculations.


How much money do I need to trade a market


We need the annualised risk per contract (again using the contract with the highest volume):

Risk per contract = 16* (daily standard deviation in price units) * M

Where M is the contract multiplier. Which we can then convert into dollar terms:

Dollar risk per contract = FX rate * risk per contract

For example for Eurodollar (December 2023, which I currently hold) the daily standard deviation in price units is about 0.027 price units, so the annualised risk per contract is around $1100 (the FX rate is irrelevant).

What is a maximum size for risk per contract?  

As an example, with an account size of $500,000, let's assume that we are only trading a single instrument (which is conservative, of course we'd actually be trading more) with annual risk of 25%. That equates to a maximum $125,000 of annual risk per contract. Now suppose we're happy to hold just a single contract (i.e. we don't get any of the benefits of continous position sizing - again this is very conservative). That means the maximum annualised risk per contract is $125,000

Clearly Eurodollar is fine since we could happily hold over 100 contracts on this basis. But we couldn't trade the full size (5 coin) bitcoin future; with annualised risk of nearly $300,000.



Measuring trading costs


I've talked about how I measure risk adjusted trading costs in some detail elsewhere, but suffice to say I'm going to use the following formula:

Cost per trade = C + M * (spread/2)

Where C is the commission in local currency (the currency the contract is priced in), M is the contract multiplier, and spread is the normal bid/ask spread.

For example for Eurodollar futures (which I trade!) C= 0.85, M = 2500, Spread = 0.005. So the total cost is $7.10.

We now want to work out what that is risk adjusted terms, so we need the annualised risk per contract:

Risk per contract = 16* (daily standard deviation in price units) * M

For example for Eurodollar (December 2023, which I currently hold) the daily standard deviation in price units is about 0.027 price units, so the risk per contract is around $1100.

We can now work out the risk adjusted cost in annual SR units. Here it is for Eurodollar:

Cost per trade, SR  = cost per trade / risk per contract = $7.10 / $1100 = 0.0065


In terms of data we're going to need for a given contract, we want:

  • The commission (which I'm just going to paste into my spreadsheet as an extra column)
  • The multiplier (which we already have)
  • The spread (we need to get this from IB)
  • The daily standard deviation in price units (we already need this for volumes)

Note that the spread may vary depending on which date of contract we are sampling. For this reason I'm going to focus only on the contract with the highest trading volume.

As a rule of thumb anything with a cost above 0.01SR is going to be too expensive (assuming quarterly rolls and a very pedestrian 6 trades per year gets us to 0.10 SR units; just below my absolute maximum 'speed limit' of 0.13: see my books for more information). Eurodollar is an expensive contract which comes in a little below this threshold. In practice I'm going to try and avoid more expensive contracts, although once again I can add them to my price inputs but not actually trade them.


Running the code to extract market volume and cost data



Using this file as the input, the code is here, and the results are here 

EDIT: I've modified this code since the original post to include some additional information and the 100 contract minimum volume.

As well as calculating the risk per contract, volume in risk terms, and risk adjusted cost, it will show the "expiry profile": the current contract calendar, plus the relative volumes of each contract (normalised so the highest volume is 1.0). We'll need this when/if we decide to add a given contract.

This will take you a while!! There are nearly 300 markets, and we need to get market data from IB for every contract that is traded (the rest of the code is pretty swift). 

Note that any market where there is no market data will appear with many NaN values. For me the main markets without data are:

  • NYBOT commodities 
  • NYSELIFFE 
  • Canadian CDE markets
  • Australia SNFE 
  • Hong Kong HKFE
  • Europe MATIF, MONEP, IDEM, MEFFRV, OMS
  • UK ICE: IPE, ICEEU, LMEOTC, ICEEUSOFT

At this point you can decide if it's worth buying additional market data from IB. I decided not to bother, since there are such a large number of contracts I'm already subscribed to but haven't yet got in my database.

For reference my data subscriptions are:

Cboe One - Trader Workstation USD 1.00 / Month 
CFE Enhanced - Trader Workstation USD 4.50 / Month 
Eurex Core - Trader Workstation EUR 8.75 / Month 
Eurex Retail Europe - Trader Workstation EUR 2.00 / Month 
Euronext Basic Bundle - Trader Workstation EUR 3.00 / Month 
Korea Stock Exchange - Trader Workstation USD 2.00 / Month 
Singapore - SGD 2.00 / Month
Osaka - 200 Yen / Month

Fee waivers:
IDEAL FX - Trader Workstation
Physical Metals and Commodities - Trader Workstation
US and EU Bond Quotes - Trader Workstation
Canadian Securities Exchange (CSE) - Trader Workstation
US Mutual Funds - Trader Workstation
US Reg NMS Snapshot - Trader Workstation
US Securities Snapshot and Futures Value Bundle - Trader Workstation 

Comes out to about £15 a month. Something like ICE would cost over $100 a month - I don't think that is worth it for my account size.

EDIT: In the original post I wasn't subscriped to the SGX and OSAKA data feeds. But as these are only about £2 a month together, it made sense to add these as possible markets.

For completeness I also ran the above methodology on my existing markets. The results are here. Although there are no problems with volume, there are a few contracts that have crept over my cost limit (Shatz which I don't trade currently, and US 2 year bonds which I do), and the Palladium contract is over my size limit. This just goes to show that even those with a limited set of markets should review them more often than every 7 years!


Deciding on a market priority order


I firstly grouped the results into 3 buckets:

  • Don't add: Markets without data subscriptions (89 markets) *
  • Add later: Markets which fail eithier the maximum cost test (0.01SR), maximum risk per contract ($125,000) or minimum volume ($1.25 million per day in risk units, and 100 contracts per day)  (96 markets)
  • Add first: Markets that passed all the filters (98 markets)

* It might be possible to get end of day data from Barchart, and still include these markets in my universe as non-traded markets, or as traded markets where execution is done 'blind' (which rules out the use of a tactical execution algo)

Incidentally most of the markets which failed my filters did so because they had low volume (45 instruments), were too expensive to trade (7 instruments), or both (41 instruments). Only three contracts were too large: Lumber, the large BTC contract, and the Ethereum contract.

Then within 'add first' I looked for market groupings that were most diversifying. Now if I was starting from scratch I'd have a different decision to make here, but as a reminder my current market list includes the following asset classes:

  • A small number of commodities
  • Government bonds
  • Volatility (just two)
  • Stock indices
  • IMM FX
  • Metals
  • Energy markets (just two)
  • STIR (just one)
Clearly my first priority should be adding new asset classes. Perusing the list some assets that are missing but which are in my potential new markets include:

  • Bitcoin and Ether
  • Real estate 
  • Interest rate swaps [will be highly correlated with bonds, but useful for fixed income RV trading one of my possible future systems]
  • Stock sector indices [will be correlated to stock indices]
  • Corporate bonds (iBoxx) [though will be correlated with bonds, and have insufficient volume]
There are also some asset classes where I'm pretty light on instruments, and in which there are plenty I could be adding. In particular there are a lot of weird and wonderful commodities I could be trading; and these are likely to be very diversifying. 

Then of course there are asset classes where I'm doing okay, but where it wouldn't hurt to add some markets I don't have (eg DAX, is pretty similar to Eurostoxx, but won't be 100% correlated).

Finally it's worth noting that there are some markets which are exact duplicates of markets I'm already trading, but with different contract sizes. So for example I'm trading the 'E-mini' S&P 500, but there is also a micro contract which I don't trade. It's probably worth picking the instrument with the best cost and volume profile and only trading one of those - ideally that would be the contract with the smallest size. 

When I checked I found that there 7 smaller sized contracts that I could use to replace my existing instruments: Mini Crude, Mini Gas, Mini Copper, Micro Gold, Mini KOSPI, Micro NASDAQ and Micro S&P 500. All are more expensive than my current contracts (though they passed the cost filter with flying colours), but the benefit of better position granulation should outweigh that. 

Now I could probably come up with some quantitative way to work out which markets were the most diversifying, but I figure I'd already punished my IB data feed enough. So instead I ranked the markets in the 'add first' group by hand.

  • Priority 'zero' (in practice will come a bit later as will require some careful surgery of my configuration files when rolling - I won't actually be downloading more prices): Mini and micro contracts that replace existing instruments (7 instruments)
  1. Bitcoin and Real estate (2 instruments)
  2. Swaps (2) 
  3. Stock sectors (22)
  4. New commodity, energy and metals (13)
  5. New currencies and Spanish BONOS, New stock indices (14)
  6. Currency crosses, and missing points off the German,Italian and US bond curve. KOSDAQ150 (8)
  7.  'Last day' crude and gas (2)
Total of 69 instruments.

Then there are some other contracts which I won't be adding:
  • Two I already trade which got included in the list by accident
  • Another 5 year IRS: I don't need two five year swaps, thanks.
  • Larger versions of contracts where I've opted to trade the micro or mini
  • Dollar NIKKEI 225 (I will trade the Yen version for reasons discussed in 'Smart Portfolios')



A case study: Bitcoin


Well let's make a start with the very first contract on the list: the small Bitcoin future, which only started trading very recently. Now it's well known that I don't like Bitcoin. Really I don't. However that doesn't preclude me from trading it. What precludes me from trading it is the hassle and risk of trading it 'cash/spot'.

I've said a ~1000 times I would trade Bitcoin futures if the risk wasn't so big on the full size contract ($278,000 in annualised risk according to my calculations). Well the small contract is only 1/10 of a coin: 1/20 of the size, so it's risk is a mere $5,000 or so (not exactly 1/20 as the measured price standard deviations are different; the one for the small coin is only using a few days of data). Volume is $14m a day and cost is just 0.0018SR units. Finally, to finish with what I can see in the analysis file contracts are monthly, but only the nearest contract appears to be liquid (though of course it is very early days).  

So the time has come for me to trade Bitcoin, or futures at least.



Backfilling prices with barchart

Before we turn on price collection in the trading system, we need to backfill the price with historic data.

Now of course as it is very new we can't backfill the price very much, but I think we can use the large contract as a proxy which has been trading for longer. Note we can't use bitcoin spot as we'd need to adjust it for borrowing and lending rates; a lot of effort even if accurate data was readily available. The large contract goes back a few years (to December 2017 to be exact; the first contract was January 2018).  Not long enough to conduct serious backtesting, but one advantage of the way I trade is that we don't need more than a year or so of data for a new market (just enough to ensure the slowest momentum is reasonably accurate).

However one issue with using IB to provide price data is that they don't give you prices for expired contracts! So even to get a year of data we're going to need to look elsewhere.

EDIT: HobbyTrader on elitetrader.com has pointed out:
For any futures instrument can you ask for a list of conid's, including those which expired within the last two years. This can be done by using the code contract.includeExpired(true); the default value is false. Of this list of conid's you can get the historical data.
 

There are many places where we could get this, but since I have a subscription to Barchart I'm going to use them.

Barchart have an API but I use the old school method (not very factory like I know) of manually downloading the prices. I'm limited to 100 downloads a day, but I'm going to need less than half of that for Bitcoin. I download hourly data, since my system now collects hourly data (although currently only uses end of day data). I end up with files covering each contract from January 2018 to July 2021.

First thing we have to do is rename the Barchart .csv files so they have the format 'BITCOIN_YYYYMMDD' which is expected by my code (note from now on the code has a dependency on pysystemtrade):

import os
from syscore.fileutils import files_with_extension_in_pathname
from syscore.dateutils import month_from_contract_letter


def strip_file_names(pathname):
file_names = files_with_extension_in_pathname(pathname)
for filename in file_names:
identifier = filename.split("_")[0]
yearcode = int(identifier[len(identifier)-2:])
monthcode = identifier[len(identifier)-3]
if yearcode>50:
year = 1900+yearcode
else:
year = 2000+yearcode
month = month_from_contract_letter(monthcode)
marketcode = identifier[:len(identifier)-3]
instrument = market_map[marketcode]

datecode = str(year)+'{0:02d}'.format(month)

new_file_name = "%s_%s00.csv" % (instrument, datecode)
new_full_name = "%s%s" % (pathname, new_file_name)
old_full_name = "%s%s.csv" % (pathname, filename)
print("Rename %s to\n %s" % (old_full_name, new_full_name))

os.rename(old_full_name, new_full_name)

return None

market_map = dict(
BT="BITCOIN")
strip_file_names("mypathname")

Now it's pretty trivial to read in the data, since my .csv data object reader is very flexible and can cope even with the slightly tortured formatting of the barchart files. In fact the main 'gotcha' this code is coping with is the fact that the Barchart files are in a different timezone to the UTC that all my data is aligned to.

import pandas as pd
from syscore.dateutils import adjust_timestamp_to_include_notional_close_and_time_offset
from sysdata.csv.csv_futures_contract_prices import csvFuturesContractPriceData, ConfigCsvFuturesPrices
from sysobjects.futures_per_contract_prices import futuresContractPrices
from sysobjects.dict_of_futures_per_contract_prices import dictFuturesContractPrices
def process_barchart_data(instrument):
config = ConfigCsvFuturesPrices(input_date_index_name="Date Time", datapath=
"/home/rob/data/barchart_csv",
                                                              input_skiprows=1, input_skipfooter=1,
input_column_mapping=dict(OPEN='Open',
HIGH='High',
LOW='Low',
FINAL='Close',
VOLUME='Volume'
))

csv_futures_contract_prices = csvFuturesContractPriceData(datapath=
datapath,
config = config)


all_barchart_data_original_ts=csv_futures_contract_prices.get_all_prices_for_instrument(instrument)
all_barchart_data=
dict([(contractid, index_to_closing(data, csv_time_offset, original_close, actual_close))
for contractid, data in all_barchart_data_original_ts.items()])

all_barchart_data=dictFuturesContractPrices([(key, futuresContractPrices(x))
for key, x in all_barchart_data.items()])

return all_barchart_data

def index_to_closing(data_object, time_offset, original_close, actual_close):
"""
Allows us to mix daily and intraday prices and seperate if required

If index is daily, mark to actual_close
If index is original_close, mark to actual_close
If index is intraday, add time_offset

:param data_object: pd.DataFrame or Series
:return: data_object
"""
new_index = []
for index_entry in data_object.index:
# Check for genuine daily data
new_index_entry = adjust_timestamp_to_include_notional_close_and_time_offset(index_entry, actual_close, original_close, time_offset)
new_index.append(new_index_entry)

new_data_object=pd.DataFrame(data_object.values,
index=new_index, columns=data_object.columns)
new_data_object = new_data_object.loc[~new_data_object.index.duplicated(
keep='first')]

return new_data_object

# slightly weird this stuff, but basically to ensure we get onto UTC time
original_close = pd.DateOffset(
hours = 23, minutes=0, seconds=1)
csv_time_offset = pd.DateOffset(
hours=6)
actual_close = pd.DateOffset(
hours = 0, minutes = 0, seconds = 0)
barchart_data = process_barchart_data("BITCOIN")
barchart_prices_final = barchart_prices.final_prices()
barchart_prices_final_as_pd = pd.concat(barchart_prices_final, axis=1)
barchart_prices_final_as_pd.plot()

We can now look at the prices:





Each colour is a different contact. Importantly there are no gaps. Let's look at the percentage change to see if there are any obvious outliers:

perc=barchart_prices_final_as_pd.diff()/barchart_prices_final_as_pd.shift(1)
perc.plot()




OK it ocasionally gets a bit tasty, but hey - it's bitcoin! And the fun and games in March 2020 isn't a surprise to anyone.

I think we can now write the individual futures prices to our database. I'm using a copy of my production database on my laptop for now; obviously be very careful with this sort of thing on a live machine: I won't be writing to my production machine until I've done plenty of testing.

from sysdata.arctic.arctic_futures_per_contract_prices import arcticFuturesContractPriceData
from sysobjects.futures_per_contract_prices import futuresContractPrices
from sysobjects.contracts import futuresContract

def write_barchart_data(instrument, barchart_prices, delete_first=False):
artic_futures_price_data = arcticFuturesContractPriceData()
# want a clean sheet
if delete_first:
artic_futures_price_data.delete_all_prices_for_instrument_code(instrument, areyousure=True)
list_of_contracts = barchart_prices.keys()
for contractid in list_of_contracts:
futures_contract = futuresContract(instrument, contractid)
artic_futures_price_data.write_prices_for_contract_object(futures_contract, barchart_prices[contractid])

write_barchart_data("BITCOIN", barchart_prices)



Multiple and adjusted prices



The next step is to build the 'multiple' price series. This is effectively a dataframe showing which contracts we're currently using for pricing, the next contract in line (the forward contract) and the contract used to calculate carry. 

So we need to make some decisions about the roll configuration. Our roll cycles are easy: we'll trade every month. When do contracts expire, relative to the first of the month? According to the CME it's on the last Friday. So as an approximation let's say 3 days before the end of the month.

When should we roll? Well it's a bit soon to tell for the BTC micro contract, but for the big contract let's look at what happened to daily volumes of the current and previous contract during the last roll:

barchart_prices['20210400'].VOLUME.resample("1B").sum().plot()
barchart_prices['20210500'].VOLUME.resample("1B").sum().plot()



It looks like we'd need to roll no more than a week or so before expiry; this is very much like a stock index future where we trade the front contract until it is just about to expire. There isn't enough volume in the next contract to roll early, or to hold the second contract consistently. 

Finally we need the carry offset: do we use the previous contract (like with commodities) or the following contract (as with stocks and bonds)? The former is better, but since we are going to be trading the first contract we're going to have to use the latter method. This isn't always going to be accurate, since there is no guarantee (unlike for say Gold) that the slope between spot and first contract (the true carry for the first contract) will be the same as between the first and second contract.

We need to add these parameters to a .csv file:

Instrument HoldRollCycle RollOffsetDays CarryOffset PricedRollCycle ExpiryOffset
BITCOIN FGHJKMNQUVXZ -4 1 FGHJKMNQUVXZ -4

Now I run this script to add the parameters to my database.

Next we need to generate a 'roll calendar'.  This basically shows when we actually roll in the backtest price series from one contract to the next.

from sysinit.futures.rollcalendars_from_arcticprices_to_csv import *
build_and_write_roll_calendar("BITCOIN")

Weird extra step required because of my python environment (shell)

cp ~/.local/lib/python3.8/site-packages/pysystemtrade-0.85.0-py3.8.egg/data/futures/roll_calendars_csv/BITCOIN.csv ~/pysystemtrade/data/futures/roll_calendars_csv/

Now I can actually generate the multiple prices:

from sysinit.futures.multipleprices_from_arcticprices_and_csv_calendars_to_arctic import *
process_multiple_prices_single_instrument("BITCOIN", ADD_TO_CSV=True)

Let's check:

from sysdata.arctic.arctic_multiple_prices import arcticFuturesMultiplePricesData
a = arcticFuturesMultiplePricesData()
price = a.get_multiple_prices("BITCOIN")

Now for the backadjusted prices:
from sysinit.futures.adjustedprices_from_mongo_multiple_to_mongo import *
process_adjusted_prices_single_instrument("BITCOIN", ADD_TO_CSV = True)
More script mumbo jumbo (not actually required for this to work, but will ensure the repo has the BITCOIN prices in .csv format for the simulation):

 cp ~/.local/lib/python3.8/site-packages/pysystemtrade-0.85.0-py3.8.egg/data/futures/multiple_prices_csv/BITCOIN.csv ~/pysystemtrade/data/futures/multiple_prices_csv/

cp ~/.local/lib/python3.8/site-packages/pysystemtrade-0.85.0-py3.8.egg/data/futures/adjusted_prices_csv/BITCOIN.csv ~/pysystemtrade/data/futures/adjusted_prices_csv/


And we check again:

from sysdata.arctic.arctic_adjusted_prices import *
a = arcticFuturesAdjustedPricesData()
a.get_adjusted_prices("BITCOIN").plot()




Bitcoin in simulation


The next step is to see what a backtest for Bitcoin looks like. Importantly this is not so we can make a decision about whether to trade it or not, based on performance. That would be implicit, in-sample, fitting and a very bad thing. No, it's so we can check all the 'plumbing' is working. 

First we need to update this file so the simulation knows about Bitcoin:

Instrument Pointsize AssetClass Currency Slippage PerBlock Percentage PerTrade Description

BITCOIN 0.1 Metals USD 5 1 0 0 Micro Bitcoin

You could argue that Bitcoin should be a different asset class, but at this stage it isn't going to make much difference.
Let's check all the rawdata is feeding through:

from systems.provided.futures_chapter15.basesystem import *
system = futures_system()
system.config.instrument_weights=dict(BITCOIN=1.0)
system.data.get_raw_price("BITCOIN")
system.data.get_instrument_raw_carry_data("BITCOIN")
system.data.get_raw_cost_data("BITCOIN")

Do a bunch of plots and make sure they are vaguely sensible:

system.rules.get_raw_forecast("BITCOIN", "ewmac64_256").plot()
system.rules.get_raw_forecast("BITCOIN", "carry").plot()
system.combForecast.get_combined_forecast("BITCOIN").plot()
system.positionSize.get_subsystem_position("BITCOIN").plot()

Let's do a sense check on the position sizing. 

system.positionSize.get_volatility_scalar("BITCOIN")
2021-04-29    12.135829
2021-04-30    11.984527
2021-05-03    12.291325
2021-05-04    11.987441
2021-05-05    12.237734

This is the position we'd take with a forecast of 10. 
system.config.notional_trading_capital
250000
system.config.percentage_vol_target
20.0
system.config.base_currency
'USD'

That equates to a risk target of $50,000 a year. If we have 12 contracts, each one must have risk of $4167. That's pretty close to the $5000 I calculated above, which was using slightly different data in any case. Now let's check the costs:

system.accounts.get_SR_cost_per_trade_for_instrument("BITCOIN")
0.0017412069830178192

Again that's pretty close to what I worked out before.

Oh, I can't resist. Did we make money?

system.accounts.pandl_for_instrument("BITCOIN").curve().plot()
It's quite a short period of time, and too short to make any judgements. But it doesn't do anything crazy, which is the important thing.

Calibrating an instrument


At this stage if I was doing things normally I'd include some optimisation to work out the correct forecast weights and instrument weight for Bitcoin. However, I'm going to do something a bit different. My long term isn't to add just Bitcoin, but to start doing things very differently, but for the purposes of this post I think it would be nice to actually take the implementation all the way through to completion; basically by adding Bitcoin to my current system. And the easiest way of doing that is to replace an existing instrument.

Remember earlier when I noted that the Palladium contract was just way too big for me now? So 
I'm going to reallocate the Palladium instrument weight (4%) to Bitcoin. Palladium is actually twice as expensive to trade as Bitcoin, so just re-using the Palladium forecast weights is going to be conservative. Plus I've categorised Bitcoin as a 'metal', so why not.


Production trading configuration


Before we go any further, we need to set up the instrument in the production system. 

First I create some instrument configuration, which will copy from this file, by running this script.

I also need to modify my production system .yaml file, basically searching and replacing PALLAD-> BITCOIN.

Finally I need to add the instrument to the IB configuration file:

Instrument IBSymbol IBExchange IBCurrency IBMultiplier MyMultiplier IgnoreWeekly
BITCOIN BRR CMECRYPTO USD 0.1 1 FALSE



Live sampling


In theory, everything should now be setup for live sampling. So in bash, first let's make sure we have the right contracts:

~/pysystemtrade/sysproduction/linux/scripts$ . update_sampled_contracts 


2021-05-06:1525.02 {'type': '', 'component': 'mongoFuturesContractData', 'instrument_code': 'AUD', 'contract_date': '20210300'}  Added contract AUD 20210300
2021-05-06:1525.03 {'type': '', 'broker': 'IB', 'clientid': 138, 'instrument_code': 'AUD', 'contract_date': '20210300'}  Contract AUD/20210300 has expired so now stopped sampling
2021-05-06:1525.05 {'type': '', 'component': 'mongoFuturesContractData', 'instrument_code': 'BITCOIN', 'contract_date': '20210200'}  Added contract BITCOIN 20210200
2021-05-06:1525.07 {'type': '', 'broker': 'IB', 'clientid': 138, 'instrument_code': 'BITCOIN', 'contract_date': '20210200'}  Contract BITCOIN/20210200 now added to database and sampling
2021-05-06:1525.08 {'type': '', 'component': 'mongoFuturesContractData', 'instrument_code': 'BITCOIN', 'contract_date': '20210700'}  Added contract BITCOIN 20210700
2021-05-06:1525.10 {'type': '', 'broker': 'IB', 'clientid': 138, 'instrument_code': 'BITCOIN', 'contract_date': '20210700'}  Contract BITCOIN/20210700 now added to database and sampling

... and so on....

Don't worth about this sort of thing, it's just because I conservatively generate a list of contracts including ones that have expired:

Error 200, reqId 14: No security definition has been found for the request, contract: Future(symbol='BRR', lastTradeDateOrContractMonth='202102', multiplier='0.1', exchange='CMECRYPTO', currency='USD')
2021-05-06:1525.25 {'type': '', 'broker': 'IB', 'clientid': 138, 'component': 'ibFuturesContractData'} [Warning] Reqid 14: 200 No security definition has been found for the request  (BRR/202102)
2021-05-06:1525.26 {'type': '', 'broker': 'IB', 'clientid': 138, 'component': 'ibFuturesContractData'} [Warning] futuresInstrumentWithIBConfigData(instrument=BITCOIN, ib_data=ibInstrumentConfigData(symbol='BRR', exchange='CMECRYPTO', currency='USD', ibMultiplier=0.1, myMultiplier=1, ignoreWeekly=False)) could not resolve contracts: No contracts found matching pattern
2021-05-06:1525.28 {'type': '', 'broker': 'IB', 'clientid': 138, 'component': 'ibFuturesContractData', 'instrument_code': 'BITCOIN', 'contract_date': '20210200'} [Warning] Contract is missing can't get expiry
2021-05-06:1525.29 {'type': '', 'broker': 'IB', 'clientid': 138, 'instrument_code': 'BITCOIN', 'contract_date': '20210200'}  Can't find expiry for BITCOIN/20210200, could be a connection problem but could be because contract has already expired




Now let's get some prices, this time from interactive brokers. Importantly these will be the price of the 0.1 coin, not the 5 coin future. My price filters will bark if the prices are too different, which is one last check:

rob@TradingPC2:~/pysystemtrade/sysproduction/linux/scripts$ . interactive_manual_check_historical_prices 

sysproduction.interactive_manual_check_historical_prices.interactive_manual_check_historical_prices:
Do a daily update for futures contract prices, using IB historical data

If any 'spikes' are found, run manual checks

:return: Nothing


Arguments:
[]


Instrument code?BITCOIN
.....

Now let's have a look at them:

rob@TradingPC2:~/pysystemtrade/sysproduction/linux/scripts$ . interactive_diagnostics 
 INTERACTIVE DIAGONSTICS


0: backtest objects
1: reports
2: logs, emails, and errors
3: View prices
4: View capital
5: View positions & orders
6: View instrument configuration


Your choice? <RETURN for EXIT> 3
30: Individual futures contract prices
31: Multiple prices
32: Adjusted prices
33: FX prices


Your choice? <RETURN for Back> 30
Instrument code?BITCOIN
Available contract dates ['20180100', '20180200', '20180300', '20180400', '20180500', '20180600', '20180700', '20180800', '20180900', '20181000', '20181100', '20181200', '20190100', '20190200', '20190300', '20190400', '20190500', '20190600', '20190700', '20190800', '20190900', '20191000', '20191100', '20191200', '20200100', '20200200', '20200300', '20200400', '20200500', '20200600', '20200700', '20200800', '20200900', '20201000', '20201100', '20201200', '20210100', '20210200', '20210300', '20210400', '20210500p', '20210600f', '20210700']
p = currently priced, c=current carry, f= current forward
Contract date? [yyyymm or yyyymmdd] (ignore suffixes)20210500
                        OPEN     HIGH      LOW    FINAL  VOLUME
index                                                          
2021-03-01 07:00:00  47615.0  47615.0  47615.0  47615.0       1
2021-03-01 09:00:00  48920.0  48920.0  48920.0  48920.0       2
2021-03-01 11:00:00  49145.0  49185.0  49145.0  49185.0       3
2021-03-01 12:00:00  49500.0  49525.0  49500.0  49525.0       2
2021-03-01 13:00:00  49580.0  49755.0  49580.0  49730.0       7
...                      ...      ...      ...      ...     ...
2021-05-06 11:00:00  57000.0  57530.0  56960.0  57465.0     202
2021-05-06 12:00:00  57455.0  58580.0  57375.0  58245.0    2389
2021-05-06 13:00:00  58140.0  58330.0  57830.0  58100.0     316
2021-05-06 14:00:00  58230.0  58230.0  57675.0  57830.0     737
2021-05-06 15:00:00  57825.0  57920.0  57185.0  57390.0    1304

[965 rows x 5 columns]
....

Instrument code?BITCOIN
.....
Contract date? [yyyymm or yyyymmdd] (ignore suffixes)20210400
                      OPEN   HIGH    LOW  FINAL  VOLUME
index                                                  
2020-12-01 11:00:00  20385  20385  20385  20385       2
2020-12-01 15:00:00  20175  20175  20175  20175       1
2020-12-01 17:00:00  19840  19840  19840  19840       1
2020-12-01 20:00:00  19790  19790  19790  19790       1
2020-12-04 14:00:00  19880  19880  19880  19880       8
...                    ...    ...    ...    ...     ...
2021-04-30 11:00:00  54435  54500  54435  54500       3
2021-04-30 12:00:00  54310  54310  54200  54240       8
2021-04-30 13:00:00  54225  54740  54225  54740      21
2021-04-30 14:00:00  54800  56000  54510  56000     123
2021-04-30 15:00:00  56190  56565  55950  56110     189

[1754 rows x 5 columns]
....
Contract date? [yyyymm or yyyymmdd] (ignore suffixes)20210600
                        OPEN     HIGH      LOW    FINAL  VOLUME
index                                                          
2021-01-05 17:00:00  34730.0  34730.0  34730.0  34730.0       1
2021-01-08 21:00:00  42030.0  42030.0  42030.0  42030.0       1
2021-01-11 15:00:00  35350.0  35350.0  35350.0  35350.0       1
2021-01-11 16:00:00  33600.0  33600.0  33600.0  33600.0       2
2021-01-11 17:00:00  35600.0  35600.0  35600.0  35600.0       1
...                      ...      ...      ...      ...     ...
2021-05-06 10:00:00  57625.0  57625.0  57395.0  57395.0       2
2021-05-06 11:00:00  57480.0  57515.0  57425.0  57515.0       3
2021-05-06 12:00:00  58340.0  58930.0  58340.0  58930.0      19
2021-05-06 14:00:00  58445.0  58450.0  58085.0  58235.0      28
2021-05-06 15:00:00  57785.0  57975.0  57470.0  57470.0      17

[1083 rows x 5 columns]


Now to update adjusted prices

rob@TradingPC2:~/pysystemtrade/sysproduction/linux/scripts$ . update_multiple_adjusted_prices 

rob@TradingPC2:~/pysystemtrade/sysproduction/linux/scripts$ . interactive_diagnostics 
0: backtest objects
1: reports
2: logs, emails, and errors
3: View prices
4: View capital
5: View positions & orders
6: View instrument configuration


Your choice? <RETURN for EXIT> 3
30: Individual futures contract prices
31: Multiple prices
32: Adjusted prices
33: FX prices


Your choice? <RETURN for Back> 31
BInstrument code?ITCOIN
                       CARRY CARRY_CONTRACT    PRICE PRICE_CONTRACT  FORWARD FORWARD_CONTRACT
index                                                                                        
2017-12-22 02:00:00  15080.0       20180300  14115.0       20180200  15080.0         20180300
2017-12-22 03:00:00  14065.0       20180300  14505.0       20180200  14065.0         20180300
2017-12-22 04:00:00  15050.0       20180300  15010.0       20180200  15050.0         20180300
2017-12-22 05:00:00      NaN       20180300  14425.0       20180200      NaN         20180300
2017-12-22 07:00:00  13540.0       20180300  13095.0       20180200  13540.0         20180300
...                      ...            ...      ...            ...      ...              ...
2021-05-06 11:00:00  57515.0       20210600  57465.0       20210500  57515.0         20210600
2021-05-06 12:00:00  58930.0       20210600  58245.0       20210500  58930.0         20210600
2021-05-06 13:00:00      NaN       20210600  58100.0       20210500      NaN         20210600
2021-05-06 14:00:00  58235.0       20210600  57830.0       20210500  58235.0         20210600
2021-05-06 15:00:00  57470.0       20210600  57390.0       20210500  57470.0         20210600

[15191 rows x 6 columns]
30: Individual futures contract prices
31: Multiple prices
32: Adjusted prices
33: FX prices


Your choice? <RETURN for Back> 32
Instrument code?BITCOIN
index
2017-12-22 02:00:00    17930.0
2017-12-22 03:00:00    18320.0
2017-12-22 04:00:00    18825.0
2017-12-22 05:00:00    18240.0
2017-12-22 07:00:00    16910.0
                        ...   
2021-05-06 11:00:00    57465.0
2021-05-06 12:00:00    58245.0
2021-05-06 13:00:00    58100.0
2021-05-06 14:00:00    57830.0
2021-05-06 15:00:00    57390.0
Name: price, Length: 15180, dtype: float64


Incidentally, if this was an instrument for which I didn't have barchart data I'd obviously have skipped straight to setting up live sampling; and then got myself as much data as IB could give me.


Live system 'backtest' and optimal trades



rob@TradingPC2:~/pysystemtrade/sysproduction/linux/scripts$ . update_system_backtests 


rob@TradingPC2:~/pysystemtrade/sysproduction/linux/scripts$ . interactive_diagnostics 
0: backtest objects
1: reports
2: logs, emails, and errors
3: View prices
4: View capital
5: View positions & orders
6: View instrument configuration


Your choice? <RETURN for EXIT> 1
10: Roll report
11: P&L report
12: Status report
13: Trade report
14: Reconcile report
15: Strategy report
16: Risk report
Your choice? <RETURN for Back> 14

=============================================================
               Optimal versus actual positions               
=============================================================

                               current        optimal  breaks
.....
medium_speed_TF_carry PALLAD       0.0   -0.017/0.017   False
medium_speed_TF_carry BITCOIN      0.0    0.517/1.111    True



Let's buy some bitcoin then


You should make sure you have trading permissions set up for Bitcoin futures. And if you are based in the UK, you need to make sure your Mfid categorisation is 'Professional' to trade crypto derivatives.


rob@TradingPC2:~/pysystemtrade/sysproduction/linux/scripts$ . update_strategy_orders 

2021-05-06:1547.12 {'type': '', 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'BITCOIN', 'instrument_order_id': ''}  Upper 1.11 Lower 0.52 Current 0 Required position 1 Required trade 1 Reference price 57390.000000  for contract 20210500
2021-05-06:1550.42 {'type': '', 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'BITCOIN', 'instrument_order_id': 30837}  Added order (Order ID:no order ID) Type best for medium_speed_TF_carry BITCOIN, qty [1], fill [0]@ price, None Parent:no parent Children:no_children to instrument order stack with order id 30837


Now the order is executed:

2021-05-10:1123.16 {'type': '', 'broker': 'IB', 'clientid': 152, 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'BITCOIN', 'contract_order_id': 30903, 'instrument_order_id': 30844}  Created a broker order (Order ID:no order ID) Type limit for medium_speed_TF_carry/BITCOIN/20210500, qty [1], fill [0]@ price, None Parent:30903 Children:no_children (not yet submitted or written to local DB)
2021-05-10:1123.18 {'type': '', 'broker': 'IB', 'clientid': 152, 'component': 'ibExecutionStackData', 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'BITCOIN', 'contract_order_id': 30903, 'broker_order_id': ''}  Going to submit order (Order ID:no order ID) Type limit for medium_speed_TF_carry/BITCOIN/20210500, qty [1], fill [0]@ price, None Parent:30903 Children:no_children to IB
2021-05-10:1123.20 {'type': '', 'broker': 'IB', 'clientid': 152, 'component': 'ibExecutionStackData', 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'BITCOIN', 'contract_order_id': 30903, 'broker_order_id': ''}  Order submitted to IB
2021-05-10:1123.21 {'type': '', 'broker': 'IB', 'clientid': 152, 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'BITCOIN', 'contract_order_id': 30903, 'instrument_order_id': 30844, 'broker_order_id': ''}  Submitted order to IB (Order ID:no order ID) Type limit for medium_speed_TF_carry/BITCOIN/20210500, qty [1], fill [0]@ price, None Parent:30903 Children:no_children
2021-05-10:1123.23 {'type': '', 'broker': 'IB', 'clientid': 152, 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'BITCOIN', 'contract_order_id': 30903, 'broker_order_id': 30844}  Managing trade (Order ID:30844) Type limit for medium_speed_TF_carry/BITCOIN/20210500, qty [1], fill [0]@ price, None Parent:30903 Children:no_children with algo 'original-best'
2021-05-10:1123.30 {'type': '', 'broker': 'IB', 'clientid': 152, 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'BITCOIN', 'contract_order_id': 30903, 'broker_order_id': 30844}  Switch to aggressive because Adverse price movement
2021-05-10:1123.31 {'type': '', 'broker': 'IB', 'clientid': 152, 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'BITCOIN', 'contract_order_id': 30903, 'broker_order_id': 30844}  Tried to change limit price to 58215.000000
2021-05-10:1123.33 {'type': '', 'broker': 'IB', 'clientid': 152, 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'BITCOIN', 'contract_order_id': 30903, 'broker_order_id': 30844}  Trade completed
2021-05-10:1123.35 {'type': '', 'component': 'mongoContractOrderStackData', 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'BITCOIN', 'contract_order_id': 30903, 'instrument_order_id': 30844}  Changed fill qty from [0] to [1] for order (Order ID:30903) Type best for medium_speed_TF_carry/BITCOIN/20210500, qty [1], fill [0]@ price, None Parent:30844 Children:[30844]
2021-05-10:1123.36 {'type': '', 'broker': 'IB', 'clientid': 152, 'instrument_code': 'BITCOIN', 'contract_date': '20210500'}  Updated position of BITCOIN/20210500 from 0 to 1; new position in db is 1
2021-05-10:1123.38 {'type': '', 'broker': 'IB', 'clientid': 152, 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'BITCOIN', 'contract_order_id': 30903, 'instrument_order_id': 30844}  Updated position of BITCOIN/20210500 because of trade (Order ID:30903) Type best for medium_speed_TF_carry/BITCOIN/20210500, qty [1], fill [0]@ price, None Parent:30844 Children:[30844] ID:30903 with fills 1
2021-05-10:1123.40 {'type': '', 'broker': 'IB', 'clientid': 152, 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'BITCOIN', 'instrument_order_id': 30844}  Updated position of medium_speed_TF_carry BITCOIN from 0 to 1 because of trade (Order ID:30844) Type best for medium_speed_TF_carry BITCOIN, qty [1], fill [0]@ price, None Parent:no parent Children:[30903] 30844 fill [1]
2021-05-10:1123.41 {'type': '', 'component': 'mongoInstrumentOrderStackData', 'strategy_name': 'medium_speed_TF_carry', 'instrument_code': 'BITCOIN', 'instrument_order_id': 30844}  Changed fill qty from [0] to [1] for order (Order ID:30844) Type best for medium_speed_TF_carry BITCOIN, qty [1], fill [0]@ price, None Parent:no parent Children:[30903]

Almost certainly the top of the market, but I am now the proud (?!) owner of one micro bitcoin future.

Apologies if this post has gone a bit tedious, but I'm trying to illustrate the number of careful checks required when you are doing this sort of exercise. Some of this work can be batched (eg copying configuration into databases), but otherwise it's definitely worth checking the data at each step. Once you have bad data in your database, it's a pain to get it out!


Summary



So, hopefully this has been interesting. For me there are some key points:
  • It's quite nice to just use raw IB_insync for simple work
  • You can see the power of 'risk adjusting everything' when creating some filters to use when deciding which market to trade.
  • There are a lot of futures markets out there which are reasonably liquid, not expensive, and not too big! Even without paying a fortune for market data.
  • The mini/micro futures are a lot cheaper and more liquid than when I first looked at them, so I really ought to be trading them.
  • It's worth rechecking that your instruments still pass filters, probably more than once every seven years
  • Actually adding markets is something you should do carefully.
  • By all means backtest a new market to make sure the sizing and 'plumbing' is correct, but don't make in sample decisions, especially based on just a few years of data.

Next steps: I will now carry on adding markets! This is obviously going to take a while, although (and I won't be able to get barchart data for everything). 

I will also be replacing the 'full fat' contracts I already trade with the micro or mini versions. That's a bit easier: it just involves changing one file, but it's safest to do this when I have zero positions (which might involve waiting for a roll to happen). 

Then I'll be moving on to the issue of trading all of these markets with just the paltry amount of capital I've got. I hope to be covering that in my next blog post.