Wednesday, 9 March 2016

Diversification and small account size


I get occasional emails asking me to cover subjects in my blog (keep them coming! I will eventually get round to them). A pretty common one runs something like this:

"I understand that diversification over instruments is the best way to improve returns- you trade almost 40 futures markets, and the likes of AHL and Winton trade hundreds. But how can someone with a small account trade enough instruments?"

This post will try and answer that question. It will also illustrate the latest feature I've recently added to pysystemtrade - dealing with costs. There is some messy code here.

David Harding (co founder of AHL and Winton) doesn't have a problem with small account size... (afr.com)



The problem


I cover this problem in much more detail in chapter 12 of my book. Briefly though suppose you have a system that trades one instrument, and which holds a few contracts at a time, depending on on the strength of your forecast. Say it has an average position of 4 contracts, and a maximum of 8.

I've said repeatedly that diversification of your trading system across multiple instruments is the most powerful source of extra return. But if you have 10 instruments, rather than one, then clearly your positions will be about 10%* of the initial value: average 0.4, and maximum 0.8.

* actually it's going to be a little more than 10% due to the diversification multiplier, but I'll come back to that.

But you can't own less than one futures contract. So except with high forecasts we'll have no position on; and our position will be one contract at best.

(I'll be talking about futures in this post, but the same problem exists elsewhere. if you spread bet, there is a minimum bet per point. With trades in shares having minimum fees it isn't economic to trade in amounts of less than £1,000 or so.)

In fact I recommend that you are holding at least 4 contracts at the maximum forecast. This implies we could hold only two instruments in this particular portfolio. Two instruments is unlikely to be enough. So we are caught between the devil of fractional contracts and the deep blue sea of insufficient diversification.


The devil of fractional contracts


Let's dig into what happens when we have an instrument for which we can only get at best a 1 lot position. The x axis here shows the forecast. The red line shows that ideally we'd be able to trade fractional positions such that we had an effective forecast equal to what was desired (with the usual capping at absolute values of 20).


However we can't trade fractional lots. In this example we have no position at all unless the forecast hits 15; then subsequently we get a position of one contract. Since one contract is equivalent to a forecast of 15 our effective forecast will be zero, or 15.

The precise forecast at which you'd move to having a position will depend on the instrument, and it's current volatility.

This presents a few problems:
  • The system is binary in nature
  • Behaviour will not be consistent across instruments or across time- it's arbitrary binary
  • Average risk targeting will be incorrect


Bad Binary


 Binary systems aren't very nice, plus they also tend to have higher costs.
  • you wouldn't adjust your position unless your forecast changes massively.
  • you wouldn't adjust your position unless volatility changes massively.
  • you won't adjust your position unless your account value falls massively
  • position adjustments will be sudden : higher costs



Arbitrary binary


The other problem is that not all instruments are the same. A 10% allocation in your portfolio to Eurodollar futures might be enough to get you an average position of a few contracts. But the same allocation in DAX wouldn't get you a single measly contract.

Things also change over time as volatility changes. The cutoff point at which your system puts on it's single position is arbitrary.


The wrong trousers risk


"Oh heck Grommet, I knew I shouldn't have used a students-t distribution"  (ethosdisability.com)


There's another more subtle problem. If we assume say a gaussian distribution for the forecast with a standard deviation of 10, then the forecast passed through the red line filter will have a standard deviation of 9.65 (it's not exactly 10 because of the capping). But the same through the blue line will have a standard deviation of just 5.5. So on average this particular instrument will have less risk than it should do.


Possible solutions


There are a number of different solutions to this problem. The first two are mentioned in my book. The third and fourth are novel.


Live with it - 'arbitrary binary'


The first solution is to just live with arbitrary binary, in exchange for a more diversified system.


Reduce diversification


The second option is to live with the reduced diversification. Perhaps it's better to have one instrument on which you adjust the position properly, rather than a more diversified portfolio of five on which you're effectively running an arbitrary binary system....?



Explicit binary


Perhaps explicit binary is better than arbitrary binary:


We go long one 'unit' with positive forecast, and vice versa. The standard deviation of the blue line is 10 here by the way. The effective forecast might not be 10 in reality; for example for the earlier graph I showed it would be 15, and the standard deviation would also be 15, which is higher than we want.


Thresholded system


Let's have a think about forecasts again. We know that p&l is proportional (in risk adjusted space) to forecast strength. So throwing away small forecasts will probably do limited damage to our returns. We can set a threshold; an absolute forecast levels below which we have no position.

Above the threshold we start buying in, though at a faster rate so we end up with the maximum position of 20 when our desired forecast is also 20.

Let's use a value of 10* for our threshold. We get something like this:

* Other values are possible - higher values will mean you have fewer positions, and lower values do not improve the maximum position so much.


The standard deviation of the blue line is lower than we want, as we've removed a chunk of the forecast distribution (around two thirds, between the -1 and +1 standard deviation points of -10 to +10). We've effectively got spare 'risk' to redistribute.This spare risk can be used to beef up the maximum forecast. 

We end up with something like this:


Here the standard deviation is the same for both red and blue lines. Comparing the red line to the blue line we've exchanged having no position for weak forecasts, for having a larger position when forecasts are strong. A larger position will allow us to have less of a binary problem.

Using forecast thresholding then is a matter of feeding your raw combined forecast into the blue line function mapping.

Of course the result of this in terms of positions will depend on the instrument. Consider for example the instrument in my earlier graph which had a single contract for every 15 units of forecast. The actual result of pushing this through the thresholding function will be as follows:


We can now get 2 positions rather than one; and there will be some limited adjustment.

Here's an instrument for which a forecast of 10 represents one contract:


As you can see the nice thing about this method is that it can be used for instruments which would otherwise have only one to three lots at maximum forecast.

However if you have an instrument with a 'natural' maximum position of 4 contracts (my recommended minimum) you shouldn't need to do this. This would be equivalent to an instrument for which with typical volatility a single contract was equivalent to 5 or fewer forecast units.


Evaluation


We need to measure the expected benefits of diversification, versus the costs of eithier (a) "living with" the problem of arbitrary binary, (b) using explicit binary or (c) thresholded systems.



Measuring diversification benefit



In this section I'm going to answer the question - "How much benefit should I expect to gain from moving from x to x+1 instruments?". I'll be doing this for the 37 futures markets which I currently trade. The results should apply elsewhere; although bear in mind that correlations would be higher (and thus the benefit lower) if for example you were trading only UK equities.

The base system I will be using is the same outlined in chapter 15 of my book; which has 3 variations of the EWMAC trend following rule, plus a carry rule.


Market selection


Selecting the order to include markets in a portfolio of a given size is a complex job which requires weighing up multiple selection criteria*, and a little bit more of an art than a science.

* I list 12 criteria in chapter six of my book.

To make things fair we're going to need to automate the market selection process, and narrow down the selection criteria to just two: asset class, and maximum position size.

The latter is the position we expect to have for the maximum forecast of 20 (see chapter 12); I'll measure this at the subsystem level for an arbitrary amount of capital.

For the first instrument we:
  1. Within each asset class pick the instrument with the largest maximum position
  2. Amongst those top ranked instruments (one per asset class) pick the instrument with the largest maximum position
For subsequent instruments:
  1. Out of asset classes for which we have a gap (fewer assets, and more of that asset class waiting to be picked):
  2. Perform steps 2 and 3 above
Doing this I end up selecting instruments in the following order:

['KR3',  'V2X',  'EDOLLAR', 'MXP', 'CORN', 'EUROSTX', 'GAS_US', 'PLAT',
 'US2', 'LEANHOG', 'GBP', 'VIX', 'CAC', 'COPPER', 'CRUDE_W',
 'BOBL', 'WHEAT', 'JPY', 'NASDAQ', 'GOLD',
 'US5', 'SOYBEAN', 'AUD', 'SP500','PALLAD',
 'KR10', 'LIVECOW', 'NZD', 'KOSPI',
 'US10', 'SMI', 'EUR',
 'OAT', 'AEX',
 'BUND',
 'BTP',
 'US20']

Notice that the number of instruments in each asset isn't constant. The first group closely resembles the set in chapter 15 of my book, although that group excludes Gas and Platinum, and has a US rather than a Korean bond future.

I then tested the following groups of instruments.

  • Sets of 1.... 8 instruments, i.e. going from one asset class to the full set
  • Sets of 15, 20, 25 instruments, i.e. with 2, 3, 4 assets in each asset class (excluding STIR and energies for which we only have 1 and 2 instruments anyway)
  • The full set of 37 instruments.
Because I'm going to be running twelve sets of backtests, I'll be using shrinkage to determine the instrument weights, as this is faster than bootstrapping.


Measuring the benefit


It's going to be hard to demonstrate the benefits of diversification here because not all the markets have data going back in history. This plot, which I've shown before, shows the number of markets with data over time.

This means that a simple test of looking at the improvement in sharpe ratio as we add instruments won't work that well.

For example the first instrument we add is KR3. This has an amazing sharpe, but then it's only been traded for a couple of years. Adding the next instrument actually makes performance worse!

There are some ways round this. For example the instrument diversification multiplier is a measure of how many undiversified bets there are in the portfolio. It's the inverse of how much risk should fall versus trading a portfolio of just one instrument.

If we assume that expected returns are the same for all instrument subsystems, then an IDM of 2.0 should imply that over the long run we'd have a sharpe ratio of twice what we'd get with one instrument.

Because the number of instruments and calculated IDM vary over time, we can scatter plot not just twelve observations, but one for every year. So each year in each of the twelve backtests we measure the average number of instruments currently traded, and the current IDM.

As you can see the IDM increases concavely, asymptoting at around 2.75. The first set of instruments (one per asset class) roughly doubles the IDM from 1.0 to 2.0. Adding another instrument from each asset class then increases it by 50% to around 2.5. The next 20 or so instruments have a marginal effect.

However the IDM is an in sample measure of diversification. We could also look at the ratio of realised risk divided by the IDM. If the IDM is over estimating how much diversification is available, then the realised risk with the IDM backed out will be too high. For example suppose we have an IDM of 3.0, but there's only enough diversification to halve the risk. Then the realised risk will be 50% above target.

Note - realised risk will also be affected by changing forecasts, particularly for a smaller number of instruments.

So if we plot the risk target of 20% annualised, divided by the risk adjusted for IDM, it will give us the effective amount of diversification. Again we can do a scatter plot; each year is a point within one of the twelve backtests:

The maximum diversification measured using effective risk comes in a little lower than with the IDM, at perhaps 2.5 (my preferred maximum for the IDM measure). Because of weak forecasts we can sometimes see this measure below 1.0. Again the first set of instruments from each asset pushes the diversification measure, and thus sharpe ratio, up by about 75%; with a diminishing improvement thereafter.

This suggests that the "true" out of sample correlation of the first eight instruments is around 0.25.



A rule of thumb


The average Sharpe ratio of each individual instrument is around 0.42. This is pretty high given we only have two trading rules: Carry and three slow variations of EWMAC.

However many instruments are only recent arrivers in the porfolio, during a period in which the system did very well indeed. This will be biasing results upwards. If we time weight the returns to correct for this then the average is a more reasonable 0.35.

With that in mind, and keeping the pictures above in mind, I'm going to use these as rules of thumb for diversification within my chapter 15 futures portfolio:

  • One instrument: SR 0.35
  • Two instruments: SR 0.45
  • Three: SR 0.50
  • Four: SR 0.54
  • Five: SR 0.57
  • Six: SR 0.58
  • Seven: SR 0.60
  • Eight instruments (one per asset class): SR 0.61
  • Fifteen instruments (two per asset class, except STIR): SR 0.65
  • All 37 instruments: SR 0.70


Measuring the badness of position sizing


To recap we've got the following options:

  1. Arbitrary binary (live with it)
  2. Explicit binary
  3. Thresholding
For each of these options I'm going to compare the p&l with and without rounded positions for some "small" capital (see below). I'll focus on net Sharpe, but I'll also look at costs - if net Sharpe doesn't change but costs are higher for a particular option then I'd be concerned.

I'm not interested in the absolute sharpe, but the relative one. I'll compare the relative damage done by bad rounding to the relative benefit from diversification.

As well as sharpe ratio I will also look at realised risk over time as this will probably be more 'jagged' with rounding. I'll look at the maximum rolling risk over time for each option (using a 20 week moving average).

The varying number of markets over time is problematic for any kind of backtest evaluation. There are a few solutions to this:

  • look at the whole account curve (which will be dominated by assets with more data). For this particular test this will make life difficult since different assets will be affected by sizing problems to different degrees.
  • take an average of the performance of each asset (eithier equally weighted, or weighted by instrument weights)
  • take an average, weighted by the amount of available data, of the performance of each asset. An easy way to do this is to 'stack up' the returns from each asset, and take a sharpe ratio from the entire set.

Here's a quick summary of my findings:
  • If you have enough capital running the system "normally" is around 20% better than using a binary or a threshold system. This comes from much lower returns and slightly lower vol; peak risk also falls a little. Much of the lower return for threshold systems comes from higher cost.
  • If you only have enough capital for a single contract at maximum position then the sharpe ratios of arbitrary binary, explicit binary, and thresholding are fairly similar. The return for arbitrary binary will be higher, but with higher vol, higher costs, and higher peak volatility. Costs are a little lower for thresholding, and lower again for explicit binary.
  • There is around a 20% penalty to Sharpe if you can only hold one contract; around 5% with a maximum position of two. This falls to zero if you can hold 3 or 4 contracts.
  •  Correlations will be lower when using thresholding; so the IDM of the whole portfolio will be higher. This will compensate for the lower return and lower vol.
 My conclusion is that if you can only hold one or two contracts then you should use a threshold filter for your forecasts. This will give you a Sharpe ratio penalty of around 20% if you can only hold one contract or 5% if you can only hold two. So an average SR of 0.36 will become 0.29 or 0.34.


Putting it together


The following graph summarises the results of diversification and small contract size.


Y axis is sharpe ratio, x axis number of assets. The blue line show the results from earlier; the diversification benefits that accrue as you add instruments in the order I've suggested. These will only be applicable if your maximum position is three, four, or more contracts. If you can only hold two contracts you'll get the red line, with one contract you'll get the yellow line.

To use this graph let's say you had a choice between trading 8 assets with a maximum position of a single contract, or trading 2 assets with a maximum position of three or more contracts. You can see that the former has a SR of about 0.49 and the latter about 0.45. So in this case diversification wins over the problems of position rounding.

Alternatively suppose you were considering going from 8 instruments, and being able to hold three or more contracts, up to the full portfolio of 37 futures but only holding a maximum of one contract. The Sharpe ratios are about 0.61 and 0.56 respectively. For this at least the diversification isn't worth it.

As a general rule adding diversification wins for the first few assets, even if it means holding maximum positions of just one or two contracts. After that the costs of discrete positions bite into your returns. You need a lot more money to make it worth adding further instruments.

Specific portfolios


To make things concrete let's consider a few different account sizes: $5,000; $10,000, $50K, $100K, $250K, $500K and a million dollars. In all cases we're using a percentage vol target of 20%, except for the very first portfolio. I also assume equal weights; for later portfolios where there is a preponderance of particular assets that might not be accurate.

Note that in many cases we have different maximum positions for different instruments; hence a straightforward application of the previous graph isn't possible.

Capital of $2,500: One instrument


This is just enough to get to a single contract on eithier the KR3 or V2X futures. I'd suggest using thresholding on these first few portfolios.

Because a single contract has a very low expected sharpe ratio the vol target of 20% is a little high given my usual insistence on using a half kelly figure. I'd run this mini portfolio at 15% annualised risk.


Capital of $5,000: Two instruments


This is just enough to get to a single contract on both the KR3 and V2X futures, assuming a 50% weight to each and an IDM of 1.27*. This is superior to sticking to one instrument, and being able to get two contracts.

* all IDM's from table 18 of my book, assuming correlation of 0.25

Capital of $10,000: Three instruments


This is just enough to get to a single contract on Eurodollar, and two on both the KR3 and V2X futures, assuming a 33% weight to each and an IDM of 1.4.

Incidentally with $10K we could also hold just KR3 or just V2X and get four contracts. However we know that this is suboptimal.

Capital of $20,000: Four instruments


This is just enough to get to a single contract on Eurodollar and MXPUSD, and three on both the KR3 and V2X futures, assuming a 25% weight to each and an IDM of 1.51.


Capital of $30,000: Five instruments


3 contracts a piece: KR3, V2X; Single contracts: Eurodollar, MXP, Corn


Capital of $50,000: Six instruments


Five contracts a piece: KR3, V2X; Two contracts: Eurodollar, MXP; Single contracts: Corn, Eurostoxx

At this stage you could stop using thresholding on the first two instruments, or for consistency keep it on all instruments.


Capital of $80,000: Seven instruments


KR3, V2X, Eurodollar, MXP, Corn, Eurostoxx, US Gas

Note: Contracts in italics do not require thresholding.


Capital of $100,000: Eight instruments


KR3, V2X, Eurodollar, MXP, Corn, Eurostoxx, US Gas, Platinum

Note: Contracts in italics do not require thresholding.


Capital of $300,000: Fifteen instruments


KR3, V2X, Eurodollar, MXP, Corn, Eurostoxx, US Gas, Platinum,
US2, Leanhog, GBP, VIX, CAC, Copper, Crude

Note: Contracts in italics do not require thresholding.


Capital of $800,000: 37 instruments


KR3, V2X, Eurodollar, MXP, Corn, Eurostoxx, US Gas, Platinum,
US2, Leanhog, GBP, VIX, CAC, Copper, Crude
Bobl, Wheat, JPY, Nasdaq, Gold,
US5, Soybean, AUD, SP500, Pallad,
KR10, Livecow, NZD, Kospi,
US10, SMI, EUR, OAT, AEX,
Bund, BTP, US20

Note: Contracts in italics do not require thresholding.


Very rich people


With a capital of $2.2 million you could hold 37 instruments with a maximum position of at least 2 contracts in each.

With a capital of $3.25 million you could hold 37 instruments with a maximum position of at least 3 contracts in each. You shouldn't need to use thresholding at this stage for any instrument.

With a capital of $4.25 million you could hold 37 instruments with a maximum position of at least 4 contracts in each.

Beyond this you could think about adding further instruments. The largest CTA's, like my ex-fund AHL have 100+ instruments in their portfolio.


Conclusion


I've hopefully answered the question "what do I do with a small account". To summarise, you're probably better off using a binary or thresholded forecast filter with as wide a range of instruments as you can manage, at least until you have $100K or so.


98 comments:

  1. There's a mini DAX on Eurex now, 5x I think.

    At a cost, there's mini everything in certificate or CFD form. Options also allow to take future sized equivalent positions for a few hundred quid, though you need some volatility model.

    ReplyDelete
    Replies
    1. Oh sure and you can also spread bet or use ETF's (maybe not on the commodities but equities and bonds). Generally though these other options are more expensive and so have to be traded slower. A similar post could be done on whether it's better to trade futures fast and be less diversified, or say trade spread bets but slower.

      Delete
  2. what is the assumed margin / leverage on the portfolios above?

    ReplyDelete
    Replies
    1. Heck, how long is a piece of string? It's whatever it takes to hit the volatility target. I can't easily compute this for every portfolio, but I can give you my own figures for the full 37 futures. Note I'm running at 25% annualised vol, rather than 20%.

      My margin runs at around 35% of capital at risk on average.

      Leverage (gross value of positions divided by capital) is around 14.6 times. But leverage is a very rough way to measure risk. About a third of that is in one position (Eurodollars), and another third in bond futures. The leverage on the rest of my portfolio runs at around 7 times.

      Delete
    2. taking the portfolio capital of $2,500, the V2TX contract requires ~1150 EUR of overnight margin, so if you used all of risk capital as margin you could hold 2 contracts. How does this tie in with the volatility target? Also, unrelated, the ticker codes you use are from where? I notice they are not the IB codes. Thanks

      Delete
    3. I don't even look at margin, because with the kind of risk targets I run I rarely use more than 50% of the available margin. Using all your capital as margin is usually too aggressive.

      The ticker codes are entirely my own invention, as I find it hard to remember the various crazy codes that I've used in my life (Bloomberg, I'm talking about you...) so when I created my own system decided to make up easy to use ones

      Delete
    4. Hi Robert - may I ask you a couple of questions?

      1) what is the typical margin required by brokers on big contracts in fixed income like the Euro$ and T-note? If I assume 10% on both of them and trading capital of $250,000, then my margins as % of Total trading capital go to more than 100%!

      2) which source(s) did you use to get historic daily prices for each of the following futures: Euro$, 5T-note, Estxx, V2TX, MXP, Corn, US Gas, Platinum,
      US2, Leanhog, GBP, VIX, CAC, Copper, Crude, KR3?

      3) for the carry forecast calculation, we will need two time series of daily prices - I guess for all of them it will be 1 month to expiry (current contract we’re invested in) and 2 month to expiry contacts, correct? Do data providers have such distinction or do they generally provide only the 1 month to expiry contract?

      4) for Euro$, I see that you suggest investing in the 40th contract month away from today and roll it over after 3 months - please correct me if I got this wrong. Do databases of daily prices have historic info on such long expiry contracts?

      Apologies if these questions are a bit too basic.

      Thanks, R

      Delete
    5. That's 4 questions :-)

      You'd have to look up the margins, but the lower the vol the lower the margin. So front eurodollar is $200 per contract for a notional exposure of 99 * 2500 = around 250,000 which is less than 0.1%. On T-notes it would be more. Compare to say VIX which is $13000 per contract on a notional exposure of $23 * $1000 = $23000, so over 50% because it is crazy volatile.

      It would originally have been quandl which no longer offers free data; if I have to backfill now I use barchart. Since I started trading I collect new data via my broker. Some of these I couldn't get historic data for, so only have limited data, for example V2TX.

      I get contract by contract data not continous data from brokers; but yes you'd need to make sure you could get continous data for different expiries.

      Again, I don't get continous data from brokers so I have no idea if it goes out as far as say 3-4 years to expiry.

      Delete
    6. Robert - thank you so much for your patience with my questions. The only way for me to “compensate” you is to buy “Leveraged trades”, the only book I left for me to buy (out of the three). Deal? :-)

      The above is clear, but I have some follow-up questions:

      1) if you did not get enough historic data for V2TX, then does it mean you did not back test it and instead you used the more popular VIX, and used that as a proxy for back testing purposes since they both measure equity vol?

      2) when you were back testing the carry factor on Euro$, were you not using continuous data for 40th and 43th contracts away from expiry? I understand that these are the sort of maturities that you advise using in “live trading” but I wonder if you used shorter term different maturity contracts when you did the initial backtest?

      Thanks!

      Delete
    7. Thanks in advance for buying Leveraged Trading...

      1) Not sure what you mean by 'for backtesting purposes'. Why do I need to backtest an instrument before I trade it? What would I do differently if I was able to 'backtest' a specific instrument? What changes would I make to how I traded it? For example, I would have been perfectly happy to add a new future with only a year or so of price history (enough to establish the forecasts for slower signals, nowhere near enough to 'backtest' it).

      2) Yes basically my stitching algo adjusts the maturity depending on whether prices are available. Probably for Eurodollar

      Delete
    8. ... probably for eurodollar initially I was only trading maybe 12 contracts out at the start of the backtest and it would have adjusted gradually to fit my ideal contract profile once prices were available.

      Delete
    9. Thanks Robert - with regards to my first Q, my thinking was as follows:

      You backtest 4 trading rules on 9 different instruments . Then a new instrument, the 10th, becomes available and passes the test of costs and diversification. You don’t know if the 4 trading rules work on this specific instrument and so I assumed you would always want to backtest the 4 approved trading rules on a new instrument before adding that same instrument into the trading system and portfolio. From your answer, it seems that one needs to be confident that approved trading rules will work on any instrument.

      Please let me know where I am getting my thinking wrong.

      Thanks

      Delete
    10. " I assumed you would always want to backtest the 4 approved trading rules on a new instrument before adding that same instrument into the trading system and portfolio"

      No. Picking instruments that only work for certain trading rules is *massive* overfitting.

      For any given trading rule, probably 40% of instruments lose money in the backtest, i.e. 'don't work'. But the differences are far from being statistically significant.

      If I've backtested something for 40 years, and then I have one year when i lose money, should I abandon my whole system? No because the extra year of data isn't statistically significant.

      If I've backtested something for 40 instruments, and then I have one instrument when i lose money, should I not trade that instrument? No because the extra instrument isn't statistically significant.

      If an instrument loses money in the backtest *and* the differences are statistically significant, well then my portfolio optimisation will downweight that instrument automatically - but it's extremely unlikely to completely remove it. And as that decision is made using only out of sample data, and in a robust fashion, it's *not* overfitting.

      Delete
    11. Robert, I have just subscribed to Barchart and I would like to ask you a question with regards to the Eurostoxx future contract.

      Essentially, I have been able to download the continuous time series of prices for the eurostoxx June contract - FXM21 - going back to 2000. Then, because I need to calculate the carry factor, I downloaded the time series of continuos prices for the September 2021 contract but I got the same identical file as the June 2021 contract, whilst in reality there should be a difference between these two. Is this assumption correct?

      If not, why are the two time series of prices of two different contracts the same?

      Thanks,
      R

      Delete
    12. You haven't downloaded the contionous series for FXM21 - there is no such thing. You've downloaded the continous series for Eurostoxx, or for the first contract to be precise. And it seems like in their interface, whatever contract you click on, if you ask for a continious series it will give you the same one: most likely the first contract. And what you also need is the continous series for the second contract. Do Barchart offer this? How do you get it? On both questions, my answer is 'I don't know'.

      Delete
    13. Thanks Robert

      I’ve emailed Barchart and I’ll see what they say. If they don’t offer the continuous times series on the second contract, is there anything else you could suggest to get this data?

      Out of curiosity, how did you deal with this issue when you first had to backtest your system? Did you not bump into the same issue?

      Any advice or suggestion will be highly appreciated.
      Thanks

      Delete
    14. See my reply above "I don't get continous data from brokers"

      I only use individual futures contract prices which I stitch together.

      Delete
    15. Robert,

      On Barchart, I was double checking the prices between the time series of continuous prices of the first contract with the actual historic prices of individual contracts and I found out there is an exact match between the prices of the two series, which means that the continuous time series of prices does not seem to be *back-adjusted*. Therefore, “continuous” might just simply mean that Barchart attach time series together without back-adjustment.

      I believe this is not what a continuous time series should be?

      Delete
    16. You're right - that isn't continous, just a bunch of prices pasted next to each other.

      Delete
  3. Can you explain what is benefit to use futures instead of ETF’s for small account?

    It looks like you can get more diversification and accurate position sizing with ETF’s

    ReplyDelete
    Replies
    1. a) Cheaper to trade (tighter spreads, no TER costs although you do have to pay to roll).

      b) Easier leverage.

      c) More liquidity (usually).

      d) Longer data history

      Delete
    2. Those are very good points.
      The question is if those benefits are greater than the benefits you can get from diversification with ETF’s ?

      Delete
    3. Yes it's a good question - you can for example slow your trading down to avoid getting hit too hard by higher costs - and it's an interesting question what damage that will do to your returns. Probably someone with less than $50K will be better off with ETF's or spread bets.

      Delete
  4. I tried to run the code from github (in smallaccountsize folder) but run into bugs and missing stuff. Are you planing to finalize it soon ?

    ReplyDelete
    Replies
    1. Hi Max
      I'm afraid for some of these posts I don't always write code that is expected to be run by others. This is an example. It's more there to be read so you can see in more detail how I did certain things.

      Delete
  5. Excellent post: relevant to ETFs since the share unit prices vary significantly, 5-250 GBP.

    ReplyDelete
  6. What is an acceptable correlation level between 2 instruments in the same portfolio?

    ReplyDelete
    Replies
    1. I don't understand the question.

      I'd try and add uncorrelated instruments (correlation<1.0) to my portfolio until I ran out of capacity.

      So technically 0.9999999999999999 is "acceptable".

      Delete
  7. The question is what the benefit of trade 2 instrument with correlation of 0.99999, it’s pretty much the same instrument that you trading twice.

    ReplyDelete
    Replies
    1. It's a very small benefit, but there is one. It depends on what the cost of the additional instrument is. If there is no cost (in terms of extra time, data feeds or whatever) then you should still add the extra thing.

      This breakeven point (at what correlation would you not bother adding more instruments) will depend on for example your AUM. A large fund can justify more instruments, as they are spreading the extra (mostly fixed) cost over a larger fund (they may also benefit from using less capacity in an existing market).

      For a smaller trader it's more likely that they are going to be limited by their account size, rather than by the extra costs of adding an instrument. If my account size doubled, I'd roughly double the number of instruments I traded, even if the last one I added had a 0.9 correlation, because it only adds about 5 minutes a month to my life to have an extra instrument.

      Delete
  8. I'm sure this is an easy question to answer but how did you compute the example of $4.25M with 37 instruments you can hold 4 contracts each? Would you mind providing an example?

    Thank you.

    ReplyDelete
  9. The amount of money is the smallest amount where I can hold 37 instruments and the instrument with the largest contract size has a natural maximum position of 4 contracts.

    ReplyDelete
  10. I'm sorry, what do you mean by natural maximum position? I would think the minimum position would matter most? For instance, if your minimum allowable position is 10% and the largest contract is $100K and you want a minimum of 4 contracts, assuming you want to hold atleast 40% in margin you would need atleast:

    ($100K * 4) / .1 = $4M * .4 = $1M.

    ReplyDelete
  11. Have you read chapter 12 of my book? It goes into much more detail than I have space and time to repeat here.

    ReplyDelete
  12. Hi Rob,

    On the signal level, did you make any research of continuous vs binary? Beyond allowing to increase/decrease risk with signal strength, did you find any evidence of higher SR / difference in higher moments ?

    I am rather fond of continuous signals but according to my brief experiments, I could not find any statistically significant difference in SR / skew / kurt between a continuous or a binary signal (gross of cost). I have been working with a few large CTA's and have seen both methodologies in practice but I have never seen a descent study of the subject so far.



    ReplyDelete
    Replies
    1. Thats exactly what the plot under the heading "putting it together" shows. Continous systems are better than binary. Don't forget large CTAs will prefer continous because of transaction costs for large binary position switches.

      Delete
  13. Hi Rob,

    Where do you get data for Korean 3Y bonds? I can't seem to find any Korean bonds on Quandl.

    Thank you.

    ReplyDelete
  14. Hi Rob,

    you advise specific instruments depending on the amount of trading capital available. Suppose that somebody reads this blog post several years in the future. Would the same instruments be used? Or would the analysis need to be repeated from scratch?

    Thank you.

    ReplyDelete
    Replies
    1. Great question. The results will change if (a) new contracts are listed, (b) existing contracts are removed or become illiquid or (c) long run average relative volatility changes a lot. If any of this is the case then yes, the results would be out of date.

      Delete
    2. Thank you for your clear reply.

      Delete
  15. Thank you for a very helpful analysis. In your book, you mention trading over 40 instruments. Are you now trading only the 37 in this post? Last year you wrote about dropping the shatz due to low volatility and low upside. If you've dropped others, what were the reasons?

    The Canadian dollar and silver are both liquid with long histories, but they're not included here. Is that due to their high correlations with other instruments?

    ReplyDelete
    Replies
    1. Yes I'm only the rading the 37, less Shatz.

      I dropped australian 3 year, 10 year and stock index because they increased their data charges.

      I dropped the US30 bond and BUXL bond because the size of the contract and low vol meant they weren't meeting my min account size thresholds.

      Because I was having general issues with minimum account size I also dropped a few other contracts which were eithier very low volatility or highly correlated with stuff I already had.

      So CADUSD and CHFUSD for example had low vol. Silver is correlated with Gold. I also dropped a couple of commodity contracts because of high correlation.

      Finally I used to hedge my equity book with both the FTSE and Eurostoxx; I dropped the FTSE again because of data charges.

      Its my intention to add some of these back to my data collection process, even if I don't trade them again.

      Delete
  16. Hi Robert,
    Is implementing your thresholding concept as simple as increasing the maximum allowable forecast from +/-20 to +/-30, and keeping a flat position in a given market until a forecast of +/-10 is achieved? Also, in the system you present in your book, you assume capital of 250k with 6 instruments. However, in your discussion here, with an account size of 100k, you recommend more than 6 instruments (with some of them "thresholded"). Does this mean you would recommend more than 6 instruments for an account size of 250k? Thanks a lot.
    Peter

    ReplyDelete
    Replies
    1. Not quite that simple. Let f be the forecast without thresholding, g be the forecast with.

      If f<10 then g=0
      If 1020 then g=30

      and the same on the short side

      In the book I didn't discuss thresholding as I didn't want to over complicate things. 250K was the minimum to trade 6 instruments without thresholding.

      But yes I'd say now that if you wanted to use thresholding then you can use the suggested portfolio sizes in this blog post.

      Delete
    2. Thanks, Robert. So, if I use your spreadsheet to arrive at a combined forecast of 18 for a particular instrument, then I would input 30 as the value for the combined forecast into the other spreadsheet that computes the qty of contracts to put on. If the combined forecast came to -5, I would use a value of zero to compute the qty of contracts, right? Thanks, again.

      Delete
    3. Yes, but why not be lazy and do it all in the spreadsheet? If D9 is the raw forecast:

      =IF(ABS(D9)<10,0,IF(ABS(D9)>20,30*SIGN(D9),(ABS(D9)-10)*3*SIGN(D9)))

      Delete
    4. Good idea. It looks like your formula above would indicate that a raw forecast of 18 should be "translated" to 24. I thought you were implying that any raw forecast over 10 would be translated to 30, so now I understand. Thanks.

      Delete
  17. Hi Rob,
    Regarding the instruments to use with a small account size: for an account of approx. 20 k USD you advise to use Eurodollar, MXPUSD, KR3 and V2X. The account size is too small to add CORN, EUROSTX, GAS_US and PLAT because their volatility is too large. I notice from my own analysis that BOBL and KR10 would also fit this account size, from a value volatility point of view. My dilemma is now : to add or not to add? Your general advice is "the more instruments the better", but these two are in the same asset class as KR3. Adding these would not create diversification in the sense of adding a new asset class. Grouping KR10 and BOBL with KR3 in one class would lower the maximum size available to KR3. It would create diversification within this one asset class though. Is this a useful diversification? Or too much "micro management"?
    What is your opinion?

    ReplyDelete
  18. Hi Rob,
    sorry for asking a second question. This time about using thresholding. On what level of the forecasting process do you use thresholding? For example: assume you use three variations of a EWMAC forecast. Do you apply thresholding on each of these three variations? Or do you combine the forecast of these three into a combined EWMAC forecast and then apply thresholding? Or do you only apply thresholding after you've combined the EWMAC forecast with other forecasts (e.e. momentum) into an overall forecast?

    ReplyDelete
    Replies
    1. Do it at the end on the combined forecast from all trading rules together.

      Delete
    2. Thank you for answering my question. Doing it at the end is also the easiest to implement in software (at least, in my case).

      Delete
  19. Hi Rob.
    First the obligatory complements on the blog and content. I've been lurking and following for a while and you are extremely open and generous in your sharing.
    So that's the mushy stuff out of the way old bean can I mention your book?

    It's that time of the year and Santa is asking what I'd like. He's all out of knees so I'm contemplating your book but a little worried on the stats side.
    I am a experienced Python Dev and trading FX for many years on and off but despite all that I am pretty poor on maths and formulas.
    I can follow your blog without too much trouble but when I hear words like alpha and gamma I always dose off into a dream about the hulk. ;)

    On a serious note. Will I be able to take something away from your book?

    Cheers Steve.

    ReplyDelete
    Replies
    1. Thanks for the kind comments.

      Well, bear in mind I'm biased (one purchase of my book by you = extra stocking filler for my kids), but there isn't too much maths in my book at all. Yes I use shorthand terms like alpha and beta, but I define them.

      I don't know if you've read a preview:

      https://books.google.co.uk/books?id=X3hxCgAAQBAJ&printsec=frontcover#v=onepage&q&f=false

      But hopefully a quick scan through should reveal there is nothing too scary inside.

      Merry Christmas, in advance.

      Delete
    2. Yep the preview looks palatable and your writing style is human rather than cyborg so stocking fillers all round.
      Merry jingles to you and yours
      Cheers Steve.

      Delete
    3. Rob also posted online various spreadsheets showing the kind of calculations being done in his trading system. If you can follow those (and they are not very complicated, I believe) you won't have any problems with the maths in reading the book.

      Delete
    4. Thanks Jeroen. I'll hunt them out. Formulas written in code are never an issue. Formulas written on paper are all Greeks to me ;)

      Delete
    5. Great news Santa brought me your book Rob. I've already read it and so much of what you describe resonates with me. Loads of it is scaringly intuitive and makes me wonder why I've not approached it from this angle before.

      Need to re-read it a couple of times and start trying to see if I can apply some of it to my unfortunately limited range of instruments.

      Great stuff, well done

      Delete
    6. Glad to hear you are enjoying it minkym.

      Delete
  20. Hi Rob, I am now paper trading your chapter 15 system with the following adjustments. 100k USD account value at 50% annual volatility, trading eurodollar 12.5%, 2 year notes 12.5%, nasdaq emini 25%, mxp 25%, platinum 6%, natural gas 6%, corn 6%, and live hogs 6%. Some of these are thresholded (per your post) to a max forecast of 30. My broker doesnt offer european futures, so I am restricted to US. I am not trading a volatility instrument because the margin requirements are so high. I've calculated my instrument diversification multiplier to be 1.6 based on a correlation matrix of the log of weekly returns going back about 8 weeks. I have noticed that the average absolute value of the forecasts for the carry trading rule has been 20, so maybe I will reduce the forecast scaler to 15 for the Carry rule. Once I start real trading, I will probably dial back my volatility target to 30%. Does this seem reasonable to you? Thanks, as always.

    ReplyDelete
  21. All fine except: "I have noticed that the average absolute value of the forecasts for the carry trading rule has been 20"

    Over what time period?

    ReplyDelete
  22. Over the past 60 trading days, is that too small of a sample maybe?

    ReplyDelete
  23. One other question, if I may: currently the system has me in 5 contracts of 2 yr. notes, and short 2 corn contracts, and short 1 lean hogs. This represents only 6k in margin requirements for an account of 100k. Should I put around 80k in some low-risk bond etf to boost returns a bit?

    ReplyDelete
    Replies
    1. HI Rob, what is your thinking here? I ask because I have read other opinions along the lines of "don't keep much cash in your brokerage account; keep it in a low risk e.g. short rates or money market fund: in the event of your broker going bankrupt, getting any securities returned is quicker than getting any cash returned (if it has not already gone)".

      Delete
    2. The risk of a problem if the broker goes bust is, I think, minimal. The extra interest you will earn is also, right now, minimal. Weighed up against that if you own an ETF you'll incur management fees and be exposed to some risk, and trading costs if you have to adjust the size of your margin.

      Having said that there is no strict right or wrong on this issue. Some big CTAs are strictly cash only. Others do indeed use short term US government bonds.

      Of course you don't have to put the other 80k (or whatever) into your brokerage account. You can keep it in another account, and only keep enough in your brokerage account to cope with an extreme shock to your p&l. You then shuffle money across regularly as required. Slightly risk and requires more work - but again this is what a large CTA would do.

      Delete
  24. Hi Robert,
    By "GAS_US" are you speaking of IB ticker RB or NG? Do you have a list of your symbols versus IB's symbols. Thanks, again.

    ReplyDelete
    Replies
    1. NG.

      https://github.com/robcarver17/pysystemtrade/blob/master/sysdata/legacycsv/instrumentconfig.csv

      Delete
  25. Hi Rob, I'm starting to paper trade your python system. Is there a fairly quick way I could update the code to accommodate thresholding on certain instruments, or is this very involved? Thanks.

    ReplyDelete
    Replies
    1. Not sure what 'thresholding is'. Is it the same as buffering - already supported system.accounts.get_buffered_position

      Delete
    2. I was referring to your adjustment of the forecast in the above post. In excel, you instructed me to do the following: "If D9 is the raw forecast:

      =IF(ABS(D9)<10,0,IF(ABS(D9)>20,30*SIGN(D9),(ABS(D9)-10)*3*SIGN(D9)))"

      Is there a quick way to add this to your python code for certain instruments?

      Delete
    3. https://github.com/robcarver17/pysystemtrade/blob/master/examples/smallaccountsize/roundingeffects.py

      Delete
    4. Thanks for the above link, Rob. I was hoping there might be a quick way to implement forecast thresholding of certain instruments by changing some parameter in the yaml config file. It appears that it's more complicated than that. I suppose I will have to brush up on my python skills, but if you have any additional guidance, I'd appreciate it. Thanks.

      Delete
  26. Rob
    Apologies if you have already explained this, but to a newcomer coming to this and currently working through your code and having read your book, a simple question:
    "how to I work out contract size in currency terms?".
    I presume you multiply the Pointsize (in InstrumentConfig.csv) by the closing price for each instrument. Is this correct? I have assumed that the Poinsize is equivalent to the IB "multiplier" which I get from the IB "product description". However, I note in some cases eg Wheat ZW the multiplier is 5000 and the point size in instrumentconfig is 50.

    ReplyDelete
    Replies
    1. The point size is how much money you stand to make or lose if the price moves by a point. There is sometimes some inconsistency about the way that IB prices come in versus what the multiplier shows. For wheat the confusion comes down to whether you quote wheat in dollars or cents. I use a price in cents which means 1 point is 50 bushels, if you price in dollars then 1 point is 5000 bushels.

      Delete
  27. Hi Rob, do you revisit the number of instruments in the event of (large enough) drawdowns. For example, one of your examples is "With a capital of $2.2 million you could hold 37 instruments with a maximum position of at least 2 contracts in each". If one were to suffer a drawdown of say 30% (not unheard of at your preferred level of target volatility) and capital is reduced, do you then start removing instruments from your portfolio? I presume at 1.32MM, that is no longer enough to support the 37 instruments?

    ReplyDelete
    Replies
    1. In theory you should do that... in practice I haven't bothered since it would be a very complex exercise and result in extra trading that would probably wipe out the benefits. To deal with this properly you'd have to set the problem out in a precise algorithim, with a generalised non linear mapping whose parameters are determined by your current capital. Then you'd get a smooth transition as your capital changes. This isn't that difficult to do, but it's not something I'm personally going to bother with.

      Delete
  28. Hi Rob, you propose adding instruments in this order as your account size grows:

    ['KR3', 'V2X', 'EDOLLAR', 'MXP', 'CORN', 'EUROSTX', 'GAS_US', 'PLAT',
    'US2', 'LEANHOG', 'GBP', 'VIX', 'CAC', 'COPPER', 'CRUDE_W',
    'BOBL', 'WHEAT', 'JPY', 'NASDAQ', 'GOLD',
    'US5', 'SOYBEAN', 'AUD', 'SP500','PALLAD',
    'KR10', 'LIVECOW', 'NZD', 'KOSPI',
    'US10', 'SMI', 'EUR',
    'OAT', 'AEX',
    'BUND',
    'BTP',
    'US20']

    I understand your methodology for this, but let's say I am considering adding SP500 to my universe. Isn't that extremely correlated to NASDAQ? Might I be better off adding something like other currencies (NZD, or EUR) or something that is less correlated to what I already have in my universe? Thank you for any thoughts you may have.

    ReplyDelete
    Replies
    1. It's a fair point but you should stick to adding something from consecutive asset classes or you'd end up overweight currencies. So maybe do KOSPI, SMI, AEX before NASDAQ, although obviously this would increase the capital required.

      Delete
  29. Probably the fifth time I came back to this post. And the fifth time I LOL'ed at the excellent David Harding image choice and its caption.

    ReplyDelete
  30. Hi Robert,

    I'm thinking about using MICRO futures to implement my portfolio. What do you think about these? It`s the volume enough? My portfolio would be Micro-Gold, micro Currencies and micro S&P500

    ReplyDelete
    Replies
    1. The micros have just been launched I think? So it's wait and see for me.

      Delete
  31. Hi Rob,
    I'm playing around with my system here mainly based on your "leveraged trading" book (excellent one as always).

    I got the opposite problem to this article here. What do you do if the futures point value or nominal value are too small? I get excellent results on Eurodollar or Fed Funds Futures for example (both STIRs - GE and ZQ on IB), but the contracts are small with a margin of just $300 per lot. Your system would then recommend a very big allocation to this (~200 contracts in my case) which then generates a lot of comission.

    Is there any way around that? I see you are trading Eurodollar. Is there a bigger contract I am missing?

    thanks!

    ReplyDelete
    Replies
    1. Small point size isn't an issue in itself, indeed it's a good thing.

      My suspicon is that you are trading one of the nearer months in Eurodollar, which are very low volatility that translates to large position sizes. Don't do this. It's very dangerous. For this reason I trade Eurodollar 3 years out, where the volatility is much higher.

      Fed funds I would avoid completely as I don't think it trades out that far.

      Just avoid stuff with really low vol (the front of most STIR contracts and until recently 2 year bond futures in most places fell into this category).

      Delete
    2. Hi Rob, I noticed the volatilit of the 3 years out Eurodollar is still low, 0.005 (2023/12, last 25 days). I was thinking of going 1 year further, where the vol is 0.008 (2024/12). How does this sound? Daily volume of the 4 years out contract is around 5k, (vs. 10k of 3 years out contract), which looks acceptable.

      Delete
  32. Hi Rob, I noticed your answer to henner about the low vol of 2 year bond futures. In my portfolio, KR3, US2, and Eurodollar (2024/12) has got vol of smaller than 1% (0.8%, 0.2%, 0.6% respectively), and I am still trading them. You would advice to replace them with something else such as longer term bonds, correct? I am curious if you keep them in your portfolio (particularly Eurodollar). Thanks!

    ReplyDelete
    Replies
    1. I still trade all of these things! Not sure where the boundary for 'very low vol is' but something like front Eurodollar I'd avoid.

      Delete
    2. Thanks for the answer! I was worried that I might be doing silly things anybody else would not do... I feel relieved.

      Delete
  33. Hi Rob! For the "effective forecast" line for thresholding, wouldn't it be better for the slope to start at (+/-6.67, 0), rather than the (+/-10, 0)? I am refering to the 4th chart on this page. With this, the area between the line and x-axis stays the same before and after thresholding, in the forecast range from -20 to +20. (with the current line the area is smaller - hence I thought we might not get the full benefit of the thresholding)

    Or have you considered occurance rate or / and some other advanced items, which I have not considered at all, to come up with the (+/-10, 0)??
    (you did mention "distribution" there..)

    I guess this is something I should not spend too much time on, but still wanted to know your thought behind... Thank you!

    ReplyDelete
    Replies
    1. You would be correct if forecasts were uniformly distributed. But they are not; they are approximately Gaussian which means there isn't much mass in the tails.

      Delete
    2. Thank you! However, doesn't that mean that the slope needs to start even closer to x=0, such as (5, 0)?? Apparently I am not a good student.

      Delete
    3. Oh, I think I got it right. Since your mass is bigger for the center, the weight for the center needs to be small. Thank you so much!

      Delete
    4. I need to lie down and have a think about this...

      Delete
    5. I just wrote I got you, but has come back to the question... I don't think I still get it. Great to know that you can give a thought on it. Thank you so much!

      Delete

Comments are moderated. So there will be a delay before they are published. Don't bother with spam, it wastes your time and mine.