Monday 18 May 2015

Systems building - futures rolling


Trading futures has one fairly substantial complication compared to many other assets. You can't just buy 'the CME Gold future'. There is no such thing. You need to select which delivery date future you are trading. Do you want to trade June 2015, July, August....?

Once you've chosen you can relax - but not for long. If you're trading June 2015 (I'm writing this in mid May), then in a few weeks time  - unless you've given up on Gold and want to close your position - you'll need to get rid of your June future and buy another one. What's the best way of doing this?

This is the second post in a series giving pointers on building systematic trading systems (you don't need to read the first; this post stands alone). However nearly all the content will be useful to non systematic futures traders. It might also help those who trade similar 'rollable' derivatives, such as spread bets.

First warning: This isn't an introductory guide to futures trading. I will be going very quickly or completely ignoring large parts of the mechanics of this asset class. You might want to read something first (like this, other alternatives are available).

Second warning: This is quite long. So you probably want to make coffee and / or visit the bathroom now.


Which contract to trade


First decision is which contract to trade. There are a number of things we need to think about.

Liquidity

The most important thing when deciding where on the curve to trade is liquidity. There are four main liquidity patterns that you'll see in the futures market. These patterns are very important for how we think about rolling these different markets, so it's worth spending some time understanding them.

Case 1 - Let's take it one year at a time


First let's have a look at Gold:


Source: CME

There's more after this, but I'm showing you the exciting bit. Focus on the 'total volume' and 'open interest - at close' columns*. Gold is an example of a contract with monthly deliveries, but where only selected rolls are liquid. We see a fairly typical pattern of liquidity declining over time, but for about a year there's enough of the stuff for most people.

* I'm using these as a proxy for liquidity. I could also look at the inside size and depth of the order book; however these won't show what was actually executed - there can be a big difference in the two measures because of spoofing.

So unless you are a massive hedge fund for which only June is liquid enough you'll probably be able to trade the June, August, October or December 2015 contracts.

Contracts with a similar pattern (reasonably decent for 3 or 4 selected deliveries, or a year or so, with gradually declining liquidity) include other metals like platinum.

With liquidity declining monotonically every month, but again reasonably good for the next four deliveries and sometimes longer, there are the VIX and V2X equity volatility futures; and US (Henry Hub) natural Gas.

Case 2 - To infinity (almost) and beyond!

Here's another contract with plenty of possibilities, Eurodollar.


To avoid terminal boredom I won't keep going, but suffice to say there are 10K lots a day of volume available until March 2020. Again unless you're a massive hedge fund that is an awful lot of contracts to choose from.

Most short term interest rate futures, including LIBOR and EURIBOR, have these long tails of liquidity into the future, although none is quite as good as Eurodollar.


Case 3 - Tis' the season to be rolling...


A lot of commodity (agricultural, energy) contracts are strongly influenced by the season of their delivery month. This also has an impact on their liquidity patterns. Here is crude oil:


And some more (boring this isn't it?):


The pattern of good, but gradually declining, liquidity in a summer (June) and winter (December) contracts continues after this, with December showing particular strength as only December crude is quoted after 2019.

You'll see a similar seasonal pattern across many commodities, although the favored 'month' will vary, and few will have such long dated delivery months as crude. US Gas is an exception, as I've already noted.


Case 4 - You want the second contract? You can't handle the second contract!


Let's move on to our final example: US 10 year treasury futures please step forward.


No I haven't missed anything out - that's your lot. All the liquidity is in the front month. This thing rolls quarterly on IMM months, and a few people have begun to move into September, but basically June 2015 is the only game in town.

Other bond, IMM currency, and equity futures show this pattern - we can only trade the front month - there is no choice.


Cost & The Spread

 

To make a massive simplification trading cost depends mainly on the bid-ask spread, and on the volatility of the market. This is a simplification which is probably fine for a relatively small trader (larger traders will need to worry about the depth of the order book), who crosses the spread each time they trade (rather than using an execution algo that could improve this price).

Lower volatility means we need to hold bigger positions to achieve a certain risk, which means higher costs. Since volatility is an important subject in its own right, I'll come back to it in a second.

Since we're making massive simplifications, let's make another one and say that the bid-ask spread will generally track the open interest and volume which we used to monitor liquidity above. So the most liquid deliveries should also be the cheapest to trade.


Volatility and kurtosis


I said above that low volatility means higher costs. It also means more leverage to hit a given risk target. Markets with low volatility also tend to have nasty kurtosis, as the low volatility has a nasty tendency of turning into high volatility when you're not looking.

Basically low volatility is pure evil. It is a devil-like* statistic, tempting you with it's promise of almost no risk, only to reveal it's true self - a horrifically ugly kurtotic distribution with a tails leading off into the far distance.


* God may not play dice. But the devil does. And he cheats.





So in case you haven't got the message: Do not trade low volatility delivery months!

Let's look at typical volatility term structure pattern*, for Eurodollar.

* I am not talking specifically about realised or implied volatility here, since most will tend to show similar patterns.

In normal times nearer STIR contracts are the most volatile, with things getting less risky as you go further out in the term structure. They are constantly at risk of a central bank changing interest rates. This also means nasty kurtosis from those 25bp or 50bp jumps when an unexpected rate move happens. Generally you want to be near the front, but not too near so the kurtosis catches you.

But we don't live in normal times. When interest rates are basically zero, and expected to be that way for a while, the front contract doesn't move. Volatility then gradually picks up. Here are the prices of the first (blue) and ninth (red) Eurodollar contracts.

Source: www.quandl.com

If you're colour blind, you can probably still take a good guess which is which.
The nicely wobbly line at the bottom is the ninth contract. Above it in blue you see an almost straight line which looks like it has been drawn freehand in Microsoft paint by someone with an almost impercetible nervous twitch? That's the front contract. I tried to measure its volatility but I got a floating point error.




Do you want to be holding this thing when Janet finally pulls the trigger?




The need to measure contango


A lot of traders, including myself use contango/rolldown/carry (pick your favourite nickname) to decide whether to be long or short. There are several ways of doing this (from best to worst):

  • Compare current contract with the next nearer delivery. Assume we're holding September 2015, we'd compare it with August 2015 (monthly cycle) or June 2015 (quarterly).
  • If there isn't a nearer delivery, compare current contract with spot. This isn't readily available for many instruments, although it is for equities and STIR.
  • If there isn't a spot price available, compare current contract with the subsequent delivery. Assume we're holding June 2015, we'd compare it with July 2015 (monthly cycle) or September 2015 (quarterly).
This means it's better, if you have the option (so not the 'front only' case 4 markets), not to hold the very nearest contract as it will make it more difficult to measure rolldown.


The best rolldown is...

On a related subject it's best to be in the contract with the best rolldown for the sign of position you're holding - long or short.

* Since the sign of position you're holding depends on the rolldown be careful you don't get trapped into an endless alice in wonderland style decision loop.

If you're long then it's better to be in a delivery month which is backwardated versus the nearer month, i.e. the nearer price is above the price of what we're currently holding, so we'd expect the contract we hold to rally over time. If we can't hold a backwardated delivery then we'd want the month with the lowest contango.

If short you'll want to be in a month which is in contago versus it's nearer counterpart, i.e. the position month price is above the nearer month, and we'd expect it to fall as the contract ages. Again if we can't find a month which is in contango, then choose the least backwardated delivery.

Let's take an example. Here's the current term structure for VIX futures.

Let's assume we could hold any of these deliveries without liquidity issues, and we aren't worried about any other selection issues (probably not true, but bear with me). First of all suppose I wanted to be long VIX. None of the months shown above are in backwardation, so we'd settle for the least contangoed (is that a word?) which would be December, with contango of only 0.11 (comparing 18.51 to the nearer price of 18.40).

Now imagine we wanted to be short. The curve gets more contangoed as we get closer to the front. The best contango is clearly in June*.

* On this occasion we do have access to a spot price, but for simplicity I'm assuming that we can't measure the contango for the May delivery, even though this would probably be even better on this occasion.

Another example. VIX again, from a particularly dull week in the markets a few years ago.

If we wanted to be long we'd opt for the second contract (not shown, but it would be the November delivery), and if want to be short the fifth looks the best (backwardation of only 34.71 - 34. 32 = 0.24).


Price action


In certain contracts some traders argue that certain months show "better" behaviour than others. So for example if you're a trend follower you might look for a trendy month (whatever that is).

I'm not hugely convinced by this, but its certainly true that nearer months often do a different merry dance of prices than later ones, perhaps a tango rather than a waltz. So as systematic traders what we should be looking for is consistent price behaviour.

It's better then, if you can, to stick to a contract which is of a roughly constant maturity. Right now for example I am the proud owner of four September 2018 Eurodollar futures, which expire in about 40 months. I'm currently thinking about rolling them to the December version, as I want to keep roughly that maturity.

Stick to a season


In a similar vein for seasonal commodities it probably makes sense (if you can) to stick to trading the winter or summer version (or spring, or autumn...). For example trading winter crude (as I do) implies holding December 2015, then moving imaginatively on to December 2016, before rolling into December 2017.... you get the idea.

One contract or two sir? (or three, or four...)


All of the above implies we're only going to hold positions in one contract at a time (except when we're rolling from one to another, but more of that in a moment). Why wouldn't we want to do this? Well normally delivery months of the same future are very highly correlated. So there isn't much diversification benefit from holding multiple dates.

If you're a small trader there is probably no value in using up your scarce risk capital to hold more than one month, unless of course you are explicatally trading spreads. If you have an enormous fund then you might have no choice but to spread your vast position over multiple delivery months.

If you're somewhere in the middle then a good place to start would be to trade different seasons of commodity contracts, like winter and summer crude, since they'll show the most differentiation.


Summary


Contract choice depends mainly on which kind of liquidity pattern we've got:

 

Case 1 and 2 (liquidity for a year, and "infinite" liquidity) 

Avoid the first contract (can't measure rolldown). Avoid low volatility and evil skew contracts. Trade a liquid contract, with a tight bid-ask, with the best roll-down for the position you've got on. If you can, keep a roughly constant maturity, to get a consistent price pattern.

You might have to compromise on some of these points. Choosing the right contract is more of an art than a science.

Case 3 (seasonal)

As above, but if possible stick to one calendar month (eg December), then roll annually.


Case 4 (only the front)

We have no choice but to hold the first contract.


When to roll - the rolling period


Okay so we've managed to find which contract we'd like to trade. At some point however we're going to prefer another contract (or at least have no choice in the matter, because the bloody thing is about to expire).

 All the considerations of contract selection are also important when considering when to switch. So we should only switch to contracts once they are liquid enough, whilst their volatility isn't too low and have the right rolldown characteristics. We'd also want to switch before the current contract becomes illiquid, low volatility or when it's rolldown is no longer suitable.

This means that the decision of when to rolling can sometimes be a difficult balancing act. If you're long, and the next contract in line has less worse backwardation, then you might want to delay the roll as long as possible; whilst rolling before you end up holding the front contract, on which you can't measure rolldown and also has horrible volatility.


Rolling period


How long do you have to roll? It depends. If we're trading Eurodollar 40 months out, then we've got a maximum of three years to roll. Except given we want to remain about 40 months out, we've got roughly 3 months to do the roll.

Similarly if we're trying to stick to trading the second VIX future, then with monthly rolls we've got about four weeks, probably less.

On seasonal rolls, like winter December crude, we'll be able to roll once next years December contract is liquid enough. Ideally this would be before the current December becomes the front month, so we can still keep measuring rolldown, i.e. before the current November expires. Again this is usually a window of a few weeks or months.

For 'front only' markets like US 10 year futures the liquidity on the second contract might be available a few weeks in advance, or at the other extreme the next Korean bond future only becomes liquid around a day before the previous one expires.


Last possible moment


When do you absolutely have to roll? There is no easy answer. It might be just before the liquidity is completely sucked out of the contract you're currently in. It might be on the very day of expiry. Or, if you're long and unless you want to take physical delivery (and your broker will let you), before the first notice date where that is relevant.

Of course for cash settled contracts you don't have to roll. If you know you're going to close your position anyway then it might make sense to save the execution cost of doing so if you're happy to take a small amount of risk on what the settlement price will be. I discuss this more below.


Price stitching

Let's have a look at VIX prices over the last few months. Assuming we wanted to stay a couple of contracts in the future we might have a picture something like this.
Ignoring the terrible excel date formatting you can hopefully see that at point 1 we switch from holding the May contract to the June. Then at point 2 we switch from the June to the July. If you mentally fill in the gaps you'll see that this naive series of price now contains two jumps, the second of which is more serious than the other.

There are various ways to deal with this problem. The first method, which I use, is probably the simplest and is known as the "Panama method". Essentially we keep the last July contract unchanged. We then shift the June contract up in parallel until it lines up exactly with the July on the roll date. The same process is applied to May. This gives us something like this:

The nasty jumps have gone - hurrah! However

  • Drift effect: The little jumps in the price are effectively corrections for rolldown. By removing them we're including that rolldown in the price. This biases the trajectory of the price series (more here).
  • Because of drift the price series we're using 20 years ago might have very different levels from what the actual price was at the time. If we go back far enough, we could even have a negative price.
  • Percentage returns won't make sense any longer. If for a bond with price 120.0 we go back far enough to when the spliced price was 1.0, then a smallish single point move would be 100% of the spliced price.
I'm personally sanguine about these problems (I don't use percentage returns with the spliced price as the denominator, or actual prices in any trading rules, nor do I care whether a long term trend was partially caused by rolldown).

But some people prefer other stitching methods such as proportional back adjustment (which also messes with price levels and volatility) or linear weighting.

In the section below on detailed implementation I'll discuss how to put together historical series of panama stitched contracts. You should be able to adapt this procedure to other methods if you use them.


Trading the roll - let me count the ways


How do we actually go from holding May 2015 to June 2015, once we've decided that is the thing to do? I can think of four ways.


Passive

A very nice way to do the roll is passively. Let's suppose we're long 5 contracts of May 2015 Gold, and we want to own June 2015.

Now suppose that we want to buy two more Gold futures, for reasons that are not important. Should we buy May gold? No, that would be stupid. Instead of having to roll 5 contracts, we'd need to roll 7. Active rolling means trading, and trading costs money (though, as we'll discuss in a moment, often less than normal trading). Why do more than you have to?

Instead it makes more sense to buy 2 June 2015 contracts. We now own 5 May and 2 June. We still only have 5 May to roll.

Let's suppose that we've regretted our buying decision, and we now want to sell those 2  Gold we just bought. Does it make sense to sell the June? No, again that wouldn't be logical. If instead we sold the May then we'd only have to roll 3 May contracts.

Similar arguments apply to short positions.

To summarise:

Once the next market is liquid
If LONG, then BUY the next market and SELL the current one
If SHORT, then SELL the next market, and BUY the current one

Another way of thinking about this is you should always add to your new position, and reduce your current position.

Passive rolling can cut the separate cost of rolling to effectively nothing if you're trading a lot. Disadvantages are that you need to have a reasonable window to roll in, and if you're not trading much you probably won't be able to do all of your roll in that window. So it's of little help if you're a slowish trader of those 'only the front' markets where we only have one or two days to roll.

If you can't passively roll before the 'I really need to roll now' moment, then you need to look at one of the active roll methods.

Spread trade


The classic way of actively rolling is to find somebody else with the opposite position, and trade the calendar spread with them. So I'd go to the market with my sell May buy June trade.

The advantage of this is you'd normally halve your execution cost (rather than paying a tick say on each leg, hopefully you'd only pay a single tick on the combo). You also have no price risk exposure on the roll trade compared to trading the legs separately.

The problem is that not all instruments have active, liquid, spread markets.


Separate legs


If you can't do a spread trade then you'll need to execute the two legs of your roll trade separately. If you have to cross the spread then this will cost you twice what a normal trade would cost.

There is also price risk involved if you don't cross the spread (and even hitting the bid/offer and paying the spread doesn't guarantee you won't still have price risk). Suppose you offer your five May gold contracts @1228.2 and bid for June@1227.6.

Then something bad happens and gold gets hammered. Your June bid is filled, but your May offer is now a hundred points off market. You now own ten Gold contracts - twice as many as you want to - in a falling market, and you've already taken a hundred point loss on them.

This isn't likely to end up being a very pleasant experience for anyone involved. So trading the legs should be avoided where possible.


Cash settle and rebuy


Very occasionally you might end up holding a contract to expiry. This is probably an absolute no-no if it requires physical settlement (unless you're a decent sized fund who deliberately arbitrages the settlement and has the merchandise to deliver, a warehouse to accept delivery, and the back office to handle the paperwork).

I did however accidentally take delivery on some currency IMM futures; that was cleared up with a few spot trades.





Clearing up after a delivery of Live cattle could be a lengthier and more problematic process.





 For cash settled contracts it might not be the end of the world to hold at expiry.
I can think of times when it might make sense to cash settle. If you know you're not going to want to own the contract, and if volatility is low, then settling could be cheaper than selling out. It often makes sense to cash settle options trades which are well ITM or OTM, as the bid-ask to trade out can often be high just before expiry.

Even if you do want to own the contract after expiry then if the alternative is trading each leg separately then cash settling means you only have to pay execution costs on half of the trade.

Obvious disadvantages of cash settlement are that if you do want to maintain the exposure then you will have some price risk, as you're unlikely to be able to buy the new exposure at exactly the same relative level as where the settlement occured.


The double roll problem


From the discussion above its hopefully clear that a roll has completed when two events have taken place:

  1. We no longer have positions in the old delivery month
  2. We are using the new month price to generate our trading signals (having stitched the prices of the old month, and all its predecessors)
Now ideally these two things would happen at exactly the same time. In the real world nothing ever happens at the same time. Also if we're passively rolling there will be a time when event 1 has only partly happened. Broadly then there are three options:

  • Before we roll positions, we switch to using the new month price.
  • During a passive position roll, we switch to using the new month price.
  • After all positions are rolled, we switch to using the new month price after stitching.
The potential issue with the 'before' option is that the new month price might not be available, if for example you're dealing with a 'front only' market where the new month won't be liquid until the day of expiry (and you want to switch the price series over the previous night).

'During' has the nice property that if you're doing a gradual passive roll then you can switch over once you have a bigger position in the new month than in the old month. You'll need to pause any trading to do the switch.

The main downside of 'after' is you could be using the wrong price to generate trades (or worse still if the contract expires, you won't be seeing any updates to the price at all). It will be important to switch as soon as possible.

Different approaches clearly make sense for different instruments, and also different trading styles (a low frequency trader can be more relaxed about the whole thing). As we'll see below my own preference is to make 'after' a compulsory method, to simplify the control process for the roll.


Rolling in practice



Theory over; here are some details of how I manage the rolling process in practice.

I don't think you can fully automate the process of rolling. It needs a lot of human judgement to decide when to go. Although in theory this could be done systematically it would need a lot of data, rules and special cases.






Also if I automated it I'd be worried that something would go horribly wrong...








So whilst everything is automated, in the sense I don't have to go and input orders myself, the computer won't start rolling until I've assessed it is the right time to do it.

Historical price stitching


If you're trading systematically then you'll need a history of prices to analyse before you start trading a new instrument; ideally tons to do backtesting, or at least enough to generate a signal on when you switch on your magic black box.

To start with lets assume we have downloaded a set of 'raw' prices for each of the contracts that cover our history, perhaps from somewhere like www.quandl.com. You have also got the expiry date of each of those contracts.

So we'll have a set of these, here showing the Feb 1980 Gold contract:

run@bilbo ~/workspace/systematic_engine/sysdiag/scripts $ . displayraw LIVE GOLD 198002 20
            PRICE  ROWID
1980-01-29  702.0  10083
1980-01-30  658.0  10084
1980-01-31  681.5  10085
1980-02-01  683.3  10086
1980-02-04  660.0  10087
1980-02-05  687.0  10088
1980-02-06  713.0  10089
1980-02-07  689.5  10090
1980-02-08  694.5  10091
1980-02-11  702.0  10092
1980-02-12  697.5  10093
1980-02-13  688.0  10094
1980-02-14  670.5  10095
1980-02-15  663.0  10096
1980-02-19  632.0  10097
1980-02-20  661.5  10098
1980-02-21  656.5  10099
1980-02-22  615.0  10100
1980-02-25  648.9  10101
1980-02-26  625.5  10102


Decide your rolling policy


Depending on what kind of rolling pattern you have you need to determine your rolling policy. The rolling policy is made up of two components.


  • How many days before expiry do we roll?
  • Which subset of available contracts do we consider in our rolling cycle?

So for example take a case 1 contract (liquidity 'for a year') with monthly expiries, of which only the quarterly IMM dates are liquid, and where we want to avoid holding the front month perhaps so we can always measure rolldown. Then we'd want to roll something like 45 days before expiry (less than 31 means we'll end up in the undesirable front), and only include the HMUZ monthly expiries in our roll cycle.

For an 'infinite' liquidity like Eurodollar I like to stay about 40 months out. At this point on the curve there are only quarterly expiries anyway, and we'd want to roll about 1216 days before expiry.

If we're in an 'only the front' contract like Korean bonds then we'd almost certainly include all contracts in the cycle. I'd probably want to roll close to expiry, unless like US 10 year bonds liquidity comes in earlier than that, or we're trying to avoid a first notice date.

In a seasonal case like Crude oil, since I'm sticking to the winter flavour, I'd only include the 'Z' contract in my roll cycle. I want to measure rolldown, and with monthly expiries I'll be doing so off the November 'X' contract, so I need to roll at least 45 days before expiry. Notice the items in red here.

run@bilbo ~/workspace/systematic_engine/sysdiag/scripts $ . showinstrument LIVE CRUDE_W

**************************************************************************************
Instrument code CRUDE_W, Light sweet crude Winter
 rollcycle Z, pricecycle FGHJKMNQUVXZ,  assetclass OilGas
 exchange NYMEX, pointvalue 1000.000000 currency USD
 lastrolloffset -60 days, firstrolloffset -90 days, carryoffset -1 priced contracts

IBSymbol CL
IB factor 1.000000 IB Price exchange NYMEX

Priced contract is 201512, forward is 201612, carry is 201511 



Set your initial stitching dates


Using your rolling policy and the history of data you have got you need to determine your initial stitching dates. Start at the beginning of your history, and work out what contract you'd have been trading at that point. Then once you're N days before expiry, switch to the next contract in the subset; then repeat.

Let's take a simple example. Suppose I had Gold prices from 1st January 1980. To ensure I can measure rolldown I want to be in the second contract, but no further. My roll cycle is GJMQVZ which are the liquid months, all of which are around 2 months, 60 days, apart. To make things simple let's ignore the possibility of having a front month not in the roll cycle to measure rolldown off, and switch 70 days before expiry.

In January I'd want to be in April Gold, so I can measure rolldown off February. 70 days before April Gold expires I would then switch to June Gold. 70 days before June Gold expires I would move to August Gold, and so on.

I save down my stitching dates into .csv files so I can manually hack them later if required (see below).


Adjust your initial stitching dates



For a switch to happen there needs to be a closing price on the stitch for both the old and the new delivery months. If there isn't then the roll will fail. The initial stitching dates might need adjusting to ensure they are all good switch dates.

First I use an automated script which tries to move the stitching date backwards and forwards until it finds a date when everything matches up. This can have constraints on it like 'I never want to trade the first contract'.

If that doesn't work, and I still have bad dates then I start manually adjusting the dates, or taking other action. I might for example throw away earlier data if I can't find good roll dates.

If there is a gap in the new contract price then you could consider using the last known good price difference to extrapolate the price of the contract to the point when you can have a good roll date. This does mean your price series won't be pure, but could be viable for instruments where calendar spreads are very stable.




There is no one true set of stitching dates




If you're in a situation where you have some freedom with your stitching dates, bear in mind that due to liquidity pressures and arbitrage the roll spread can move quite a lot going into expiry, so you'd get different sizes of roll jumps baked into your price series. Changing your stiching dates will have a subtle effect on your price series and so on the results of any backtesting.


Create your stitched price series



Once all your roll dates are good you can create your stitched price series. You need to implement the algorithim of your choice to work backwards and repeatedly through the data.

A couple of technical points. Stitching can be done on the fly whenever you need a stitched price, or like me you can precalculate and update the stitched series whenever you get a new price for the current contract. The latter is faster, but does mean you are stuck with one set of stitching dates. You can also store your stitching dates seperately, or in the same table as the stitched price series.

Here is what my stitched price table looks like for the dutch equity market:

>>> atable=adjusted_price_table("LIVE", "AEX")
>>> atable.read_prices_for_code("AEX")[pd.datetime(2015,5,10):pd.datetime(2015,5,12)]
                         ADJ  CARRY CARRY_CONTRACT  FORWARD FORWARD_CONTRACT  \
2015-05-11 08:01:14  489.675    NaN         201506      NaN           201506  
2015-05-11 09:01:29  489.675    NaN         201506      NaN           201506  
2015-05-11 10:02:52  487.850    NaN         201506      NaN           201506  
2015-05-11 11:03:04  487.825    NaN         201506      NaN           201506  
2015-05-11 12:04:23  487.950    NaN         201506      NaN           201506  
2015-05-11 13:05:35  489.775    NaN         201506      NaN           201506  
2015-05-11 14:13:16  489.225    NaN         201506      NaN           201506  
2015-05-11 15:40:35  489.825    NaN         201506      NaN           201506  
2015-05-11 15:40:36  489.825    NaN         201506  489.825           201506  
2015-05-11 15:40:37  489.825    NaN         201507      NaN           201507  

2015-05-11 16:40:55  490.600    NaN         201507      NaN           201507  
2015-05-11 23:00:00  490.300    489         201507  489.000           201507  

                          FX    PRICE PRICE_CONTRACT  ROW_IDS 
2015-05-11 08:01:14  1.12494  490.175         201505     2618 
2015-05-11 09:01:29  1.12494  490.175         201505     2619 
2015-05-11 10:02:52  1.12494  488.350         201505     2620 
2015-05-11 11:03:04  1.12494  488.325         201505     2621 
2015-05-11 12:04:23  1.12494  488.450         201505     2622 
2015-05-11 13:05:35  1.12494  490.275         201505     2623 
2015-05-11 14:13:16  1.12494  489.725         201505     2624 
2015-05-11 15:40:35  1.12494  490.325         201505     2625 
2015-05-11 15:40:36  1.12494  490.325         201505     2626 
2015-05-11 15:40:37  1.12494  489.825         201506     2627 

2015-05-11 16:40:55  1.12494  490.600         201506     2628 
2015-05-11 23:00:00  1.12494  490.300         201506     2629 




I've highlighted the roll here in red. So before the roll (row 2625) we were trading May '15 as our current PRICE_CONTRACT. The next contract to roll into was June (FORWARD_CONTRACT). There is also a CARRY_CONTRACT which is what we use to measure rolldown from, and in this case it's the same as the forward (because AEX is a 'front only' instrument).

After we've rolled (row 2627) things have moved on, the PRICE_CONTRACT is now June, the FORWARD_CONTRACT is now July.

I'll return to the mechanics of the roll, and explain the rest of this dataframe, in a second.


States


At the kernel of how I manage rolls is the concept of an instrument roll state. Possible states are as follows:

  • NOT rolling. We only have positions in the current delivery month. We're taking our price from the current month.
  • PASSIVE rolling. We opportunistically buy or sell the current or new month to reduce our current position. We may have positions in the current or the new month. We're taking our price from the current month.
  • IMMEDIATE rolling. We'll try and issue a spread order to do the entire roll. We may have positions in the current or the new month. We're taking our price from the current month.
  • FORCELEG rolling. We'll try and issue seperate trades for each leg of the entire roll. We may have positions in the current or the new month. We're taking our price from the current month.
  • FINISHED rolling. We have no positions in the current month. We're taking our price from the current month. Once we've switched our price feed, and done our stitching, we can put this back into a 'NOT' state.

This is simplified due to my insistence on only changing the source of prices after the roll of positions has been done. The state transition matrix is:

NOT -> PASSIVE
NOT -> IMMEDIATE
NOT -> FORCELEG

PASSIVE -> IMMEDIATE
PASSIVE -> FORCELEG
PASSIVE -> FINISHED

IMMEDIATE -> FORCELEG
IMMEDIATE -> FINISHED

FORCELEG -> IMMEDIATE
FORCELEG -> FINISHED

FINISHED -> NOT

Again you'd need more complex transitions if you allowed switching of the price feed to the new month at any point in the process, rather than after positions are rolled as I do.

I run checking reports which make sure that the positions we have are consistent with the state, and email me exceptions, such as a position in the next month when we're in a 'NOT' state, or a position in a 'bad' month which is neithier current or new.

Reporting


Although I don't fully automate rolls, I do use a number of reports to help me decide when to roll.


I generate this report 3 times a day:

Roll status with action needed

       code                              msg  near_expiry_days  position price_contract  relative_volume rollstatus                 suggest
28      VIX           Roll positions ongoing                 8        -2         201506         0.276182      ALLOW                CONTINUE
40   GAS_US           Roll positions ongoing                15        -1         201507         0.146196      ALLOW                CONTINUE
42  EDOLLAR           Roll positions ongoing                 8         4         201809         0.878177      ALLOW                CONTINUE
20      AEX  Roll near on price, not liquid.                 3         0         201505         0.004865        NOT  START ROLL...NOTLIQUID
21      CAC  Roll near on price, not liquid.                 3         0         201505         0.007042        NOT  START ROLL...NOTLIQUID



Sorry about the wrapping. This report shows me:

  • Futures that are currently rolling ('ALLOW') and whether the roll is passive ('ALLOW') or aggressive.
  • Futures that are 'NOT' rolling, but which have less than 4 weeks before the point where I'd like to roll them.
It's also possible to produce a similar table for all futures. Essentially the report works out what it thinks you should be doing ('suggest') and compares this with the current status.

The suggestion is based purely on volume and the number of days to the point where we'd normally want to roll, and does not take other liquidity measures into account, nor any other criteria. However it's a useful tool to keep track of markets for which a roll of one kind or another might be in order, or for which we need to step the rolling up a gear and go from passive to active rolling.

For example in the report above we see three markets on which we should CONTINUE rolling, and two where we would ideally START but the new month isn't yet liquid enough (volumes are less than 1% of the current month).

Before rolling I'd also look at a more detailed report for that instrument. Here's Australian STIR:

run@bilbo ~/workspace/systematic_engine/sysdiag/scripts $ . rollstatus LIVE AUSSTIR
For AUSSTIR current roll status is ALLOW; Suggested action is to CONTINUE
Roll positions ongoing
Roll positions ongoing
Contracts are
{'CARRY': '201603', 'FORWARD': '201609', 'PRICE': '201606'}
Positions are:
{'FORWARD': 0, 'PRICE': 8}
There are 387 days before price contract expires and 296 before carry
The suggested start roll period is 14.000000 days away
Volume data:
{'fiveday': {'201612': 2017, '201603': 10622, '201509': 14866, '201606': 7127, '201609': 4771, '201512': 18507, '201506': 8155}, 'oneday': {'201612': 841, '201603': 3809, '201509': 3171, '201606': 3200, '201609': 2197, '201512': 4740, '201506': 1018}, 'twoday': {'201612': 1637, '201603': 8801, '201509': 6979, '201606': 7518, '201609': 3880, '201512': 13166, '201506': 5006}, 'tenday': {'201612': 2691, '201603': 12441, '201509': 17871, '201606': 8721, '201609': 6231, '201512': 20925, '201506': 10516}}
As of dates:
{'FORWARD': Timestamp('2015-05-16 23:00:00'), 'PRICE': Timestamp('2015-05-16 23:00:00')}

Price matrix
                     201506  201509  201512  201603  201606  201609  201612
2015-05-16 23:00:00   97.87   97.93   97.97   97.96  97.910   97.84   97.76


Market liquidity (size, size, bid, ask) - priced contract
[586, 142, 97.87, 97.88]
Forward contract:
[1122, 275, 97.79, 97.80]

From this I can glean that I should ideally be done rolling within the next couple of weeks, which is over a year from the price expiry as befits an 'infinite' instrument. Looking at the liquidity picture, M16 averaged 8,155 contracts a day over the last five business days, and U16 did 4,771 which is less but still more than enough.

The new month ('forward contract') is trading with a 1 tick spread, in decent size.

I am long this thing to the tune of 8 contracts, so I want the best backwardation. The backwardation for M16 is 97.96 - 91.91 = 0.05 and for U16 is 97.91 - 91.84 = 0.07. So the backwardation will actually slightly improve when we switch. There is no reason not to do this roll.

Rolling execution algo


My execution process will behave differently according to the roll state:

  • NOT rolling. Buy and sell only the current month.
  • PASSIVE rolling. Opportunistically buy or sell the current or new month, as above. When (if) we have no positions left in the current month change state to FINISHED.
  • IMMEDIATE rolling. Issue a spread order to do the entire roll. When (if) completely filled change status to FINISHED.
  • FORCELEG rolling. Issue seperate trades for each leg of the entire roll, assuming both meet certain liquidity requirements. When (if) completely filled change status to FINISHED.
  • FINISHED rolling. Buy and sell only the new month.

Live price stitching

Let's return to the extract of stitched price data for AEX that we looked at above. However I am now going to wind the clock back to just before the roll began:


>>> atable=adjusted_price_table("LIVE", "AEX")
>>> atable.read_prices_for_code("AEX")[pd.datetime(2015,5,10):pd.datetime(2015,5,12)]
                         ADJ  CARRY CARRY_CONTRACT  FORWARD FORWARD_CONTRACT  \

2015-05-08 23:00:00  488.750  488.25        201506   488.250          201506
2015-05-11 08:01:14  490.175    NaN         201506      NaN           201506  
2015-05-11 09:01:29  490.175    NaN         201506      NaN           201506  
2015-05-11 10:02:52  488.350    NaN         201506      NaN           201506  
2015-05-11 11:03:04  488.325    NaN         201506      NaN           201506  
2015-05-11 12:04:23  488.450    NaN         201506      NaN           201506  
2015-05-11 13:05:35  489.275    NaN         201506      NaN           201506  
2015-05-11 14:13:16  489.725    NaN         201506      NaN           201506  
2015-05-11 15:40:35  490.325    NaN         201506      NaN           201506  
                          FX    PRICE PRICE_CONTRACT  ROW_IDS

2015-05-08 23:00:00  1.12964  488.750         201505     2617 
2015-05-11 08:01:14  1.12494  490.175         201505     2618 
2015-05-11 09:01:29  1.12494  490.175         201505     2619 
2015-05-11 10:02:52  1.12494  488.350         201505     2620 
2015-05-11 11:03:04  1.12494  488.325         201505     2621 
2015-05-11 12:04:23  1.12494  488.450         201505     2622 
2015-05-11 13:05:35  1.12494  490.275         201505     2623 
2015-05-11 14:13:16  1.12494  489.725         201505     2624 
2015-05-11 15:40:35  1.12494  490.325         201505     2625 


Focus on the red row. At this point we've done our position rolling, and we're ready to move from collecting K16 for pricing ('PRICE CONTRACT') to collecting M16 ('FORWARD CONTRACT').

Remember to do a roll we need to have a price in both the old and new contracts. But there is no FORWARD price, since I'm only collecting end of day. In this common case I offer the user the option to extrapolate from the last good price difference, or to put their own forward price in.

In row 2617 the last good price difference was 0.50 (488.75 versus 488.25). So the extrapolated forward price is 490.335 - 0.50 = 489.825. We add this as a pseudo row:


                         ADJ  CARRY CARRY_CONTRACT  FORWARD FORWARD_CONTRACT  \
2015-05-08 23:00:00  488.750  488.25        201506   488.250          201506
2015-05-11 08:01:14  490.175    NaN         201506      NaN           201506   

... removed ...
2015-05-11 14:13:16  489.725    NaN         201506      NaN           201506  
2015-05-11 15:40:35  490.325    NaN         201506      NaN           201506   

2015-05-11 15:40:36  490.325    NaN         201506  489.825           201506   

 
                          FX    PRICE PRICE_CONTRACT  ROW_IDS

2015-05-08 23:00:00  1.12964  488.750         201505     2617 
2015-05-11 08:01:14  1.12494  490.175         201505     2618  

... removed ...
2015-05-11 14:13:16  1.12494  489.725         201505     2624 
2015-05-11 15:40:35  1.12494  490.325         201505     2625 

2015-05-11 15:40:36  1.12494  490.325         201505     2626 

Notice this is a clone of the previous row, apart from the imputed forward price (in bold) and the slightly incremented date (to respect the unique datetime key in the table).

This works well for me, although for high frequency you'd need to use a smaller date increment, and you might want to mark the row with a flag to indicate it's not a true price.

Now for the actual roll. First of all we're going to add another pseudo row.

                         ADJ  CARRY CARRY_CONTRACT  FORWARD FORWARD_CONTRACT  \
2015-05-08 23:00:00  488.750  488.25        201506   488.250          201506
2015-05-11 08:01:14  490.175    NaN         201506      NaN           201506   

... removed ...
2015-05-11 14:13:16  489.725    NaN         201506      NaN           201506  
2015-05-11 15:40:35  490.325    NaN         201506      NaN           201506   

2015-05-11 15:40:36  490.325    NaN         201506  489.825           201506   
2015-05-11 15:40:37  489.825    NaN         201507      NaN           201507  

 
                          FX    PRICE PRICE_CONTRACT  ROW_IDS

2015-05-08 23:00:00  1.12964  488.750         201505     2617 
2015-05-11 08:01:14  1.12494  490.175         201505     2618  

... removed ...
2015-05-11 14:13:16  1.12494  489.725         201505     2624 
2015-05-11 15:40:35  1.12494  490.325         201505     2625 

2015-05-11 15:40:36  1.12494  490.325         201505     2626 
2015-05-11 15:40:37  1.12494  489.825         201506     2627 

Notice that the new PRICE is the same as the FORWARD in the prior row 2626,  because M16 is now the PRICE_CONTRACT. N16 is now the FORWARD_CONTRACT, for which we don't yet have a valid price. The ADJ price is the same as the PRICE; this will always be the case in this data frame

Finally we need to resplice the adjusted price series. Since there has been no real price movement between rows 2626 and 2627 (and also the previous row, although this isn't always the case) the ADJ should be the same in both cases. To achieve this the ADJ in row 2626 needs to be 0.50 lower. We also need to move all previous ADJ right back to the start of our data history down by 0.50.

Here is the final result, with the new ADJ highlighted in bold, all of which are 0.50 less than the previous entries.

                         ADJ  CARRY CARRY_CONTRACT  FORWARD FORWARD_CONTRACT  \
2015-05-08 23:00:00  489.250  488.25        201506   488.250          201506 2015-05-11 08:01:14  489.675    NaN         201506      NaN           201506   
...removed ...
2015-05-11 14:13:16  489.225    NaN         201506      NaN           201506  
2015-05-11 15:40:35  489.825    NaN         201506      NaN           201506  
2015-05-11 15:40:36  489.825    NaN         201506  489.825           201506  
2015-05-11 15:40:37  489.825    NaN         201507      NaN           201507  


                          FX    PRICE PRICE_CONTRACT  ROW_IDS 
2015-05-08 23:00:00  1.12964  488.750         201505     2617  
2015-05-11 08:01:14  1.12494  490.175         201505     2618  
.... removed ...
2015-05-11 14:13:16  1.12494  489.725         201505     2624 
2015-05-11 15:40:35  1.12494  490.325         201505     2625 
2015-05-11 15:40:36  1.12494  490.325         201505     2626 
2015-05-11 15:40:37  1.12494  489.825         201506     2627 


This re-adjustment of all previous rows in the table can take a while, during which the database is locked. I need to pause my trading and sampling processes (process control will form another post in this series) to make sure they don't try and access the price data.

From a technical point of view I've found that partitioning my spliced price data so that each instrument occupies one file on disk is more efficient (the readjustment is quicker; in the order of five minutes for 30 years of history rather than hours) as well as isolating so only a single file is locked at a time, and all other instruments are unaffected.

Fin


That is pretty much everything you need to know about futures rolling, or at least everything I can think of right now. However if I've missed anything out, or something is unclear, then comment below or email me.

This is the second in a series of posts on building systematic futures trading systems.

The first post is here.
The next post is here.

94 comments:

  1. Thank you for the very interesting article. Can you please clarify few points regarding the carry crude oil futures carry trade example:
    - because the price in contango you short Dec CL contract and long Nov CL contract?
    - When do open this trade?
    - What is the carryoffset -1 in the CL futures example? Why it’s -1?
    - Priced contract is $201512 , is it the price for Nov CL contract?
    - Also, in your book as well as in this article you mentioned that contango or backwardation is the main decision factor for being long/short. Do I understand it right?
    Thank you
    Max

    ReplyDelete
    Replies
    1. Max,
      Sorry I don't really follow many of your questions, but I'll try and clarify things.

      "Can you please clarify few points regarding the carry crude oil futures carry trade example:"

      - because the price in contango you short Dec CL contract and long Nov CL contract?

      This would be a spread trade. I'm not discussing spread trading in this post, which is a whole new ball game.

      - When do open this trade?

      I don't understand the question; I don't have discrete "trades" only a continous position target.

      - What is the carryoffset -1 in the CL futures example? Why it’s -1?

      This is a parameter telling my system to use the contract immediately before the priced contract to calculate carry. Eg if we're in Dec 16 we should use Nov 15 to calculate carry (Dec 16 vs Nov 15)

      Using -1 is preferable; unless it isn't liquid enough, or we're already holding the front month. See the description of the carry rule in appendix B.

      -Priced contract is $201512 , is it the price for Nov CL contract?

      No, that's a yyyymm timestamp, telling me my priced contract is currently december 2015.

      - Also, in your book as well as in this article you mentioned that contango or backwardation is the main decision factor for being long/short. Do I understand it right?

      No, that's wrong. The carry rule I describe in chapter 7 does exactly that. But you'd have other rules as well, eg trend following rules.

      GAT

      Delete
  2. Thank you for your reply. Hope you will keep posting more articles like this as well as looking forward to new articles on pysystemtrader

    ReplyDelete
  3. Hi Rob, with the Panama method you have to rewrite the whole price history once you move to a new contract. Do you see any problems with instead keeping history constant and adjusting just the new contract?

    ReplyDelete
    Replies
    1. Mathematically it would make no difference. However I personally like the fact that the current adjusted price is equal to the price of the contract I'm already trading.

      Delete
  4. Hi Rob, When using the Panama method, don't the percentage return distortions that arise going back in time create problems when calculating return volatility and correlations--at least with contracts where substantial drift is introduced and particularly those where the price approaches zero or goes negative? Do you adopt a different method where this is the case?

    Thanks, John

    ReplyDelete
    Replies
    1. Yes it would cause problems. For this reason when I calculate anything that is a percentage, I use the following Pt - Pt-1 / P*t-1 where Pt and Pt-1 are the adjusted price, and P* is the price of the actual contract I was trading at the time. pysystemtrade does this automatically.

      Delete
    2. From the user guide....there is a special method in the data class called `daily_denominator_price`. This tells the code what price to use for the P* in the calculation above. In the base class this defaults to the stitched price (but in the futures class, described below, it uses the raw price of the current contract).

      Delete
    3. Hi Robert - maybe I am asking too much but I’ll try. Would you be able to upload an excel sheet on “systematic trading” webpage and show how all the adjustments are made and how to calculate the daily price vol with stitched price series, as you describe here? Thanks!

      Delete
    4. Vol calculation is covered by one of the existing sheets here https://www.systematicmoney.org/systematic-trading-resources

      https://docs.google.com/spreadsheets/d/1NJw1cnnsVCG6QxXutylmUJ-0naV8QZHonZZ6rCtTRo8/edit?usp=sharing

      Delete
  5. I wrote this Python script for the VIX example; it was able to reproduce the graphs exactly. Needs a pip install for Quandl and an Quandl API key.

    https://gist.github.com/burakbayramli/5d263f611da41fae985a#file-vixstitch-py

    Note: I delete gists periodically, so the link can go away at some point.

    ReplyDelete
  6. Quick question: I tried to replicate the Eurodollar front month / 9th month volatility graph, but I could not get the volatility between 98 and 98.5 you've shown for the 9th contract.

    My method was: go to Quandl, download historical data for EMM2015 (june 2015) and EMU2015 (september 2015), and plot the Settlement columns. When that did not give the range you showed, then I tried all other months as well, N,Q,V,X,Z - I do not see that exact volatility. There is volatility, but all prices seem to be above 99.0.

    Regards,

    ReplyDelete
  7. Dear Rob, I have several questions after reading your article:

    1. If i want to backtest the sytem say on 20 different futures, does that mean i need to pick traded contract for each instrument depending on its liquidity pattern and then define roll policy and stich each instrument separately? This sounds like a very laborious task!

    2. Can I use setttlement price for stiching instead of close? Quandl seem to have big gaps on "closes" for most of the data the offer for free.

    3. I didn't quite get the idea behing being long the most backwarded contract, how this relates to the combined forecast i might have on this instrument? I suggest this contract will get the biggest carry forecast and thus when combined with forecasts from other rules the combined forecast will be biased upwards for the instrument.

    4. Why is it better to trade next contract and measure carry against nearer contract rather than always trade the front month and measure carry against the next contract?

    5. What instruments have consistent carry and why do they? Why others don't? Should I use carry rule only on those who have?

    Thank You!
    Peter

    ReplyDelete
    Replies
    1. 1. Yes you have to do this. Or you could subscribe to a premium service on quandl.com that offers continuous contracts. Or you could piggyback off the data in pysystemtrade.

      2. You can use any price you like, as long as you are confident that the two contracts that are being stitched are synchronised at that point.

      3. A systematic way to think about it is this: suppose we have a system that had to choose between all the contracts (for a single instrument) we could possible trade, but could only choose one. You would choose the strongest forecast (highest conviction). The forecast's on those contracts are likely to be very similar, but the one that is strongest is probably going to be where the carry and other signals are facing "the same way" and where carry is strongest. Short of calculating the actual forecast for each contract choosing the contract with the best carry for our position sign is a good approximation for this ideal.

      4. Because the carry you actually earn is always between a further out contract and a nearer contract (or spot if unavailable); the former 'ages' to become the latter.

      5. http://www.bi.edu/InstitutterFiles/Samfunns%C3%B8konomi/Papers/Fall%202013/Pedersen.pdf

      table 3. p.53

      Carry seems to be pretty consistent. Statistically I've never seen evidence that carry shouldn't be included for a particular instrument (in general I find this particular kind of over fitting doesn't produce worthwhile results).

      Delete
  8. Dear Robert, thank you for the response!

    However I still didn't quite grasp the idea.

    If I want to be long most backwarded and short most contangoed contract, this means i will need to check with my most current forecast to decide to which contract to roll to?

    This implies my roll policy won't be consistent due to constant changes of my combined forecast and term structure, is this so in real life?

    Thanks!

    ReplyDelete
    Replies
    1. Yes your roll policy might change depending on the shape of the curve, and your current position. Particularly in something like VIX, the curve is sometimes weird and your roll policy could be very different.

      Delete
    2. Hi Rob,
      So when you are building your back history of stitched prices, in your opinion do you think its better to then stick with one contract (Winter Crude or the far out euro dollar) rather than in each period try and figure out the best contract to be in?
      Then in reality when you're actually trading, your stitched price series could be a combination of roll policies as you adjust to the optimum contract?
      Thanks

      Delete
    3. It depends on the market. In crude I'd rather stick to Winter. In Eurodollar right now it's a balance between volume (better at the front) and volatility (better further out). In VIX / V2X it could vary from month to month as the curve changes shape. However in a backtest we can't do something like that without a lot of work, so we have to stick to one policy.

      Delete
  9. Hi Rob, in userguide.md you wrote "However when working out percentage returns we absolutely don't want to use the 'stitched' price as the denominator. For positive carry assets stitched prices will increase over time; this means they will be small or even negative in the past and the percentage returns will be large or have the wrong sign."

    Could you ellaborate on this little more? In your carry datasets, you have one column for PRICE for current contract, and another for CARRY. Both come from different contracts, and they would be stitched seperately, using some roll dates. PRICE-CARRY / denom would generate a forecast, and a return can be calculated. When would this break down?

    Thanks,

    ReplyDelete
    Replies
    1. The stitched price is the one we get by backadjusting futures prices. If carry is positive then stitched prices will have a drift term added to them over time. So imagine you have something like US 10 year bonds, currently trading @130ish. Look at the price of US10. At the end of the series it matches the current contract. But in 1986 it's close to zero (1.08 on 1986-02-27 for example) when the actual contract was around 98. At the start of the series in 1982 it is -37. But the actual contract price was 72. The difference of 119 points is 34 years of accumulated carry.

      Now to work out % returns let's suppose the price moves by 1 point. If I use the stitched price series I will get 1/130 for todays move, about .77%. The same move in 1986 it should be 1/98 = 1%. But if I use the stitched price as the denominator I get 1/1.08 = 93%! In 1982 a point move should be worth 1/72 = 1.4%. Using the stitched price gets a negative number!

      Delete
    2. Thanks for the clarification.

      Delete
  10. Hi Rob,

    You wrote "If you're long then it's better to be in a delivery month which is backwardated versus the nearer month, i.e. the nearer price is above the price of what we're currently holding, so we'd expect the contract we hold to rally over time (..) If short you'll want to be in a month which is in contago versus it's nearer counterpart"

    This seems to suggest that depending on forecast (long/short) I would have to change which contract I look at for carry. But I look at VIX_carrydata.csv in pysystemtrade, PRICE_CONTRACT is always one ahead of CARRY_CONTRACT, it never changes. Did I misunderstand?

    ReplyDelete
    Replies
    1. Yes you misunderstood! We always calculate carry by taking the difference between the month we're holding and a nearer contract (ideally - Read appendix B of my book again). This is the same regardless of if we are long or short.

      Delete
  11. Yes I saw that (appendix B), and the CARRY/PRICE columns were in-line with that explanation. Anyway; the excerpt from above start with "On a related subject", so I guess this is a nice-to-have but harder approach? I will go with the simpler one :) Thanks!

    ReplyDelete
  12. Hi Rob,
    I note that for carry purposes, you mention it is always better be in the second periodic contract and to compare this to the first periodic contract if it is available.
    In pysystemtrade I see that e.g., for SP500, we trade ES1 and compare it to ES2, as ES2 is never liquid enough.
    Why wouldn't you compare ES1 to the SP500 cash index in pysytemtrade in this case? This goes for all of the equity indices in pysytemtrade. The volatility indices could also be compared to cash. Why is that not done?
    Thanks!

    ReplyDelete
    Replies
    1. It's fine to do that. However you need to go to the bother of getting the spot data downloaded from somewhere. For equities the benefit would be minimal as the carry is likely to be very similar (for VIX / V2X might be more helpful as the curve is often weird, but then here we can get more contracts).

      Delete
    2. Hi Rob,
      For situations where the second futures contract is used for carry (e.g., with SP500 we trade ES1 and use ES2 for carry), do we need to make any adjustments in pysystemtrade so the carry rule knows to look at the second contract instead of the first contract? Some instruments will trade the second/third month and use the first for carry (e.g., VIX, V2X), whereas others do the reverse (ES1/2 as mentioned above). Are these configured differently?

      Delete
    3. You don't have to do anything. The code understands the difference.

      Delete
    4. Thanks so much.

      For your IB account, can you make use of cross-margining? I ask as I note that pysystemtrade does not take posted margin into account, and at times the margin required to finance positions equals or exceeds account value, which is unrealistic. The only way this could work is if you cross margin.
      How do you account for margins in your own system and can you cross margin? What is your average margin to equity?

      Delete
    5. "at times the margin required to finance positions equals or exceeds account value, "

      That depends on your annual risk target. If it's too high then yes you may run out of margin.

      With a 25% annual risk target my average margin to equity is about 30% and I'd never expect it to exceed 50%. This is why I never bothered including margin figures - doing a backtest of margin requirements is hard.

      That suggests that something like a 50% annualised volatility might be the maximum.

      I just have the standard "Portfolio Margin" with interactive brokers.

      Delete
    6. I see. Apologies, I had miscalculated margins by adding the initial + maintenance margin, but after having read your reply and re-read initial + maintenance margin notes on IB I see that the figures you mention above seem correct.

      On another note, have you approached a level where you can leave your system for a period of time? One thing I am concerned about with starting such a trading strategy is that I would never be able to take a holiday/be sick/leave a stable internet connection for 24hrs. How do you handle this?

      Delete
    7. I just go on holiday. I'm doing 3 weeks this summer. I've backtested my strategy to see the effect on Sharpe Ratio of a 3 week delay in trading. It's about 10%. I can live with this. I've been on holiday and lost my internet, and then not traded for a couple of weeks.

      If I was trading something much faster, or with much higher risk, then I'd probably turn it off.

      Otherwise the solution is a virtual server and checking in every day with your smart phone. Not for me.

      Delete
    8. Seems a healthy approach!

      I assume then you close down all existing positions when doing so?

      Delete
    9. No - I leave the system running with all positions open. If anything happens the worst case scenario is I won't trade for a few weeks. This is what I've backtested.

      Delete
  13. Hi Rob, for crude oil you said "you roll at least 45 days before expiry". The showinstrument output for crude oil though says "lastrolloffset -60 days, firstrolloffset -90 days". Should that say lastrolloffset -45 days? And in that case what would firstrollloffset be?

    Thanks,

    ReplyDelete
  14. Those figures aren't used any more when I decide when to roll. Instead I get a message to roll 20 days before the expiry of the carry contract, which of course is around 50 days before price expires.

    ReplyDelete
  15. Thanks. Based on your posts and the data in _carrydata.csv files, I tried to come up with some instrument values. Any glaring ommisions?

    instrument,carryoffset,rollcycle,rolloffset
    CL,-1,Z,50
    EDOLLAR,-3,HMUZ,440
    US5,-9,HMUZ,50
    EUROSTX,3,HMUZ,90
    V2X,-1,FGHJKMNQUVXZ,60
    MXP,3,HMUZ,90
    CORN,-3,Z,365

    ReplyDelete
  16. Looks about right. My full config will one day be on github.

    ReplyDelete
  17. Hi Rob,

    I was thinking about buying the month with the steepest Backwardation and shorting the steepest Contango -

    I thought futures were priced very efficiently, and diff slope could be due to valid reasons (e.g. more expected dividends this month than next, for index futures).

    Also if this point is valid, then it implies there are arbitrage opportunities long/shorting different months then? (think its called calendar spreads?) so this must be a very profitable strategy if the anomaly is valid.

    ReplyDelete
    Replies
    1. This strategy will work, in the sense that it is profitable, but it's not a pure arbitrage as it isn't guaranteed to converge. In something like equity futures the slope difference will just be down to dividends (and I wouldn't do it here since the back months aren't liquid enough, same for bond futures).

      In other places there are good reasons why there are profits here. In STIR futures it's a version of the term premium you can earn. In commodities it will be driven by supply / demand and inventory costs. In volatility it's differences in the term structure of vol.

      You'll need large leverage to do something like this as obviously the two legs will be highly correlated and execution is vital to get right.

      Delete
  18. Hi Rob,

    Nice article which gives some good insights into rolling strategies.
    I've some questions about your article.

    1. you point out a relative advanced (at least for me) rolling methodology. But it's not clear for me if you do all this steps both in real live trading as in simulation. If I want to do this all in simulation, it's a complex task to build this up, especially the part when you go from passive trading to active (spread or separate legs).
    In my opinion this could be realistic to build :
    - in live : obviously we do all the steps (with some manual interaction)
    - in simulation : prepare data (stitching and adjust prices on daily basis, so nothing intraday), passive trading and just one form of active trading (= separated legs because this is the most conservative choice for costs) when reaching expiry date.
    Or didn't you go so for in simulation ?

    2. is there a reason why you didn't mention Open Interest for rolling decisions? For so far I can see you mainly look to the expiry date and then to the volume (over x days).

    3. The method to choose the right contract depending on your position (long or short) is also a difficult one to program and also not easy in live trading I think. Has this a big advantage compared with the classic contract selection ?

    Kris

    ReplyDelete
    Replies
    1. 1. Exactly, in simulation I'd do (a) a very basic thing that just uses the backadjusted stitched series (current implementation of pysystemtrade) which will mean roll costs won't be considered and (b) a slightly more complex thing that assumes we trade the entire position on one specific roll date

      2. Open interest and volume tend to be pretty correlated; I'm a small trader rather than a big fund so I'm more interested in the volume rather than worrying I'll be more than 10% of the OI on a thin contract.

      3. It does add something, it won't double your Sharpe, but "every little helps" and it's quite consistently good. However there aren't many contracts in practice where you can do this - the VIX/V2X are the only ones where you genuinely have a choice of several contracts and where you also get big changes in curve shape over the liquid months.

      Delete
  19. Thank you for the information

    ReplyDelete
  20. Hi - I am trying to figure out why the breakout system setup you walked through in a different post loses a large amount of money on Sep 20 1982, in corn. The price jumps significantly on that date, which seems to coincide with a roll taking place (the contracts in the equivalent carry prices file switch from 1982 to 1983). I would expect the prices to be stitched together but they do not seem to be. Is this price jump correct, in which case there may be some aspect of price stitching that I am missing? Or is this a mistake in the stitching algorithm?

    ReplyDelete
    Replies
    1. There is no price jump so the algo must have screwed up. I've corrected the csv files in pysystemtrade. When I come to include stitching in pysystemtrade I'll check to see that the error doesn't get reproduced.

      Thanks for spotting that - it really shows the benefit to me of open sourcing stuff!

      Delete
    2. Hi, there seems to be a similar issue with US10 on 1999-10-26, the carry data loses its price for one of the contracts, and the price data itself shows a large jump. Out of interest how do you handle situations like this in your own trading? i.e. if you realize you had some bad data which you were using to formulate weights etc do you re run? Or do these not have much of an impact on the end result (given number of instruments, amount of data being used etc).
      How's the new book coming along - do you need any help proof reading?

      Delete
    3. I'll check that out - thanks.

      These things shouldn't make much different on the end results. Indeed adding random noise to your prices is a good way of testing how robust everything is.

      New book going well I think, I'm probably okay for proofreaders, but thanks for the offer.

      Delete
  21. When you do trade/roll is there a specific time during the day or trading session that you prefer to trade at? For example towards the end of the day?

    ReplyDelete
    Replies
    1. No specific time. The order manager program also does the actual trading or rolling and runs throughout the day.

      Delete
  22. Robert,

    I'm trying to make my own rolling process. In first instance I want to use this process to create my own continuous contracts.

    In the following link you will find a graphical view of my idea : http://www.screencast.com/t/MCXTv3VyAf

    The point is that I make first a difference between contracts with 'fixed' specified contracts (front only or seasonal) or 'flexible' contracts (1yr liquidity or infinite liquidity).

    When the rollwindow is entered, the proces is as follows :

    For fixed contracts :
    1. check Volume5D
    2. check Open Interest.
    3. If both OK --> activate rolling process

    For Flexible contracts :
    1. Make a selection from all contracts where the Volume5D is OK. We have now a subset of contracts
    2. From this subset, make a selection where the Open Interest is OK. We have now a new (maybe smaller) subset of contracts
    3. From this subset, select the contracts with highest volatility. We have now e new (maybe smaller) subset of contracts
    4. choose the contract wich is closest to the current maturity. So when we are current trading 9 months further, we select the contract which is closest to this 9 months.
    5. when for example the new contract is 12 months in further, we must change the roll window. If it is the same, the roll window is not changed.
    6. when the contract is selected --> activate rolling process

    I think I can program this but I would like to know if this is a good and realistic process.

    Kris

    ReplyDelete
    Replies
    1. Yeah it sounds feasible, although massively more complex than what I do myself!

      Delete
    2. Robert,

      Indeed, it was some reasonable job (after-work) but from now I can create my own continious contracts following the process I described above. For the moment it's based on Quandl data but in the future I want also get data from IB for a double-check or when Quandl data is delayed.

      If interested in parts of my code (written in vb.net) for some ideas, please let me know.

      Kris

      Delete
  23. Hi Rob,
    thanks a lot for the detailed explanation.
    I think there might be an issue with Eurodollar data in https://github.com/robcarver17/systematictradingexamples/blob/master/data/EDOLLAR_carrydata.csv

    Seems like nearer price is almost always (more than 85% of days since 1983.) lower (in contango) than traded price, which means we should be short. This in opposite from what Quandl data shows, I was able to verify (using 6-7 datapoints from 2006-2013) that traded price is exactly as in the data from Quandl, but nearer price is always off.
    This causes the carry forecast to be negative (as nerpu is negative) which you can easily reproduce using your own code here https://github.com/robcarver17/systematictradingexamples/blob/master/carry.py
    In your other post (http://qoppac.blogspot.com/2016/01/correlations-weights-multipliers.html) you've also show a graph where carry for EDOLLAR is usually positive, which would make more sense as well.

    Additionally, in your previous post in this series, you've showed example for Eurodollar price from your DB where carry (nearer) price is above traded, opposite from what's in the csv file, that row is:
    2015-04-21 23:00:00 97.9050 97.985

    I checked for other instruments such as crude oil and corn, and wasn't able to figure out where does the nearer price come from, but it's never from the actual price of the nearer contract on that given date. Could you explain how you calculate nearer price?

    Thanks!

    ReplyDelete
    Replies
    1. I think you are misreading the columns. For example for the last row in eurodollar carry the carry contract (201906) has a price of 98.23 which is higher than the price for the further out (201909) priced column price of 98.16.

      Delete
    2. Ah I see the issue now. I was looking at the prices in systematictradingexamples repo, I just saw that there are prices in the pysystemtrade repo I think those might be more up to date.

      Thanks!

      Delete
  24. Hi Rob, If I look at your legacy csvs, for EUROSTX_price, I see that the price for 2013-03-19 is 2290, but the price for that date in the carry version of the file is 2577, which matches what I see in Quandl. Are you only applying the stitching adjustments to the price file and not the carry file?
    On another topic, I plan on stitching in the following way: each day, I will check whether it's a good day to roll. If so, I will take the difference between yesterday's close of the contract I was holding (let's say 95), and yesterday's close of the contract I want to hold (let's say 105). I will then add 10 to every price in my price history since the beginning. Does this make sense? Thanks, as always, for your guidance. -Peter

    ReplyDelete
    Replies
    1. Yes the whole point of the carry file is it contains, raw unstitched data.

      Your approach to stitching is correct.

      Delete
  25. Hi Rob, thank you for yet another amazing post. I’m asking this question in relation to the EWMAC strategy you mentioned in your first book: I understand that if you use the Panama stitching method there will be price-drift, and while that won’t affect the calculation of vol for vol-adjusting the forecasts (since we use the absolute movement instead of % movement), it will effectively affect the integrity of a trend-following system since prices have an unintended drift, am I right? My system incorporates the use of a 250 EMA so I’m wondering if the effect of the drift will be pronounced.

    Also, if there’s an ‘unintended’ drift due to rollover-adjustment then the trend following system effectively captures trend-following signals and carry signals - am I right to say that?

    Appreciate it if you could help clarify! Thanks Rob!

    ReplyDelete
    Replies
    1. You are correct - stitching means that accumulated carry is included in the backadjusted price series which lead to biases in slower momentum signals - and this is indeed a subject which I've agonised over in the past. So I've tried stripping out the carry component and then trend following what's left. But all that happens is you end up putting more money on the explicit carry rule, and you end up with exactly the same system, more or less. In an institutional setup this is (cynically) a way to get more carry into your system than your explicit trend following mandate would allow. My current feeling is that it's simpler to just trend follow the backadjustement with the drift, and in any case I am comfortable with a higher allocation to carry (both explicit and implicit) than I could have got away with in an institutional context.

      Delete
  26. Hi Rob, thank you for your amazing blog and I am grateful for your prompt reply to everyone.

    I have one question regarding carry signal which puzzles me for a while. Hope you could shed some light on it.
    I discovered that there is "seasonality" in the carry signal of equity indices futures because of the dividend payout period. For instance, to trade CAC40 futures and trade quarterly contracts only in this case. The spread between Generic 1 and Generic 2 futures series during Q1 is always higher than during other quarters (a significant level shift). Just to be more precise, the carry signals calculated using June - March Contracts is higher than the signals calculated using other quarterly contracts. This causes a jump in carry signals at the beginning of Q1 and a gap down at the end of Q1. How do you deal with the "seasonality" issue? or it is OK to leave carry signals being calculated as simple as the spread?

    Much appreciate if you could help answer this!

    ReplyDelete
    Replies
    1. This seasonality occurs in many places- bonds, commodities...

      There is no perfect solution, but my own answer is to use a smoothed version of carry by taking a moving average of the raw signal. A lookback of 90 days will take care of quarterly seasonality. A lookback of 365 days will do the same for annual. Both also reduce the responsiveness of the carry signal, so in practice I use a mixture of different smooths.

      Delete
    2. Thanks Rob.That's a great solution.

      Delete
    3. Hi Rob,

      What do you think about solving the seasonality problem by instead calculating the carry signal using a wider forward looking span?

      E.g., CL_carry = CL_front_month / CL_1_year_forward

      Of course, the 1 year forward version would be less volatile than the 1 month ahead version so it would have to be scaled up appropriately. But it seems to me that it could give better information about the current payoff to carry than using stale prices.

      Delete
  27. Hi Rob,
    I'm currently writing a version of the stitching logic from scratch using my own data and tried to compare the results with your most recent .csv files in github.
    I started with CRUDE_W (i only have data starting in 2006) and found an incorrect behavior in your adjusted prices. To be clear i'm talking about this file:
    pysystemtrade\data\futures\adjusted_prices_csv\CRUDE_W.csv
    Your adjusted price time-series jumps from 74.88 to 90.98 between Nov. 13th 2006 and Nov 15th 2006. There is a roll happening around those days and, looking at raw data, the futures moved less than a 1USD between those dates. So i suspect there is something wrong in the roll logic. FWIW, the incorrect offset of 15USD disappears around 1 year later when the next roll happens.
    Just thought you might want to know this as it might impact other periods or futures.
    One1Hedge

    ReplyDelete
    Replies
    1. FWIW, I generated adjusted prices with my stitching function and reran a BT on the following perimeter:
      ['AEX', 'AUD', 'BOBL', 'BTP', 'BUND', 'CAC', 'COPPER', 'CORN', 'CRUDE_W', 'EDOLLAR', 'EUR', 'EUROSTX', 'GAS_US', 'GBP', 'GOLD', 'JPY', 'KOSPI', 'KR10', 'KR3', 'LEANHOG', 'LIVECOW', 'MXP', 'NASDAQ', 'NZD', 'OAT', 'PALLAD', 'PLAT', 'SHATZ', 'SMI', 'SOYBEAN', 'SP500', 'US10', 'US2', 'US20', 'US5', 'V2X', 'VIX', 'WHEAT']

      Here are the results:
      Before (=files currently in github)
      [('min', '-22.89'), ('max', '18.2'), ('median', '0.05709'), ('mean', '0.07933'), ('std', '1.356'), ('skew', '-0.2759'), ('ann_mean', '20.31'), ('ann_std', '21.7'), ('sharpe', '0.936'), ('sortino', '1.17'), ('avg_drawdown', '-12.1'), ('time_in_drawdown', '0.9306'), ('calmar', '0.4202'), ('avg_return_to_drawdown', '1.679'), ('avg_loss', '-0.8402'), ('avg_gain', '0.8803'), ('gaintolossratio', '1.048'), ('profitfactor', '1.203'), ('hitrate', '0.5345'), ('t_stat', '6.354'), ('p_value', '2.182e-10')], ('You can also plot / print:', ['rolling_ann_std', 'drawdown', 'curve', 'percent', 'cumulative']

      After (=with new stitching)
      [('min', '-14.78'), ('max', '18.61'), ('median', '0.06289'), ('mean', '0.09877'), ('std', '1.284'), ('skew', '0.6726'), ('ann_mean', '25.29'), ('ann_std', '20.54'), ('sharpe', '1.231'), ('sortino', '1.71'), ('avg_drawdown', '-8.702'), ('time_in_drawdown', '0.9102'), ('calmar', '0.6024'), ('avg_return_to_drawdown', '2.906'), ('avg_loss', '-0.8152'), ('avg_gain', '0.8909'), ('gaintolossratio', '1.093'), ('profitfactor', '1.261'), ('hitrate', '0.5358'), ('t_stat', '8.355'), ('p_value', '7.286e-17')]

      Delete
  28. Rob,
    I was able to isolate the problems by using only your data (from folders multiple_prices_csv and adjusted_prices_csv).
    I found a few incorrect behaviors by comparing your adjusted_prices_csv files to the time series i generated with my stitching function.

    1. If i am correct, the adjusted prices should be built by using only the last price per day, usually the one you record with the timestamp 23:00:00. It looks like the stitching function doesn't remove the intraday duplicates for historical prices. I was able to spot this with the following example on SP500.
    From multiple_prices_csv\SP500.csv:
    DATETIME,PRICE_CONTRACT,FORWARD_CONTRACT,CARRY_CONTRACT,PRICE,FORWARD,CARRY
    2018-03-12 16:41:01,201803,201806,201806,2785.125,-5.75,
    2018-03-12 16:41:02,201806,201809,201809,-5.75,,
    2018-03-12 23:00:00,201806,201809,201809,2792.5,2799.5,2799.5
    2018-03-13 23:00:00,201806,201809,201809,2767.5,2774.75,2774.75
    2018-03-14 15:03:31,201806,201809,201809,2779.375,,
    The data recorded at 2018-03-12 16:41:02 is incorrect, it shows PRICE=-5.75.
    This incorrect intraday value is used when rebuilding the adjusted prices and propagated to all dates before:
    From adjusted_prices_csv\SP500.csv:
    DATETIME,price
    2018-03-12 15:32:22,20.375
    2018-03-12 16:41:00,10.5
    2018-03-12 16:41:01,10.5
    2018-03-12 16:41:02,10.5
    2018-03-12 23:00:00,2808.75
    2018-03-13 23:00:00,2783.75
    You can see the adjusted price drops because of that incorrect value.

    2. The problem i mentioned in my previous post happens many times and not just on CRUDE_W. I believe this is happening when the "default" roll date is not available in the multiple_price csvs.
    For example: in my previous example Tuesday Nov 14th 2006 is missing in multiple_price_csv\CRUDE_W. But there are other incorrect jumps too so there is probably another condition for this to happen.
    Some other examples with big jumps in the adjusted prices and a corresponding missing date in multiple_price:
    CORN: jump on 20-Sep-1994
    SOYBEAN: jump on 19-Aug-2011
    GAS_US: drop on 18-apr-2006
    etc.
    If you chart the adjusted prices you can spot the big jumps but there are a lot of smaller ones that you couldn't catch easily.

    3. Also, the multiple_price file for US10 is incorrect from 28-July-1999 to 25-Oct-1999.
    It looks the PRICE for the correct contract was not available and has been replaced by the FORWARD price which leads to a big jump during that period.
    1999-10-25 00:00:00,199912,200003,200003,108.0625,96.3046875,96.3046875
    1999-10-26 00:00:00,200003,200006,200006,96.203125,,
    1999-10-27 00:00:00,200003,200006,200006,96.625,,
    You can see the FORWARD has been moved to the PRICE field and there is no FORWARD value during that period.

    I wrote a .py module that helps visualizing if you are interested. It's not long but not sure i should paste the code here.
    One1Hedge

    ReplyDelete
    Replies
    1. Thanks for this, I hope to look at this in the next few days.

      Delete
    2. Thank you Rob. If that helps, that's what I used to build the panama stitching on my side:

      def FUT_compute_adjusted_prices(price_df):

      # 1. Backward Panama stitching: we sort dates from most recent to oldest
      price_df = price_df.sort_values(by=['DATETIME'], ascending=False).reset_index(drop=True)

      # 2. Get roll dates from change in contract
      price_df.loc[:, 'IS_ROLL_DATE'] = False
      price_df.loc[price_df['PRICE_CONTRACT'] != price_df['PRICE_CONTRACT'].shift(1), 'IS_ROLL_DATE'] = True
      price_df.loc[0, 'IS_ROLL_DATE'] = False # resetting first line to False

      # 3. Compute roll offsets
      price_df.loc[:, 'ROLL_OFFSET'] = 0
      price_df.loc[price_df['IS_ROLL_DATE'], 'ROLL_OFFSET'] = price_df['FORWARD'] - price_df['PRICE']
      price_df.loc[:, 'ROLL_OFFSET_CUMSUM'] = price_df['ROLL_OFFSET'].cumsum()

      # 4. Compute adjusted prices
      price_df.loc[:, 'PRICE_ADJ'] = price_df['PRICE'] + price_df['ROLL_OFFSET_CUMSUM']
      price_df = price_df.sort_values(by=['DATETIME'], ascending=True)

      return price_df

      Delete
    3. "If i am correct, the adjusted prices should be built by using only the last price per day,"

      No, I want to preserve intraday data in the adjusted prices, as long as it is correct!

      The crude oil issue has gone away with a data refresh I had already done but not committed to git, but some of the other problems remain. I need to do a complete rebuild of the price series with up to date data. This will take a few days.

      Delete
  29. Hi Rob,

    I get bothered by unstable carry forecast for KR3 or GBP in my system (for which I need to trade nearer contract). Let's say I get max 20 carry forecast, and it drops to 15 next day, and it comes back again to 20 on the next next day. It seems this is due to the unstable closing price of the second cotract, which is not liquid enough. One thing I noticed while I am writing this - I am comparing the last single day price only between the 2 contracts. Do you increase the observation, say last 5 days, and take the average?

    ReplyDelete
    Replies
    1. Yes I take a variety of smooths for carry, the slowest is 10 days but I go up to 125 days.

      Delete
    2. I quickly implemented the 10 days average solution, and the unstability is gone. Will look into the longer days too. Thank you so much!

      Delete
    3. Now I implemented the 4 different timeframe for carry smoothing thanks to your advice. 10, 30, 60, and 125 days (got those numbers from your post on EliteTrader forum). I treat them as 4 variations of carry rule and give 0.125 weight each (total 0.5).

      However I noticed that you wrote that you use fixed 90 days for carry smoothing in this post (https://qoppac.blogspot.com/2017/06/some-more-trading-rules.html), and hence a bit confused.

      Does this simply mean that your methodology has improved over time, and the 90 days one is old dated, or am I missing something? Thank you in advance!

      Delete
    4. The 90 day was just there to keep the article simple. In my own system I have always used a mixture of smooths.

      Delete
    5. Thank you again! I am making advancement slowly. One step at a time but steadily. Probably it will take years to reach to the point where I want to reach technically, but that is also fine. I owe you a lot.

      Delete
  30. Hi Rob, I hope you’re well.

    I’d like to ask you a question. I have designed a trading system which uses a risk on/ risk off indicator to decide when it needs to be in the risky asset or risk free asset.
    However, there are many round trips per year - circa 9-10 - even after applying a moving average to slow it down. I now have a dilemma: which instrument to use to implement, ETFs or Index Funds. The problem I have with ETFs is the bid ask spread: when I factor in 10 bps, most of the returns are eaten away by that. However, if I were to use index funds, I would not have this issue BUT because their settlement period Is T+4, I might not be able to trade sometimes as quickly as instructed by the system - this still remains a “market timing” process.

    May I kindly ask what your thoughts or suggestions are?

    Thanks so much,
    Ric

    ReplyDelete
    Replies
    1. If you're trading 9 times a year, then a 4 day delay shouldn't damage your expected returns too much. Of course it's possible to calculate the effect by adding a 4 day delay to your backtest and seeing what happens.

      Having said all I don't understand why the settlement period is an issue, since you can normally trade with uncleared funds on most platforms.

      Delete
  31. Broken link: "This biases the trajectory of the price series (more here)."
    Current link: http://www.seykota.com/tribe/tsp/Continuous/index.htm
    Link should be: https://www.seykota.com/tribe/TSP/Continuous/index.htm
    (i.e it seems to be case sensitive)

    ReplyDelete
  32. Hey Rob, I have a question that is doing my head in. I have chosen to back adjust futures using the ratio method, as opposed to the difference/panama method. Then, for a large enough notional, I can calculate my P&L by using percentage differences (which is important for other purposes in my backtesting framework). One issue I have is in calculating the costs. If I am using the percentage method to back-adjust, do I need to back-adjust the costs? In my case, costs are expressed in percentage terms as: (half-bid-ask / contract size). (I am marking to mid so use half of bid-ask as the spread). In this formula, should the half-bid-ask be back-adjusted? Should the contract size be back-adjusted?

    ReplyDelete
    Replies
    1. Hi. This is why I just use risk normalised costs, it's much simpler. If you use percentage costs, then it probably makes sense to use the current bid/ask, divided by the current futures price, and assume that percentage is constant in the past. Because adjusted prices can go negative, you'll end up with weird results in the past.

      Delete
  33. Hi Rob,
    I'm reviewing the back-adjustments for Gold, and want to make sure I understand how the process works.
    Your rollconfig.csv has the following:
    RollOffsetDays -30
    ExpiryOffset 26
    So this implies that the Roll date = -3 from the 1st of the month.
    So for December 2020, this is 28 Nov, a Saturday - so, I assume the next best date is 27 Nov.
    I've downloaded data from barchart.com and the volumes leading to this date are as follows:
    20201200 20201200 20210200 20210200
    Date Last Volume Last Volume
    30/11/2020 1775.7 3145 1780.9 242726
    27/11/2020 1781.9 71804 1788.1 282346
    25/11/2020 1805.5 205754 1811.2 133601
    24/11/2020 1804.6 367879 1810.9 126761
    23/11/2020 1837.8 319657 1844.1 108427
    20/11/2020 1872.4 186166 1878.2 46608

    Your roll calendar for Gold has 23 Nov. Does this mean you manually edited that date, or was a particular rule applied? So I can better understand the process, what would be the reasons for not choosing 27 Nov?
    How much difference does it make?

    Secondly, why is CarryOffset 1 and not -1 in the config for Gold? Your roll calendar file means that we're not in the nearest contract. I don't know if your Roll Calendar has been generated using the config, but the carry contract is the same as the next contract, when it seems it could be the nearest contract. Or have I not understood how Carry works correctly?

    ReplyDelete
    Replies
    1. For newer roll dates (and it depends on the instrument) they were generated by me manually choosing to roll on that date on my live system. This isn't an automated process that strictly follows the date offsets; it might just be that I fancied rolling earlier.

      It won't make that much difference, but actually this is something I'm going to explore in my new book (just started writing).

      Carry could be the nearer contract, but for Metals the futures curve has a fairly steady slope so it won't matter. Having carry as the further out contract means I have the option to delay rolling for some reason.

      Delete
    2. Thanks for the quick response,
      David

      Delete
  34. Hi Rob,
    A query about rolldown, stitching, trading rules and forecasts:

    If long, then you want to be in the most backwardated contract vs nearer month, and if short, then most contagoed.
    Does that mean:
    #1. you run all of your trading rules for the instrument on a stitched timeseries ending in the most backwardated month, and produce a combined forecast: the "long forecast"

    #2. you also run all of your trading rules on a stitched timeseries ending in the most contagoed month, and produce another combined forecast: the "short forecast"

    #3. If "long forecast" is positive and > abs("short forecast"), you go long the backwardated month? If short forecast is negative and abs(short forecast) > abs(long forecast), you go short the contagoed month? ...but then what if short forecast is positive and > long forecast ...?? Or long forecast is weak negative and short forecast is stronger and positive, etc?

    I imagine I'm over-complicating things as you regularly say you like to keep things simple. But I assume you want to be running the Trading Rules on a time series that includes the contract you actually want to trade, and you'll get different forecasts out of #1 and #2, above.

    Also, as forecasts switch from +ve to -ve does that you mean you switch contract months?

    ReplyDelete
    Replies
    1. Neithier #1 or #2; I don't select contracts in this way in my backtest since it would be lethally complicated.

      "Also, as forecasts switch from +ve to -ve does that you mean you switch contract months?"

      No, not that eithier

      Basically I just use price and current position (forecast) as an input into my live trading contract decision when to roll. Once I've rolled I wouldn't go back to a prior contract, that would be a nightmare.

      If I switch from a contract that is backwardated to one that is contango, then my raw carry signal will change on that day. But because I smooth that, it will be a few days before this starts to affect my forecast.

      I hope that makes sense.

      Delete
    2. "If I switch from a contract that is backwardated to one that is contango, then my raw carry signal will change on that day":
      because as you roll, you update the carry contract in the system?

      And can the smoothing lag cause any issues if it is still being influenced by the previous carry data from the previous contract? Could the carry forecasts be "wrong" until it catches up?

      Delete
    3. Yes, and Yes.

      But carry is very noisy; being 'wrong' for a few days is a small price to pay for a cleaner signal (in fact I smooth for much longer periods than a few days in many cases)

      It's also not so likely that the carry will switch sharply between contracts, with the possible exception of natgas and any other seasonal commodity where I can't hold a constant month.

      Delete
    4. Thanks for clarifying. My perspective is currently from spread trading, where the only contracts available are nearer months, so for instance, I can't roll WTI from Dec 21 to Dec 22, but only to Jan/Feb/Mar. So (with your help!), I've gradually been improving my intuition around this

      Delete
  35. Hi Rob,

    I've been running the trading system in Excel for a couple of months on 5 instruments, and the Carry rule on the Bund was generating significant losses: Carry weighting was 50%, and it resulted in about 15% realised loss on the Risk Capital allocated for the Bund, on an Account level Risk Target of 23%
    So I've switched off Carry for the Bund until I understand it better.

    For the Bund, you are using the next contract for Carry, according to the Roll Calendar csv. I've just rolled Dec 21 to Mar 22, with Dec expiring tomorrow. The June contract has no volume - so can it's price be trusted for Carry? Are there any alternate instruments that could be used for the Bund's Carry?

    ReplyDelete
    Replies
    1. There is no volume, but the price is marked to model; so it should match the theoretical carry on the CTD bond. This is probably okay with financials which is good because for most of them we can only trade the 1st contract. It's more speculative on commodities, but we can normally trade further out and use a nearer contract for carry calculations.

      Alternatives are compare to spot price (lot of work, need to know what CTD is and conversion factors) or use the yield curve (lot of work, extra data).

      Delete
  36. Hi Rob, lots of valuable information here.

    Basic question:
    Assuming I have a system generating daily signals on the settlement or closing price, when and how would ou send the orders considering that you don't want to trade too long after the ref price used for signal generation, and the liquidity drop when the next day session restarts.

    Thank you

    ReplyDelete
    Replies
    1. "considering that you don't want to trade too long after the ref price used for signal generation"

      Actually I don't care that much. In my backtests I assume I delay all my trades for a day after the relevant closing price is captured. It makes bugger all difference to the p&l. I have to delay for a week or more before there is even a tiny difference. This is because I am trading slow signals. So to answer your question, I trade the next day once the markets are liquid.

      Delete

Comments are moderated. So there will be a delay before they are published. Don't bother with spam, it wastes your time and mine.