My last blog post was about a new method for a daily dynamic optimisation of portfolios with limited capital, to allow them to trade large numbers of instruments.

(Although I normally write my blog posts to be self contained, you'll definitely have to read the previous one for this to make any sense!)

Subsequent to writing that post I implemented the method, and quickly ran into some problems. Basically the dammn thing was trading too much. Fortunately there is a single parameter which controls trading speed in the model - the *shadow cost*. I turned up the shadow cost to a very high value and spent a few days investigating what had went wrong.

Broadly speaking, I found that:

- My measure of turnover wasn't suitable for 'sparse' portfolios, i.e. portfolios where most instruments have a zero position at any given time
- My measure of risk adjusted costs also wasn't suitable
- The true turnover of the dynamic optimised portfolio was much higher than I'd realised
- The true trading costs were thus also much higher
- Thus the results in my previous post were misleading and wrong

After discussing with Doug (the original source of this idea) I realised I'd been missing a step: *postion buffering*. This is something I did in my original static system to slow down my trading behaviour, but which was missing here.

In this post I explain:

- Some other bad things I did
- Why my approximations for trading costs and turnover were wrong, and how I fixed them
- Why increasing the shadow cost doesn't help
- Why position buffering is a good thing and how it works in the old 'static' system
- How to create a position buffering method for dynamic optimisation
- How I calibrated the position buffering and shadow cost for the dynamic model
- How I used shrinkage to make the results more stable
- A more appropriate benchmark
- Some revised results using the new method, and more accurate statistics

## Some other bad things I did

- Not thinking more carefully about the interpretation of the shadow cost
- Not using common code between sim and production (this didn't serious cause problems, but did make it harder to work out what was going ong)
- Not paper trading for long enough

## My (bad) approximations for trading costs and turnover

- Work out the current cost of trading
- Divide that by current vol to calculate the cost in SR units
- Calculate the turnover in contracts
- Calculate the normalised turnover; dividing the historic turnover in contracts by the current average position, and the taking an average
**The average position assumes we have a constant forecast of +10**- Multiply the normalised turnover
- I do a similar calculation for rolling costs, assuming I roll an average position worth of contracts N times a year (where N is the number of rolls each year)

- Work out the current cost of trading in actual $$$ terms
- For each historic trade, multiply that by the number of contracts traded
- For rolling costs, assume we roll a number of contracts equal to the number of contracts we held in the period before the roll (gives a more reliable figure)
- Deflate these figures according to the difference in price volatility between then and now
- So for example, if the vol of S&P 500 is 20% with a price of 5000, that's a price vol of 1000
- If that figure was 200 at some point in the past, then we'd divide the historic cost by 5

## Why shadow cost doesn't help - much

**starts at zero**(in the absence of corner case constraints) and then starts changing position

**with the sign of the optimal unrounded position**.

## Position buffering in a static system

*Doug isn't doing exactly what I've decided to do, but the principal of applying a buffer was his idea so is due credit*

**the edge of the buffer**.

**average position**(the position I'd get with an average forecast of +10).

## Position buffering in a dynamic system

**tracking error /unrounded**). If this is less than the buffer, we stick with the current positions. No optimisation is needed. This step doesn't affect what happens next, except to speed things up by reducing the number of times we need to do a full blown optimisation.

**tracking error/rounded)**. By construction, this tracking error will be lower than tracking error/unrounded.

**) and the buffer (**

*x***b)**:

*max((x - b)/x, 0)*## Calibrating buffer size and shadow cost

**1.25%**(not 2.5% as you might expect as this buffer is expressed differently).

- Sharpe Ratio 1.0
- Target annual standard deviation 25%
- Target annual return = 1.0 * 25% = 25%
- Suppose we're running at half the risk we want, so our tracking error will be 12.5%
- In this case we'll also be missing out on ~12.5% of annual return
- If SR = 1, then tracking error is effectively in annualised return units

**shadow cost of 250**would annualise costs and put them in the same units as the tracking error.

## Shrinkage and the mystery of the non semi-definite matrix

## How should we assess this change?

- The correlation in returns between an unrounded and the rounded optimised portfolio
- The level of costs paid in SR units
- The total portfolio level turnover (see notes above)

## The static benchmark

instrument_list = ['DOW', 'MUMMY','FTSECHINAA', 'OAT', 'NIFTY', 'KOSDAQ','SP500_micro', 'NIKKEI',

'BOBL', 'KR10','KR3','EDOLLAR', 'US10U','US3',

'SOYOIL', 'WHEAT', 'LIVECOW', 'LEANHOG',

'CNH', 'YENEUR','RUR','BRE', 'JPY', 'MXP', 'NZD','CHF',

'BITCOIN', 'IRON','SILVER', 'GOLD_micro' ,

'CRUDE_W_mini', 'GASOILINE','GAS_US_mini',

'VIX'

]

- Equities
- Bonds/STIR
- Ags
- Currencies
- Metals
- Energies
- Vol

## Results versus benchmark

**0.986**.

**1.287**net. So that's a SR cost loss of ~5bp a year. In other words, we'd expect to lose 1/20 of our vol target (1/20 of 25% = 1.25%) in costs annually; the

**actual average cost is 1.7%**a year. The portfolio level turnover is

**51.1**times a year.

Orange: Unrounded, Blue: Optimised |

Another way of visualising the costs: If I plot the costs with the sign flipped, and multiplied by 10, and then add on the net account curve; you can see how we are losing less than half (so less than 5%) of our gross performance. Also the compounded costs are steady and linear, indicating a nice consistency over time.

- In recent years it has lower costs
- But probably higher turnover
- The historic net Sharpe Ratio is effectively identical

The turnover isn't a massive deal, as long as we have accurately estimated our trading costs: we'll do a few more trades, but on average they will be in cheaper markets.

However, to reiterate, the results of the static system are very much more a matter of luck in market selection particularly for the early backtest when it only has a few markets. Overall I'd still rather have the dynamic system - with the opportunity to catch market movements in over 100 markets and counting - than the static system where if I am missing a market that happens to move I will kick myself.

I wonder why to adjust your trades to the edge of the buffer and not to the optimal position. If you stay on the edge all the time you increase the number of adjustements, am I wrong?

ReplyDeleteYup, you're wrong. Trading to the edge of the buffer is optimal because if you trade to the middle you are doing larger traders, and on average you trade more contracts even though you are doing slightly fewer trades. Here is a relatively non technical paper on the subject https://arxiv.org/pdf/1603.06558.pdf

Deleteanother question, what about instruments weights? considering that the dynamic portfolio will select only a subset of instruments, the total weight will not sum to 1.

DeleteIf I have 100 instrument position that produces std dev risk of 20%, and I end up with 1 dynamically optimised position that is 100x larger than the original portfolio, is the effective total instrument weight on that position 1% or 100%?

Deletethanks, from the text in this article and the book chapter it was not clear to me the rationale of staying on th edge, but it is clear now.

ReplyDeleteHi Rob, what do you think of adding instruments that are very similar and highly correlated but not necessarily identical or interchangeable into a dynamic optimisation? For example if I have Arabica Coffee contracts from ICE/US, is there a benefit to adding Robusta Coffee contracts from ICE/London? Also, do you think that there is a benefit to adding futures that have daily data from multiple exchanges, for example if I have Gold futures daily data from COMEX, would there be a benefit from adding Gold futures from JPX, HKFE into the dynamic optimisation? Thanks in advance Rob! Love reading all your write-ups on your blog!

ReplyDelete" For example if I have Arabica Coffee contracts from ICE/US, is there a benefit to adding Robusta Coffee contracts from ICE/London?" yes I do this. Unlike other optimisations, DO Is pretty robust to very high correlations since it isn't inverting the matrix.

DeleteAhh I see. But do you think it would be worthwhile for me to have say for example both the China A50 contracts from SGX and HKFE in the DO? or should I just stick to the one that has the lowest trading cost?

DeleteNo clearly if they are basically the same instrument there is no point. I would go for lowest cost that meets liquidity thresholds; as per my report https://github.com/robcarver17/reports/blob/master/Duplicate_markets_report

DeleteWhen backtesting rob_system for different notional account values, are the simulated stats system.accounts.optimised_portfolio().percent.stats() or simply system.accounts.portfolio().percent.stats() already accounts for the optimisedPositions?

ReplyDeleteoptimised_portfolio() is after optimisation, portfolio() is before.... I would have thought was obvious

ReplyDeleteNot obvious to a dolt!

DeleteI thought because it was accessing the dynamic system functions to run, perhaps there was already a position optimisation.

Hi Rob,

ReplyDeletejust re-read this post in connection with the DO section in the book. One problem of the algo stems from it always starting at zero. Though a little more complex, would it be useful to let the algo start at the current portfolio positions and loop through an incremental increase AND decrease of each instrument in the universe?

Other people have suggested this, one obvious disadvantage is you could end up stuck with the wrong sign position on an instrument for a while if expensive I would need to test

Delete