My last blog post was about a new method for a daily dynamic optimisation of portfolios with limited capital, to allow them to trade large numbers of instruments.

(Although I normally write my blog posts to be self contained, you'll definitely have to read the previous one for this to make any sense!)

Subsequent to writing that post I implemented the method, and quickly ran into some problems. Basically the dammn thing was trading too much. Fortunately there is a single parameter which controls trading speed in the model - the *shadow cost*. I turned up the shadow cost to a very high value and spent a few days investigating what had went wrong.

Broadly speaking, I found that:

- My measure of turnover wasn't suitable for 'sparse' portfolios, i.e. portfolios where most instruments have a zero position at any given time
- My measure of risk adjusted costs also wasn't suitable
- The true turnover of the dynamic optimised portfolio was much higher than I'd realised
- The true trading costs were thus also much higher
- Thus the results in my previous post were misleading and wrong

After discussing with Doug (the original source of this idea) I realised I'd been missing a step: *postion buffering*. This is something I did in my original static system to slow down my trading behaviour, but which was missing here.

In this post I explain:

- Some other bad things I did
- Why my approximations for trading costs and turnover were wrong, and how I fixed them
- Why increasing the shadow cost doesn't help
- Why position buffering is a good thing and how it works in the old 'static' system
- How to create a position buffering method for dynamic optimisation
- How I calibrated the position buffering and shadow cost for the dynamic model
- How I used shrinkage to make the results more stable
- A more appropriate benchmark
- Some revised results using the new method, and more accurate statistics

## Some other bad things I did

- Not thinking more carefully about the interpretation of the shadow cost
- Not using common code between sim and production (this didn't serious cause problems, but did make it harder to work out what was going ong)
- Not paper trading for long enough

## My (bad) approximations for trading costs and turnover

- Work out the current cost of trading
- Divide that by current vol to calculate the cost in SR units
- Calculate the turnover in contracts
- Calculate the normalised turnover; dividing the historic turnover in contracts by the current average position, and the taking an average
**The average position assumes we have a constant forecast of +10**- Multiply the normalised turnover
- I do a similar calculation for rolling costs, assuming I roll an average position worth of contracts N times a year (where N is the number of rolls each year)

- Work out the current cost of trading in actual $$$ terms
- For each historic trade, multiply that by the number of contracts traded
- For rolling costs, assume we roll a number of contracts equal to the number of contracts we held in the period before the roll (gives a more reliable figure)
- Deflate these figures according to the difference in price volatility between then and now
- So for example, if the vol of S&P 500 is 20% with a price of 5000, that's a price vol of 1000
- If that figure was 200 at some point in the past, then we'd divide the historic cost by 5

## Why shadow cost doesn't help - much

**starts at zero**(in the absence of corner case constraints) and then starts changing position

**with the sign of the optimal unrounded position**.

## Position buffering in a static system

*Doug isn't doing exactly what I've decided to do, but the principal of applying a buffer was his idea so is due credit*

**the edge of the buffer**.

**average position**(the position I'd get with an average forecast of +10).

## Position buffering in a dynamic system

**tracking error /unrounded**). If this is less than the buffer, we stick with the current positions. No optimisation is needed. This step doesn't affect what happens next, except to speed things up by reducing the number of times we need to do a full blown optimisation.

**tracking error/rounded)**. By construction, this tracking error will be lower than tracking error/unrounded.

**) and the buffer (**

*x***b)**:

*max((x - b)/x, 0)*## Calibrating buffer size and shadow cost

**1.25%**(not 2.5% as you might expect as this buffer is expressed differently).

- Sharpe Ratio 1.0
- Target annual standard deviation 25%
- Target annual return = 1.0 * 25% = 25%
- Suppose we're running at half the risk we want, so our tracking error will be 12.5%
- In this case we'll also be missing out on ~12.5% of annual return
- If SR = 1, then tracking error is effectively in annualised return units

**shadow cost of 250**would annualise costs and put them in the same units as the tracking error.

## Shrinkage and the mystery of the non semi-definite matrix

## How should we assess this change?

- The correlation in returns between an unrounded and the rounded optimised portfolio
- The level of costs paid in SR units
- The total portfolio level turnover (see notes above)

## The static benchmark

instrument_list = ['DOW', 'MUMMY','FTSECHINAA', 'OAT', 'NIFTY', 'KOSDAQ','SP500_micro', 'NIKKEI',

'BOBL', 'KR10','KR3','EDOLLAR', 'US10U','US3',

'SOYOIL', 'WHEAT', 'LIVECOW', 'LEANHOG',

'CNH', 'YENEUR','RUR','BRE', 'JPY', 'MXP', 'NZD','CHF',

'BITCOIN', 'IRON','SILVER', 'GOLD_micro' ,

'CRUDE_W_mini', 'GASOILINE','GAS_US_mini',

'VIX'

]

- Equities
- Bonds/STIR
- Ags
- Currencies
- Metals
- Energies
- Vol

## Results versus benchmark

**0.986**.

**1.287**net. So that's a SR cost loss of ~5bp a year. In other words, we'd expect to lose 1/20 of our vol target (1/20 of 25% = 1.25%) in costs annually; the

**actual average cost is 1.7%**a year. The portfolio level turnover is

**51.1**times a year.

Orange: Unrounded, Blue: Optimised |

Another way of visualising the costs: If I plot the costs with the sign flipped, and multiplied by 10, and then add on the net account curve; you can see how we are losing less than half (so less than 5%) of our gross performance. Also the compounded costs are steady and linear, indicating a nice consistency over time.

- In recent years it has lower costs
- But probably higher turnover
- The historic net Sharpe Ratio is effectively identical

The turnover isn't a massive deal, as long as we have accurately estimated our trading costs: we'll do a few more trades, but on average they will be in cheaper markets.

However, to reiterate, the results of the static system are very much more a matter of luck in market selection particularly for the early backtest when it only has a few markets. Overall I'd still rather have the dynamic system - with the opportunity to catch market movements in over 100 markets and counting - than the static system where if I am missing a market that happens to move I will kick myself.