After a summer break, this is post (checks notes) #11 in my 2026 series on portfolio optimisation. Up until now I've been focusing on the optimisation of forecast weights -the weights of signals to predict a given instrument - on a pre cost basis. So now it seems that we should introduce the question of costs.
But if you are a faithful follower of this blog, you may be groaning! I have covered this subject at length before in three substantial posts:
In this ten year old (!) post which is literally called 'optimising with costs'
... and this isn't to ignore the numerous posts I've done which talk about costs in some way.
So this post is going to be a bit of a mish mash. It will be a bit of an abstract 'let us think about this from all possible angles'. It will be a bit of a literature review of previous posts. And there will be at least one new idea for dealing with costs in portfolio optimisation ("no rob haven't we suffered enough").
My trading strategy and speed and costs
My strategy consists of lot's of little simple things, but layered together the interactions can get a little complex.
Remember we begin by combining forecasts trading at different speeds for a given instrument.
Then we allocate an instrument weight. This gives me the ideal portfolio I would trade without worrying about pesky integer contracts. Next, I run a dynamic optimisation ('DO') which does worry about integer contracts. But it also worries about costs. It removes instruments which I've decided are too expensive to trade (as well as those which aren't sufficiently liquid, require too much leverage, or which I can't legally trade). It applies a buffer to the overall trading, which slows down the system's turnover. And it applies a trading cost penalty. The effect of all these cumulatively was discussed in one of the prior posts.
Let us know think about how this affects our decision making. For concreteness, let's assume we have three instruments in the worlds weirdest portfolio: S&P 500 (with 0.00025 SR points of costs per trade, amongst the cheapest I have), Cheese (0.008 SR points, just inside the 0.01 threshold where I'd stop trading it) and Sugar #16 (0.09 SR points, well above my limit). So bear in mind that our trades in Cheese will be severely penalised by the trading cost penalty in the DO. And we won't actually trade Sugar at all, since it will be excluded at the DO stage.
How should we allocate forecast weights to say a fast, medium and slow trend following rule variation within each of these three instruments? Here are some of the things we need to consider:
- Should I have some heuristic pre-filtering where I won't use rules where turnover*SR cost>speed limit? How would that work with Sugar where none of the rules would meet that speed limit? [in my current system, I pre-allocate a set of the slowest rules]
- What is the likely pre-cost SR of these rules, and does it make sense to do some kind of pooling to estimate them? This was discussed in the previous post in this 2026 series.
- How does the internal correlation of these rules affect things? For example, we know that fast/slow are more diversifying than medium which will be 0.8 correlation with both.
- How should we apply trading costs in the optimisation? Should we apply them so we end up putting less into faster variations for Cheese than for S&P 500? [The exact methodology of how costs are applied in optimisation is a seperate subject I will discuss below]
- How would that work with Sugar, when all of our trading rules would probably have negative SR after costs?
- Should we get more diversification by, for example, putting more in fast SP500 trend, even if that doesn't actually make money; as otherwise we wouldn't be able to trade fast trend at all?
Next, how should we allocate instrument weights? I haven't yet addressed this in my posting series, but because of the layers of interaction it does need to be discussed.
- What is the likely pre-cost SR of instrument performance, and is it effectively flat so it can be set to be equal across instruments?
- Should we apply trading costs to instrument performance, so we end up allocating less to Cheese and more to S&P?
- How does that work with Sugar, where we would probably end up negative SR after costs even if the forecast weights were just 100% in the slowest system?
- Given I am going to apply cost penalties to Milk anyway, does it matter if we apply trading costs?
- Given we won't actually trade Sugar because of it's trading costs, it's instrument weight will just determine how it influences the positions we do actually trade. So what implications does that have for applying costs?
Applying trading costs in optimisations
Much of this was discussed here, but for the sake of completeness here are the options (including a new one):
- Heuristic: apply a maximum cost threshold. This can be applied before, and in addition to, any of the methods below.
- Ignore costs, use gross returns. These could be pooled.
- Use net returns. Note if you are pooling across instruments, that implies you should probably pool gross returns then subtract instrument specific costs (I did consider the alternative of pooling costs, or pooling net returns, but this option makes more sense).
- Just use costs - set gross returns to some arbitrary equal number.
- Subtract costs from gross returns, and multiply costs by a factor >1. This reflects the fact costs are more certain than gross returns. It will make the results a little more like if you just used costs (option 4). This has the problem of finding the optimal value of 'factor'.
- Calculate weights using gross returns, and adjust subsequently for cost levels using some inverse-optimisation-Black-Litterman-Shtick. This is handy if you are pooling weights across instruments, rather than pooling returns. You can get the pooled weights with gross returns and then subtract costs.
- NEW (applies only if you are using shrinkage, my preferred option): Get your gross returns, which could be pooled. Shrink the SR estimates. Then subtract costs from the shrunk SR. This reflects the fact costs are more certain than gross returns but without the need to work out a factor as in option 5. If we are then using the averaged of unpooled, pooled and asset class pooled; we can do this for each and then take our average.
Some of these seem silly (2, 4). Some of them seem over complicated (5, 6).
Then we have the thorny issue of how that would work with a super expensive instrument like Sugar, in which case it's likely that we'd fail both pre-optimisation filtering (1), and also end up with negative SR after applying costs (3 and 7).
Right now I handle this with forecast weights by using a small subset of really slow trading rules for an instrument which fails the speed limit heuristic test. And I don't use SR at all, which means I ignore costs apart from the speed limit test. And I handle it with instrument weights by not using SR at all, which means once again that I'm basically ignoring costs.And things with SR cost over 0.01 will not actually be traded in DO, irrespective of their instrument weights.
Blimey Rob, it's complicated isn't it
So there is sort of a double standard here. And a bit of a mess. The cliff edge of 0.01 SR instrument costs where we start doing something completely different is a bit distasteful. Probably, as instruments get more expensive, there should be an impact on forecast weights but not neccessarily in a complete 1:1 way. Although this does go against the statement that costs are more predictable than returns. And perhaps the same is true of instrument weights; rather than keeping them but not trading them, perhaps we should be dialling them down somewhat.
Here is the core dilemna, with DO we are (kind of) not trying to predict the returns of any individual instrument, but find the most cost effective way to implement a portfolio which represents our opinion on pre-cost price movements. But my previous research found that DO isn't completely effective at reducing the trading speed of expensive instruments.
Having given this much thought, I think my new favourite thing to do is this for forecast weights:
- Set a ceiling on SR costs used for the optimisation at 0.01 SR
- For pooled optimisations, pool gross returns across all instruments or within asset class, but use the costs for the individual instrument concerned to calculate net returns.
- Shrink correlations and gross SR estimates
- Then subtract the per instrument cost from the shrunk SR (#7)
- Do the optimisation on the shrunk SR net of costs and shrunk correlations
- Instruments which have current SR costs of greater than 0.01 would be ignored in the dynamic optimisation
Note this makes the cliff edge less problematic. I change my instrument overrides based on current costs much more frequently than my forecast weights. Consider something (instrument A) which has SR cost of 0.02 when I do the forecast weight optimisation, and instrument B with SR 0.005 at that time. We will end up trading the former as if it has a SR cost of 0.01 - so a little too quickly, but it will initially be removed from the forecast weight optimisation. Then suppose A and B swap, so B has SR 0.02 costs and A has 0.005. At this point we're going to be trading A a little too slowly, but fine; and B too quickly. Once I update my overrides though B will be excluded from the dynamic optimisation. I'm still 'trading' it too quickly, but I'm not actually trading it so I can relax.
Although we aren't worrying too much about instrument weights at this stage, we can use a similar approach for those except there is no pooling:
- Set a ceiling on SR costs used for the optimisation at 0.01 SR
- Get gross returns for each instrument
- Shrink correlations and gross SR estimates
- Then subtract the per instrument cost from the shrunk SR (#7)
- Do the optimisation on the shrunk SR net of costs and shrunk correlations
- Instruments which have current SR costs of greater than 0.01 would be ignored in the dynamic optimisation
A summary:
Although not strictly within the remit of this post I think it is useful to summarise what my 'gold standard' forecast weight optimisation looks like after all the thinking and analysis I've done this year. It's effectively a nested set of functions. Here is some code that illustrates it:
op_shr_deduct_costs = ForecastWeightsOptimiserShrinkageDeductCosts()
opcluster_shr_deduct_costs = ForecastWeightsOptimiserClustering()
opcluster_shr_deduct_costs.set_optimiser(op_shr_deduct_costs)
op_rw_shr_cluster_deduct_costs = ForecastWeightsOptimiserReverseWeights() ## Clustering must go INSIDE reverse weights
op_rw_shr_cluster_deduct_costs.set_optimiser(opcluster_shr_deduct_costs)
op =ForecastWeightsOptimiserAcrossInstrumentsAveragePoolingGrossReturnsDeductCosts()
op.set_optimiser(op_rw_shr_cluster_deduct_costs)
wts = rolling_forecast_weights_optimiser_across_instruments(optimiser=op,
dict_of_instrument_spec=dict_of_instrument_spec,
forecast_pandl_across_instruments_and_forecasts=forecast_pandl_across_instruments_and_forecasts)
Going from the outside, inwards:
- The outermost wrapper (rolling_forecast_weights_optimiser_across_instruments) does rolling pure out of sample optimisation, rerunning each year on a 30 year expanding then rolling window (in this post I recommended a 40 year EMA but that's equivalent to a 28 year SMA and this is nicer to code). We'll estimate both SR and correlations using this window.
- Inside that is the pooling code ForecastWeightsOptimiserAcrossInstrumentsAveragePoolingGrossReturnsDeductCosts which does three optimisations for each instrument with different sets of gross returns, but using the costs of the actual instrument.
- The gross returns of the instrument
- The pooled gross returns of the asset class it belongs in
- The pooled gross returns across all instruments
- We then take an average of those three weights. This approach is justified here.
- The optimisation inside that is a reverse weights wrapper (ForecastWeightsOptimiserReverseWeights). The reverse weights wrapper finds assets with negative gross p&l and reverses the sign on them. Once the weights are found, it puts a minus sign in front of the weights. This means we have no prior on whether a forecast has (pre-cost) positive p&l, thus reducing tacit fitting and opening up the possibility of finding strategies we didn't realise would work.
- Inside that is ForecastWeightsOptimiserClustering. That's a clustering, with a target cluster size of six, as discussed here.
- Within each cluster we use ForecastWeightsOptimiserShrinkageDeductCosts. As noted above that means we shrink the correlation and gross returns SR (use shrinkage for optimisation with SR shrinkage 0.5, correlation 0.75 unless you have less than five years of data in which case use higher shrinkage) and then deduct costs from the gross returns SR.
The instrument weights code will be exactly the same, but without the pooling step.
Next blogpost will be to do some playing around with this model on a relatively small tractable portfolio.
