Why optimising for the left tail of outcomes is like playing chess...

A short one today.

Suppose I offer you N trading strategies to choose between. We can run some kind of bootstrap of their historical returns to get a final distribution of CAGR, with all the caveats implied by that. Or suppose I ask you to pick the correct level of leverage on a trading strategy. Again we can bootstrap historical returns at different leverage levels, giving us a series of N discrete possible leverage levels. How do we do that? Both are equivalent to choosing which of N distributions of possible outcomes we prefer.

The strict expectation method is to pick the option with the distribution that has the best median value, eg 50% distribtional point. As I've discussed before, that is equivalent to the Kelly optimal.

The gambling addict method would be to pick the one with a very good right tail. We might go for the option that has the best 1% right tail. That would be a high leverage option, something with high natural risk like crypto, or a short gamma strategy that doesn't manage to blow up in the best possible alternative histories that generated the bootstrap. Nobody should be a gambler, except sometimes.

The conservative method, which I favour, is the opposite. We might go for the option that has the best 1%, 10% or 25% tail; eg the option that will have the best outcome if bad things happen. 

We can imagine an extreme version of the conservative method where we pick the option with the best worst outcome, i.e. the best minimum value. In fact if we're bootstrapping enough times the lower bound of all distributions will be a final portfolio value very close to zero (with compounded returns and absolute returns>-100% we can't mathemtically below zero) and hard to distinguish; so in practice the most realistic extreme version would be something like picking the option with the most optimal 1% left tail point. Given that is likely to be negative, that would be the option with the least worst 1% tail.

(Frankly more extreme outcomes are highly uncertain, and will probably go beyond what a historical bootstrapping exercise could predict)

The least worst option? That is also the MinMax of game theory, Nash equilibria and all chess playing software.

Credit: Chatgpt, authors own prompt

If you view the market as your adversary, you want to have the least bad outcome in the worst possible state.

Now I am not suggesting you optimise for the 1% point, but they do say that there are no old, bold traders. Good traders know how to quantify and when to take risks, but they also understand the importance of keeping their capital mostly intact in the future. So somewhere around the 10% or 25% point seems appropriate to me. 

Optimisation with costs (oh please seriously Rob, not again...) and a sort of summary

After a summer break, this is post (checks notes) #11 in my 2026 series on portfolio optimisation. Up until now I've been focusing on the optimisation of forecast weights -the weights of signals to predict a given instrument - on a pre cost basis. So now it seems that we should introduce the question of costs.

But if you are a faithful follower of this blog, you may be groaning! I have covered this subject at length before in three substantial posts:

In this ten year old (!) post which is literally called 'optimising with costs'

In this 2023 Geri Halliwell (yes really) themed post on whether performance is identical across instruments on a pre-cost or post cost basis 

In this follow up 2023 movie franchise themed post about the effects of various layers of my strategy on costs

... and this isn't to ignore the numerous posts I've done which talk about costs in some way.

So this post is going to be a bit of a mish mash.  It will be a bit of an abstract 'let us think about this from all possible angles'. It will be a bit of a literature review of previous posts. And there will be at least one new idea for dealing with costs in portfolio optimisation ("no rob haven't we suffered enough"). 

My trading strategy and speed and costs

My strategy consists of lot's of little simple things, but layered together the interactions can get a little complex.

Remember we begin by combining forecasts trading at different speeds for a given instrument. 

Then we allocate an instrument weight. This gives me the ideal portfolio I would trade without worrying about pesky integer contracts. Next, I run a dynamic optimisation ('DO') which does worry about integer contracts. But it also worries about costs. It removes instruments which I've decided are too expensive to trade (as well as those which aren't sufficiently liquid, require too much leverage, or which I can't legally trade). It applies a buffer to the overall trading, which slows down the system's turnover. And it applies a trading cost penalty. The effect of all these cumulatively was discussed in one of the prior posts.

Let us know think about how this affects our decision making. For concreteness, let's assume we have three instruments in the worlds weirdest portfolio: S&P 500 (with 0.00025 SR points of costs per trade, amongst the cheapest I have), Cheese (0.008 SR points, just inside the 0.01 threshold where I'd stop trading it) and Sugar #16 (0.09 SR points, well above my limit). So bear in mind that our trades in Cheese will be severely penalised by the trading cost penalty in the DO. And we won't actually trade Sugar at all, since it will be excluded at the DO stage. 

How should we allocate forecast weights to say a fast, medium and slow trend following rule variation within each of these three instruments? Here are some of the things we need to consider:

• Should I have some heuristic pre-filtering where I won't use rules where turnover*SR cost>speed limit? How would that work with Sugar where none of the rules would meet that speed limit? [in my current system, I pre-allocate a set of the slowest rules]
• What is the likely pre-cost SR of these rules, and does it make sense to do some kind of pooling to estimate them? This was discussed in the previous post in this 2026 series. 
• How does the internal correlation of these rules affect things? For example, we know that fast/slow are more diversifying than medium which will be 0.8 correlation with both.
• How should we apply trading costs in the optimisation? Should we apply them so we end up putting less into faster variations for Cheese than for S&P 500? [The exact methodology of how costs are applied in optimisation is a seperate subject I will discuss below]
• How would that work with Sugar, when all of our trading rules would probably have negative SR after costs? 
• Should we get more diversification by, for example, putting more in fast SP500 trend, even if that doesn't actually make money; as otherwise we wouldn't be able to trade fast trend at all?

Next, how should we allocate instrument weights? I haven't yet addressed this in my posting series, but because of the layers of interaction it does need to be discussed.

• What is the likely pre-cost SR of instrument performance, and is it effectively flat so it can be set to be equal across instruments?
• Should we apply trading costs to instrument performance, so we end up allocating less to Cheese and more to S&P?
• How does that work with Sugar, where we would probably end up negative SR after costs even if the forecast weights were just 100% in the slowest system?
• Given I am going to apply cost penalties to Milk anyway, does it matter if we apply trading costs?
• Given we won't actually trade Sugar because of it's trading costs, it's instrument weight will just determine how it influences the positions we do actually trade. So what implications does that have for applying costs?

Applying trading costs in optimisations

Much of this was discussed here, but for the sake of completeness here are the options (including a new one):

1. Heuristic: apply a maximum cost threshold. This can be applied before, and in addition to, any of the methods below.
2. Ignore costs, use gross returns. These could be pooled. 
3. Use net returns. Note if you are pooling across instruments, that implies you should probably pool gross returns then subtract instrument specific costs (I did consider the alternative of pooling costs, or pooling net returns, but this option makes more sense).
4. Just use costs - set gross returns to some arbitrary equal number.
5. Subtract costs from gross returns, and multiply costs by a factor >1. This reflects the fact costs are more certain than gross returns.  It will make the results a little more like if you just used costs (option 4). This has the problem of finding the optimal value of 'factor'.
6. Calculate weights using gross returns, and adjust subsequently for cost levels using some inverse-optimisation-Black-Litterman-Shtick. This is handy if you are pooling weights across instruments, rather than pooling returns. You can get the pooled weights with gross returns and then subtract costs.
7. NEW (applies only if you are using shrinkage, my preferred option): Get your gross returns, which could be pooled. Shrink the SR estimates. Then subtract costs from the shrunk SR. This reflects the fact costs are more certain than gross returns but without the need to work out a factor as in option 5. If we are then using the averaged of unpooled, pooled and asset class pooled; we can do this for each and then take our average.

Some of these seem silly (2, 4). Some of them seem over complicated (5, 6). 

Then we have the thorny issue of how that would work with a super expensive instrument like Sugar, in which case it's likely that we'd fail both pre-optimisation filtering (1), and also end up with negative SR after applying costs (3 and 7). 

Right now I handle this with forecast weights by using a small subset of really slow trading rules for an instrument which fails the speed limit heuristic test. And I don't use SR at all, which means I ignore costs apart from the speed limit test. And I handle it with instrument weights by not using SR at all, which means once again that I'm basically ignoring costs.And things with SR cost over 0.01 will not actually be traded in DO, irrespective of their instrument weights.

Blimey Rob, it's complicated isn't it

So there is sort of a double standard here. And a bit of a mess. The cliff edge of 0.01 SR instrument costs where we start doing something completely different is a bit distasteful. Probably, as instruments get more expensive, there should be an impact on forecast weights but not neccessarily in a complete 1:1 way. Although this does go against the statement that costs are more predictable than returns. And perhaps the same is true of instrument weights; rather than keeping them but not trading them, perhaps we should be dialling them down somewhat. 

Here is the core dilemna, with DO we are (kind of) not trying to predict the returns of any individual instrument, but find the most cost effective way to implement a portfolio which represents our opinion on pre-cost price movements. But my previous research found that DO isn't completely effective at reducing the trading speed of expensive instruments.

Having given this much thought, I think my new favourite thing to do is this for forecast weights:

• Set a ceiling on SR costs used for the optimisation at 0.01 SR
• For pooled optimisations, pool gross returns across all instruments or within asset class, but use the costs for the individual instrument concerned to calculate net returns.
• Shrink correlations and gross SR estimates
• Then subtract the per instrument cost from the shrunk SR (#7)
• Do the optimisation on the shrunk SR net of costs and shrunk correlations
• Instruments which have current SR costs of greater than 0.01 would be ignored in the dynamic optimisation

Note this makes the cliff edge less problematic. I change my instrument overrides based on current costs much more frequently than my forecast weights. Consider something (instrument A) which has SR cost of 0.02 when I do the forecast weight optimisation, and instrument B with SR 0.005 at that time. We will end up trading the former as if it has a SR cost of 0.01 - so a little too quickly, but it will initially be removed from the forecast weight optimisation. Then suppose A and B swap, so B has SR 0.02 costs and A has 0.005. At this point we're going to be trading A a little too slowly, but fine; and B too quickly. Once I update my overrides though B will be excluded from the dynamic optimisation. I'm still 'trading' it too quickly, but I'm not actually trading it so I can relax.

Although we aren't worrying too much about instrument weights at this stage, we can use a similar approach for those except there is no pooling:

• Set a ceiling on SR costs used for the optimisation at 0.01 SR
• Get gross returns for each instrument
• Shrink correlations and gross SR estimates
• Then subtract the per instrument cost from the shrunk SR (#7)
• Do the optimisation on the shrunk SR net of costs and shrunk correlations
• Instruments which have current SR costs of greater than 0.01 would be ignored in the dynamic optimisation

A summary: 

Although not strictly within the remit of this post I think it is useful to summarise what my 'gold standard' forecast weight optimisation looks like after all the thinking and analysis I've done this year. It's effectively a nested set of functions. Here is some code that illustrates it:

op_shr_deduct_costs = ForecastWeightsOptimiserShrinkageDeductCosts()
opcluster_shr_deduct_costs = ForecastWeightsOptimiserClustering()
opcluster_shr_deduct_costs.set_optimiser(op_shr_deduct_costs)
op_rw_shr_cluster_deduct_costs = ForecastWeightsOptimiserReverseWeights() ## Clustering must go INSIDE reverse weights
op_rw_shr_cluster_deduct_costs.set_optimiser(opcluster_shr_deduct_costs)  
op =ForecastWeightsOptimiserAcrossInstrumentsAveragePoolingGrossReturnsDeductCosts()
op.set_optimiser(op_rw_shr_cluster_deduct_costs) 
wts = rolling_forecast_weights_optimiser_across_instruments(optimiser=op,
                                                      dict_of_instrument_spec=dict_of_instrument_spec,
                                                      forecast_pandl_across_instruments_and_forecasts=forecast_pandl_across_instruments_and_forecasts)

Going from the outside, inwards:

• The outermost wrapper (rolling_forecast_weights_optimiser_across_instruments) does rolling pure out of sample optimisation, rerunning each year on a 30 year expanding then rolling window (in this post I recommended a 40 year EMA but that's equivalent to a 28 year SMA and this is nicer to code). We'll estimate both SR and correlations using this window.
• Inside that is the pooling code ForecastWeightsOptimiserAcrossInstrumentsAveragePoolingGrossReturnsDeductCosts which does three optimisations for each instrument with different sets of gross returns, but using the costs of the actual instrument.
• The gross returns of the instrument
• The pooled gross returns of the asset class it belongs in
• The pooled gross returns across all instruments
• We then take an average of those three weights. This approach is justified here.
• The optimisation inside that is a reverse weights wrapper (ForecastWeightsOptimiserReverseWeights). The reverse weights wrapper finds assets with negative gross p&l and reverses the sign on them. Once the weights are found, it puts a minus sign in front of the weights. This means we have no prior on whether a forecast has (pre-cost) positive p&l, thus reducing tacit fitting and opening up the possibility of finding strategies we didn't realise would work. 
• Inside that is ForecastWeightsOptimiserClustering. That's a clustering, with a target cluster size of six, as discussed here.
• Within each cluster we use ForecastWeightsOptimiserShrinkageDeductCosts. As noted above that means we shrink the correlation and gross returns SR (use shrinkage for optimisation with SR shrinkage 0.5, correlation 0.75 unless you have less than five years of data in which case use higher shrinkage) and then deduct costs from the gross returns SR.

The instrument weights code will be exactly the same, but without the pooling step.

Next blogpost will be to do some playing around with this model on a relatively small tractable portfolio.

Writing a non fiction book

 I get asked for advice from prospective authors a lot. The purpose of this post is to consolidate that advice into one handy to read place. If you came here for mean variance optimisation in the presence of uncertainty - skip this post.

Am I qualified to answer this?

(This is more from people finding this post randomly). I'd say yes. I've written five books, with total sales of over 40,000 copies (more statistics later) in the eleven years since my first book came out.

A list of my books is here: https://www.systematicmoney.org/. 

But bear in mind my experience is with a single publisher, writing in a specific genre. So I'm not an expert. This is very much my personal take. 

Why write a book?

Broadly speaking, I'd say people write non fiction books:

1. to make money
2. to indirectly make money by getting people to buy their course, hire them as consultants, to help get a job (probably only academics get jobs from writing books and not even then!)
3. to boost their fragile egos
4. out of a burning desire to teach as many people as possible

Note that 2 and 3 are all about writing a book to boost or burnish your profile in some way; whilst 1 and 2 are about making money. You are unlikely to make a lot of money directly from book royalties - it's a very long tailed, positive skewed call option. 

Of these I'd say 3 and 4 were definitely my main motivations up front; I have had a bit of 1 and 2. 

It's really important to understand why you are writing this book, as it will determine what sort of book you write, and what route you go down. For example:

1. to have an outside option of making a ton of money write a more popular book through a 'big' publisher or self publish; for a lower standard deviation option write a less accessible, more technical book.
2. for course selling you probably want an accessible book that you self publish and give away, for more esoteric technical purposes like consulting and academia you want a fancy sounding book through a reputable publisher.
3. and 4. If you want lots of people to worship you write an accessible book, but you will probably want to do it through a decent publisher to get them to listen to you.

What area and level are you pitching at?

Often trite advice that is given is to imagine exactly who you are writing the book for. It helps if it's a real person! That person is your market. As already noted there is an obvious relationship between market size and accessibility. There is also a relationship with price. Books likely to sell in smaller numbers will have their prices set higher by publishers to cover the fixed costs. 

(Of course you can ignore this advice if you self publish, but you'd be stupid to do so...)

I'd describe my books as 'mid market' in terms of technicality, accessibility and price; ranging from LT (£30) up to AFTS (£60). Something like 'the blue book' would be an example of a highly technical with a very limited market. It's over £100 and has probably sold only a few copies outside of universities. Something like this would be 'mass market' (disclosure: I am in a chapter of that book). It's £30 and has probably sold a zillion copies. You can pick up paperback 'trading' books for £15 to £20.

Publish or self publish

I've published through a publisher for my entire career, whilst my mate Andreas has self published his last few books (disclosure: I wrote a chapter of one of those books).

The main benefit of self publishing is financial; you keep the net sales proceeds rather than just getting a royalty (more of that later). However you will have to pay all the up front costs, plus the production costs on each book. The maths of that means that self publishing won't break even until you've sold a fair amount of books. It's also more work. Although you can outsource most of that work it's a hassle up front to find decent people to do it (editors, designers ...), and they are not cheap. It costs money to do quality work.

There is nothing worse than a self published book that has been done on the cheap; the lack of quality really shines through. People might put up with that for a cheap fiction novel in kindle version, but I think they are less likely to accept it for a non fiction book; particularly in finance. I think it would be okay if you're creating something to give away to sell your dodgy forex course, otherwise quality is important.

I'd say for a first book definitely go through a publisher (which Andreas did, FWIW). Then you can make your own mind up afterwards.

Which publisher?

I've been with Harriman house since the beginning. They are now part of McMillan. They have been great for me, but there are obviously other very good publishers; in particular for more technical books Wiley are the G.O.A.T. whilst O'Reilly are good for tech focused stuff. The academic imprints are also good for textbook adjacent work.

If you're unsure about a publisher, have a look at their other books. There is a specific publisher I am constantly being spammed by on LinkedIn. Their books are of very poor quality. People are probably going to assume yours is as well.

Also, watch out for scams. There is a real trend for authors and potential authors to be scammed. I get these messages about once a week. There is an account of one attempt here.

Money, sales ... what can I expect?

Let me share with you some figures for my 1st book, Systematic Trading (ST) which came out in October 2015. It's sold a total of 20k copies which averages out to 1800 per year. In fact the sales are lumpy, from 1k to 3k copies in a given year. Generally books have a big initial sale period then slow down. ST's initial sales weren't spectacular as I wasn't very well known to begin with. ST did then have a slowing down period, but has since accelerated in the last few years. 

ST is an outlier however; my 2nd book Smart Portfolios didn't sell as well (my only book on investing rather than trading... hmm) and my 3rd book initially sold at a similar rate to ST but has since slowed down. My 4th book had really strong year one sale, much better than ST, but is now selling at a similar rate. Whether that's because I'm more well known now, or because it was a book with a naturally bigger market, I'm not sure.

According to the internet anything selling more than 5k copies in it's lifetime is 'very good', whilst over 10k lifetime copies would be a 'strong success'. Even if all my books were at end of life stage, which hopefully are not, all are in the 'very good' category whilst ST is a 'strong success'. I am a relatively succesful author in terms of my point on the distribution. Most books don't sell that well. There are many, many books with near zero sales.

To repeat you'd have a chance of higher sales with a more accessible book; but also a higher chance of very low sales. These sort of books are very much shit or bust. Sometimes there is a breakthrough and an esoteric book will sell a lot of copies to people who pretend to read it, but that's rare.

My sales are split around 35:65 between kindle and hardback; but I get higher royalties per book from Kindle sales (zero cost of production). Mass market books would have more ebook sales; more technical books would have more hardback.

In an ideal world all copies of my book were sold at full price by the publisher. However books get discounted and book sellers take a cut. As a result the net revenue is lower than the cover price. Books tend to get discounted more as they get older, and more expensive books also get discounted more as a percentage.

Book royalties are eithier paid as a % of the full price, or as a higher % of the net revenue. I can't disclose my royalty rate, but you can ask your favourite LLM for some figures. 

Anyway, in real terms I've made £223,000 from book royalties in the last eleven years. Again, that's lumpy - it will bump when a new book comes out - and it's also been higher in recent years as I've been selling four books at once rather than just one. Because most books slow down in sales (though not, so far, ST...) you really need to keep writing one every few years if you want to maintain a constant income. 

I would say each book takes me about a year of working on average 70% of the time. Call it 2000 hours. That means I've earned about £53 per hour from writing my first book and about £16 per hour from my 2nd and 3rd (my fourth lies in the middle; whilst my fifth isn't out yet). Obviously over time those figures can only go up assuming the books continue to sell. My books do require a fair bit of work and you could write a simpler book in less than 2000 hours; but to repeat they are relatively good sellers. 

I'd imagine the median author does not earn minimum wage. The lower figures there are slightly above that, but not by much! Of course even the higher figure is only a fraction of what I would be paid if I had continued working in a hedge fund...

Bear in mind I also do other things, mainly for free, which promote my books. That bumps the hourly rate down (and I have a few expenses like web hosting fees...). On the other hand, I have perhaps derived some other financial benefits from writing which don't show up in those figures. My hourly rate from consulting is many multiples of the figures above, and arguably I wouldn't have got most of that consulting work without writing the books.

Summary

Write a book! But don't do it for the money. If you have any more questions please comment below and I'll put the answers in the body of the post.