Tuesday, 3 November 2026

Writing a non fiction book

 I get asked for advice from prospective authors a lot. The purpose of this post is to consolidate that advice into one handy to read place. If you came here for mean variance optimisation in the presence of uncertainty - skip this post.


Am I qualified to answer this?

(This is more from people finding this post randomly). I'd say yes. I've written five books, with total sales of over 40,000 copies (more statistics later) in the eleven years since my first book came out.

A list of my books is here: https://www.systematicmoney.org/. 

But bear in mind my experience is with a single publisher, writing in a specific genre. So I'm not an expert. This is very much my personal take. 


Why write a book?

Broadly speaking, I'd say people write non fiction books:

  1. to make money
  2. to indirectly make money by getting people to buy their course, hire them as consultants, to help get a job (probably only academics get jobs from writing books and not even then!)
  3. to boost their fragile egos
  4. out of a burning desire to teach as many people as possible

Note that 2 and 3 are all about writing a book to boost or burnish your profile in some way; whilst 1 and 2 are about making money. You are unlikely to make a lot of money directly from book royalties - it's a very long tailed, positive skewed call option. 

Of these I'd say 3 and 4 were definitely my main motivations up front; I have had a bit of 1 and 2. 

It's really important to understand why you are writing this book, as it will determine what sort of book you write, and what route you go down. For example:

  1. to have an outside option of making a ton of money write a more popular book through a 'big' publisher or self publish; for a lower standard deviation option write a less accessible, more technical book.
  2. for course selling you probably want an accessible book that you self publish and give away, for more esoteric technical purposes like consulting and academia you want a fancy sounding book through a reputable publisher.
  3. and 4. If you want lots of people to worship you write an accessible book, but you will probably want to do it through a decent publisher to get them to listen to you.


What area and level are you pitching at?

Often trite advice that is given is to imagine exactly who you are writing the book for. It helps if it's a real person! That person is your market. As already noted there is an obvious relationship between market size and accessibility. There is also a relationship with price. Books likely to sell in smaller numbers will have their prices set higher by publishers to cover the fixed costs. 

(Of course you can ignore this advice if you self publish, but you'd be stupid to do so...)

I'd describe my books as 'mid market' in terms of technicality, accessibility and price; ranging from LT (£30) up to AFTS (£60). Something like 'the blue book' would be an example of a highly technical with a very limited market. It's over £100 and has probably sold only a few copies outside of universities. Something like this would be 'mass market' (disclosure: I am in a chapter of that book). It's £30 and has probably sold a zillion copies. You can pick up paperback 'trading' books for £15 to £20.


Publish or self publish

I've published through a publisher for my entire career, whilst my mate Andreas has self published his last few books (disclosure: I wrote a chapter of one of those books).

The main benefit of self publishing is financial; you keep the net sales proceeds rather than just getting a royalty (more of that later). However you will have to pay all the up front costs, plus the production costs on each book. The maths of that means that self publishing won't break even until you've sold a fair amount of books. It's also more work. Although you can outsource most of that work it's a hassle up front to find decent people to do it (editors, designers ...), and they are not cheap. It costs money to do quality work.

There is nothing worse than a self published book that has been done on the cheap; the lack of quality really shines through. People might put up with that for a cheap fiction novel in kindle version, but I think they are less likely to accept it for a non fiction book; particularly in finance. I think it would be okay if you're creating something to give away to sell your dodgy forex course, otherwise quality is important.

I'd say for a first book definitely go through a publisher (which Andreas did, FWIW). Then you can make your own mind up afterwards.


Which publisher?

I've been with Harriman house since the beginning. They are now part of McMillan. They have been great for me, but there are obviously other very good publishers; in particular for more technical books Wiley are the G.O.A.T. whilst O'Reilly are good for tech focused stuff. The academic imprints are also good for textbook adjacent work.

If you're unsure about a publisher, have a look at their other books. There is a specific publisher I am constantly being spammed by on LinkedIn. Their books are of very poor quality. People are probably going to assume yours is as well.

Also, watch out for scams. There is a real trend for authors and potential authors to be scammed. I get these messages about once a week. There is an account of one attempt here.


Money, sales ... what can I expect?

Let me share with you some figures for my 1st book, Systematic Trading (ST) which came out in October 2015. It's sold a total of 20k copies which averages out to 1800 per year. In fact the sales are lumpy, from 1k to 3k copies in a given year. Generally books have a big initial sale period then slow down. ST's initial sales weren't spectacular as I wasn't very well known to begin with. ST did then have a slowing down period, but has since accelerated in the last few years. 

ST is an outlier however; my 2nd book Smart Portfolios didn't sell as well (my only book on investing rather than trading... hmm) and my 3rd book initially sold at a similar rate to ST but has since slowed down. My 4th book had really strong year one sale, much better than ST, but is now selling at a similar rate. Whether that's because I'm more well known now, or because it was a book with a naturally bigger market, I'm not sure.

According to the internet anything selling more than 5k copies in it's lifetime is 'very good', whilst over 10k lifetime copies would be a 'strong success'. Even if all my books were at end of life stage, which hopefully are not, all are in the 'very good' category whilst ST is a 'strong success'. I am a relatively succesful author in terms of my point on the distribution. Most books don't sell that well. There are many, many books with near zero sales.

To repeat you'd have a chance of higher sales with a more accessible book; but also a higher chance of very low sales. These sort of books are very much shit or bust. Sometimes there is a breakthrough and an esoteric book will sell a lot of copies to people who pretend to read it, but that's rare.

My sales are split around 35:65 between kindle and hardback; but I get higher royalties per book from Kindle sales (zero cost of production). Mass market books would have more ebook sales; more technical books would have more hardback.

In an ideal world all copies of my book were sold at full price by the publisher. However books get discounted and book sellers take a cut. As a result the net revenue is lower than the cover price. Books tend to get discounted more as they get older, and more expensive books also get discounted more as a percentage.

Book royalties are eithier paid as a % of the full price, or as a higher % of the net revenue. I can't disclose my royalty rate, but you can ask your favourite LLM for some figures. 

Anyway, in real terms I've made £223,000 from book royalties in the last eleven years. Again, that's lumpy - it will bump when a new book comes out - and it's also been higher in recent years as I've been selling four books at once rather than just one. Because most books slow down in sales (though not, so far, ST...) you really need to keep writing one every few years if you want to maintain a constant income. 

I would say each book takes me about a year of working on average 70% of the time. Call it 2000 hours. That means I've earned about £53 per hour from writing my first book and about £16 per hour from my 2nd and 3rd (my fourth lies in the middle; whilst my fifth isn't out yet). Obviously over time those figures can only go up assuming the books continue to sell. My books do require a fair bit of work and you could write a simpler book in less than 2000 hours; but to repeat they are relatively good sellers. 

I'd imagine the median author does not earn minimum wage. The lower figures there are slightly above that, but not by much! Of course even the higher figure is only a fraction of what I would be paid if I had continued working in a hedge fund...

Bear in mind I also do other things, mainly for free, which promote my books. That bumps the hourly rate down (and I have a few expenses like web hosting fees...). On the other hand, I have perhaps derived some other financial benefits from writing which don't show up in those figures. My hourly rate from consulting is many multiples of the figures above, and arguably I wouldn't have got most of that consulting work without writing the books.


Summary

Write a book! But don't do it for the money. If you have any more questions please comment below and I'll put the answers in the body of the post.

Friday, 4 September 2026

Not another one! My fifth book...

Well yes, I am pleased to announce that as of this week I completed final proof reading of my new book "The Art And Science of Trading"(AAST). It joins the list of my existing books with their  own acronyms: Systematic Trading (ST), Smart Portfolios (SP), Leveraged Trading (LT) and Advanced Futures Trading Strategies (AFTS).

Due to the inherent delays involved in global dead tree supply chains the book won't actually be officially published until 1st December, although if you pre-order now you will probably get your copy a little earlier. 

I also approved the cover art, which looks like this:



Briefly, what is this book about?

This is effectively four books at once (I know, cool trick right?!). Firstly, it effectively forms a counterpart to my third book Leveraged Trading (LT). LT is an 'introductory' book to trading properly with leveraged instruments; things like futures or spreadbets. Essentially, AAST is the delta-one counterpart to LT. It's an introductory level to trading properly without leverage - so where you can't take on bigger positions than your account value, nor can you go short. It covers the classic delta one instruments used by retail traders: shares, ETFs and ..... crypto (I'll address that particular block chain sized elephant in a moment).

Note: Yes I did cover shares and ETFs in 'Smart Portfolios' (book#2) but from an investment rather than a trading perspective.

Secondly, it's the most introductory book I've ever written, and aimed at more of a mass market (is that because I'm trying to educate as many people as possible, or is it because I want to sell a gazillion copies rather than modest five figures.... well a little of both). So there is a lot more hand holding than there is even in Leveraged Trading; plus more careful explanations of why you can't trust 'gurus', and I've  also stripped back my usual introductory trend trading strategy to the simplest possible rule set it can be without being dangerous (essentially using Occams razor to the point where you are about to cut yourself with it, but not quite). For example, the trend rule - which you can have for free as loyal readers of this blog - is 'has the price gone up or down'?

Thirdly, and this is in the title, I think there is a bunch of 'science' that all traders need to know about, even retail traders - nay, especially retail traders. And actually I go into far more detail about explaining this intuitively than I have in any of my previous books. Familiar concepts like proper robust backtesting, judging outcomes by distributional uncertainty, and of course my old mate the Kelly criteria are all in here. These are all explained in an accessible way from first principles.

Finally, and this is also in the title, it's about the art and science of trading. I came up with the idea of semi-automatic trading in my 1st book, and I used it again in my 3rd. So if you are one of those gifted people who can intuitively predict price direction better than a systematic method, that's lovely, but you should still do that within a systematic framework for position sizing and risk management. Essentially by putting a set of science based guard rails around your trading methods, you will make the most of a good method, and protect yourself from the worst excesses of a bad one. This also means someone using a crap 'systematic' method like <insert obscure japanese sounding charting method name here> won't lose too much money, and if they're lucky enough might even make some.

The relationship between my 5 books is explained quite well on this web page https://www.systematicmoney.org/which-book-should-i-buy where you can find this neat diagram:



A brief aside: Crypto. Rob-seriously?

Let me quote from page 25 of AAST:

I should state my own prejudices up front: I am not a fan of cryptocurrencies, which are used primarily for gambling and criminal activities such as money laundering, and which consume a great deal of electrical power, with obvious implications for the climate crisis. In an ideal world I would ignore them completely. Although I am retained as a research advisor for a crypto hedge fund, and I trade Bitcoin and Ethereum futures, I do not hold any crypto in my personal portfolio.

But the point of this book is to guide traders into using certain scientific rules to reduce their chances of losing money, and nobody needs those rules more than those trading crypto. The message of this book as it pertains to crypto can be summarised as ‘I would not do this stupid thing if I were you, but if you insist, this is the least stupid way to do it.’

(is this because I'm trying to educate as many crypto people as possible, or is it because putting crypto on the cover will increase the sales by a factor of a gazillion.... well a little of both)


Do I need to buy it?

If you're reading this blog, there's a good chance you tick all of these boxes:

  • I consider myself something of a financial expert, actually
  • I only trade leveraged instruments, like futures
  • I'm a fully systematic trader, not interested in this 'intuitive trading' sounds like woke nonsense to me.
  • I've already read your other books and I have a firm grasp on the concepts, thanks very much
So if you tick all of these boxes then you probably don't need to buy this book. Of course you may still want to, because:
  • You have already derived $x000's or even $x,000,000's of benefit from my existing books and blog at minimal cost, and chucking me another $x0 seems only fair
  • My writing style is so awesome that people will happily read my books for fun
  • It's probably worth being reminded of certain basic concepts and ideas
  • You never know, there could even be some deep insight within the book that even you haven't thought of before. As one reviewer said "Despite being an experienced market practitioner, this book really gave me new ideas when it comes to trading". Thanks Saeed, the next burger is on me!
  • You have a friend or relative who doesn't tick all these boxes, but has an interest in trading. And... Christmas is coming!
However if you are someone who ticks one or more of these boxes:
  • Is a relative beginner at this trading nonsense;
  • Is unfamiliar with, or struggled to understand, books like Leveraged Trading, Systematic Trading and Advanced Futures Trading Strategies;
  • Trades non leveraged instruments like ETFs, stocks or crypto;
  • Trades in a non systematic way
... then I'd suggest that you should seriously consider a purchase of AAST. The worst outcome would be that you'd read it, decided trading isn't as easy as you thought, and then decide not to trade. You'd be out the price of a book rather than losing hundreds or thousands of <insert local currency unit here>. 

Of course if you do decide to trade, then owning a copy of AAST will seriously improve your chances of successful trading. Naturally, actually reading and doing what it suggests will help even more.

Has anyone read this nonsense, and liked it?

Five people! And only three of them are mates of mine! Here are some nice things they wrote without me having to give them money (or got AI to write, I didn't ask):


Saeed Amen, Author of Trading Thalesians; Co founder Turnleaf Analytics (ex Lehmans FX, ex Thalesians)

I’ve always enjoyed Rob’s trading books, because they’re always very practical. This new book is useful both for novice and experienced traders. I like the way he creates a framework in the book for managing risk that is applicable to both discretionary and systematic traders. Crucially he takes you through the whole lifecycle of a trade, from position sizing to execution and the broader question of how to manage different assets in a portfolio. Despite being an experienced market practitioner, the book really gave me new ideas when it comes to trading.

Andreas F. Clenow, Chief Investment Officer and author of multiple financial bestsellers (his words!)

Rob Carver is an outlier among financial authors. He has managed billions at tier one institutions and he is a world class quant professional. Seeing his name on the cover of a book should be enough to buy it. I have all his previous books in my office and reference them often in my own work. This time, Rob brings his institutional perspective to tackle equities, ETFs, and crypto in the same structured and well-researched manner as his previous work. In particular, I enjoyed his ruthless analysis of crypto returns, day trading, and the separation of internet hype from the brutal reality of real-world, professional trading. I am clearing space on my office shelf for another book which I will read and reference many times, and so should you.


Alex Spiroglou, CFTe, Dip(TA) ATAA Semi-systematic macro trader

A highly insightful and practical read for anyone looking to improve their trading. Rob Carver cuts through the noise and shares a thoughtful approach grounded in experience, discipline and real-world market understanding. The book offers plenty to reflect on, whatever your trading style or experience level.


Tony Guida, Head of Quant Research AtonRa Partners

In a few words: I really liked it. It reads well and fast, and you got the ordering right. In my world almost all the effort goes into the opening signal and almost all the P&L comes from what surrounds it. Very few authors are willing to say that out loud, because "here is how to size a position" sells fewer books than "here is how to pick winners". The costs chapter and the trade management rules are what I would hand to a junior on day one. A few other things I liked. Chapter 12, on adjusting positions and stops when risk changes, is the chapter almost nobody writes and everybody needs. The forecast-based stop losses in 13 are the right way round: the stop follows the conviction rather than a round number on a chart. And I appreciated that the diversification section treat it as an operational problem, instrument substitution, and portfolio construction, not as a slogan about not putting eggs in one basket.

I should declare my bias. I would have happily read another two hundred page of formulas and python. But I am clearly not the main target here, and that is the point. Putting it in spreadsheets is what makes the rules usable by the people who need them, and the rules do not get worse for being written in Excel.

My humble endorsement: “A discretionary trader and a systematic PM can read this book and end up agreeing with each other, which almost never happens unless the building is on fire or bonuses are at stake. Carver's argument is that you can keep your intuition for choosing what to trade, provided everything about how you trade it follows rules. Thirty years of experience compressed into procedures that fit in a spreadsheet, with no equations and no black boxes. He has written something a beginner can act on and a professional cannot argue with.”


Sam Beatson, PhD, EMBA, FRSA Lecturer in Finance, Risk and Banking Nottingham University Business School and CEO, MarketNous

Robert Carver has managed to combine a rigorous approach with accessibility, without pretending that trading in markets is anything other than demanding and challenging. The Art and Science of Trading is the perfect primer to the wider Carver canon, but also stands strongly in its own right as a practical and intellectually grounded guide to trading.

One of the book’s most useful contributions is the distinction between the art of finding trades and the science of managing them. Carver gives intuition, judgement and experience a legitimate place in trade-finding, while making a persuasive case that position sizing, costs, risk and trade management demand much stricter, rules-based discipline.

He shares his expertise on the less glamorous but essential aspects of trading: randomness, overfitting, transaction costs and the difficulty of distinguishing genuine skill from good fortune. Rather than looking for shortcuts, the reader is encouraged to question apparently impressive results, understand why a strategy should work, and think carefully about whether an edge is likely to survive beyond the historical data in which it was discovered.

In characteristic Carver style, you’ll find answers grounded in evidence, scepticism about any easy path to market profits, and an abundance of practical lessons for new and experienced traders alike.”

Seems appropriate to say thank you

Yes, I do have a list of thank yous:
  • The panel of Elite Traders who helped me pierce my Anglocentric bubble and get a better understanding of the international landscape for retail traders: Alexey Chursanov, ‘Elder’, ‘Kernfusion’, Matthijs Vákár, Professor Paul Calluzzo, Goran Peretin, Dr Sebastian Schmidt, and Vikram Mundkur.
  • A couple of key crypto people from Twitter/X: Crunchster and Scott Phillips. They didn't know they were contributing to a book but their posts were very helpful in my research process.
  • My sister Vicki and my 'sister' Debbie who forced me to write their names into the book despite me insisting it wasn't a novel. This isn't really a thank you, it's documentary evidence I will need for the police report.
  • The expert reading panel: Burak Yenigun (of Stylus Digital) and Riccardo Ronco (of recently retired). This is an unpaid position requiring many hours of reading the first draft of my prose. Which is awful. I can barely read it myself. And the only reward is a signed copy and passive aggressive responses from me ignoring nearly all their comments. Why do they do it? To get this brief mention in a blog post only 50 people will read? I guess.
  • Various random people from my publisher Harriman house: Myles, Victoria, both Charlottes, Sally, Lucy and Suzanne
  • Editors: Comissioning - Craig Pearce, Copy - Dominic Fenn, Proof - Amy Webber (who also as assistant editor has assidously guided this book on it's journey from concept to physical copies). This is a paid position requiring many hours of reading the 2nd, 3rd, 4th..... 89th drafts of my prose. And you can't get AI to do it. Because AI would be like "This is great! Just needs more em-dashes and it would be perfect".
  • The people who live in my house with me (some or all of the time) and put up with all my nonsense.

Is that it then? No more books? <slightly pleading voice>

Ha! If only. No, I'm currently in the early stages of thinking about book#6 which will provisionally be  a relatively advanced book about backtesting. And I'll probably stick AI on the cover to get some more sales.


Note: for those that don't understand the 'not another one' it's a UK political reference to 'Brenda from Bristol' 

https://www.reuters.com/article/world/us/not-another-one-brenda-from-bristol-speaks-for-britons-sick-of-elections-idUSKBN17L1FX/

To see Brenda live, watch the first 5 seconds of this video:




Friday, 3 July 2026

Jumping back in the pool(ing): testing pooling by asset class and portfolio weight distance

This is post #10 in my 2026 series on portfolio optimisation. Time for a quick recap. I'm not going to revisit every post but instead summarise what I now think one should be doing when optimising forecast weights before costs (I haven't yet incorporated costs, nor thought about instrument weights).

(I also confirmed in my very first post it was better to estimate forecast then instrument weights, rather than doing them jointly).

That doesn't seem like much value for the thousands of words I've written, and it's also not a million miles from what I would have down without all this research. A few things haven't worked out: random based methods (bayesian and monte carlo) which don't account for the reduced predictability of real returns compared to synthetic data; formal structural breaks on estimates; grouping and pooling instruments according to forecast SR; shorter EWM windows for SR estimates; and shrinking weights rather than inputs. 

I should probably move on now to looking at costs, and instrument returns, but like a dog with a particularly tasty bone or a cat pulling on an especially interesting piece of string; I can't quite let go of the idea that we should be able to improve on pooling everything.


Prior art

Let's run through the options we potentially have for pooling:

  1. We could cluster things together that have similar characteristics, such as by asset class. 
  2. We could do it on an estimate by estimate basis. We could compare the distribution of returns for say carry10 on US10 year bonds, and on US2 year bonds; and say "Well these distributions aren't significantly different. Let's pool the returns together". 
  3. Or we could look at the estimates of SR across different rules. You could have a vector of the SR for carry10, carry20 and so on. And you'd look at that vector of estimates, and calculate <some measure of> distance between them, and if the distance is low enough, then you'd pool the returns for all the rules for those two instruments.  I covered this in my previous post in this series. 
  4. Or we could do it on portfolio weights. We could for example fit the weights for 2 year bonds, and for 10 year bonds, and then see if they were significantly different. We could then pool the returns if they were not that different. I have also looked at this before.
  5. We don't pool at all, and fit each instrument individually. That sounds terrifying, but remember we're shrinking with our fitting.
  6. We pool everything. So far that seems to the best option, and the one I've used in the past. 

Note that we also have the option of:

  • A pooling the returns before estimating the statistics and then the weights
  • B not pooling the returns, and then pooling the weights
I'm not keen on B because it produces 'over robustness' when combined with a shrinkage methodology. Basically we throw away too much information and end up too close to equal weights.

So returning to the numbered list:

  1. By asset classes - is untried, although it resembles what we used to at AHL when we were organised into asset class teams, each of which fitted their own strategies.
  2. Grouping per estimate: I have objections to in terms of computational time and statistical unpleastness, discussed in the previous post.
  3. Grouping per vector of estimates: I tried this in the previous post. It wasn't effective, and also produced weird undesirable groups.
  4. Grouping by weights: I have tried before in a limited test with some success.

So that leaves us with 1 and 4 as candidates, along with the standard options of #6 full pooling and #5 no pooling at all - fitting each instrument's forecast weights purely on it's own data:

  • Unpooled
  • All instruments pooled
  • Asset class pooled
  • Grouping by portfolio weights


By asset classes - method

This is pretty trivial; the eight asset classes in my system are:

  • Stock indices (58 instruments in my dataset including duplicates and expired instruments like Eurodollar to avoid survivorship bias)
  • Sector stocks (eg 'EU oil companies') 36 instruments
  • Vol 4
  • FX 43
  • Bonds and STIR 39
  • Energies 20
  • Agricultural 39
  • Metals 21 (includes two crypto futures)

So for a given portfolio we fit those instruments in the same asset class together.


Grouping by weights

Well this is easy, as I already did this here and under the heading "get instrument groupings" it tells us we can use k-means clustering and there is even some code there for me to copy and paste. An important difference between this and the grouping by SR vector is that correlations will also be taken into account, at least in an implicit way.

One open question that remains is whether the grouping is done on portfolio weights that have been derived using a shrinkage method, or on weights that haven't (just using naive mean variance). I felt it was better to use 'purer' weights which hadn't used shrinkage so we don't end up discarding useful differences.

As I did in my prior post in this series, let's run the grouping exercise on my entire portfolio. Partly for laughs, and partly to see if the grouping makes sense. How many groups/clusters should we use? Well there are 7 substantive asset classes, excluding vol:

Cluster 0, length 16

BTP3, CANOLA, EU-FOOD, FED, GBPCHF, GOLD_micro, HIGHYIELD, LIVECOW, OMX, R1000, SILVER, SP500_micro, US-INDUSTRY, WHEAT, YENEUR, ZAR

Ags: 3, Metals: 2, Equity: 3, FX: 3, Sector: 2, Bond: 3


Cluster 1, length 29

BRENT_W, CNH, COAL-GEORDIE, COCOA, COFFEE, COTTON, ETHER-micro, EU-AUTO, EU-DJ-TELECOM, EU-DJ-UTIL, EU-MEDIA, EU-REALESTATE, EU-TECH, FTSETAIWAN, GBP, HEATOIL, KOSPI_mini, MILK, MSCIEAFA, MSCISING, OATIES, OJ, RUBBER, SEK, SMI, SONIA3, US-DISCRETE, US2, VNKI

OilGas: 3, Ags: 7, Metals: 1, Equity: 5, FX: 3, Sector: 7, Vol: 1, Bond: 2


Cluster 2, length 20

CAD10, CH10, CHF, CHFJPY, COPPER-micro, CZK, FTSECHINAA, FTSEINDO, IRON, JGB, JGB-SGX-mini, JP-REALESTATE, MUMMY, NIKKEI, SGX, SOYBEAN_mini, SOYOIL, TOPIX, US-ENERGY, US-HEALTH

Ags: 2, Metals: 2, Equity: 6, FX: 3, Sector: 3, Bond: 4


Cluster 3, length 12

AUD_micro, EU-INSURE, FTSE100, FTSECHINAH, GASOIL, HANG_mini, HOUSE-US, NASDAQ_micro, NOK, NZD, US-STAPLES, US-TECH

Sector: 4, Equity: 4, FX: 3, OilGas: 1


Cluster 4, length 16

ALUMINIUM, AUDJPY, BITCOIN, BOBL, BONO, BUND, BUXL, CORN, DOW, GBPJPY, MILKDRY, OAT, RICE, ROBUSTA, SHATZ, STEEL

Ags: 4, Metals: 3, Equity: 1, FX: 2, Bond: 6


Cluster 5, length 79

BB3M, BBCOMM, BRE, BTP, BUTTER, CAD, CHEESE, CHINAA-CON, COAL, COCOA_LDN, COPPER_LME, COTTON2, CRUDE_ICE, CRUDE_W_micro, DJSTX-SMALL, DX, EU-BANKS, EU-CHEM, EU-CONSTRUCTION, EU-MID, EU-OIL, EU-TRAVEL, EURCAD, EURCHF, EURIBOR-ICE, EUROSTX, EUROSTX-SMALL, EUR_micro, FANG, FEEDCOW, FTSE250, GAS-PEN, GASOILINE, GAS_US_mini, GICS, GILT, HANGENT_mini, IBEX_mini, IG, INR, IRS, JPY, KOSDAQ, KR10, KR3, LEAD_LME, LEANHOG, LUMBER-new, MIB, MILKWET, MILLWHEAT, MSCIEMASIA, MSCITAIWAN, MSCIWORLD, MXP, NICKEL_LME, PALLAD, PLAT, REDWHEAT, SARONA, SGD, SMI-MID, SOFR, SOYMEAL, SP400, SPI200, SUGAR11, SUGAR16, SUGAR_WHITE, TIN_LME, TWD, US10, US20, US5, V2X, VIX_mini, WHEAT_ICE, WHEY, ZINC_LME

OilGas: 6, Ags: 18, Metals: 7, Equity: 16, FX: 12, Sector: 6, Vol: 2, Bond: 12


Cluster 6, length 32

AEX, BOVESPA, BRENT-LAST, CAC, CAD2, CAD5, CLP, DAX, EU-BASIC, EU-DIV30, EU-HEALTH, EU-HOUSE, EU-RETAIL, EUA, EURAUD, EURO600, FTSEVIET, GBPEUR, HANGTECH, KRWUSD_mini, MSCIASIA, PLN, RUSSELL, SWISSLEAD, US-FINANCE, US-MATERIAL, US-PROPERTY, US-REALESTATE, US-UTILS, US10U, US3, US30

OilGas: 2, Equity: 11, FX: 5, Sector: 9, Bond: 5

There doesn't seem much congruency with asset classes there. Is this "the data speaking to us", or are we just data mining with a very sharp spade? Let's find out.


Testing

In my older post, here, I did a rather simplistic 'one shot' test on a subset of my available instruments and forecast rules (albeit on a rolling out of sample basis). But I have a rather more exhaustive way of doing things I've been using in this series. 

I cycle through different lengths of in sample (5 years, 10 years, 20 years) and out of sample (1 year and 5 years) lengths of time. For shorter time periods that will allow me to subsample different historic periods. For speed and to get some alternative paths I'm not going to consider all the instruments. Instead I will randomly subsample 50 instruments randomly out of the 214 available. 

Then for a given set of returns I will eithier use fully pooled, asset class pooled, portfolio weight pooled, or unpooled returns. Then I will optimise for each instrument based on the relevant returns, using the shrinkage method with SR shrinkage of 0.5 and correlation of 0.75. Finally I will take the equally weighted across instruments portfolio SR for the 50 instruments, out of sample. 

In previous posts I've discussed a more honest way of backtesting, where we include the opposite of a given trading rule to avoid implicit fitting; and then only bring positive SR rules into the optimisation. All the results here will use that methodology exclusively.


5 years in sample, 1 year out of sample

You should hopefully recognise this format from before. Each row is a fitting option. The first column shows the median SR across the many, many runs of random resampling. The second column shows the t-test p-value from comparing the best option with the others. NaN means this is the best option. A low number in this column, say below 0.01 or 0.05, indicates that the best option is statistically significantly better than the other option.

                           SR  pvalue

unpooled                 0.046     0.0

all pooled               0.425     0.0

asset class pooled       0.557     NaN

weight distanced pooled  0.184     0.0

That is ... pleasing. The least robust method is worse. More robust methods do better. And we get a significant improvement from pooling within asset classes. OK the portfolio weight distancing isn't so good, but we haven't got huge amounts of data to form our portfolio weights with so maybe they are a little unstable.


5 years in sample, 5 years out of sample

                            SR  pvalue
unpooled                 0.418     0.0
all pooled               0.391     0.0
asset class pooled       0.731     NaN
weight distanced pooled  0.376     0.0

Unpooled does a little better here, but asset classes are still the way to go.

10 years in sample, 1 year out of sample

                            SR      pvalue
unpooled                 0.664     NaN
all pooled               0.417   0.000
asset class pooled       0.635   0.127
weight distanced pooled  0.415   0.000

OK interestingly unpooled is making a comeback, but it still isn't significantly better than asset class pooled.

10 years in sample, 5 years out of sample

                            SR  pvalue
unpooled                 0.855     0.0
all pooled               0.575     0.0
asset class pooled       1.028     NaN
weight distanced pooled  0.559     0.0

Asset class is again asserting it's dominance with unpooled a close second.

20 years in sample, 1 year out of sample

                            SR  pvalue
unpooled                 0.571     0.0
all pooled               0.469     0.0
asset class pooled       1.127     NaN
weight distanced pooled  0.410     0.0

OK this is getting a bit silly. I feel like the dad whose kid at sports day is winning everything, proud but also getting a little embarrassed. "Now come on jonny, let one of the other kids win the next one". 

Interestingly it does seem with more data that unpooled is the way to go for a second option.


20 years in sample, 5 years out of sample

                            SR  pvalue
unpooled                 0.251     0.0
all pooled               0.569     NaN
asset class pooled       0.551     0.0
weight distanced pooled  0.553     0.0

"Well done Jonny. Everyone knows you could have won it if you wanted but it's good to show good sportmanship"

So all pooled finally gets it's day in the sun albeit with a slim advantage over the other two pooled methods. Bear in mind only 65 instruments have sufficient history here; with only 18 having two distinct blocks of 25 years so there won't be much genuine variation if we choose 50. So this could be a fluke. Jonny will tell you that it is.


But Rob, What about averaging?

At this point, given the choice between the complexity of weight distancing, and the simplicity and efficiency of asset class pooling; I'm inclined to go with the latter. And it's what we were doing at AHL all those years ago (not because of empirical evidence but because it suited the organisational structure...).

However there is another option which I talked about in the original asset class pooling post, using a blend. Here we take an average of the portfolio weights selected with different methodologies. So that would be an average of:

  • Unpooled
  • All pooled
  • Asset class pooled
Blending weights in this way is a way to improve robustness. It's arguably the correct thing to do, since otherwise we'd be making an in sample choice of methodology - 'meta implicit fitting' if you will. One of my favourite research shops, Resolve asset management, are very keen on doing this. One potential downside is it might be producing 'over robustness' given we're using weights that have already had shrinkage. But let's find out.


5 years in sample, 1 year out of sample

Note these numbers won't be exactly the same as those above, since they're a different set of random experiments. They would eventually converge but it would take millions of runs.

And also, just for fun, I've added an extra column. I started off this series of posts talking about the importance of considering other points of the distribution but I've quietly dropped that and only been quoting the median. For balance then, I've added the 25% SR point as well as the median. The pvalue is as before.

                    SR median  SR 25%  pvalue
unpooled                0.025  -0.569     0.0
all pooled              0.475  -0.147     0.0
asset class pooled      0.620   0.045     0.0
average                 0.650  -0.067     NaN

Averaging is the winner - just - but asset class is better at the more conservative point.

5 years in sample, 5 years out of sample

                    SR median  SR 25%  pvalue
unpooled                0.428   0.164     0.0
all pooled              0.430   0.243     0.0
asset class pooled      0.771   0.537     NaN
average                 0.664   0.430     0.0

A clear win for asset class pooled here. Averaging suffers from it's association with the less performative unpooled / all pooled.

10 years in sample, 1 year out of sample

                    SR median  SR 25%  pvalue
unpooled                0.681   0.034     0.0
all pooled              0.452   0.067     0.0
asset class pooled      0.664   0.167     0.0
average                 0.935   0.256     NaN

This time averaging takes the win, helped by the good performance of unpooled.

10 years in sample, 5 years out of sample

                    SR median  SR 25%  pvalue
unpooled                0.887   0.641     0.0
all pooled              0.570   0.406     0.0
asset class pooled      1.066   0.833     NaN
average                 1.005   0.780     0.0

Asset class pooled is still the winner, but averaging gives a good job.

20 years in sample, 1 year out of sample

                    SR median  SR 25%  pvalue
unpooled                0.538  -0.209     0.0
all pooled              0.515   0.154     0.0
asset class pooled      1.167   0.539     NaN
average                 0.809   0.252     0.0

Asset class pooled by more of a margin now.

20 years in sample, 5 years out of sample

                   SR median  SR 25%  pvalue
unpooled                0.243   0.084     0.0
all pooled              0.531   0.414     NaN
asset class pooled      0.513   0.391     0.0
average                 0.461   0.355     0.0

As before 'all pooled' is the winner, whilst average is dragged down by the poor performance of unpooled. But as I said above, with these longer periods it's hard to know if it's just down to flukey instrument selection.

What to do...

There is enough evidence above to justify asset class pooling as the dominant choice. But equally, I don't think there is enough to discard averaging. And there is something so neat about averaging. We combine three quite disparate source of data together, so we're protected if one of them doesn't work out. It's robustness writ large! It can be justified without any in sample fitting - whereas one could argue that the selection of asset class pooling is an implicit in sample 'meta parameter' choice.

I think we're now (finally) ready to fit our forecast weights, and with costs. This is exciting for me, as whatever comes out I will be using as my new weights. This will be more of a 'literature review' since I've talked about optimising with costs in some detail and at some length before.