Monday, 16 November 2026

AI for systematic traders - this is the least stupid way to do it.

 Anyone who has spent more than 5 nano seconds following me on social media or listening to me waffle on during podcasts will know there are a few things that I am not a fan of. One is crypto. The other is AI. 

These two things have at least one thing in common. They are terrible for the environment, increasing the usage of energy at a time when we aren't using 100% renewable energy, and using other resources as well. 

Both Donald Trump and Elon Musk also like them, and they also sit firmly inside my personal 'not a fan of' box.

And yet, in my fifth book that is being released very shortly, that word 'crypto' is on the cover. Why? Let me explain myself in my own words from the book:

"I should state my own prejudices up front: I am not a fan of cryptocurrencies, which are used primarily for gambling and criminal activities such as money laundering, and which consume a great deal of electrical power, with obvious implications for the climate crisis. In an ideal world I would ignore them completely. Although I am retained as a research advisor for a crypto hedge fund, and I trade Bitcoin and Ethereum futures, I do not hold any crypto in my personal portfolio.

But the point of this book is to guide traders into using certain scientific rules to reduce their chances of losing money, and nobody needs those rules more than those trading crypto. The message of this book as it pertains to crypto can be summarised as “I would not do this stupid thing if I were you, but if you insist, this is the least stupid way to do it.”


In a similar vein, I am not a fan of AI, although unlike crypto I will admit to using it in a very limited way. Mostly for LOLs and making silly pictures, but yes ocassionally for more serious purposes. But I do appreciate that there are a lot of other people out there who are using AI; and perhaps more to the point unlike crypto I do believe AI is useful although it is clearly IMHO massively overhyped in terms of what it can achieve in the quant finance space specifically. But there are an awful lot of people using it in very stupid ways. Hence the subtitle of this post.


A silly picture - thanks chatgpt. See Appendix for prompts and transcript.


The 1st past of this post will be a brief and breezey criticism of the current use cases of AI by people who do similar things to me (develop trading strategies that can then be automated). Then I'll briefly explain the limited set of things I would trust AI to do. Finally I will outline a framework for using AI in a systematic trading strategy reserach process. 


Some bad ways to use AI 

Let's begin with the dumbest way to use AI for trading purposes, with a prompt like "Tell me what stocks to buy" on a generic public model. I hope I don't need to elucidate to the wise and educated people on this blog why that would be a bad idea. 

One level about that would be using the same prompt on a model that has been locally trained on some financial data. Generic machine learning classification tends to do very badly on financial data, mainly due to non stationarity. It's not clear to me why LLMs should do any better. 

Going up another level, would be a prompt like "Tell me what trading strategy to use, and why". Congratulations, you have just invented in sample fitting but in a completely opaque way! Plus you can't trust the code that's been used to produce the back test you have just been shown. Interestingly, AIs don't seem to be as good at the insample research loop (develop model -> test-> change model -> retest) that humans do so well.

That brings me to the writing of code. Any code that you use to handle your money and/or other people's money should be 100% trustworthy and fully understood by you, whether it be backtesting or production code. You can never say that is true of AI code.

Agentic AI? Well..





(The second image is a real screen shot from someone I know, who - for obvious reasons - will remain anonymous)

I am expecting this part of the post to get a lot of replies, mostly critical and starting with "Ok boomer..." (typical AI mistake, I am gen-X). I will ignore ones that say "Ah but that is because you should be using model XYZ" or "No you should be using a different prompt" or "Ah but you just need to do X, Y and Z". LLMs have inherent flaws that can never be completely fixed by prompting or using different models or using an agentic framework; and will always be opaque - a very bad thing indeed.

Some good ways to use AI 

Idea generation: I am not averse to using AI for idea generation. This could work with eithier a publically trained AI or a local model. 

Summarising research: Next after idea generation, it makes a lot of sense if you are doing a lot of research to have an AI summarise for you all the academic papers in field X (you shouldn't ask it to sumarise book Y, since the contents of that book will have been stolen from it's hard working authors). Of course AI's writing style is bloody irritating so I'm not sure how much of it you could read without screaming. I find that asking it do summarys in bullet points removes a lot of the pain. The other obvious problem is hallucinating, so you need to be prepared to get your metaphorical hands dirty and read the underyling research yourself. Still it's a good starting point, and better than just typing "phrase" into google scholar.


There are another ten pages of this, and in fairness although there were a few minor errors it pointed me to some papers I wasn't aware of

Finding data sources: arguably this is something a search engine could do, but we're pretty much at the point where AI is better at search than a search engine is. But don't just say 'give me the data', find out where the date comes from and get it yourself.

Writing small amounts of code: I've used AI to write small bits of code. Small bits because you can easily verify they are correct. It's particularly useful in a situation where the code is relatively isolated from everything else, and it's not a big deal if it fails. A recent example would be that for the framework described at the end of the post, I wanted a configuration file that could only be added to and not otherwise edited, and decided to implement this using hashes (yeah I know that makes the file like a single node blockchain, the irony isn't lost on me). 


I could have easily written this myself in an hour or so, but I was in a hurry and got AI to write it. It ran perfectly and although the style of the code wasn't something I would have written myself, it's small enough that I can easily check it's correctness in much less than an hour (so net time saved), and if it goes wrong it isn't going to result in me accidentally buying 10,000 contracts of Bund futures.

Checking code: In theory at least, I don't have a big problem with AI checking your code and suggesting changes. I've had other people check my code with AI, and to be fairness it did find some issues. The downsides here is that the language of code checking in AI is bloody awful (as is obvious from the AI contributions to pysystemtrade), and you often need another AI to interpret what it is saying, which seems to be a net waste of time. 

You should absolutely not merge the changes from the AI without checking them. The same code check someone did for me resulted in the AI unintentionally introducing an absolutely massive bug. So what I am suggesting here is definitely not a 'write code with AI', 'check code with AI' doom loop without human intervention; or using AI so extensively that you can't actually properly check all the code written or suggested changes. That would be insane, and it's incredible to me that there are people doing it.

Writing tests: I hate writing tests, as anyone involved in my open source psystemtrade project will know. Tests are small bits of code with a well defined objective that if they go wrong won't affect the rest of your codebase - perfect for AI. Note that I wouldn't extend this to writing documentation, another thing I am really lazy at (although unlike tests, I do actually do it), since AI writing isn't something I am a fan of; and there is a real benefit to writing your own in my opinion.

Stealing code: Like crypto, AI is also good for doing crime and stuff.


A framework for using AI (and also human brains)

With the above in mind, what is a safe framework for using AI in trading research? In fact this is exactly the same framework that should be used in an ideal environment for testing human generated ideas. We still have the same three versions of overfitting we're trying to avoid: explicit, implicit and tacit. It's just that the overfitting is wrapped in an opaque blob so we don't even know if it's happening or not. 

Idea generation, data scoping, initial research: As noted above this could be done by AI, or by a person. It's fine.

The end point of this exercise will be the following:

  • Some trustworthy data. To be trustworthy the data must have come direct from the source not via the AI. You should have checked the specs of the data to make sure there is no in-sample fitting hidden inside it (this is a problem often for sell-side produced 'strength indices', and more recently I've had this issue with certain Glassnode crypto data) and no forward looking data (a problem for eg macro data published with a lag and then revised, and company earnings that certainly don't come on the final day of the quarter they relate to).
  • A piece of parameter-isable code implementing the strategy idea (i.e. producing a forecast) given the data as an input and incorporating parameters that require calibration like the fast_span and slow_span below:


(As an aside, the code produced here was a bit rubbish as it also included some backtesting fluff and didn't just produce the forecast but this is a 'skill issue' that better prompting or a skill.md file can probably solve).

Note that the parameter-isable code is small and therefore it's correctness can be verified. However it may well incorporate tacit fitting.

That is the last time we will touch AI in this process. Everything else that happens will happen in a robust testing framework. Importantly that robust framework must not have been written with AI or be run be agents (with the caveat that small parts of the code may be AI written, AI may have been used to suggest improvements, and the tests may be written with AI).

A good way of explaining this is to think about the harness+AI that produces AI agents, something like this:


From https://www.bankofengland.co.uk/speech/2026/november/jon-hall-speech-at-cf-imperial-workshop which I attended actually

Instead of a model inside a harness (and inside a sandbox here) the model only gets to play with one thing; the parameter-isable python code that ends up getting tested as a forecasting rule. Everything else in the robust framework around it is off limits to the AI.

The list of requirements for the robust tester is again not different from what a proper tester would have for human generated ideas or strategies:

  • Generating the parameter fitting space eithier by using a very limited amount of real "burn in" data, fake data or theory. If real data is used, performance isn't considered, only correlation and turnover.
  • As alluded to above when I discussed the config file hash key thing, an add only system where new forecasting rules once tested can't be removed from the backtest. This avoids implicit fitting in the form of 'research loops' where the results of backtests are used to refine forecasting rules, or remove them entirely. As we're not ever going to remove or drop a rule, we don't need to worry about nonsense like Whites Reality Check and we're not at risk of dropping rules that may work but whose natural profitability is too low to pass an arbitrary statistical hurdle.
  • Backward looking explicit fitting of the combination of optimal parameters and forecasting rules.
  • The fitting is robust - it knows about statistical significance and trading costs (as discussed in my recent series of posts, the last one is here).
  • Both positive and negative versions of a strategy to avoid tacit in sample fitting. This avoids the problem that both smart humans and AI already know what's worked, eg medium speed trend following.

What I haven't done of course is specify exactly how this robust fitting engine will work. That is something I am currently working on; and will be spelt out more in (a) future blog post(s). It will also be the subject of my next (6th) book, title to be confirmed, publication date probably at least 2 years away...


Appendix: chat transcript for image

Me: I want a picture of renowned systematic trading author robert carver sat at his computer swearing because AI has just deleted all his files

Chat:







Me: that doesn't look like him and that book cover is completely wrong

Chat: You're right — the likeness and the book cover both need correcting. The image invented a generic older author rather than capturing Robert Carver. Let's get it right this time. Could you upload: 1. A photograph of Robert Carver that you'd like me to use as the likeness reference. 2. The actual cover of his book, Systematic Trading.

I'll recreate the scene with: - A much closer facial resemblance. - The genuine book cover, not an AI-invented version. - Robert visibly furious, shouting and swearing at his computer. - A screen displaying: "AI HAS DELETED ALL YOUR FILES".

Would you prefer photorealistic comedy or an exaggerated satirical caricature?

Me: <Attached Images> I am robert carver and I have the copyright to these images. photorealistic

Chat:








Me: WTF is going on with my hands, they look like those of an 80 year old man? What OS is that - should be linux? And why is my book so big - should be smaller

Chat:

Much better on the Linux front! This version has: - Ubuntu-style Linux desktop, with Tux visible. - A smaller, more realistic-sized book on the bookshelf. - A ThinkPad rather than a generic desktop monitor. - More natural-looking hands and gestures. - Your actual appearance and clothing as the reference. The AI has still managed to delete everything. At least it hasn't deleted your book. 😄

NOTE: Smiley face was inserted by AI not me. I am not that whimsical

Wednesday, 4 November 2026

Why optimising for the left tail of outcomes is like playing chess...

A short one today.

Suppose I offer you N trading strategies to choose between. We can run some kind of bootstrap of their historical returns to get a final distribution of CAGR, with all the caveats implied by that. Or suppose I ask you to pick the correct level of leverage on a trading strategy. Again we can bootstrap historical returns at different leverage levels, giving us a series of N discrete possible leverage levels. How do we do that? Both are equivalent to choosing which of N distributions of possible outcomes we prefer.

The strict expectation method is to pick the option with the distribution that has the best median value, eg 50% distribtional point. As I've discussed before, that is equivalent to the Kelly optimal.

The gambling addict method would be to pick the one with a very good right tail. We might go for the option that has the best 1% right tail. That would be a high leverage option, something with high natural risk like crypto, or a short gamma strategy that doesn't manage to blow up in the best possible alternative histories that generated the bootstrap. Nobody should be a gambler, except sometimes.

The conservative method, which I favour, is the opposite. We might go for the option that has the best 1%, 10% or 25% tail; eg the option that will have the best outcome if bad things happen. 

We can imagine an extreme version of the conservative method where we pick the option with the best worst outcome, i.e. the best minimum value. In fact if we're bootstrapping enough times the lower bound of all distributions will be a final portfolio value very close to zero (with compounded returns and absolute returns>-100% we can't mathemtically below zero) and hard to distinguish; so in practice the most realistic extreme version would be something like picking the option with the most optimal 1% left tail point. Given that is likely to be negative, that would be the option with the least worst 1% tail.

(Frankly more extreme outcomes are highly uncertain, and will probably go beyond what a historical bootstrapping exercise could predict)

The least worst option? That is also the MinMax of game theory, Nash equilibria and all chess playing software.


Credit: Chatgpt, authors own prompt

If you view the market as your adversary, you want to have the least bad outcome in the worst possible state.

Now I am not suggesting you optimise for the 1% point, but they do say that there are no old, bold traders. Good traders know how to quantify and when to take risks, but they also understand the importance of keeping their capital mostly intact in the future. So somewhere around the 10% or 25% point seems appropriate to me. 



Optimisation with costs (oh please seriously Rob, not again...) and a sort of summary

After a summer break, this is post (checks notes) #11 in my 2026 series on portfolio optimisation. Up until now I've been focusing on the optimisation of forecast weights -the weights of signals to predict a given instrument - on a pre cost basis. So now it seems that we should introduce the question of costs.

But if you are a faithful follower of this blog, you may be groaning! I have covered this subject at length before in three substantial posts:

In this ten year old (!) post which is literally called 'optimising with costs'

In this 2023 Geri Halliwell (yes really) themed post on whether performance is identical across instruments on a pre-cost or post cost basis 

In this follow up 2023 movie franchise themed post about the effects of various layers of my strategy on costs

... and this isn't to ignore the numerous posts I've done which talk about costs in some way.

So this post is going to be a bit of a mish mash.  It will be a bit of an abstract 'let us think about this from all possible angles'. It will be a bit of a literature review of previous posts. And there will be at least one new idea for dealing with costs in portfolio optimisation ("no rob haven't we suffered enough"). 


My trading strategy and speed and costs

My strategy consists of lot's of little simple things, but layered together the interactions can get a little complex.

Remember we begin by combining forecasts trading at different speeds for a given instrument. 
Then we allocate an instrument weight. This gives me the ideal portfolio I would trade without worrying about pesky integer contracts. Next, I run a dynamic optimisation ('DO') which does worry about integer contracts. But it also worries about costs. It removes instruments which I've decided are too expensive to trade (as well as those which aren't sufficiently liquid, require too much leverage, or which I can't legally trade). It applies a buffer to the overall trading, which slows down the system's turnover. And it applies a trading cost penalty. The effect of all these cumulatively was discussed in one of the prior posts.

Let us know think about how this affects our decision making. For concreteness, let's assume we have three instruments in the worlds weirdest portfolio: S&P 500 (with 0.00025 SR points of costs per trade, amongst the cheapest I have), Cheese (0.008 SR points, just inside the 0.01 threshold where I'd stop trading it) and Sugar #16 (0.09 SR points, well above my limit). So bear in mind that our trades in Cheese will be severely penalised by the trading cost penalty in the DO. And we won't actually trade Sugar at all, since it will be excluded at the DO stage. 

How should we allocate forecast weights to say a fast, medium and slow trend following rule variation within each of these three instruments? Here are some of the things we need to consider:
  • Should I have some heuristic pre-filtering where I won't use rules where turnover*SR cost>speed limit? How would that work with Sugar where none of the rules would meet that speed limit? [in my current system, I pre-allocate a set of the slowest rules]
  • What is the likely pre-cost SR of these rules, and does it make sense to do some kind of pooling to estimate them? This was discussed in the previous post in this 2026 series. 
  • How does the internal correlation of these rules affect things? For example, we know that fast/slow are more diversifying than medium which will be 0.8 correlation with both.
  • How should we apply trading costs in the optimisation? Should we apply them so we end up putting less into faster variations for Cheese than for S&P 500? [The exact methodology of how costs are applied in optimisation is a seperate subject I will discuss below]
  • How would that work with Sugar, when all of our trading rules would probably have negative SR after costs? 
  • Should we get more diversification by, for example, putting more in fast SP500 trend, even if that doesn't actually make money; as otherwise we wouldn't be able to trade fast trend at all?

Next, how should we allocate instrument weights? I haven't yet addressed this in my posting series, but because of the layers of interaction it does need to be discussed.

  • What is the likely pre-cost SR of instrument performance, and is it effectively flat so it can be set to be equal across instruments?
  • Should we apply trading costs to instrument performance, so we end up allocating less to Cheese and more to S&P?
  • How does that work with Sugar, where we would probably end up negative SR after costs even if the forecast weights were just 100% in the slowest system?
  • Given I am going to apply cost penalties to Milk anyway, does it matter if we apply trading costs?
  • Given we won't actually trade Sugar because of it's trading costs, it's instrument weight will just determine how it influences the positions we do actually trade. So what implications does that have for applying costs?

Applying trading costs in optimisations

Much of this was discussed here, but for the sake of completeness here are the options (including a new one):

  1. Heuristic: apply a maximum cost threshold. This can be applied before, and in addition to, any of the methods below.
  2. Ignore costs, use gross returns. These could be pooled. 
  3. Use net returns. Note if you are pooling across instruments, that implies you should probably pool gross returns then subtract instrument specific costs (I did consider the alternative of pooling costs, or pooling net returns, but this option makes more sense).
  4. Just use costs - set gross returns to some arbitrary equal number.
  5. Subtract costs from gross returns, and multiply costs by a factor >1. This reflects the fact costs are more certain than gross returns.  It will make the results a little more like if you just used costs (option 4). This has the problem of finding the optimal value of 'factor'.
  6. Calculate weights using gross returns, and adjust subsequently for cost levels using some inverse-optimisation-Black-Litterman-Shtick. This is handy if you are pooling weights across instruments, rather than pooling returns. You can get the pooled weights with gross returns and then subtract costs.
  7. NEW (applies only if you are using shrinkage, my preferred option): Get your gross returns, which could be pooled. Shrink the SR estimates. Then subtract costs from the shrunk SR. This reflects the fact costs are more certain than gross returns but without the need to work out a factor as in option 5. If we are then using the averaged of unpooled, pooled and asset class pooled; we can do this for each and then take our average.
Some of these seem silly (2, 4). Some of them seem over complicated (5, 6). 

Then we have the thorny issue of how that would work with a super expensive instrument like Sugar, in which case it's likely that we'd fail both pre-optimisation filtering (1), and also end up with negative SR after applying costs (3 and 7). 

Right now I handle this with forecast weights by using a small subset of really slow trading rules for an instrument which fails the speed limit heuristic test. And I don't use SR at all, which means I ignore costs apart from the speed limit test. And I handle it with instrument weights by not using SR at all, which means once again that I'm basically ignoring costs.And things with SR cost over 0.01 will not actually be traded in DO, irrespective of their instrument weights.


Blimey Rob, it's complicated isn't it

So there is sort of a double standard here. And a bit of a mess. The cliff edge of 0.01 SR instrument costs where we start doing something completely different is a bit distasteful. Probably, as instruments get more expensive, there should be an impact on forecast weights but not neccessarily in a complete 1:1 way. Although this does go against the statement that costs are more predictable than returns. And perhaps the same is true of instrument weights; rather than keeping them but not trading them, perhaps we should be dialling them down somewhat. 

Here is the core dilemna, with DO we are (kind of) not trying to predict the returns of any individual instrument, but find the most cost effective way to implement a portfolio which represents our opinion on pre-cost price movements. But my previous research found that DO isn't completely effective at reducing the trading speed of expensive instruments.

Having given this much thought, I think my new favourite thing to do is this for forecast weights:
  • Set a ceiling on SR costs used for the optimisation at 0.01 SR
  • For pooled optimisations, pool gross returns across all instruments or within asset class, but use the costs for the individual instrument concerned to calculate net returns.
  • Shrink correlations and gross SR estimates
  • Then subtract the per instrument cost from the shrunk SR (#7)
  • Do the optimisation on the shrunk SR net of costs and shrunk correlations
  • Instruments which have current SR costs of greater than 0.01 would be ignored in the dynamic optimisation
Note this makes the cliff edge less problematic. I change my instrument overrides based on current costs much more frequently than my forecast weights. Consider something (instrument A) which has SR cost of 0.02 when I do the forecast weight optimisation, and instrument B with SR 0.005 at that time. We will end up trading the former as if it has a SR cost of 0.01 - so a little too quickly, but it will initially be removed from the forecast weight optimisation. Then suppose A and B swap, so B has SR 0.02 costs and A has 0.005. At this point we're going to be trading A a little too slowly, but fine; and B too quickly. Once I update my overrides though B will be excluded from the dynamic optimisation. I'm still 'trading' it too quickly, but I'm not actually trading it so I can relax.

Although we aren't worrying too much about instrument weights at this stage, we can use a similar approach for those except there is no pooling:
  • Set a ceiling on SR costs used for the optimisation at 0.01 SR
  • Get gross returns for each instrument
  • Shrink correlations and gross SR estimates
  • Then subtract the per instrument cost from the shrunk SR (#7)
  • Do the optimisation on the shrunk SR net of costs and shrunk correlations
  • Instruments which have current SR costs of greater than 0.01 would be ignored in the dynamic optimisation

A summary: 

Although not strictly within the remit of this post I think it is useful to summarise what my 'gold standard' forecast weight optimisation looks like after all the thinking and analysis I've done this year. It's effectively a nested set of functions. Here is some code that illustrates it:

op_shr_deduct_costs = ForecastWeightsOptimiserShrinkageDeductCosts()
opcluster_shr_deduct_costs = ForecastWeightsOptimiserClustering()
opcluster_shr_deduct_costs.set_optimiser(op_shr_deduct_costs)
op_rw_shr_cluster_deduct_costs = ForecastWeightsOptimiserReverseWeights() ## Clustering must go INSIDE reverse weights
op_rw_shr_cluster_deduct_costs.set_optimiser(opcluster_shr_deduct_costs)
op =ForecastWeightsOptimiserAcrossInstrumentsAveragePoolingGrossReturnsDeductCosts()
op.set_optimiser(op_rw_shr_cluster_deduct_costs)
wts = rolling_forecast_weights_optimiser_across_instruments(optimiser=op,
dict_of_instrument_spec=dict_of_instrument_spec,
forecast_pandl_across_instruments_and_forecasts=forecast_pandl_across_instruments_and_forecasts)

Going from the outside, inwards:
  • The outermost wrapper (rolling_forecast_weights_optimiser_across_instruments) does rolling pure out of sample optimisation, rerunning each year on a 30 year expanding then rolling window (in this post I recommended a 40 year EMA but that's equivalent to a 28 year SMA and this is nicer to code). We'll estimate both SR and correlations using this window.
  • Inside that is the pooling code ForecastWeightsOptimiserAcrossInstrumentsAveragePoolingGrossReturnsDeductCosts which does three optimisations for each instrument with different sets of gross returns, but using the costs of the actual instrument.
    • The gross returns of the instrument
    • The pooled gross returns of the asset class it belongs in
    • The pooled gross returns across all instruments
  • We then take an average of those three weights. This approach is justified here.
  • The optimisation inside that is a reverse weights wrapper (ForecastWeightsOptimiserReverseWeights). The reverse weights wrapper finds assets with negative gross p&l and reverses the sign on them. Once the weights are found, it puts a minus sign in front of the weights. This means we have no prior on whether a forecast has (pre-cost) positive p&l, thus reducing tacit fitting and opening up the possibility of finding strategies we didn't realise would work. 
  • Inside that is ForecastWeightsOptimiserClustering. That's a clustering, with a target cluster size of six, as discussed here.
  • Within each cluster we use ForecastWeightsOptimiserShrinkageDeductCosts. As noted above that means we shrink the correlation and gross returns SR (use shrinkage for optimisation with SR shrinkage 0.5, correlation 0.75 unless you have less than five years of data in which case use higher shrinkage) and then deduct costs from the gross returns SR.
The instrument weights code will be exactly the same, but without the pooling step.

Next blogpost will be to do some playing around with this model on a relatively small tractable portfolio.