Wednesday, 18 January 2017

Playing with Docker - some initial results (pysystemtrade)

This post is about using Docker - a containerisation tool - to run automated trading strategies. I'll show you a simple example of how to use Docker with my python back testing library pysystemtrade to run a backtest in a container, and get the results out. However this post should hopefully be comprehensible to non pysystemtrade and non python speaking people as well.

PS: Apologies for the long break between posts: I've been writing my second book, and until the first draft is at the publishers posts will be pretty infrequent.

The logo of docker is a cute whale with some containers on its back. It's fun, but I'm worried that people will think it's okay to start using whales as a cheap alternative to ships. Listen people: It's not okay.

What the Heck is Docker and why do I need it?

As you'll know if you've read this post I currently run my trading system with two machines - a live and a backup. A couple of months ago the backup failed; some kind of disk failure. I humanely dispatched it to ebay (it is now in a better place, the new owner managing to successfully replace the disk: yes I am a software guy and I don't do hardware...).

A brief trip to ebay later and I had a new backup machine (actually I accidentally bought two machines, which means I now have a machine I can use for development). A couple of days ago I then had to spend around half a day setting up the new machine.

This is quite a complicated business, as the trading system consists of:

  1. An operating system (Linux mint in my case)
  2. Some essential packages (ssh; emacs, as I can't use vi; x11vnc as the machine won't normally have a monitor attached; git to download useful python code from github or my local network drive)
  3. Drive mountings to network drives
  4. The interactive brokers gateway
  5. A specific version of Python 
  6. A bunch of python libraries that I wrote all by myself
  7. A whole bunch of other python libraries like numpy, pandas etc: again all requiring specific versions to work (my legacy trading system is in frozen development so it is keyed to an obsolete version of various libraries which have since had pesky API changes)
  8. Some random non python package dependencies 
  9. A directory structure for data
  10. The actual data itself 

All that needs to be setup on the machine; which at times can be quite fiddly; for example many of the dependencies have dependencies and sometimes a google is required. Although I have a 'build' file consisting of a document file mostly saying "do this, then this..." it can still be a tricky process. And some parts are... very... slow...

While moaning about this problem on twitter I kept hearing about something called Docker. I had also seen references to Docker in the geeky dead trees based comic I occasionally indulge myself with, and most recently at this ultra geeky linux blog written by an ex colleague.

At it's simplest level Docker allows the above nightmare to be simplified to just the following steps:

  1. An operating system. Scarily this could be ANY operating system. I discuss this below.
  2. Some essential packages (these could be containerised but probably not worth it)
  3. Drive mountings to network drives
  4. The interactive brokers gateway (could also be put inside Docker; see here).
  5. Install Docker
  6. Run a docker container that contains everything in steps 5 to 10

This will certainly save time every time I setup a new trading server; but unless you are running your own data centre that might seem a minimal time saving. Actually there are numerous other advantages to using Docker which I'll discuss in a second. But first lets look at a real example.

Installing Docker

Installing Docker is, to be fair, a little tricky. However it's something you should only need to do once; and it will in the long run prevent you from much pain installing other crud. It's also much easier than installing say pandas.

Here is how it's done: Docker installation

It's reasonably straightforward, although I found to my cost it won't work on a 32 bit linux distro. So I had to spend the first few hours of yesterday reinstalling the OS on my laptop, and on the development machine I intend to use for running pysystemtrade when it gets closer to being live for at least .

Not a bad thing: I had to go through the pain of reinstalling my legacy code and dependencies to remind me of why I was doing this, took the opportunity to switch to a new IDE pycharm, and as a bonus finally wiped Windows 10 off the hard disk (I'd kept it 'just in case' but I've only used it twice in the last 6 months: as the rest of my household are still using Windows there are enough machines lying around if I need it).

The documentation for Docker is mostly excellent, although I had to do a little bit of googling to work out how to run the test example I'm going to present now (that's mostly because there is a lot of documentation and I got bored - probably if I'd read it all I wouldn't have had to google the answer).

Example of using Docker with pysystemtrade

The latest version of pysystemtrade on github includes a new directory: pysystemtrade/examples/dockertest/.  You should follow along with that. All the command line stuff you see here is linux; windows users might have to read the documentation for Docker to see whats different.

Step one: Creating the docker image (optional)

The docker image is the starting state of your container. Think of a docker image as a little virtual machine (Docker is different from a true virtual machine... but you can google that distinction yourself) preloaded with the operating system, all the software and data you need do your thing; plus the script that your little machine will run when it's loaded up.

Creating a docker image essentially front loads - and makes repeatable - the job of setting up the machine. You don't have to create your own images, since you can download them from the docker hub - which is just like the git hub. Indeed you'll do that in step two. Nevertheless it's worth understanding how an image is created, even if you don't do it yourself.

If you want to create your own image you'll need to copy the file Dockerfile (in the directory of pysystemtrade/examples/dockertest/) to the parent directory of pysystemtrade on your computer. For example the full path name for me is /home/rob/workspace3/pysystemtrade/... so I would move to the directory /home/rob/workspace3/. Then you'll need to run this command:

sudo docker build -t mydockerimage .

Okay; what did that just do? First let's have a look at the Dockerfile:

FROM python
MAINTAINER Rob Carver <>
RUN pip3 install pandas
RUN pip3 install pyyaml
RUN pip3 install scipy
RUN pip3 install matplotlib
COPY pysystemtrade/ /pysystemtrade/
CMD [ "python3", "/pysystemtrade/examples/dockertest/" ]

Ignoring the second line, this does the following (also read this):

  • Loads a base image called python (this defaults to python 3, but you can get earlier versions). I'll talk about base images later in the post.
  • Loads a bunch of python dependencies (the latest versions; but again I could get earlier versions if I wanted)
  • Copies my local version of the pysystemtrade library into the image
  • Ensures that python can see that library
  • Runs a script within the pysystemtrade 

If you wanted to you could tag this image and push it on the docker hub, so that other people could use it (See here). Indeed that's exactly what I've done: here.

Note: Docker hub gives you one free private image by default, but as many public images as you need.

Step two: Running the container script

You're now ready to actually use the image you've created in a container. A container is like a little machine that springs into life with the image pre-loaded, does it stuff in a virtual machine like way, and then vanishes.

If you haven't created your own image, then you need to run this:

sudo docker run -t -v /home/rob/results:/results robcarver17/pysystemtrade

This will go to docker hub and get my image (warning this may take a few minutes).

OR with your own local image if you followed step one above:

sudo docker run -t -v /home/rob/results:/results mydockerimage

In both cases replacing  /home/rob/results with your own preferred directory for putting the backtest output. This stuff after the '-v' flag mounts that directory into the docker container; mapping it to the directory /results. Warning: the docker image will have complete power over that directory, so it's best to create a directory just for this purpose in case anything goes wrong.

The docker image will run, executing this script:

from systems.provided.futures_chapter15.basesystem import futures_system
from matplotlib.pyplot import show

resultsdir="/results"system = futures_system(log_level="on")
system.pickle_cache("", resultsdir+"/dockertest.pck")

This just runs a backtest, and then saves the result to /results/dockertest.pck. The '-t' flag means you can see it running. Remember the /results directory is actually a volume mapped on to the local machine. After the image has closed it will have created a file on the local machine called   /home/rob/results/dockertest.pck

Step three: Perusing the results

Now in a normal local python sessions run the file

from systems.provided.futures_chapter15.basesystem import futures_system
from matplotlib.pyplot import show

system = futures_system(log_level="on")
system.unpickle_cache("", resultsdir+"/dockertest.pck")
# this will run much faster and reuse previous calculationsprint(system.accounts.portfolio().sharpe())

Again you will need to change the resultsdir to reflect where you mapped the Docker volume earlier. This will load the saved back test, and recalculate the p&l (which is not stored in the systems object cache).

Okay... so what (Part one: backtesting)

You might be feeling a little underwhelmed by that example, but there are many implications of what we just did. Let's think about them.

Backtesting server

Firstly, what we just did could have happened on two machines: one to run the container, the other to analyse the results. If your computing setup is like mine (relatively powerful, headless servers, often sitting around doing nothing) that's quite a tasty prospect.

Backtesting servers: Land of clusters

I could also run backtests across multiple machines. Indeed there is a specific docker product (swarm) to make this easier (also check out Docker machine).

Easy setup

Right at the start I told you about the pain involved with setting up a single machine. With multiple machines in a cluster... that would be a real pain. But not with docker. It's just a case of installing essential services, docker itself, and then launching containers. 

Cloud computing

These multiple machines don't have to be in your house... they could be anywhere. Cloud computing is a good way of getting someone else to keep your machines running (if I was running modest amounts of outside capital, it would be the route I would take). But the task of spinning up, and preparing a new cloud environment is a pain. Docker makes it much easier (see Docker cloud).

Operating system independent

You can run the container on any OS that can install Docker... even (spits) Windows. The base image essentially gets a new OS; in the case of the base image python this is just a linux variant with python preloaded. You can also get different variants which have a much lower memory overhead.

This means access to a wider variety of cloud providers. It also provides redundancy for local machines: if both my machines fail I am only minutes away from running my trading system. Finally for users of pysystemtrade who don't use Linux it means they can still run my code. For example if you use this Dockerfile:

FROM python
MAINTAINER Rob Carver <>
RUN pip3 install pandas
RUN pip3 install pyyaml
RUN pip3 install scipy
RUN pip3 install matplotlib
COPY pysystemtrade/ /pysystemtrade/

sudo docker build -t mydockerimage .
sudo docker run -t -v -i mydockerimage

... then you will be inside an interactive python session with access to the pysystemtrade libraries. Some googling indicates it's possible to run ipython and python notebooks inside docker containers as well, though I haven't tried this myself.

Okay... so what (Part two: production)

Docker also make running production automated trading systems much easier: in fact I would say this is the main benefit for me personally. For example you can easily spin up one or more new trading machines agnostic of OS either locally or on a cloud. Indeed using multiple machines in my trading system is one of the things I've been thinking about for a while (see this series of tweets: one, two, three and so on).


Docker makes it easier to adopt a microservices approach where we have lots of little processes rather than a few big ones. For example instead of running one piece of code to do my execution, I could run multiple pieces one for each instrument I am trading. Each of these could live in it's own container. Then if one container fails, the others keep running (something that doesn't happen right now).

The main advantage of Docker over true virtual machines is that each of those containers would be working off almost identical images (the only difference being the CMD command at the end; in practice you'd have identical images and put the CMD logic into the command line); Docker would share this common stuff massively reducing the memory load of running a hundred processes instead of one.

Data in container images

In the simple version of pysystemtrade as it now exists the data is effectively static .csv files that live inside the python directory structure. But in the future the data would be elsewhere: probably in databases. However it would make sense to keep some data inside the docker image, eg static information about instruments or configuration files. Then it would be possible to easily test and deploy changes to that static information.

Data outside container images

Not all data can live inside images; in particular dynamic data like market prices and system state information like positions held needs to be somewhere else.

Multiple machines means multiple places where data can be stored (machine A, B or a local NAS). Docker volumes allow you to virtualise that so the container doesn't know or care where the data it's using lives. The only work you'd have to do is define environment variables which might change if data is living in a different place to where it normally is, and then launch your container with the approriate volume mappings.

Okay there are other ways of doing this (a messy script of sim links in linux for example) but this is nice and tidy.

You can also containerise your storage using Docker data volumes but I haven't looked into that yet.

Message bus

I am in two minds about whether using a message bus to communicate between processes is necessary (rather than just shared databases; the approach I use right now). But if I go down that route containers need to be able to speak to each other. It seems like this is possible although this kind of technical stuff is a little beyond me; more investigation is required. 

(It might be that Swarm might remove the need for a message bus in any case; with new containers launched passing key arguments)

Still at a minimum docker containers will need to talk to the IB Gateway (which could also live in a container... see here) so it's reassuring to know that's possible. But my next Docker experiment will probably be seeing if I can launch a Gateway instance (probably outside a container because of the two factor authentication I've grumbled about before unless I can use x11vnc to peek inside it) and then get a Docker container to talk to it. This is clearly a "gotcha" - if I can't get this to work then I can't use Docker to trade with! Watch this space.


At the moment my scheduling is very simple: I launch three big processes every morning using cron. Ideally I'd launch processes on demand; eg when a trade is required I'd run a process to execute it. I'd launch price capturing processes only when the market opens. If I introduce event driven trading systems into my life then I'd need processes that launched when specific price targets were reached.

It looks like Docker Swarm will enable this kind of thing very easily. In particularly because I'm not using python to do the process launching I won't violate the IB multiple gateway connection problem. I imagine I'd then be left with a very simple crontab on each machine to kick everything into life, and perhaps not even that.


Security isn't a big deal for me, but there is something pleasing about only allowing images access to certain specific directories on the host machine.

Development and release cycle

Finally Docker makes it easier to have a development and release cycle. You can launch a docker container on one machine to test things are working. Then launch it on your production machine. If you have problems then you can easily revert to the last set of images that worked. You don't have to worry about reverting back to old python libraries and generally crossing your fingers and hoping it all works.

You can also easily run automated testing; a good thing if I ever get round to fixing all my tests.

Geeky note: You can only have one private image in your docker hub account; and git isn't ideal for storing large binaries. So another source control tool might be better for storing copies of images you want to keep private.


Development of pysystemtrade - as the eventual replacement to my legacy trading system - is currently paused whilst I finish my second book; but I'm sure that Docker will play a big part in it. It's a huge and complex beast with many possibilities which I need to research more. Hopefully this post has given you a glimpse of those possibilities.

Yes: there are other ways of achieving some of these goals (I look forward to people telling me I should use puppet or god knows what), but the massive popularity of Docker tells you why it's so good; it's very easy to use for someone like me who isn't a professional sysadmin or full on linux geek, and offers a complete solution to many of the typical problems involved with running a fully automated system.

PS You should know me better by now but to clear: I have no connection with Docker and I am receiving no benefit pecuniary or otherwise for writing this post.

Monday, 5 September 2016

Systematic risk management

As the casual reader of this blog (or my book) will be aware, I like to delegate my trading to systems, since humans aren't very good at it (well, I'm not). This is quite a popular thing to do; many systematic investment funds are out there competing for your money; from simple passive tracking funds like ETF's to complex quantitative hedge funds. Yet most of these employ people to do their risk management. Yes - the same humans who I think aren't very good at trading.

As I noted in a post from a couple of years ago, this doesn't make a lot of sense. Is risk management really one of those tasks that humans can do better than computers? Doesn't it make more sense to remove the human emotions and biases from anything that can affect the performance of your trading system?

In this post I argue that risk management for trading systems should be done systematically with minimal human intervention. Ideally this should be done inside an automated trading system model.

For risk management inside the model, I'm using the fancy word endogenous. It's also fine to do risk management outside the model which would of course be exogenous. However even this should be done in a systematic, process driven, way using a pre-determined set of rules.

A systematic risk management approach means humans have less opportunity to screw up the system by meddling. Automated risk management also means less work. This also makes sense for individual traders like myself, who can't / don't employ their own risk manager (I guess we are our own risk managers - with all the conflicts of interest that entails).

This is the second in a series of articles on risk management. The first (which is rather old, and wasn't originally intended to be part of a series) is here. The next article will be about an exogenous risk management tool I use called the system envelope. The final article will be about endogenous risk management, explain the simple method I use in my own trading system, and show an implementation of this in pysystemtrade.

What is risk management?

Let's go back to first principles. According to wikipedia:

"Risk management is the identification, assessment, and prioritization of risks (defined in ISO 31000 as the effect of uncertainty on objectives) followed by coordinated and economical application of resources to minimize, monitor, and control the probability and/or impact of unfortunate events[1] or to maximize the realization of opportunities. Risk management’s objective is to assure uncertainty does not deflect the endeavour from the business goals. "

This slightly overstates what risk management can achieve. Uncertainty is almost always part of business, and is a core part of the business of investing and trading. It's often impossible to minimise or control the probability of something happening, if that something is an external market event like a recession.

Still if I pick out the juicy parts of this, I get:

  • Identification, assessment and priorization of risks
  • Monitoring of risks
  • Minimize and control the impact of unfortunate events 
This suggests that risk management can be boiled down to the following

  1. Identify some important risks.
  2. Work out a way to measure them
  3. Set levels at which action should be taken, and specify an action to take.
  4. Monitor the risk measurements
  5. Take action if (when) the measurements exceed critical levels
  6. When (if) the situation has returned to normal, reverse the action

I would argue that only steps 1,2 and 3 are difficult to systematise. Steps 4 to 6 should be completely systematic, and if possible automated, occuring within the trading system.

Types of risk

It's very easy to forget that there are many types of risk beyond the usual; "the price will fall when we are long and we will lose our shirts". This is known as market risk and whilst it's the most high profile flavour there are others. Pick up any MBA finance textbook and you'll find a list like this:

  • Market risk. You make a bet trade which goes against you. We quantify this risk using a model.
  • Credit / counterparty risk. You do a trade with a guy and then they refuse to pay up when you win.
  • Liquidity risk. You buy something but can't sell it when you need to.
  • Funding risk. You borrow money to buy something, and the borrowing gets withdrawn forcing you to sell your position.
  • (Valuation) Model risk.You traded something valued with a model that turned out to be wrong. Might be hard to distinguish from market risk (eg option smile: is the Black-Scholes model wrong, or is it just that the correct price of OTM vol is higher?).
  • (Market) Model risk. You trade something assuming a particular risk model which turns out to be incorrect. Might be hard to distinguish from market and pricing model risk ("is this loss a 6 sigma event, or was our measurement of sigma wrong?"). I'll discuss this more later.
  • Operational / IT / Legal risk. You do a trade and your back office / tech team / lawyers screw it up.
  • Reputational risk. You do a trade and everyone hates you.

Looking at these it's obvious that some of them are things that are hard to systematise, and almost impossible to automate. I would say that operational / IT and Legal risks are very hard to quantify / systematise beyond something like a pseudo objective exercise like a risk register. It's also hard for computers to spontaneously analyse the weaknesses of valuation models, artifical intelligence is not quite there yet. Finally reputation: computers don't care if you hate them or not.

It's possible to quantify liquidity, at least in open and transparent futures markets (it's harder in multiple venue equity markets, and OTC markets like spot fx and interest rate swaps). It's very easy to program up an automated trading system which, for example, won't trade more than 1% of the current open interest in a given futures delivery month. However this is beyond the scope of this post.

In contrast it's not ideal to rely on quantitative measures of credit risk, which tend to lag reality somewhat and may even be completely divorced from reality (for example, consider the AAA rating of the "best" tranche of nearly every mortgage backed security issued in the years up to 2007). A computer will only find out that it's margin funding has been abruptly cut when it finds it can't do any more trading. Humans are better at picking up and interpreting whispers of possible bankruptcy or funding problems.

This leaves us with market risk - what most people think of as financial risk. But also market model risk (a mouthful I know, and I'm open to using a better name). As you'll see I think that endogenous risk management can deal pretty well with both of these types of risk. The rest are better left to humans. So later in the post I'll outline when I think it's acceptable for humans to override trading systems.

What does good and bad risk management look like?

There isn't much evidence around of what good risk management looks like. Good risk management is like plumbing - you don't notice it's there until it goes wrong, and you've suddenly got "human excrement"* everywhere

*Well my kids might read this blog. Feel free to use a different expression here.

There are plenty of stories about bad risk management. Where do we start... perhaps here is a good place:

Nick Leeson. Bad risk management in action, early 90's style. Source: Daily Mail

Generally traders are given a small number of risk management parameters they have to fit within.

For example my first job in finance was working as a trader for Barclays Capital. My trading mandate included a maximum possible loss (a mere million quid if I remember correctly), as well as limits on the greeks of my position (I was trading options). I also had a limit on everyones favourite "single figure" risk measurement, VAR.

Bad traders will eithier willfuly, or through ignorance, bend these limits as much as possible. For example if I return to the list of trading losses above, it's topped by this man:

Howie. The 9 billion dollar man. Not in a good way. Source:

Howie correctly called the sub prime mortgage debt collapse. He bet on a bunch of mortgage related derivative crap falling. But to offset the negative carry of this trade (which caused a lot of pain to other people doing this trade) he bought a bunch of higher rated mortgage related derivatives. For boring technical reasons he had to buy a lot more of the high rate stuff.

On paper - and presumably according to Morgan's internal models - this trade had minimal risk. It was assumed that the worst that would happen would be that house prices stayed up, and that the long and short side would remain high. Hopefully though Howie would get it right - the crap would fall, and the good stuff would keep it's value.

However it turned out that the good stuff wasn't that good eithier; the losses on the long position ended up dwarfing the gains on the short position. The risk model was wrong.

(The risk management team did [eventually] warn about this, but Howie succesfully argued that the default rate they were using to model the scenario would never happen. It did.)

Risk management embodied by trading systems

From the above discussion we can derive my first principle of risk management:

Good traders do their own risk management 

(and by trader here I mean anyone responsible for making investment decisions, so it includes fund managers of all flavours, plus people who think of themselves as investors rather than traders).

Good traders will take their given risk limits as a starting point. They will understand that all risk measurements are flawed. They will think about what could go wrong if the risk model being used was incorrect. They will consider risks that aren't included in the model.

Similarly good trading systems already do quite a lot of risk management. This isn't something we need to add, it's already naturally embodied in the system itself.

For example in my book I explain how a trading system should have a predetermined long term target risk, and then how each position should be sized to achieve a particular target risk according to it's perceived profitability (the forecast) and the estimated risk for each block of the instrument you're trading (like a futures contract) using estimates of return volatility. I also talk about how you should use estimates of correlations of forecasts and returns to achieve the correct long run risk.

Trading systems that include trend following rules also automatically manage the risk of a position turning against them. You can do a similar thing by using stop loss rules. I also explain how a trading system should automatically reduce your risk when you lose money (and there's more on that subject here).

All this is stuff that feels quite a lot like risk management. To be precise it's the well known market risk that we're managing here. But it isn't the whole story - we're missing out market model risk. To understand the difference I first need to explain my philosophy of risk in a little detail.

The two different kinds of risk

I classify risk into two types - the risk encompassed by our model of market returns; and the part that isn't. To see this a little more clearly have a look at a picture I like to call the "Rumsfeld quadrant"

The top left is stuff we know. That means there isn't any risk. Perhaps the world of pure arbitrage belongs here, if it exists. The bottom left is stuff we don't know we know. That's philosophy, not risk management.

The interesting stuff happens on the right. In green on the top right we have known-unknowns. It's the area of quantifiable market risk. To quantify risk we need to have a market risk model.

The bottom right red section is the domain of the black swan. It's the area that lies outside of our market risk model. It's where we'll end up if our model of market risk is bad. There are various ways that can happen:

  • We have the wrong model. So for example before Black-Scholes people used to price options in fairly arbitrary ways. 
  • We have an incomplete model. Eg Black-Scholes assumes a lognormal distribution. Stock returns are anything but lognormal, with tails fatter than a cat that has got a really fat tail.
  • The underlying parameters of our market have changed. For example implied volatility may have dramatically increased.
  • Our estimate of the parameters may be wrong. For example if we're trying to measure implied vol from illiquid options with large bid-ask spreads. More prosically we can't measure the current actual volatility directly, only estimate it from returns.

An important point is that it's very hard to tell (a) an extreme movement within a market risk model that is correct from (b) an extreme movement that isn't that extreme, it's just that your model is wrong. In simple terms is the 6 sigma event (should happen once every 500 million days) really a 6 sigma event?

Or is it really a 2 sigma event it's just that your volatility estimate is out by a factor of 3? Or the unobservable "true" vol has changed by a factor of 3? Or does your model not account for fat tails because 6 sigma events actually happen 1% of the time? You generally need a lot of data to make a Bayesian judgement about what is more likely. Even then it's a moving target because the underlying parameters will always be changing.

This also applies to distinguishing different types of market model risk. You probably can't tell the difference between a two state market with high and low volatility (changing parameter values), and a market which has a single state but a fat tailed distribution of returns (incomplete model); and arguably it doesn't matter.

What people love to do, particularly quants with Phd's trapped in risk management jobs, is make their market models more complicated to "solve" this problem. Consider:

On the left we can see that less than half of the world has been explained by green, modelled, market risk. This is because we have the simplest possible multiple asset risk model - a set of Gaussian distributions with fixed standard deviation and correlations. There is a large red area where we have the risk that this model is wrong. It's a large area because our model is rubbish. We have a lot of market model risk.

However - importantly - we know the model is rubbish. We know it has weaknesses. We can probably articulate intuitively, and in some detail, what those weaknesses are.

On the right is the quant approach. A much more sophisticated risk model is used. The upside of this is that there will be fewer risks that are not captured by the model. But this is no magic bullet. There are some disadvantages to extra complexity. One problem is that with more parameters they are harder to estimate, and estimates of things like higher order moments or state transition probabilities will be very sensitive to outliers.

More seriously however I think these complex models give you a false sense of security. To anyone who doesn't believe me I have just two words to say: Gaussian Copula. Whilst I can articulate very easily what is wrong with a simple risk model it's much harder to think of what could go wrong with a much weirder set of equations.

(There is an analogy here with valuation model risk. Many traders prefer to use Black-Scholes option pricers and adjust the volatility input to account for smile effects, rather than use a more complex option pricer that captures this effect directly)

So my second principle of risk management is:

Complicated risk model = a bad thing

What I prefer to do is use a simple model of returns as part of my trading system. Then I handle market model risk systematically: either endogenously within the system, or exogenously.

Risk management within the system (endogeonous)

The disadvantage of simpler models is their simplicity. But because they're simple, it's also easy to write down what their flaws are. And what can be written down easily can, and should, be added to a trading system as an endogenous risk management layer.

Let's take an example. We know that the model of fixed Gaussian volatility is naive (and I am being polite). Check this out (ignore the headline, which is irrelevant and for which there is no evidence):

S&P 500 vol over time. Source: Seeking Alpha

Now I could deal with this problem by using a model with multiple states, or something with fatter tails. However that's complicated (=bad).

If I was to pinpoint exactly what worries me here, it's this: Increasing position size when vol is really low, like in 2006 because I know it will probably go up abruptly. There are far worse examples of this: EURCHF before January 2015, Front Eurodollar and other STIR contracts, CDS spreads before 2007...

I can very easily write down a simple method for dealing with this, using the 6 step process from before:
  1. We don't want to increase positions when vol is very low.
  2. We decide to measure this by looking at realised vol versus historical vol
  3. We decide that we'll not increase leverage if vol is in the lowest 5% of values seen in the last couple of years
  4. We monitor the current estimated vol, and the 5% quantile of the distribution of vol over the last 500 business days.
  5. If estimated vol drops below the 5% quantile, use that instead of the lower estimated vol. This will cut the size of our positions.
  6. When the vol recovers, use the higher estimated vol.
Here is the implementation of this idea in pysystemtrade (Default values can be changed here).

It's easy to imagine how we could come up with other simple ways to limit our exposure to events like correlation shocks, or unusually concentrated positions. The final post of this mini series will explain how my own trading system does it's own endogenous risk management, including some new (not yet written) code for pysystemtrade.

Systematic risk management outside the system (exogeonous)

There is a second category of risk management issues. This is mostly stuff that could, in principle, be implemented automatically within a trading system. But it would be more trouble than it's worth, or pose practical difficulties. Instead we develop a systematic process which is followed independently. The important point here is that once the system is in place there should be no room for human discretion here.

An example of something that would fit nicely into an exogenous risk management framework would be something like this, following the 6 step programme I outlined earlier:

  1. We have a large client that doesn't want to lose more than half their initial trading capital - if they do they will withdraw the rest of their money and decimate our business.
  2. We decide to measure this using the daily drawdown level
  3. We decide that we'll cut our trading system risk by 25% if the drawdown is greater than 30%, by half at 35%, by three quarters at 40% and completely at 45% (allowing some room for overshoot).
  4. We monitor the daily drawdown level
  5. If it exceeds the level above we cut the risk capital available to the trading system appropriately
  6. When the capital recovers, regear the system upwards

[I note in passing that:

Firstly this will probably result in your client making lower profits than they would have done otherwise, see here.

Secondly this might seem a bit weird - why doesn't your client just stump up only half of the money? But this is actually how my previous employers managed the risk of structured guaranteed products that were sold to clients with a guarantee (in fact some of the capital was used to buy a zero coupon bond). These are out of fashion now, because much lower interest rates make the price of the zero coupon bonds far too rich to make the structure work.

Finally for the terminally geeky, this is effectively the same as buying a rather disjointed synthetic put option on the performance of your own fund]

Although this example can, and perhaps should, be automated it lies outside the trading system proper. The trading system proper just knows it has a certain amount of trading capital to play with; with adjustments made automatically for gains or losses. It doesn't know or care about the fact we have to degear this specific account in an unusual way.

In the next post I'll explain in more detail how to construct a systematic exogenous risk management process using a concept I call the risk envelope. In this process we measure various characteristics of a system's backtested performance, and use this information to determine degearing points for different unexpected events that lie outside of what we saw in the backtest.

For now let me give you another slightly different example - implied volatility. Related to the discussion above there are often situations when implied vol can be used to give a better estimate of future vol than realised vol alone. An example would be before a big event, like an election or non farm payroll, when realised vol is often subdued whilst implied vols are very rich.

Ideally you'd do this endogenously: build an automated system which captured and calculated the options implied vol surface and tied this in with realised vol information based on daily returns (you could also throw in recent intraday data). But this is a lot of work, and very painful.

(Just to name a few problems; stale and non synchronous quotes, wide spreads on the prices of OTM options give you very wide estimates of implied vol, non continuous strikes, changing underlying mean the ATM strike is always moving....)

Instead a better exogenous system is to build something that monitors implied vol levels, and then cut positions by a proscribed amount when they exceed realised vol by a given proportion (thus accounting for the persistent premium of implied over realised vol). Some human intervention in the process will prevent screwups caused by bad option prices.

Discretionary overrides

Ideally all risk managers at systematic funds could now be fired, or at least redeployed to more useful jobs.

Risk manager working on new career. Source: wikipedia

But is it realistic to do all risk management purely systematically, either inside or outside a system? No. Firstly we still need someone to do this stuff...

  1. Identify some important risks.
  2. Work out a way to measure them
  3. Set levels at which action should be taken, and specify an action to take.
... even if stages 4-6 should still be done by computers.

Secondly there are a bunch of situations in which I think it is okay to override the trading system, due to circumstances which the trading system (or predetermined exogenous process) just won't know about.

I've already touched on this in the discussion related to types of risk earlier, where I noted that humans are better at dealing with hard to quantify more subjective risks. Here are some specific scenarios from my own experience. As with systematic risk management the appropriate response should be to proportionally de-risk the position until the problem goes away or is solved.

Garbage out – parameter and coding errors

If an automated system does not behave according to its algorithm there must be a coding bug or incorrect parameter. If it isn't automated then it's probably a fat finger error on a calculator or a formula error on a spreadsheet. This clearly calls for a de-risking unless it is absolutely clear that the positions are of the correct sign and smaller than the system actually desires. The same goes for incorrect data; we need to check against what the position would have been with the right data.

Liquidity and market failure

No trading system can cope if it cannot actually trade. If a country is likely to introduce capital controls, if there is going to be widespread market disruption because of an event or if people just stop trading then it would be foolish to carry on holding positions.

Of course this assumes such events are predictable in advance. I was managing a system trading Euroyen interest rate futures just before the 2011 Japanese earthquake. The market stopped functioning almost overnight.

A more pleasant experience was when the liquidity in certain Credit Default Swap indices drained away after 2008. The change was sufficiently slow to allow positions to be gradually derisked in line with lower volumes.

Denial of service – dealing with interruptions

A harder set of problems to deal with are interruptions to service. For example hardware failure, data feed problems, internet connectivity breaking or problems with the broker. Any of these might mean we cannot trade at all, or are trading with out of date information. Clearly a comparison of likely down time to average holding period would be important.

With medium term trading, and a holding period of a few weeks, a one to day outage should not unduly concern an individual investor, although they should keep a closer eye on the markets in that period. For longer periods it would be safest to shut down all positions, balancing the costs of doing this against possible risks.

What's next

As I said I'll be doing a couple more posts on this subject. The next one will talk about specific exogenous systematic risk management. The final post will explain how I use endogenous risk management within my own trading system.

Monday, 27 June 2016

Obligatory Brexit ex-post (Survival, Skew, Politics & Data, )

This is a two part post which blends my favourite subjects: finance, economics, and politics. It's about the stunning victory of the "leave" campaign that has left the UK facing a period of serious uncertainty.

Part one is an ex-post update on whether my portfolio was as robust to Brexit as I hoped it would be in my last post. This is based on market moves on Friday; of course things will continue to move as the full ramifications are digested.

Part two explains how Brexit was a classic negative skew event in more ways than one, how behavioural finance can help explain that, how it was mispriced by the markets, and how I used that information to make a little bit of money.

Say hello to our next Prime Minister. Source: The Sun

Did I survive Brexit?


Let me expand.

Trading in GBPUSD

I made no money at all trading cable, the asset most exposed to Brexit. In fact I lost some. When I wrote my last post I had no position in GBPUSD IMM futures. However the swings in sterling sentiment were reflected in first a sell:

11242 GBP 201609 2016-06-15 02:38:16 -1 1.412600

And then I closed that as GBP went stronger in the post Jo-Cox* bounce in remains polling figures.

* It feels wrong to be talking about her tragic death in a grubby post which is mainly, at least at the moment, about money. However it seems likely that what happened to her was clearly linked to the referendum campaign. I had the privilege of briefly seeing her in person. She was on the select committee which I appeared before last year (16:20:00 on the timestamp of the video feed onwards), and I was very impressed with her performance. I'll be doing the politics of this later.

11296 GBP 201609 2016-06-17 18:54:08 1 1.434800

As with a typical trend following system in a range bound market I sold low and bought high. And a further opening buy almost exactly 24 hours before the result would become clear, as the world decided remain was a shoe in (more about this below).

11461 GBP 201609 2016-06-23 02:51:41 1 1.483000

Then, of course it all went wrong. The pound got smashed. With a reversing price and volatility going through the roof, the system closed out it's position.

11509 GBP 201609 2016-06-24 03:54:26 -1 1.366400

Although the price continued to collapse it to around 1.33 then recovered, and closed at pretty much the same level. As I'm writing this the pound is drifting lower, but I have no position to take advantage of this.

I lost the equivalent of nearly 2% of my portfolio on that one lousy GBP contract, about 1.25% of which was lost on the day.

The rest of my futures portfolio

Fortunately, as I said in my last post:

"It doesn't even look like a strong "risk on" or "risk off" theme, though I look to be a little bit more "risk off"."

Yeah, I got that right. I made it all back on the rest of my portfolio. In order of importantce: US bonds and Eurodollar, German bonds (makes sense), Korean bonds (who'd have though it - Korea a safe haven from the UK...), French bonds (the irony) and JPYUSD (though I gave that all back on AUDUSD).

I had no position in Gold, but you can't have everything.

I lost a bit on short positions in VIX and V2X.

Net-net on the main futures portfolio I made a little under 2.5%. Not as much as Crispin Odey is supposed to have made, but then I would imagine he would have hurt a lot more than me if the result had been as expected.

The point here isn't so much "ooh look I made money isn't my system clever" - it is pure luck how much anyone makes on a single day, but that my risk was well managed, such that a 6 sigma event in the markets translated into roughly a 2 sigma portfolio move based on my expected risk going into the vote.

Equities, currency and hedging

Again, from the post:

As I've said before my futures trading account is funded by a long stock position, which is in a mixture of UK stocks and a European ETF IDVY; hedged by a Eurostoxx futures short. This is a more interesting little portfolio; ignoring the effects on stocks if the pound gets hammered then in GBP terms this portfolio will go up in value - there will be a gain in the value of the ETF but the futures hedge won't move since it's unaffected by currency movements.

I was close here. The Eurostoxx did get hammered, more than the FTSE so I got that wrong. However it did indeed lose less in GBP terms because of the devaluation.  Also the UK shares I hold in this account didn't do half badly. Net-net the long stocks plus hedge added another 2% or so.

I also have a long only, buy and hold, portfolio of UK stocks and ETF's covering bonds and equities across the globe.

Here are some more quotes from my Brexit post:

The ETF's are not currency hedged, so a GBP devaluation following brexit would benefit them

This is exactly what happened. The only ETF in my main portfolio that fell in £ terms was a high yield Eurostoxx beast, which suffered from the huge fall in banking shares which wiped out the EURGBP revaluation benefit. All my other ETF's were up in £ terms.

The effect on stocks is a little tricker to predict, but I would imagine my UK stocks would hurt - perhaps 5% at worse? A huge one day drop which will probably be partly reversed, but not the end of the world.

The FTSE100 was down 3.2% at the close, the FTSE 250 7.2%. So on average 5% was a good guess. Some of my UK stocks did much better (Glaxo was up), others much worse (Intermediate Capital Group was done over 14%). 

Overall in my long only account I was flat.

This isn't genius trading, but at least if the book sales fall short then I could probably get a job as a risk analyst.

All of this is on a T+1 analysis. We don't really know how things will pan out over the next N years whilst this little mess gets sorted out.

Brexit as negative skew

Cameron, David

I tweeted a lot on Friday. Here is one of my first efforts:

"Cameron took on the classic short vol trade - high probability of success, small upside, huge downside. And lost.

That is pretty self explanatory. Cameron acted like a muppet running a hedge fund.

"Muppet" is a term of abuse in the south east of England meaning "Idiot". I don't mean an actual Muppet. I have a lot of respect for Kermit. I think he'd make an excellent hedge fund manager. Source:

Your typical muppet HF manager will keep selling OTM options. They kept expiring without value, leaving him with the premium. He begins to confuse luck with skill; he thinks he was invincible. This is the classic behavioural finance effect of overconfidence. Then one day, he blows up.

Why was the referendum like an OTM option?

Firstly, it looked like a forgone conclusion. Promising a referendum was a cheap way to beat down the nascent UKIP vote in last years election (the vote still hit historic highs, but thanks to our bonkers electoral system they were unable to achieve concentrated support to win more than one seat in parliament). It would be so easy, just a half baked (and suspiciously quick) negotiation with the EU to get superficial concessions, hold the referendum, bish-bash-bosh with the handy side effect of destroying his main political rival (By which I mean Boris Johnson. Not the labour party).

Yes there were some tense moments, but he never looked ruffled, or looked like he ever thought he would lose. This looked like a high probability event, but the downside, which doesn't seem to have been properly considered, was enormous - both politically and personally.

Secondly, it had a small payoff if it succeeded. Without the referendum I think it is incredibly unlikely that Boris would be a serious contender for PM.

Thirdly it had a large cost if it went wrong. Both personally, politically and economically.

Overconfidence often seems to affect people from privileged backgrounds.

The whole country is paying for the fact that Cameron has had a smooth path through life with nothing going really wrong. There are many articles in the press pointing out how he's been lucky with several political gambles over the years. Even having to go into coalition with the Lib Dems was a blessing, giving him an excuse to keep the right wing of the party at bay whilst holding Nick Clegg in front of himself to take all the flak for the politics of austerity.

All this made him think he was invincible. Similar (dare I say?) to the careers of many financial traders and asset managers, many of whom come from privileged backgrounds and who sailed through life without failing at anything*, who confused luck with skill, became infected with the fatal disease of hubris; before the inevitable blow up.

* Full disclosure: The author of this blog was privately educated and was admitted to the worlds best university (the first generation in his family to go on to higher education, make of that what you will), but soon found he wasn't as clever as he thought he was, and left after one year. A series of random unconnected jobs followed. On first coming into contact with the financial sector with a job as a trader he hated it. So yes, I've been lucky, to have screwed up my life royally many many times.

Finally both hedge fund managers and David Cameron have the advantage of a fairly soft landing if things go wrong. Collecting 2 and 20 for a few years on a few hundred million or more can set you up for life. Cameron has more than enough personal wealth, his wife still works, and even with his ignominious exit could easily rack up a few million quid a year "advising" banks and private equity firms (a case of "don't do as I did, do as I say"?).

A fair payoff for negative skew

Now, taking negative skew bets isn't automatically a bad thing, if they're fairly priced. The minimum one should accept is something that gives you zero expectation. Ideally you'd get more than this, a lot more, since most reasonably people have a utility function that hates negative skew.

So if the probability of an event is P, and the payoff is a, with a loss of b if the event doesn't happen:

b>a  (definition of negative skew)

aP >  (1-P)b

NOT a black swan. A white one with one grey feather

Before I continue, I'll just point out that Brexit was NOT a black swan - an inconceivable event with a tiny probability of happening. Even at the widest odds the implied probability of the event was more than 10%.

The City, The Bookmakers and The Polls

It wasn't just Cameron who got the probability of this event wrong, it was everyone.

On the day before the referendum implied currency volatility was pricing in something like a 20% chance of Brexit happening:

" on BBG, prob derived from FX options for leave is 19% - but guessing the spread is huge to do the arb trade!!!"

twitter  with thanks to Saeed Amen

Similarly on the betting markets the decimal odds on betfair were also implying roughly a 20% chance, although this was moving around (see image - not up to date, but gives you an idea of the volatility)

It was clear to me that 20% was far too low. See these opinion polls.

Source: Financial Times

Notice that:

  • The vote was on a knife edge; with neither side getting a clear advantage (say 60:40) in any poll
  • There is considerable variation across time
  • There is considerable variation across polls
  • The undecided figure, although narrowing over time, was still averaging about 6% at the end - easily enough to swing it one way or another.
  • Polls have margins of error (MOE), which are shamefully absent from public discourse. Most polls of the size done in the referendum campaign would have MOE of 3-5%. Polls of polls have a smaller MOE; maybe a couple of percent.

Personally I thought on Thursday that given the final poll of polls figure of 48:46, with undecided at 6%, and an MOE of a couple of percent.... well this wasn't an event with just a 20% chance of happening. It was pretty close to a coin flip; but let's call the odds of remain P=60%. But the ratio of the two payoffs in the betting market was more than 5.0 (decimal odds of 1.2 and 6.0 respectively). So the expectation of the two options is:

0.6 x 1.2 = 0.72 
0.4 x 6.0 = 2.4

The second option (betting on leave) is clearly more attractive than the second.

Why this apparent mispricing? Well the markets (betting and FX) move depending on weight of money. Mostly wealthy well educated people of a higher social class play in both markets. Exactly the sort of people who voted remain. They ignored the polls and focused on their own prejudices. Again this is behavioural finance at work. Could be one of several effects, but lets go with narrative bias. The "story" that Brexit was terrible and nobody would vote for it was more attractive than the statistical evidence in the polls.

That is the specific effect. But generally people are really bad at understanding data when measured with uncertainty; and this is particularly dangerous when the outcome is binary - people will naturally try and assign probabilities of 100% and 0% to binary options.

My little flutter

So how best to exploit this? A straightforward way would be to sell GBP eithier in USD or EUR. Or do the same thing via options, although note that buying OTM options is an expensive business due to the spread. You could also bet on the outcome.

However although all of these bets have positive expectation, it will also on average lose money (assuming there was say a 60% chance of remain). Since we can only bet on brexit once this is somewhat annoying.

I opted for a combination trade (and here); a bet on Betfair on Brexit, and a purchase of GBP against EUR (so a bet on remain). I sized the latter trade so that my expected payout in the event of eithier remain or leave would be the same.

Actually I probably got the size wrong. I doubt that GBPEUR would have rallied enough with a remain vote to make up for my loss on Betfair. The resulting depreciation of GBPEUR was also relatively small; as it turned out Brexit is bad for Europe as well and all the capital fleeing to safe havens went to America - it was GBPUSD that really got crucified.

As a result I ended up making much more than I expected (also here) on a leave vote. However this was a fun trade done for relatively small amounts of money. This kind of global macro discretionary trading isn't my forte. It seems like a lot of work.


Generally betting on a coin toss isn't a good idea, unless the market is mispricing the chances of heads as just 20%.

However these opportunities are few and far between, and require a lot of work and analysis. Systematic trading is far easier. A properly structured (diversified, risk managed) investment portfolio or trading system will not be vulnerable to shocks like Brexit. 

So I made some money. I take absolutely no pleasure in this because I genuinely feel it was the wrong decision which will have, and already has had, terrible consequences (both economically, and in a sharp rise in racism). There is already discussion that Brexit may never actually happen; I hope this is the case.