Systems building - execution

People often get systematic and automated trading mixed up. The latter is a subset of the first. You can't have a system which is fully automated if it relies on discretionary input, no matter how small. But you can have a system which needs a human to make it run, even though there is no discretion, and its fully systematic.

The main area where humans are often used with systematic trading is in execution. It's relatively common for people to create software that analyses prices and produces a trade, which the person then has to execute.

(At the large hedge fund I used to work for, we used a mixture of automated execution algos, and real people to execute trades. FWIW the best humans were better than the best algos, on certain kinds of markets)

This post isn't for you. It's only for those who want to automate their execution as well, and avoid the stress of actually trading.

If you don't want to be like this guy, then read the rest of this post.

This is the fourth post in a series giving pointers on the nuts and bolts of building systematic trading systems. You may also want to look at my post on building a simple execution algo, which I'll refer to occasionally here.

Event driven versus Sample driven

As I've discussed before in this series there are two ways of running a trading system. The first method is event driven; you wait for something to happen (price hitting a particular level, or the next tick arriving) and then you take the appropriate action (buying or selling, or cancelling an existing order perhaps).

The second method, which I use, is sample driven (and there is no agreed nomenclature here, so you may hear it called other things). Here we periodically check / 'sample' the data we are interested in (eg prices), calculate what we want our optimal position to be, and then trade to achieve it.

The former system is most suited to high frequency trading, and to relatively simple algorithims (if you have to spend a minute analysing the tick event, then you lose any benefit from running an event driven system). It doesn't work as well with, for example, trading relative value systems.

Anyway this isn't the place to get into a debate about which is better. For my application (relatively slow trading) sample driven trading is more natural. There are also advantages in that debugging is easier, and code can be simpler.

Much of this post will be relevant to event driven systems, but those running sample systems will get more out of it.

Key principles

There are a number of factors, some mutually exclusive, to consider when designing a system to do execution. In particular there is a potential conflict between robustness and speed. To be precise, if I'm in a situation where the system might do something potentially dangerous such as an incorrect trade, I would rather delay the trade until the situation can be resolved; even if the delay could be several days.

Because I want a fully automated system, that literally should be able to run for months without being touched or watched (rolls apart), this robustness is incredibly important.

The optimal position

To be clear then before the part of my program that does execution comes into the picture something else has created a theoretically optimal position, which was true at some point in time, and with some data (in practice then, for my simple technical system, at some price).


I'll talk more in a later post about whether this is a separate process, or all one big happy function, and how separate processes 'talk' to each other.

Essentially then the job of the execution component of my system is to compare the optimal position with the current position; and if they are out of line, try and do a trade that brings the actual position back to what we want it to be.

If you want more detail on what I mean by 'out of line' you'll have to read my forthcoming book, www.systematictrading.org, chapter 11.

What can go wrong

This all sounds very simple, but in fact there are a number of things that can go wrong:

• Optimal position is  unavailable
• The price on which the optimal is predicated is out of date
• We don't know what our current position is
• Our current position is out of date, as we've recently done a trade that will change it
• We're running two processes and they will both issue the same order, ending up with the position 'overshooting' the optimal
• The market is closed
• The market isn't liquid enough to do the adjusting trade required
• We might, for some reason, not want to do the adjusting trade

Writing exhaustive lists of 'what can go wrong' like this is vital before you write a robust process. We need to make sure all our processes deal with these issues. Also my bias towards purely automated systems that run themselves, on which a delay in trading can be shrugged off, influences how these problems are solved.

Getting the current position: Storage and lining up

I won't discuss where the optimal position is stored, as this will be the subject of a future post. Just for now imagine there is a database table populated with it (as indeed is the case in my own system). However just to note that if we can't read the optimal position then we probably have a fairly broken database (or perhaps a file lock), and we shouldn't try and do any trading.

There are two main places you can store your current position:

• Your own database
• Your brokers database

In the latter case you'd need to use the brokers API to get positions. Note that unfortunately neithier of these methods is 100% reliable*. I've had issues the IB API** giving me the wrong position (usually eithier missing one instrument entirely, or doubling up on the real position). I've also had issues with fills arriving which I've missed, thus making my own database estimate of the current position wrong.

* The NASA way of avoiding conflicts of this kind would be to have three sources of current position and institute some kind of voting algorithm. Sadly I can't think of a further, independent, source of position.

** References to my broker IB (Interactive Brokers), can clearly refer to other brokers as well.

The solution I use is to make my database the primary source of current position. I then crosscheck this 3 times a day with the position the broker thinks I have. If there is a difference of opinion between me and the broker then I change the trading state to STOP (see more below about different states). Genuine differences of opinion are rare, and more often than not caused by a position cash settling or being forced to close by IB when for some reason I haven't closed it myself before expiry (discussed more here).

I also avoid creating new orders when I think, or know, I have an existing order (of which more below).

Lessons from the database managers

Let's take a few of those problems I outlined above and rephrase them slightly:

• Our current record is out of date, as we've recently done a transaction that will change it
• We're running two processes and they will both issue the same transaction, ending up with the record being incorrect.

These problems then are exactly those faced by people who create database systems. The solution that they, and anyone involved in worrying about files being updated by multiple users / processes, is locking.

If I google (images) 'database' this is the first thing that comes up. It would be much cooler if databases actually came with holograms of pieces of paper flying in and out of your server.

I use locking in the following way. Locks are applied at an instrument level (eg to the whole of the Gold future, regardless of which contract we're trading). The normal status of an instrument is unlocked. This means we can issue a new order related to that instrument. Once an order is issued however the instrument becomes locked.

In a locked status no new orders can be issued for an instrument. Because I am super paranoid I'll also check with the IB API that there are no outstanding orders; even if the instrument is unlocked if the API reports the existence of an outstanding order (which shouldn't happen... but...) means I can't do any more.

The lock will only be removed once the order is completely filled and the current position updated to reflect that, or the order is cancelled by my software (of which more later). Because I can miss fills, and because of other gremlins, I have a periodic (hourly) process which checks to see if there are any instruments that are locked, but with no outstanding orders reported by the IB API. If there are, then they are automatically unlocked.

This may seem like paranoia. Why not just check for oustanding orders from the IB API, and don't trade if they exist? Because there is a tiny period of time where IB will say 'yeah, no outstanding orders' but the relevant fill hasn't yet arrived (or the fill may be missed somehow) and my current position hasn't been updated, meaning I might try and issue the same trade again. By periodically removing locks that apparently aren't needed I do open myself up to this happening, but it's much less likely than if I used the presence of IB outstanding orders as my only check to see if it is safe to trade.

Enemy of the state

I won't always want to trade an instrument, or do certain types of trade. For this reason I have a number of trading states which an instrument can be in:

• OK - allow all kinds of trades
• NOOPEN - do not open any new trades, but allow any existing orders to continue.
• NOINCREASE - do not do any trades that will increase our position, but allow closing trades. I use this a lot; for example when I recently wanted to get out of my German 2 year bond position.
• NOCLOSE - do not do any trades that will reduce our position. It's hard to think of a situation where this would make sense, but I hate asymmetry.
• STOP - do not open any new trades, and cancel any existing orders. Once existing orders are cancelled, this migrates to 'NOOPEN'
• CLOSE - immediately try and issue a trade to close the entire position (but which will respect trading limits; see below).

Locks and states are different; locks are applied and can be removed by software. The state can be changed by software (such as when opinions on current positions vary); but can only be reset to 'ok' by human hands.

Rolling rolling rolling...

Applicable to future traders only (or spread bets, options... others where positions can't be held indefinitely and need to be 'rolled over' to the next maturity). You should have already read my post on the mechanics of rolling. Suffice to say that we can be:

• Not rolling, in which case we'll buy and sell the current contract
• Passively rolling, when we'll issue reducing orders in the current contract, and increasing orders in the new one
• Actively rolling with a spread trade, when we'll issue a single order to (if long) sell the current contract and buy the new one
• Actively rolling with an individual leg trade, when we'll issue two orders to close the current contract, and open a position in the new one

The important point here is that the code first needs to identify the roll state, work out which combination of orders makes sense, and then needs to have the logic to issue one or two orders in the current and/or new contract; or a spread order.

Open all hours

There is no point trying to trade if nobody wants to trade with you. There are five reasons I can think of why you might not be able to trade:

• The market is closed today (a regular closure, depending on the day)
• The market isn't open right now (a regular closure, depending on the hour).
• The market is unexpectedly closed.
• There isn't enough liquidity in the market to trade
• You have been banned from the market, or blown up and lost all your money; or upset your broker.

Don't copy this guy if you don't want to be banned from the market

Leaving aside the last point; for the first two points we can of course download and create tables storing market holidays and market opening times. Beware this is a lot of work; and you need to then waste your life worrying about daylight savings changes. This doesn't deal with point three of course. 

For point four you're going to have to check the liquidity before you trade. I'll discuss below exactly what a liquidity check might involve.

I pretty much choose the laziest possible option; which is to treat the first four points all in the same way as a lack of liquidity. What I do in practice then is have a table of when I know markets should be open and liquid(ish). During the times specified in that table if I feel the need to trade I will periodically check to see if that market is liquid. If it is, then I go ahead and trade (if trading is needed). If not, then I don't. Not liquid could mean the market is on holiday today, or has closed early for some reason, or it might just not be liquid.

Note that if I was even lazier, and just checked all markets 'all'* the time, then I'd be frequently checking a lot of closed markets which is a waste of processing power and bandwidth.

* My execution process runs from 10pm to 8pm the next day, local UK time. It is kicked off Sunday evening, Monday evening .... Thursday evening. The two hours of down time is used to run reports and create backups.

Liquid liquid everywhere and not a drop to drink

I said above we need to check for liquidity. There are many ways to measure liquidity of course. In my simple world, where I only have level 1 data (and L2 in any case could be corrupted by spoofers), I use the following simple test:

• Is the inside bid-ask spread in the market smaller than X (where X is some value I've set for each market, which depends on how I'm executing the order)?
• Is the size available at the bid and ask larger than the trade I want to do?

Note that if the market is closed then the values for bid-ask spread and size will be NAN and NAN respectively, so by definition there isn't enough liquidity.

I'll discuss in a second how the method of execution affects the required maximum spread. Just to note that if the size available is smaller than the trade I want to do (but is bigger than zero), and I want to do more than one contract, the execution code will reduce the size of the trade so its possible given the available size.

There is a further check I will do at this stage. Remember from above the theoretically optimal position, which was true at some point in time, and with ... some price. So I need to have a concept of staleness, and a concept of maximum price movement.

To put it another way, given the speed of my system, I would be concerned if I'd calculated my optimal position based on a price from more than a hour ago. By the way this will happen if you're running seperate optimal position calculation and trading processes as I am; and the latter kicks off in the morning before the former has had a chance to update the price. It won't happen if you have a single pipeline process 'collect price -> calculate optimal position -> create order', or if you're using event driven systems. Again this kind of stuff will be the subject of another post.

Secondly I'd be concerned if the price had moved since I checked it. If for example a market had moved more than 4 daily standard deviations in price in the hour or less since I last grabbed a price, then it's quite likely my optimal position is out of date, and I will want to recalculate it before trading. Otherwise the risk is I might buy based on the old price, and then quickly sell again, once a new price has percolated through the system.

The messy business of actually executing

When you actually execute you can:

• Issue a market order, that will most probably cross the spread
• Use your own algo, as I do (much, much more is here).
• Use a brokers algo. IB have a good set of algos for equities, fewer for futures.
• Use a third party algo. Unfortunately the good ones are only available to institutional traders. I can recommend QB for interest rate products.

I will usually use my own algo, although for manual trades (see later) I give myself the option of using a market order. Eventually I'd expect to have a suite of algos for different instruments; so I have the ability to do this ready in my database, although I don't yet use it. Algo evaluation is a science; just like evaluating other trading systems it has very noisy data and again only large institutions can realistically collect enough data to make meaningful comparisions (and possibly not even then).

As I alluded to above I have different liquidity requirements depending on whether I am issuing a market order or using my algo. If I'm issuing a market order I want the spread to be tight (normally a single tick, or whatever the minimum increment is, unless the market normally trades a little wider), since I know for sure I'll be paying half of it. If I'm using my algo I am less choosy; a much larger spread might mean I capture more of it when I submit my initial lot at the bid (if I'm buying) or offer (selling). This is true to a point - very large spreads mean the market is probably dangerously thin and I wouldn't try and issue an order.

Safety first

The scariest thing about fully automated trading is the doomsday scenario; what if this thing just goes bad and trades like crazy?

What my automated system might say if I tried to stop it trading. Although, just to be clear, my name isn't Dave.

Although I've been ultra careful there is still an outside chance something could go wrong. To avoid this I use daily, and 'lifetime' trade limits.

So before I actually issue an order the software will check to see how many contracts I'd done today already, and whether I will break that cumulative limit. If so it will reduce the trade (sometimes down to zero). I also have 'lifetime' limits. In practice if I'm going on holiday I will set the lifetime limits to a reasonably large level of trading I'd expect to do over that period (clearly less than N x daily limit, where N is the number of days I'm away, otherwise they won't bind), and then when I get back I'll reset them to large numbers.

Limits are set by looking at values in my backtest, and so are contract dependent.

In my current setup active roll trades (as spreads or legs) are exempt from trade limits, although once done they count towards the limit (often meaning you can't do any more trading that day after a roll). It would probably be better to count rolls seperately towards another, larger, limit. That is on my, very long, to-do list.

Note that IB API also has limits, but they are not instrument dependent. Unless you're trading only stocks, or  just one future, this makes them pretty useless. So I turn them off.

Correcting mistakes

Stuff happens. And when it happens you need to fix the mess it creates.

The first kind of stuff, which I've already mentioned, is where my position and the IB API position get out of line. Normally what has happened here is that I issued an order which was filled, and I missed seeing the fill; or IB closed out or settled a position which I hadn't got around to rolling in time.

To deal with this my software has the ability to do a manual fill or a balancing order. The former is where I update an existing, unfilled, order in my database to include the fill that actually happened. The latter is where I create an entirely new order in the database to reflect the trade that happened. After I've done this I should be in a situation where my, and my brokers, record of position are perfectly matched.

The other kind of stuff is sometimes I might want to trade outside of the normal execution process. For example suppose I have a contract I have to roll today. I've tried passive rolling, issuing spread orders, and even individual leg orders. None of these have filled, so what I will do is issue a manual trade from the command line, and then manually override the normal execution method to do a market order. Unlike a manual fill or a balancing order a manual trade will be sent to IB for execution, as well as going in the database.

Done

Hurray. Instead of being tied to your computer, following its robotic commands to execute trades, you can now do something else instead.

In the next post I'll look at working out how much lovely moolah you're making.

This is the fourth post in a series on building systematic trading systems.

The first three posts are:

http://qoppac.blogspot.co.uk/2015/04/system-building-data-capture.html
http://qoppac.blogspot.co.uk/2015/05/systems-building-futures-rolling.html
http://qoppac.blogspot.co.uk/2015/06/systems-building-deciding-positions.html

The next post is here:

http://qoppac.blogspot.co.uk/2015/08/systems-building-accounting.html