Friday 25 April 2014

Streaming prices from IB API swigibpy

There is now a native python API. I've written an updated version of this post, here, which uses the native API.

In my last post ( we looked at getting snapshot historical prices. In this one we will look at streamed prices - 'market tick data'. Rather than wait until the historical price feed ends with streaming prices we need to tell the streamer when to stop.

No stream rises higher than its source

Get the source code from this git repository. The files you will want to examine are and

No stream drives anything without being confined

The example code is similar to historical data; we make one of these weird client objects containing a server 'callback' connection, make one of these slightly less weird contract objects (here it is for December 2016 Eurodollar futures) and then shove one into a request function for the other.

from, __main function:

    callback = IBWrapper()
    ibcontract = IBcontract()
    ibcontract.secType = "FUT"

    print "Bid size, Ask size; Bid price; Ask price"
    print ans

This should produce something like the following (assuming the market is open and you have a IB gateway or TWS server open.:

Bid size, Ask size; Bid price; Ask price
[10836, 12362, 98.665, 98.67]

Of course in the dull familar story inside the client object functions that actually get the data we've actually got hidden event driven code again.

from, IBclient.get_IB_market_data() method:

    def get_IB_market_data(self, ibcontract, seconds=30,  tickerid=MEANINGLESS_ID):         
        Returns granular market data
        Returns a tuple (bid price, bid size, ask price, ask size)
        ## initialise the tuple
        # Request a market data stream

        <SNIP   ... more code to come>

Ah yes its the usual stuff of setting up space within the self.cb callback instance to get the data and then call the tws server request function (strictly speaking its one of those 'EClientSocket' whatdoyoucallits again).

Only dead fish swim with the stream...


We now look inside the server callback object which gets populated as an instance self.cb. As before there are a few EWrapper functions which get triggered whenever the market data arrives.

There are in fact several methods for 'tickString', 'tickGeneric', 'tickSize' and 'tickPrice'; it seems a bit stochastic (quant speak: english translation completely bloody random and arbitrary) which of these methods gets called when a tick arrives (a tick could be an update to a price or to a quoted size on the top level of the order book). Lets look at the most generic of these:
    def tickGeneric(self, TickerId, tickType, value):

        ## update generic ticks

        if int(tickType)==0:
            ## bid size
        elif int(tickType)==3:
            ## ask size

        elif int(tickType)==1:
            ## bid
        elif int(tickType)==2:
            ## ask

All the code does is identify which type of tick it is and then populate the appropriate part of marketdata. Obviously we will end up overwriting the values already in marketdata tuple but you could store them for some kind of averaging if you want.

Once in the stream of history you can't get out 


If we just let that baby run we'd be receiving streams of prices until the cows came home. So what we do back in the client world is say STOP I've had enough after a preset amount of time (we could also STOP when the N'th tick has arrived, or when there all the slots in marketdata are filled, which would be easy enough to code up).

from, IBclient.get_IB_market_data() method:

        <Continuing from where we were before. Repeating the last call for clarity>



        while not finished and not iserror:
            if (time.time() - start_time) > seconds:
        ## marketdata should now contain some interesting information
        ## Note in this implementation we overwrite the contents with each tick; we could keep them
        if iserror:
            print "Error: "+self.cb.error_msg
            print "Failed to get any prices with marketdata"
        return marketdata

So we return the last values we have populated marketdata with before we stopped (although again I could take the medians).

By the way it can be a bit dangerous to average prices too much; for example if you sample prices throughout the day and then take an average as your input into your trading algorithm you will underestimate the actual amount of volatility in the market. Similarly if you are trading high frequency stuff you will be using the active state of the order book and averaging average real time bars is probably not going to be a very wise thing to do. Over this short time period relative to my typical trading speed however its probably okay as mostly all we are going to be removing is a little illusory volatility caused by 'bid-ask' bounce.

Also even with this averaging its still worth running your prices through a 'jump detector' to make sure you don't trade off dirty prices showing spuriously large moves; I see these about once a month for each instrument I trade!

Much much more on this subject in this post

Islands in the stream...

That is it for prices. I use the historical data function whenever I start trading a particular contract but also every day as it gets close prices. This makes my system self recovering since even if it drops for a few days I will end up with daily prices at least being infilled. Also often non actively traded contracts still have close prices, useful if you are using intra contract spreads as a data input. Just be careful how you treat intraday and closing prices if you append them together.

Much much more on this subject in this post

I use market data to get intraday prices, and when I am just about to trade to check the market is liquid enough for what I want to do (or even just to check it is open since I don't bother keeping a holidays calendar for all my markets - I wouldn't want to spend more than 10 minutes of my time a day running this system now would I?). Plus it allows me to dis-aggregate my trading costs into what is coming from the inside spread, the cost of processing / execution delays and having to drop deeper into the order book.

Next on the menu will be placing an order! I will leave the trivial task of building a system which decides what the orders will be to the reader (hint: you might want to use the price in some way).

This is the third in a series of posts. The first two posts are:

The next post is: 

Friday 4 April 2014

If fund managers are overpaid then so are chefs (not really)

There is pretty clear evidence that financial investment products do not justify the performance that they offer versus the fees they charge. Generally this is attributed to the greed of overpaid fund managers (and to declare an interest, I was a member of that tribe until last year). It may surprise people to learn that forcing fund managers to earn the same income as say a nurse would not make a huge difference. To realise why we need to think about restaurants instead. 

(I can promise that this bizzare analogy will make sense in about 2 pages time but you are just going to have to trust me until then)

The rip off restaurant meal

I took my wife and children out to a restaurant for mothers day. I can't remember exactly what the bill came to; it was a niceish place in the outskirts of London but we only had a couple of glasses of wine between us so probably about £20 per head. If we had gone and cooked that same meal at home it would probably have only cost two or three pounds a head to buy the food. Were I to be a really ruthless economist and factor in the cost of gas, elecricity and a prorata share of the cost of our mortgage.... it would still be a lot short of £20 per head.

You might assume that the reason it costs so much is that there is a greedy overpaid chef in the kitchen. My brother in law works as an executive chef and whilst he does okay, last time I checked he did not have a second home in the Swiss Alps or a helicopter (He does work ridiculously long hours so he has that in common with at least some people in the finance industry). In fact the hospitality industry is notirousliy badly paid and the proportionate cost of the Chefs wages in that restaurant would only have been a small fraction of my £20. So then the meal was a complete rip off wasn't it?

Its the frictions, stupid

To get back to that £20 a head we need to factor in rent, business rates, VAT, the cost of other staff both front and back of house, deprecation, national insurance, insurance insurance, stationary, all kinds of costs you only find out about once you have run your own business.... and a profit to the owners of the restaurant that might just stretch to £1 per head (although possibly not even that). The difference between the cost of paying the chef plus the pure 'raw' cost of the food, and the total cost of the meal, might be described as a frictional cost.

You might conclude then that it makes no economic sense to pay anyone to do anything, and indeed for it to happen there must be something else going on.

Firstly we could need to bring in the concept of additional utility enjoyed from not cooking the meal yourself. You may enjoy an activity more if someone else does it for you or it may be that the person doing it enjoys it more than you (or at least dislikes it least). Before you start sniggering I am not thinking about what you are thinking about. I am thinking of things like cooking (nicer if someone else does it or the cook may be very passionate about doing it wheras you just want to eat) or plumbing (plumber may dislike it less than you).

Secondly clearly for people who have very high incomes (like greedy fund managers), or for above average income people buying a service provided by somone with a very low income does make economic sense; i.e. there are favourable pay differentials.

Also if you are buying something you simply cannot do yourself then it would make sense to buy it in of course. Like fixing a dishwasher - something most people cannot do. If you are a dishwasher engineer reading this right now, presumably from the comfort of your yacht in the cote d'azure, allI will say is I hope your bilge pump fails so you know how it feels.

(Readers should know by now that by examples are entirely arbitrary and totally unconnected to my real life. And that I have a very highly ironic tone of writing.)

A weaker form of this is that the service is something you can do, but the other person can do better than you. Many other forms of DIY fall into this bracket, at least for me. Painters and decorators all over the country are not worried about not getting my business. I can lift up a paint brush and move it over a door frame but the end result is not attractive.

Finally there are often what economists call economies of scale – it is often cheaper to provide things on a larger scale. Buying 25 kilos of prime steak every day for your restaurant will give you a lower price than buying a quarter a kilo a week.

Lets be honest this is a terrible analogy

Okay the fund management industry is not the same as running a restaurant. When you buy an ISA you are never offered a bread roll or a little bowl of olives. Also in the fund business the amounts of money are larger in absolute terms. A michelin star chef can only deal with a couple of hundred covers before the quality starts to slip; a top fund manager can look after hundreds of millions of pounds as easily as looking after ten million. More of the revenue does to go pay staff, and that revenue is shared more disproportionally than in the restaurant business with fund managers, traders and other big swinging dicks getting a large percentage than the head chef of a restaurant normally would.

So there are even now far more people in finance earning six, seven, eight and very occasionally nine figure incomes than in restaurants (discounting the celebrity chefs who do little actual cooking and so can 'run' a dozen restaurants; and even Gordon Ramsey would not break into a list of the top 100 wealthiest hedge fund managers). This is not then a case of favourable pay differentials except perhaps for the ultra high net worth customers.

Let us take a very simple financial product as an example, an index tracker. This is the most simple product that exists; the equivalent of a salad or pate on toast. To cook this particular dish at home you need access to information about the index, eg the FTSE 100. You need to know what shares are in the index and in what weights. This is available for free on various websites if its for personal use. You then need to buy the shares; and every quarter when the index is changed you might need to buy or sell a bit. You can get an online execution only broker to do this for you for between £5 and £15 per trade and perhaps a similar sized quarterly account fee.

To pay someone to do this for you wouldn't cost that much; its only a few hours a year and frankly it isn't rocket science – you can do this yourself. However arguably not everyone is a finance geek like me and so there may well be additional utility from paying someone else.

The other problem with doing this is that unless you have a lot of money you will need to hold very small, uneconomic amounts of the very smallest companies in the index; where it would cost you a very large proportionate amount to trade them each quarter. So there are economies of scale to outsourcing this activity unless you have a certain minimum level of wealth.

(You could just hold the top 30 shares of course and still get very similar performance, or hold them in equal weights and get better performance, but that is for another post and still requires larger amounts of money to invest than most people have)

So instead you go and buy a FTSE 100 tracker, eithier as a unit trust or ETF. The people running it will try and introduce some rocket science; algorithims to reduce tracking error and 'smart' execution benchmarking against VWAP and other such nonsense. You might have half a dozen people working on this 'alpha generating' activity all pretty well paid. It is all a bit futile since tracking an index of this nature is like joining a queue of people all trying to get into the same shop at the same time and a bit of jostling in the queue isn't going to help. Its better to join a different queue entirely, but thats another story. I don't think then personally that this is something the rocket scientists can do a lot better than Joe Public.

However spread over a several hundred million pound fund these extra rocket scientists are hardly going to add very much to the cost of the fund. If anything there should be some further economies of scale savings since the institutions will be paying lower dealing commissions (though due to their size their total execution bill might be higher). But then we do have that huge slew of frictional costs.

There is a lot of administration in looking after other peoples money; and admin doesn't automatically scale well to looking after larger amounts of money unlike the actual fund management activity. If all those accounts belong to smaller retail customers having a billion pounds than a million means 1000x more accounts; possibly not 1000x more staff but certainly a good deal more. Advertising takes up a big chunk although you wouldn't have thought a simple tracker would need to advertise much, but they still do. Then there are compliance costs; the greater the distrust we have in the finance sector the more people they need to employ to make sure the other people aren't breaking the rules. And they would also have to make a profit; though outside of hedge funds the profits in fund management have never been as good as in say the investment banking industry (rightly so they aren't committing their own capital but just earning off other peoples).

And its also about the intermediation, idiot

One category of costs to the fund management business we haven't talked about is what are euphimistically known as distribution costs. Many retail customers didn't directly invest in funds but went through financial advisors. Until the FCA reformed the industry rather than pay a fair price for this advice the funds paid commissions to the advisors directly, making it quite dificult for the advisors to work out which was really the best fund and which just paid the best commission. These are so called intermediation costs; the costs of not dealing directly with the fund manager but with some intermediary.

(I can show you with a few equations and some data that the best predictor of the performance of any investment are its costs, which are known with a much higher degree of certainty than any other variable. Therefore this was exactly the wrong thing to do. But you don't need the maths, this is common sense.)

Similar kinds of intermediation costs exist with funds of funds (funds that invest in other funds which exist in both the retail and the institional world), 'smart platforms' (that invest in index funds according to some algorithim), with institutional investment consultants, insurance companies, private banks and I could go on. This layering of intermediaries may be adding some value, but any value is dwarfed by the fact you have to pay restaurant costs to each layer. Each layer has their admin, tax, compliance and other costs to pay.

I won't bang on about it because there has been plenty of recent media discussion on this. If I can take just one example from the industry I used to work in; the systematic 'CTA' flavour of hedge funds. Historically these were mostly invested in by rich individuals who went through private banks, financial advisors and/or fund of funds. More recently 'normal' people have been going via so called UCITS funds which are essentially a loophole to allow people to invest in hedge funds except they are often very expensive (loopholes don't come cheap). These multiple intermediations were fine when the products were making outsized returns but once returns dropped (even if still relativelyattractive on a return:risk basis compared to other investments) they resulted in investors seeing poor performance. High returns (especially if they are mediocre returns in an enviroment of low inflation) have a great way of masking high fees.

So the cry goes up 'something must be done'. But what?


At least nothing by the government.

This isn't something the government can do for you, although there should be much better transparency around fees and fund manager fees. If you have been reading carefully you will see I don't think increased regulation is the answer. Regulation won't remove restaurant costs. If anything the government might need to think about removing regulations that encourage or force institutional investors to go through intermediaries that add no value.

I am wary of overburdening teachers with more iniatives but we don't help ourselves when we produce consumers that are not all comfortable with maths and have low levels of financial literacy.

The industry itself can also help by providing cheaper simpler more transparent products that do not involve intermediation. But they won't do that without pressure from customers; attempts by the government to force this through in the past have failed (does anyone remember CAT standards?).

It is something you have to do for yourself. Because people are getting older, because of the demise of the paternalistic final salary pension in the private sector; with the end to compulsory annuitisation and with the state pension unlikely to ever cover more than a very modest standard of living. These things all mean you are going to have to take much more responsibility for your own savings and investments. You need to do more stuff yourself, and only outsource when economies of scale really make it much cheaper or impossible to do yourself.

It takes much less time than you think. It is not as complicated as you think; much of the complexity that exists is unncessary. It is easier than you think- not as hard as fixing a dishwasher. And it is not as dull and boring as you might think. It might even be as much fun as cooking.

Tuesday 1 April 2014

Getting prices out of the IB API with swigibpy / python


There is now a native python API for interactive brokers. An updated version of this post using the native API can be found here

Okay so you have managed to run the time telling code in my last post.

Now we will do something a bit more interesting, get some market prices. Arguably it is still not that interesting, and this stuff will never be as interesting as a decent book or a good bottle of red wine, but we've all got to get the money to buy those books and bottles and this is better than many other options for achieving that.

Let us peruse the IB ABI instructions...

Whenever you are trying something new out with the API you need to identify the EClientSocket (command to ask the server for stuff) and EWrapper (command run by the wrapper client when stuff is presented unto it) functions. So Reading The Frigging Manual (to be precise the C++ section) is worth doing.

And indeed there is not just one but four places in the weird and mysterious world of EClientSocket where it looks like prices could be found:

  • Market Data
  • Historical Data
  • Real Time Bars
  • Market Depth

Market Data: Returns a stream of price ticks (updated quotes and trades). Because it is a stream you have to ask for it, and then ask for it to stop lest you be buried under a huge pile of ticks. It is level 1, eg 'top of the order book' data.

Historical Data: Returns a one off chunk of data, both prices and volumes, with a given look back history and frequency.

Real Time Bars: Returns a stream of averages of the trades (or bid, ask, midpoint) over an interval, currently only 5 seconds. So its kind of like pre-clumped tick data arriving at regular intervals. Its a stream, so you have to ask for it... you get the idea.

Market Depth: Returns a stream of updates to the order book, both prices and size. This requires a requires a level 2 subscription. Its a stream...!

Because real time bars, market data and market depth deliver very similar information I won't discuss them all (and since I am a tightwad I don't have any Level 2 data subscriptions!). In this post we'll go through the 'one shot' historical data; and in the next one we'll look at one kind of streaming data (market data).

Before we begin

You need to get another python library, pandas. This should also get numpy (for numerical analysis) and matplotlib (for drawing nice pictures) if you didn't have those in your python installation already.

Historical Data AKA Those Who Cannot Remember the Past Are Doomed...

First get the code from this repo somewhere on a server far far away... the files we'll be using today are test2_IB and wrapper_v2 (with a guest appearance from

 Getting some price data 

 Just run the test2_IB file (assuming you still have an IB connection available as per the previous post). You should get something like this:

                    close    high        low        open  volume
2013-12-12  6334.0  6338.5  6338.5  6338.5       1
2014-01-02  6610.5  6625.0  6625.0  6625.0       1
2014-01-06  6621.0  6632.5  6628.0  6632.5       8
2014-01-07  6644.5  6658.5  6632.0  6632.0       2


2014-03-25  6537.0  6569.0  6467.0  6474.5  110350
2014-03-26  6544.5  6587.5  6498.5  6556.5  108996
2014-03-27  6530.5  6538.0  6502.0  6507.5   94293
2014-03-28  6555.0  6576.0  6527.5  6536.5   91171
2014-03-31  6569.0  6602.0  6556.5  6563.5   50112

Obviously these are the prices for the FTSE 100 June 2014 futures contract. If you get an error you might need to change the expiry date for the contract. I'll show you how to do this tommorrow.

(Note before mixing these with any kind of other prices or doing proper backtesting I would personally append a 'fixed' end of date timestamp to them eg 23:59:59 to ensure there is no slight look forward bias. This is discussed more here.)

  The IB contract object or How IB Knows What We Want

Some more code from the test2_IB file:

    ibcontract = IBcontract()
    ibcontract.secType = "FUT"


(Note if you got an error above here is your chance to change the expiry to the current one. So for example as I'm revising this in July 2015 the next quarterly FTSE expiry will be "201509".)

Here is another 'gotcha' - the IB contract object (when I say 'gotcha' it can be stuff I've literally spent days banging my metaphorical head against and in some cases my actual head). This is a complete description which identifies a particular thing you can trade. Most of the data you need is available from the product listing pages. Some problems I have (there may be more associated with other asset classes) mainly in not giving enough information to uniquely identify a contract:

  • Getting the exchange name right - it needs to be exactly the same as on the product page. 
  • Having to specify a currency. This seems to be a problem for CAC40 and AEX futures (specify ibcontract.currency="EUR") and silver (USD).
  • Having to specify a multiplier effectively contract movement in cash terms per price point (For CAC ibcontract.multiplier="10", AEX is 200 and SILVER is 5000).
Note you don't need to specify the 'ConId' (a long meaningless number) which changes per contract roll, thank goodness.

(By the way I don't actually use this contract object in the rest of my code. Instead I have another class which contains more useful information, and then a convert_to_ibcontract function. This function handles the horrible special cases above and also translates between meaningless names like Z and better names like JohnBoy - my own pet name for the FTSE100 future *).

 (* This is a joke)

There's actually even more horribleness about these objects but that is for another day.

All this is Fascinating Rob But How do we Actually get the Prices?!

Here is some code from the IB client object in

    def get_IB_historical_data(self, ibcontract, durationStr="1 Y", barSizeSetting="1 day", tickerid=MEANINGLESS_NUMBER):
        Returns historical prices for a contract, up to today
        tws is a result of calling IBConnector()

        # Request some historical data.
                tickerid,                                          # tickerId,
                ibcontract,                                   # contract,
                today.strftime("%Y%m%d %H:%M:%S %Z"),       # endDateTime,
                durationStr,                                      # durationStr,
                barSizeSetting,                                    # barSizeSetting,
                "TRADES",                                   # whatToShow,
                1,                                          # useRTH,
                1                                           # formatDate
        while not finished and not iserror:
            if (time.time() - start_time) > MAX_WAIT:
        if iserror:
            print self.cb.error_msg
            raise Exception("Problem getting historic data")
        return results

The general structure of this function should be obvious to anyone who read the last post. We:

  • call some init functions in the callback instance (self.cb) so there is somewhere for error messages and our price data to go
  • Ask the IB API server for something with tws.reqHistoricalData()
  • Wait patiently until a callback finished flag is set, or we get an error, or we get bored 
  • Complain about an error, or return the results as appropriate 

The tws.reqHistoricalData() asks the server for historical price data. We can set length of time (a year in this case, although there is unlikely to be that much for a futures contract that rolls quarterly and where nobody trades the back months, being like sad lonely people at a part that no one dances with) durationStr, the end time, the bar size (in this case days; not actually the size of the kind of bar you get drinks from) and some other things you can look at the documentation yourself to find out about. Just to note not all the combinations of time period and bar length are permitted - you can't get second by second prices for a year (see this link.)

About the only thing that might not be obvious is tickerid. You might be the kind of wild crazy person who requests loads of historical data for different contracts. In which case you would get data coming back randomly and you wouldn't know which contract it was for. So you have to provide some kind of label for the request if you request loads of things at once. Note you can't actually do that because if you request more than 1 set of data about every 10 seconds you get a 'pacing violation'  (basically, don't take the mickey).

Oh Yeah I remember now, we have to have an EWrapper function on the server side... 

From the IBWrapper class in wrapper_v2:

    def init_historicprices(self, tickerid):
        if "data_historicdata" not in dir(self):
        setattr(self, "data_historicdata", histdict)
        setattr(self, "flag_historicdata_finished", False)

    def historicalData(self, reqId, date, openprice, high,
                       low, close, volume,
                       barCount, WAP, hasGaps):

        if date[:8] == 'finished':
            setattr(self, "flag_historicdata_finished", True)
            historicdata.add_row(date=date, open=openprice, high=high, low=low, close=close, volume=volume)

Essentially to write this stuff you look in the API reference to see what the wrapper function returns. In this case the wrapper function beastie is called repeatedly with the same arguments for each row of data, before you get one final and awesomely inelegant final call to say that we are finished. To compensate for this inelegance I use a little class object in to tidily gather the rows into a pandas dataframe.

That then is everything you need to know about historical data. I will cover market data, i.e. the wonderful world of streaming data, in the next post.

This is the second in a series of posts.

The previous post was:

The next post is: