Getting prices out of the IB API with swigibpy / python

NOTE:

There is now a native python API for interactive brokers. An updated version of this post using the native API can be found here

Okay so you have managed to run the time telling code in my last post.

Now we will do something a bit more interesting, get some market prices. Arguably it is still not that interesting, and this stuff will never be as interesting as a decent book or a good bottle of red wine, but we've all got to get the money to buy those books and bottles and this is better than many other options for achieving that.

Let us peruse the IB ABI instructions...

Whenever you are trying something new out with the API you need to identify the EClientSocket (command to ask the server for stuff) and EWrapper (command run by the wrapper client when stuff is presented unto it) functions. So Reading The Frigging Manual (to be precise the C++ section) is worth doing.

And indeed there is not just one but four places in the weird and mysterious world of EClientSocket where it looks like prices could be found:

• Market Data
• Historical Data
• Real Time Bars
• Market Depth

Market Data: Returns a stream of price ticks (updated quotes and trades). Because it is a stream you have to ask for it, and then ask for it to stop lest you be buried under a huge pile of ticks. It is level 1, eg 'top of the order book' data.

Historical Data: Returns a one off chunk of data, both prices and volumes, with a given look back history and frequency.

Real Time Bars: Returns a stream of averages of the trades (or bid, ask, midpoint) over an interval, currently only 5 seconds. So its kind of like pre-clumped tick data arriving at regular intervals. Its a stream, so you have to ask for it... you get the idea.

Market Depth: Returns a stream of updates to the order book, both prices and size. This requires a requires a level 2 subscription. Its a stream...!


Because real time bars, market data and market depth deliver very similar information I won't discuss them all (and since I am a tightwad I don't have any Level 2 data subscriptions!). In this post we'll go through the 'one shot' historical data; and in the next one we'll look at one kind of streaming data (market data).

Before we begin

You need to get another python library, pandas. This should also get numpy (for numerical analysis) and matplotlib (for drawing nice pictures) if you didn't have those in your python installation already.

Historical Data AKA Those Who Cannot Remember the Past Are Doomed...

First get the code from this repo somewhere on a server far far away... the files we'll be using today are test2_IB and wrapper_v2 (with a guest appearance from IButils.py).

 Getting some price data 

 Just run the test2_IB file (assuming you still have an IB connection available as per the previous post). You should get something like this:

                    close    high        low        open  volume
2013-12-12  6334.0  6338.5  6338.5  6338.5       1
2014-01-02  6610.5  6625.0  6625.0  6625.0       1
2014-01-06  6621.0  6632.5  6628.0  6632.5       8
2014-01-07  6644.5  6658.5  6632.0  6632.0       2

.....

2014-03-25  6537.0  6569.0  6467.0  6474.5  110350
2014-03-26  6544.5  6587.5  6498.5  6556.5  108996
2014-03-27  6530.5  6538.0  6502.0  6507.5   94293
2014-03-28  6555.0  6576.0  6527.5  6536.5   91171
2014-03-31  6569.0  6602.0  6556.5  6563.5   50112

Obviously these are the prices for the FTSE 100 June 2014 futures contract. If you get an error you might need to change the expiry date for the contract. I'll show you how to do this tommorrow.

(Note before mixing these with any kind of other prices or doing proper backtesting I would personally append a 'fixed' end of date timestamp to them eg 23:59:59 to ensure there is no slight look forward bias. This is discussed more here.)

  The IB contract object or How IB Knows What We Want

Some more code from the test2_IB file:

    ibcontract = IBcontract()
    ibcontract.secType = "FUT"
    ibcontract.expiry="201406"
    ibcontract.symbol="Z"
    ibcontract.exchange="LIFFE" 

(Note if you got an error above here is your chance to change the expiry to the current one. So for example as I'm revising this in July 2015 the next quarterly FTSE expiry will be "201509".)

Here is another 'gotcha' - the IB contract object (when I say 'gotcha' it can be stuff I've literally spent days banging my metaphorical head against and in some cases my actual head). This is a complete description which identifies a particular thing you can trade. Most of the data you need is available from the https://www.interactivebrokers.com/ product listing pages. Some problems I have (there may be more associated with other asset classes) mainly in not giving enough information to uniquely identify a contract:

• Getting the exchange name right - it needs to be exactly the same as on the product page. 
• Having to specify a currency. This seems to be a problem for CAC40 and AEX futures (specify ibcontract.currency="EUR") and silver (USD).
• Having to specify a multiplier effectively contract movement in cash terms per price point (For CAC ibcontract.multiplier="10", AEX is 200 and SILVER is 5000).
   

Note you don't need to specify the 'ConId' (a long meaningless number) which changes per contract roll, thank goodness.

(By the way I don't actually use this contract object in the rest of my code. Instead I have another class which contains more useful information, and then a convert_to_ibcontract function. This function handles the horrible special cases above and also translates between meaningless names like Z and better names like JohnBoy - my own pet name for the FTSE100 future *).

 (* This is a joke)


There's actually even more horribleness about these objects but that is for another day.

All this is Fascinating Rob But How do we Actually get the Prices?!

Here is some code from the IB client object in wrapper_v2.py:


    def get_IB_historical_data(self, ibcontract, durationStr="1 Y", barSizeSetting="1 day", tickerid=MEANINGLESS_NUMBER):
      
        """
        Returns historical prices for a contract, up to today
      
        tws is a result of calling IBConnector()
      
        """

        today=datetime.datetime.now()

        self.cb.init_error()
        self.cb.init_historicprices(tickerid)
          
        # Request some historical data.
        self.tws.reqHistoricalData(
                tickerid,                                          # tickerId,
                ibcontract,                                   # contract,
                today.strftime("%Y%m%d %H:%M:%S %Z"),       # endDateTime,
                durationStr,                                      # durationStr,
                barSizeSetting,                                    # barSizeSetting,
                "TRADES",                                   # whatToShow,
                1,                                          # useRTH,
                1                                           # formatDate
            )
      
        start_time=time.time()
        finished=False
        iserror=False
      
        while not finished and not iserror:
            finished=self.cb.flag_historicdata_finished
            iserror=self.cb.flag_iserror
          
            if (time.time() - start_time) > MAX_WAIT:
                iserror=True
            pass
          
        if iserror:
            print self.cb.error_msg
            raise Exception("Problem getting historic data")
      
        historicdata=self.cb.data_historicdata[tickerid]
        results=historicdata.to_pandas("date")
      
        return results
The general structure of this function should be obvious to anyone who read the last post. We:

• call some init functions in the callback instance (self.cb) so there is somewhere for error messages and our price data to go
• Ask the IB API server for something with tws.reqHistoricalData()
• Wait patiently until a callback finished flag is set, or we get an error, or we get bored 
• Complain about an error, or return the results as appropriate 

The tws.reqHistoricalData() asks the server for historical price data. We can set length of time (a year in this case, although there is unlikely to be that much for a futures contract that rolls quarterly and where nobody trades the back months, being like sad lonely people at a part that no one dances with) durationStr, the end time, the bar size (in this case days; not actually the size of the kind of bar you get drinks from) and some other things you can look at the documentation yourself to find out about. Just to note not all the combinations of time period and bar length are permitted - you can't get second by second prices for a year (see this link.)

About the only thing that might not be obvious is tickerid. You might be the kind of wild crazy person who requests loads of historical data for different contracts. In which case you would get data coming back randomly and you wouldn't know which contract it was for. So you have to provide some kind of label for the request if you request loads of things at once. Note you can't actually do that because if you request more than 1 set of data about every 10 seconds you get a 'pacing violation'  (basically, don't take the mickey).

Oh Yeah I remember now, we have to have an EWrapper function on the server side... 

From the IBWrapper class in wrapper_v2:

    def init_historicprices(self, tickerid):
        if "data_historicdata" not in dir(self):
            histdict=dict()
        else:
            histdict=self.data_historicdata
       
        histdict[tickerid]=EMPTY_HDATA
        setattr(self, "data_historicdata", histdict)
        setattr(self, "flag_historicdata_finished", False)

    def historicalData(self, reqId, date, openprice, high,
                       low, close, volume,
                       barCount, WAP, hasGaps):
       

        if date[:8] == 'finished':
            setattr(self, "flag_historicdata_finished", True)
        else:
            historicdata=self.data_historicdata[reqId]
            date=datetime.datetime.strptime(date,"%Y%m%d")
            historicdata.add_row(date=date, open=openprice, high=high, low=low, close=close, volume=volume)
Essentially to write this stuff you look in the API reference to see what the wrapper function returns. In this case the wrapper function beastie is called repeatedly with the same arguments for each row of data, before you get one final and awesomely inelegant final call to say that we are finished. To compensate for this inelegance I use a little class object in IButils.py to tidily gather the rows into a pandas dataframe.

That then is everything you need to know about historical data. I will cover market data, i.e. the wonderful world of streaming data, in the next post.




This is the second in a series of posts.


The previous post was:
http://qoppac.blogspot.co.uk/2014/03/using-swigibpy-so-that-python-will-play.html


The next post is:
http://qoppac.blogspot.co.uk/2014/04/streaming-prices-from-ib-api-swigibpy.html