Tuesday, 1 April 2014

Getting prices out of the IB API with swigibpy / python

NOTE:

There is now a native python API for interactive brokers. An updated version of this post using the native API can be found here

Okay so you have managed to run the time telling code in my last post.

Now we will do something a bit more interesting, get some market prices. Arguably it is still not that interesting, and this stuff will never be as interesting as a decent book or a good bottle of red wine, but we've all got to get the money to buy those books and bottles and this is better than many other options for achieving that.


Let us peruse the IB ABI instructions...

Whenever you are trying something new out with the API you need to identify the EClientSocket (command to ask the server for stuff) and EWrapper (command run by the wrapper client when stuff is presented unto it) functions. So Reading The Frigging Manual (to be precise the C++ section) is worth doing.

And indeed there is not just one but four places in the weird and mysterious world of EClientSocket where it looks like prices could be found:



  • Market Data
  • Historical Data
  • Real Time Bars
  • Market Depth


Market Data: Returns a stream of price ticks (updated quotes and trades). Because it is a stream you have to ask for it, and then ask for it to stop lest you be buried under a huge pile of ticks. It is level 1, eg 'top of the order book' data.

Historical Data: Returns a one off chunk of data, both prices and volumes, with a given look back history and frequency.

Real Time Bars: Returns a stream of averages of the trades (or bid, ask, midpoint) over an interval, currently only 5 seconds. So its kind of like pre-clumped tick data arriving at regular intervals. Its a stream, so you have to ask for it... you get the idea.

Market Depth: Returns a stream of updates to the order book, both prices and size. This requires a requires a level 2 subscription. Its a stream...!


Because real time bars, market data and market depth deliver very similar information I won't discuss them all (and since I am a tightwad I don't have any Level 2 data subscriptions!). In this post we'll go through the 'one shot' historical data; and in the next one we'll look at one kind of streaming data (market data).


Before we begin



You need to get another python library, pandas. This should also get numpy (for numerical analysis) and matplotlib (for drawing nice pictures) if you didn't have those in your python installation already.


Historical Data AKA Those Who Cannot Remember the Past Are Doomed...

First get the code from this repo somewhere on a server far far away... the files we'll be using today are test2_IB and wrapper_v2 (with a guest appearance from IButils.py).

 Getting some price data 

 Just run the test2_IB file (assuming you still have an IB connection available as per the previous post). You should get something like this:

                    close    high        low        open  volume
2013-12-12  6334.0  6338.5  6338.5  6338.5       1
2014-01-02  6610.5  6625.0  6625.0  6625.0       1
2014-01-06  6621.0  6632.5  6628.0  6632.5       8
2014-01-07  6644.5  6658.5  6632.0  6632.0       2


.....

2014-03-25  6537.0  6569.0  6467.0  6474.5  110350
2014-03-26  6544.5  6587.5  6498.5  6556.5  108996
2014-03-27  6530.5  6538.0  6502.0  6507.5   94293
2014-03-28  6555.0  6576.0  6527.5  6536.5   91171
2014-03-31  6569.0  6602.0  6556.5  6563.5   50112




Obviously these are the prices for the FTSE 100 June 2014 futures contract. If you get an error you might need to change the expiry date for the contract. I'll show you how to do this tommorrow.

(Note before mixing these with any kind of other prices or doing proper backtesting I would personally append a 'fixed' end of date timestamp to them eg 23:59:59 to ensure there is no slight look forward bias. This is discussed more here.)

  The IB contract object or How IB Knows What We Want

Some more code from the test2_IB file:

    ibcontract = IBcontract()
    ibcontract.secType = "FUT"

    ibcontract.expiry="201406"
    ibcontract.symbol="Z"
    ibcontract.exchange="LIFFE"
 


(Note if you got an error above here is your chance to change the expiry to the current one. So for example as I'm revising this in July 2015 the next quarterly FTSE expiry will be "201509".)

Here is another 'gotcha' - the IB contract object (when I say 'gotcha' it can be stuff I've literally spent days banging my metaphorical head against and in some cases my actual head). This is a complete description which identifies a particular thing you can trade. Most of the data you need is available from the https://www.interactivebrokers.com/ product listing pages. Some problems I have (there may be more associated with other asset classes) mainly in not giving enough information to uniquely identify a contract:

  • Getting the exchange name right - it needs to be exactly the same as on the product page. 
  • Having to specify a currency. This seems to be a problem for CAC40 and AEX futures (specify ibcontract.currency="EUR") and silver (USD).
  • Having to specify a multiplier effectively contract movement in cash terms per price point (For CAC ibcontract.multiplier="10", AEX is 200 and SILVER is 5000).
     
Note you don't need to specify the 'ConId' (a long meaningless number) which changes per contract roll, thank goodness.

(By the way I don't actually use this contract object in the rest of my code. Instead I have another class which contains more useful information, and then a convert_to_ibcontract function. This function handles the horrible special cases above and also translates between meaningless names like Z and better names like JohnBoy - my own pet name for the FTSE100 future *).

 (* This is a joke)


There's actually even more horribleness about these objects but that is for another day.


All this is Fascinating Rob But How do we Actually get the Prices?!

Here is some code from the IB client object in wrapper_v2.py:


    def get_IB_historical_data(self, ibcontract, durationStr="1 Y", barSizeSetting="1 day", tickerid=MEANINGLESS_NUMBER):
      
        """
        Returns historical prices for a contract, up to today
      
        tws is a result of calling IBConnector()
      
        """

        today=datetime.datetime.now()

        self.cb.init_error()
        self.cb.init_historicprices(tickerid)
          
        # Request some historical data.
        self.tws.reqHistoricalData(
                tickerid,                                          # tickerId,
                ibcontract,                                   # contract,
                today.strftime("%Y%m%d %H:%M:%S %Z"),       # endDateTime,
                durationStr,                                      # durationStr,
                barSizeSetting,                                    # barSizeSetting,
                "TRADES",                                   # whatToShow,
                1,                                          # useRTH,
                1                                           # formatDate
            )
      
        start_time=time.time()
        finished=False
        iserror=False
      
        while not finished and not iserror:
            finished=self.cb.flag_historicdata_finished
            iserror=self.cb.flag_iserror
          
            if (time.time() - start_time) > MAX_WAIT:
                iserror=True
            pass
          
        if iserror:
            print self.cb.error_msg
            raise Exception("Problem getting historic data")
      
        historicdata=self.cb.data_historicdata[tickerid]
        results=historicdata.to_pandas("date")
      
        return results

The general structure of this function should be obvious to anyone who read the last post. We:

  • call some init functions in the callback instance (self.cb) so there is somewhere for error messages and our price data to go
  • Ask the IB API server for something with tws.reqHistoricalData()
  • Wait patiently until a callback finished flag is set, or we get an error, or we get bored 
  • Complain about an error, or return the results as appropriate 

The tws.reqHistoricalData() asks the server for historical price data. We can set length of time (a year in this case, although there is unlikely to be that much for a futures contract that rolls quarterly and where nobody trades the back months, being like sad lonely people at a part that no one dances with) durationStr, the end time, the bar size (in this case days; not actually the size of the kind of bar you get drinks from) and some other things you can look at the documentation yourself to find out about. Just to note not all the combinations of time period and bar length are permitted - you can't get second by second prices for a year (see this link.)

About the only thing that might not be obvious is tickerid. You might be the kind of wild crazy person who requests loads of historical data for different contracts. In which case you would get data coming back randomly and you wouldn't know which contract it was for. So you have to provide some kind of label for the request if you request loads of things at once. Note you can't actually do that because if you request more than 1 set of data about every 10 seconds you get a 'pacing violation'  (basically, don't take the mickey).

Oh Yeah I remember now, we have to have an EWrapper function on the server side... 

From the IBWrapper class in wrapper_v2:

    def init_historicprices(self, tickerid):
        if "data_historicdata" not in dir(self):
            histdict=dict()
        else:
            histdict=self.data_historicdata
       
        histdict[tickerid]=EMPTY_HDATA
        setattr(self, "data_historicdata", histdict)
        setattr(self, "flag_historicdata_finished", False)

    def historicalData(self, reqId, date, openprice, high,
                       low, close, volume,
                       barCount, WAP, hasGaps):
       

        if date[:8] == 'finished':
            setattr(self, "flag_historicdata_finished", True)
        else:
            historicdata=self.data_historicdata[reqId]
            date=datetime.datetime.strptime(date,"%Y%m%d")
            historicdata.add_row(date=date, open=openprice, high=high, low=low, close=close, volume=volume)

Essentially to write this stuff you look in the API reference to see what the wrapper function returns. In this case the wrapper function beastie is called repeatedly with the same arguments for each row of data, before you get one final and awesomely inelegant final call to say that we are finished. To compensate for this inelegance I use a little class object in IButils.py to tidily gather the rows into a pandas dataframe.

That then is everything you need to know about historical data. I will cover market data, i.e. the wonderful world of streaming data, in the next post.




This is the second in a series of posts.


The previous post was:
http://qoppac.blogspot.co.uk/2014/03/using-swigibpy-so-that-python-will-play.html


The next post is:
http://qoppac.blogspot.co.uk/2014/04/streaming-prices-from-ib-api-swigibpy.html

22 comments:

  1. Hey Rob,

    sometimes when I do have an open position in the contract that I want to get the details, when I call client.get_contract_details(ibcontract) it returns the following error message:
    ....get_contract_details
    raise Exception("Problem getting details")

    Exception: Problem getting details

    Do you know whats going on?

    Regards

    ReplyDelete
    Replies
    1. Can you post (or email me) the full stack trace please?

      Delete
    2. I got the same error just now. Had to change the ibcontract.secType/symbol/exchange as I don't have subscription to FTSE data. I changed test2_IB to the following:

      ibcontract = IBcontract()
      ibcontract.secType = "STK"
      # ibcontract.expiry=""
      ibcontract.symbol="AAPL"
      ibcontract.exchange="NASDAQ"



      Here's the traceback on ipython:

      Traceback (most recent call last):

      File "", line 1, in
      runfile('C:/Python27/User/swigibpy/ibswigsystematicexamples-master/test2_IB.py', wdir='C:/Python27/User/swigibpy/ibswigsystematicexamples-master')

      File "C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 685, in runfile
      execfile(filename, namespace)

      File "C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 71, in execfile
      exec(compile(scripttext, filename, 'exec'), glob, loc)

      File "C:/Python27/User/swigibpy/ibswigsystematicexamples-master/test2_IB.py", line 19, in
      ans=client.get_IB_historical_data(ibcontract)

      File "sysIB\wrapper_v2.py", line 150, in get_IB_historical_data
      raise Exception("Problem getting historic data")

      Exception: Problem getting historic data

      Delete
    3. VitC - to be clear, do you still have a problem or did changing the contract work?

      Delete
    4. Just fixed the problem. For IBcontract() to work for equities (US equities in my case), it needs input of secType, symbol, exchange, primaryExchange, and currency.

      For Excel DDE the primaryExchange can 'sometimes' be left blank.

      Thanks much Rob for the help!

      p.s., where can I find a more comprehensive API doc for Interactive Brokers? Its C++ API doc is sparse and limited.

      Delete
    5. "p.s., where can I find a more comprehensive API doc for Interactive Brokers? Its C++ API doc is sparse and limited."

      Not that I know of. I agree, it's poor. That's why so much trial and error is needed to get things working.

      Delete
  2. Hi Rob, Receiving the following error while running test2. Thanks.
    Traceback (most recent call last):
    File "/Users/G/Desktop/sysIB/test2_IB.py", line 20, in
    ans=client.get_IB_historical_data(ibcontract)
    File "/Library/Python/2.7/site-packages/sysIB/wrapper_v2.py", line 129, in get_IB_historical_data
    1 # formatDate
    TypeError: Required argument 'chartOptions' (pos 10) not found
    >>>

    ReplyDelete
    Replies
    1. Have you got the latest code from the git repo? https://github.com/robcarver17/ibswigsystematicexamples/blob/master/sysIB/wrapper_v2.py

      Looking at your line numbers I'm not sure you have.

      (A while ago IB changed their API to require an additional, dummy, argument when calling certain functions. This looks like the error I got before I updated the code to cope with that)

      Delete
    2. Thank you. Should have kept looking before asking as I saw your response to an identical question on the next post. I appreciate your effort in explaining swigibpy!

      Delete
  3. I'm curious about obtaining hourly bars. Is there a relatively straightforward way to do this by modifying your example code? I've been modifying your code and working with the IB API reference guide with no luck. I frequently receive "ValueError: unconverted data remains: 19:00:00". I've also added "strftime" to the end of this line: date=datetime.datetime.strptime(date,"%Y%m%d").strftime("%Y-%m-%d %H:%M:%S %Z"). A simple point in the right direction would be immensely helpful. Thanks again for your time!

    ReplyDelete
  4. WOW, figured it out several hours later. All it took was commenting out the date line. I need to learn to be more lazy.

    Wrapper_v2:

    def historicalData(self, reqId, date, openprice, high,
    low, close, volume,
    barCount, WAP, hasGaps):

    if date[:8] == 'finished': #if date[:8]
    setattr(self, "flag_historicdata_finished", True)
    else:
    historicdata=self.data_historicdata[reqId]
    #date=datetime.datetime.strptime(date,"%Y%m%d").strftime("%Y-%m-%d %H:%M:%S %Z")
    historicdata.add_row(date=date, open=openprice, high=high, low=low, close=close, volume=volume)

    ReplyDelete
    Replies
    1. Glad you worked it out. Let me know if you need anymore help.

      Delete
  5. Hi Rob - thank you so much for this resource! Extremely helpful! I was wondering how you deal with rolls as IB doesn't provide first notice days through the contract details. Is there another source you use to get first notice days or do you just roll a fixed number of days before expiration?

    ReplyDelete
    Replies
    1. Hi Cherkassky

      I roll a specific number of days before expiration. So it's down to knowing the FND offsets. Normally there is a week or so (talking about US and european bonds) when it is liquid enough to roll before the FND; I prefer to roll earlier and play safe.

      Delete
  6. Take a look at this tool. I've been using it for months and very happy with it. It lets you download historical data for any duration supported by IB API with one click. It takes care of long-duration requests by breaking them down into multiple API calls behind the scenes, and downloads historical data for multiple contracts in parallel.

    I decided to use this tool after spending days trying to work around various IB Java API issues - pacing violation errors, response size limitations, etc... Works for stocks, futures, and options. Saved me a lot of time, highly recommend it.

    http://www.tradinggeeks.net/downloads/ib-data-downloader/

    ReplyDelete
  7. Does anyone else that works on their own automated trading program live around Minneapolis? I'd like to hang out and talk about shit DJLOISELATYAHOODOTCOM

    ReplyDelete
  8. Hi Rob, thanks for all the info on Swigby. I have no problem requesting historical prices for US contracts like Corn, using the below parameters:
    ibcontract.secType = "FUT"
    ibcontract.expiry="201712"
    ibcontract.symbol="ZC"
    ibcontract.exchange="ECBOT"
    But I can't seem to figure out how to request prices for ESTX50 and V2TX on the DTB exchange. It gives me an error with the below parameters. Any ideas? Thanks.
    ibcontract.secType = "FUT"
    ibcontract.expiry="201712"
    ibcontract.symbol="ESTX50"
    ibcontract.exchange="DTB"
    ibcontract.currency="EUR"

    ReplyDelete
  9. Sorry, should have supplied the error message in my previous message: Exception: Problem getting historic data

    ReplyDelete
    Replies
    1. you need to supply the full error trace please

      Delete
  10. Please delete the post prior to this, as I wasn't connected to the IB API. Here is the error I intended to show you:
    IB error id 999 errorcode 200 string No security definition has been found for the request
    True

    Traceback (most recent call last):
    File "/home/peter/ibswigsystematicexamples/sysIB/test2_IB.py", line 21, in
    ans=client.get_IB_historical_data(ibcontract)
    File "/home/peter/ibswigsystematicexamples/sysIB/wrapper_v2.py", line 148, in get_IB_historical_data
    raise Exception("Problem getting historic data")
    Exception: Problem getting historic data
    >>>

    ReplyDelete
  11. Sorry to waste your time, Rob. I fiddled some more with my market data permissions, and now I figured it out.

    ReplyDelete