Using swigibpy so that Python will play nicely with Interactive Brokers API

The interactive brokers API is as far I know the only way that a non institutional client can access financial markets in a way which makes properly programmatic, fully flexible, fully automated trading possible. 

(I am ignoring broker supplied 'front ends' that may allow you to automate trades, and some Janet and John environments that provide the ability to implement algorithms in weird proprietary languages  for a limited number of instruments)

This is a good thing.

Now for the not so good things. Firstly the API will only run in Excel, Visual Basic Active X, Java, C++, and C#. That is a bit like buying fuel for your car that will only work if you have a toy car, an electric toy car, a BMW or one of two kinds of Land Rover, one of which is only driven in Iceland. There is no support for the 'wimpy-quants' languages of choice, R, Matlab and Python.

NOTE:

There is now a native python API for interactive brokers. An updated version of this post using the native API can be found here.

(You can of course use something like Ninja Trader as a front end, but if you're reading this you're not that much of a wimp are you?)
 
The good news is there are things like IbPy and swigibpy out there to help us Python users, with my choice of swigibpy (see the December 2013 post for why). A quick Google reveals the existence of similar packages for R and Matlab. This mostly sit between your more high level language and the C++ API.

Secondly the documentation isn't great, either for the IB API itself or swigibpy. So it takes quite a lot of fiddling around and trial and error to get things working. There are some limited examples out there on the web for example here is one of the best for IBpy; however they mostly cover the absolute basics and focus on trading individual equities. They can be also very much of the 'like here is some code, and it seems to work' without any explanation that would allow you to understand and adopt the code for your own purposes.

Because I am a nice guy and a public servant to boot I intend in this and subsequent posts to provide enough working documentation to get you up and running with an interface to IB that you can use as a starting point to build an automated trading system in python. This may also be of use to those using IB API in other non swigibpy environments.

(Assumptions: I am using Python 2.7.4; older or newer versions may break. I will assume you know your way around Python to the extent of being able to create a simple package and modules and run them. My command line examples will be for Linux but similar things ought to be possible. Also that you understand the dangers and risks of trading futures; but you will need to sign a lot of disclaimers before IB let you do this for real so I will let them worry about that.)

Getting a test account, downloading the IB TWS

To play with IB without signing up for real you will need a test account. By the way if you are serious and you get a real IB account you can also request an additional account for simulated trading.

(The test account is not very realistic, eg prices can be total garbage. The simulated account is much better although for some reason you don't always get L1 and L2 data to all the data feeds your real account is signed up to. If you are going to do any systematic trading in the near future I highly recommend signing up to IB and using a proper simulated account. It doesn't cost anything if you don't trade or use any additional data feeds.)

We aren't going to bother downloading the TWS software, which is a rather heavy front end useful for trading yourself; but the much lighter and more stable 'Gateway'.

(I advise you to also download the TWS API at some point to have a play, but I don't recommend it for day to day running of a strategy since it seems to be very unstable due to the great lardy weight of fancy ultra bloated GUI that it has to support.)

1.  Go to https://www.interactivebrokers.co.uk/en/main.php
2. Click on trading menu, API solutions
3. Under IB API click on more info
4.  Click on IB gateway software
5. Under UNIX clicks on IB gateway for Unix
6. Follow the instructions.

The last section of the instructions asks you to type some gobbledygook to run 'TWS' (they mean gateway). Don't be a fool and create a shell script that does it for you. Then you just have to type something like . runGateway. I have a similar script for TWS.

1. Select IB API radio button
2.  Under username put 'edemo' and under password put 'demouser'.

 If all goes well you will eventually see a screen with a green bar showing a connected status at the top. This program acts as a server to pass on your instructions from the API connection to the great IB server, wherever that is. You now need to configure your API to accept connections.

1. Click on the Configure menu. Go to API settings
2. Socket port - should be 4001.
3. Trusted IP addresses - should include 127.0.0.1. If it doesn't you will need to add it.
4. Read only API - you can leave this checked for now but when you come to submit trades should be unchecked.
5. Go to precautions. You might want to suppress market cap warnings here when you start trading in earnest.
6. Go to presets. Again check you are happy with the limits shown.

Unlike TWS the gateway is automatically set up to accept connections so this is all you need to do.

(There is nothing special about 4001 so you can change it but be sure to remember the number and only use a radically different number if you realise you might break Linux in the process. Check http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers and make sure your game of Microsoft Ants isn't running. I run two or more Gateways in parallel each connecting to live and test accounts. I associate a different socket (I use 4001 up to about 4005) with each account and gateway session.

127.0.0.1 is just 'this machine'. If your code is running on the same machine as the Gateway you are fine. Otherwise eg if you are on a network you will have to include the IP address of any other machines that might connect to the Gateway.)

You might want to peruse the official IB API manual. The more interesting part is the C++ section. I find it very useful if I am having trouble sleeping.

 

Downloading required python libraries

 

swigibpy

Remember swigibpy is the way that python gets to talk to the c++ API for IB. Go to this page and follow the download instructions. Either method shown works for me.

(This might fail on a missing Python.h file if you have too minimal a Linux build. Just do sudo apt -get install python-dev if you have the same problem. )

Let's write some code - okay lets get some code I've already written

Pull the source from here. The two files of interest are test1_IB.py and wrapper.py.

 

Running the example

Once you have got the two files above into a package you can just run test1_IB. If it works properly you should see something like this:

Getting the time...
1395845174

So we now seem to have acquired a very complicated way of telling unix time!

Also I appear to have committed the deadly sin of 'like here is some code, and it seems to work' without any explanation. So lets dig into this a bit more.

 

All you wanted to know about IB connections but frankly couldn't care less

 

    callback = IBWrapper()
    client=IBclient(callback)

class IBclient(object):
    def __init__(self, callback):
        tws = EPosixClientSocket(callback)
        (host, port, clientid)=return_IB_connection_info()
    <snip - code missing here>


def return_IB_connection_info():
    """
    Returns the tuple host, port, clientID required by eConnect
   
    """
   
    host=""
   
    port=4001
    clientid=999
   
    return (host, port, clientid) 

(Pedantic point - Although the code says tws here, it really means gateway. Actually the client doesn't know or care what is on the other end of the connection.)

As I said above you can have any number of IB servers (gateway or TWS session) running; so I use a function to tell me what host, port and clientid to use when connecting (here the function is rather dull). As briefly alluded to above each IB server will listen out for clients on a particular port (so these should be unique to the server). You can also have numerous clients connecting to the same server. For example I have one client picking up prices; another receiving accounting information, another managing orders, various one off clients getting diagnostic information and one doing the washing up.

If a clientid is already in use by a connected session then IB will cry. It doesn't seem to be possible to kill a connection without the python code terminating; there is a method tws.eDisconnect() but using it makes no difference, nor does setting tws=None or indeed anything else short of finding the thread running the python and killing it.

What this means if you have a series of functions you will call in a given python session, all of which connect to TWS, you can't do something like this:

    def main_function():
       
        functiona()
       
        functionb()
       
    def functiona():
        callback = IBWrapper()
        client=IBclient(callback)
       
        client.do_something()
       
    def functionb():
        callback = IBWrapper()
        client=IBclient(callback)       
        client.do_something_else()

 ... as the call to functionb will fail probably with error 509. Instead you need to do something like this:

    def main_function():        callback = IBWrapper()
        client=IBclient(callback)       
        functiona(client)
       
        functionb(client)
       

    def functiona(client):
        client.do_something()
       
    def functionb(client):
        client.do_something_else()
       
Back to the story.

    callback = IBWrapper()

    client=IBclient(callback) 

class IBclient(object):
    def __init__(self, callback):
        tws = EPosixClientSocket(callback)
        (host, port, clientid)=return_IB_connection_info()
        tws.eConnect(host, port, clientid)

        self.tws=tws 
        self.cb=callback
    <snip - code missing here>

The callback instance of IBWrapper is the beast which we have to write to handle messages coming back from the server; more of that in a second.

We have to create the connection object as an instance of EPosixClientSocket initialising it with a callback function. We then call the eConnect function so that our tws object is a fully fledged connection. If that sounds like Outer Mongolian to you, don't worry. You shouldn't ever need to mess with this stuff. The point is once we have our client object (instance of IBclient) we can make it do cool things, like tell the time.

Telling the time - the hard way

We now call the speaking clock method of the IBclient:

    print client.speaking_clock()

### how many seconds before we give up
MAX_WAIT=30 
    <snip - code missing here>


class IBclient(object): 
    <snip - code missing here>
 
    def speaking_clock(self):
        print "Getting the time... "
       
        self.tws.reqCurrentTime()
       
        start_time=time.time()
       
        self.cb.init_error()
        self.cb.init_time()

        iserror=False
        not_finished=True

        while not_finished and not iserror:
            not_finished=self.cb.data_the_time_now_is is None
            iserror=self.cb.flag_iserror

            if (time.time() - start_time) > MAX_WAIT:
                not_finished=False
               
            if iserror:
                not_finished=False
   
        if iserror:
            print "Error happened"
            print self.cb.error_msg
           
        return self.cb.data_the_time_now_is


tws.reqCurrentTime() is an example of a classy class EClientSocket functions, from the official IB API manual. These are the functions that ask the server to think about doing something.

 

Things that are interesting here #1: the wait loop 

 The issue we have here is that the IB API is very much set up as an event driven process. So not like normal sequential code like function A calling function B and function A returning the answer back. No instead we have function A just kind of hanging around waiting for function B and then somehow by magic function B just happens.

That is not the way I do stuff. I don't have to - I am running fairly slow trading systems not intraday high frequency stuff that needs to react to every tick for which an event driven system makes sense. Also it goes against my ethical beliefs. Why should function A have to wait for function B to run? What makes function B so special? Its just rudeness.

(Also I've never found debugging event driven code to be particularly easy .... )

So what we have to do is make the client-server relationship appear sequential, at least to anything sitting outside the wrapper module. That also means we need to handle the conditions of the thing not finishing in a reasonable time and finishing with an error.

Things that are interesting here #2: The contents of self.cb

All the things we are pulling out of self.cb (which is the callback we passed in when we initialised the IBclient object) set somewhere else as if by magic. Actually they get set when the IB server summons the callback, calling the appropriate method. In the official IB API manual these are the very classy Class EWrapper Functions.

There are three kinds of classy ClassE Wrapper functions / methods:

 

Pointless methods that don't do anything

class IBWrapper(EWrapper):
    """
        Callback object passed to TWS, these functions will be called directly
    by TWS.

    """
    <snip - code missing here>
       
    def nextValidId(self, orderId):
        pass
   
    def managedAccounts(self, openOrderEnd):
        pass


These methods are there because the IB server isn't very gracious about not being able to call certain methods, and they don't get inherited from EWrapper (why not?!?! yes why bloody not?!). When you start doing more interesting stuff you will find you might need more of these; you will start seeing errors like this (which you can reproduce by removing the managedAccounts method):


NotImplementedError: SWIG director pure virtual method called EWrapper::managedAccounts

 

Method that handle errors

class IBWrapper(EWrapper):
    """
        Callback object passed to TWS, these functions will be called directly
    by TWS.

    """
    def init_error(self):
        setattr(self, "flag_iserror", False)
        setattr(self, "error_msg", "")

    def error(self, id, errorCode, errorString):
        """
        error handling, simple for now
      
        Here are some typical IB errors
        INFO: 2107, 2106
        WARNING 326 - can't connect as already connected
        CRITICAL: 502, 504 can't connect to TWS.
            200 no security definition found
            162 no trades

        """

        ## Any errors not on this list we just treat as information
        ERRORS_TO_TRIGGER=[201, 103, 502, 504, 509, 200, 162, 420, 2105, 1100, 478, 201, 399]
      
        if errorCode in ERRORS_TO_TRIGGER:
            errormsg="IB error id %d errorcode %d string %s" %(id, errorCode, errorString)
            print errormsg
            setattr(self, "flag_iserror", True)
            setattr(self, "error_msg", True)
          
        ## Wrapper functions don't have to return anything
    <snip - code missing here>

Note the two different methods; init_error is just my own method to ensure the callback has somewhere to store the error. Wheras error() is mandated by the IB API - if you took it out or renamed it the thing would break.

You need to have at least the two levels of error handling as shown here since a lot of the IB "errors" are really just IB telling you its alive; kind of like a child whining you need to be able to know when to ignore them. Once you get serious you may even want to have a third level of errors for when things really get nasty.

The way we handle errors is to make our waiting process finish before its proper finishing time with an error flag set. Subtle point; if an error appears when no process is waiting does anyone hear it? If a tree falls in an empty forest... does a bear crap on it? Well we'll still print the error.... arguably if there is no process to fail I wouldn't care.... for example if we temporarily lost the internet connection because, just suppose as a completely arbitrary example, you had builders in the house and they tripped the fusebox for the 5th time that morning... but if we weren't waiting for a price at the time I wouldn't be that bothered. Well I'd be livid with the builders but I wouldn't want the python client-server code to throw its toys out of the pram so I had to restart it.

That is one reason why I don't raise an exception here but let the calling process do it.

Methods that actually do something vaguely useful

Of which we only have two here; again one which is my init_ method, and the other is the IB mandated version:

    def init_time(self):
        setattr(self, "data_the_time_now_is", None)

     def currentTime(self, time_from_server):

        setattr(self, "data_the_time_now_is", time_from_server)

So lets reiterate what happens here.

    <In the client function>
    def speaking_clock(self):
        print "Getting the time... "
       
        self.tws.reqCurrentTime()
       
        start_time=time.time()
       
        self.cb.init_error()
        self.cb.init_time()




    <in the IBWrapper function, of which IBclient self.cb is an instance>
    def init_time(self):
        setattr(self, "data_the_time_now_is", None)

    <back in the client function speaking_clock>

        iserror=False
        not_finished=True




        while not_finished and not iserror:
   
    <at some point in this while loop, in the IBWrapper function, of which IBclient self.cb is an instance this will get called at some point ... hopefully.... >
     def currentTime(self, time_from_server):

        setattr(self, "data_the_time_now_is", time_from_server)


    <and back in the client function speaking_clock, in the while loop>

            not_finished=self.cb.data_the_time_now_is is None
            iserror=self.cb.flag_iserror

            if (time.time() - start_time) > MAX_WAIT:
                not_finished=False
               
            if iserror:
                not_finished=False         if iserror:
            print "Error happened"
            print self.cb.error_msg
           
        return self.cb.data_the_time_now_is

We set data_the_time_now in the callback to accept a value and then ask the server tws.reqCurrentTime(); then somewhere in the ether the IBWrapper instance method currentTime gets called by the server with the parameter time_from_server; we change the value of data_the_time_now in the callback instance, and this terminates the loop. All this assumes we don't get an error condition, or the server falls asleep and the process hits its MAX_WAIT. Pretty much everything else we do with the IB API is a variation on this particular theme so if that makes sense, you are now an expert.

(Note that the parameter names in the EWrapper method function definitions don't need to match those in the manual; in the manual it uses time which is already the name of an imported module.)

 

And We Are Done

Although this example is very simple, like the author, it does illustrate most of the 'gotchas' from working with swigibpy / Python / IB API. Subsequent posts will expand on this example to cover the full lifecycle of getting a price, generating an order, getting a fill, finding out what positions we have and working out whether we have made enough money to buy a decent laptop.




This is the first in a series of posts. The next post is:
http://qoppac.blogspot.co.uk/2014/04/getting-prices-out-of-ib-api-with.html