Interactive brokers native python API

Until quite recently interactive brokers didn't offer a python API for their automated trading software.

Instead you had to put up with various 3rd party solutions, one of which swigibpy I use myself. Swigibpy wrapped around the C++ implementation. I wrote a series of posts on how to use it, starting here.

Although swigiby has been very good to me its always better to use official solutions if they exist. And lo and behold after many people begging them we finally have a native python API, which was released a few weeks ago. I'll also be utilising this API in pysystemtrade when I get round to adding the 'talk to broker' part.

I've decided to reissue the same posts I wrote before to help relatively inexperienced programmers (like myself) get up to speed with using the API. The official documentation for the API is ... well ... tricky ("Our API components are aimed at experienced professional developers willing to enhance the current TWS functionality. And willing to read our minds") [okay, I added the last part myself].

Because I'm lazy the posts are identical, even with the same bad jokes, except where the two implementations differ. So if you're familiar with my earlier series of posts this should be a breeze.

(Assumptions: I am using Python 3.5.2; older or newer versions may break. I will assume you know your way around Python to the extent of being able to create a simple package and modules and run them. My command line examples will be for Linux but similar things ought to be possible. Also that you understand the dangers and risks of trading futures; but you will need to sign a lot of disclaimers before IB let you do this for real so I will let them worry about that.)

Getting a test account, downloading the IB TWS

To play with IB without signing up for real you will need a test account. By the way if you are serious and you get a real IB account you can also request an additional account for simulated trading.

(The test account is not very realistic, eg prices can be total garbage. The simulated account is much better although for some reason you don't always get L1 and L2 data to all the data feeds your real account is signed up to. If you are going to do any systematic trading in the near future I highly recommend signing up to IB and using a proper simulated account. It doesn't cost anything if you don't trade or use any additional data feeds.)

We aren't going to bother downloading the TWS software, which is a rather heavy front end useful for trading yourself; but the much lighter and more stable 'Gateway'. Here is how you do it for unix:

(I advise you to also download the TWS API at some point to have a play, but I don't recommend it for day to day running of a strategy since it seems to be very unstable due to the great lardy weight of fancy ultra bloated GUI that it has to support.)

1.  Go to https://www.interactivebrokers.com/en/?f=%2Fen%2Fcontrol%2Fsystemstandalone-ibGateway.php%3Fos%3Dunix
2. Follow the instructions.

Note the link may break; you might have to google or search for the gateway on the IB website. The last section of the instructions asks you to type some gobbledygook to run 'TWS' (they mean gateway). Don't be a fool and create a shell script that does it for you. Then you just have to type something like . runGateway. I have a similar script for TWS.

1. Select IB API radio button
2.  Under username put 'edemo' and under password put 'demo123'.

If all goes well you will eventually see a screen with a green bar showing a connected status at the top. This program acts as a server to pass on your instructions from the API connection to the great IB server, wherever that is. You now need to configure your API to accept connections.

1. Click on the Configure menu. Go to API settings
2. Socket port - should be 4001.
3. Trusted IP addresses - should include 127.0.0.1. If it doesn't you will need to add it.
4. Read only API - uncheck this here.
5. Go to precautions. You might want to suppress market cap warnings here when you start trading in earnest. Make sure you know what you are doing.
6. Go to presets. Again check you are happy with the limits shown.

Unlike TWS the gateway is automatically set up to accept connections so this is all you need to do.

(There is nothing special about 4001 so you can change it but be sure to remember the number and only use a radically different number if you realise you might break Linux in the process. Check http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers and make sure your game of Microsoft Ants isn't running. You could run two or more Gateways in parallel each connecting to live and test accounts; and associate a different socket (I use 4001 up to about 4005) with each account and gateway session.

127.0.0.1 is just 'this machine'. If your code is running on the same machine as the Gateway you are fine. Otherwise eg if you are on a network you will have to include the IP address of any other machines that might connect to the Gateway.)

Downloading required python libraries 

Install the python API:

http://interactivebrokers.github.io/#

Install the python libraries:

/IBJts/source/pythonclient $ python3 setup.py install

Note: The test cases, and the documentation refer to a python package called IBApi, but the actual package is called ibapi. Go figure.

You might want to peruse the official IB API manual. 

Running the example

Run the code in this gist.

If it works properly you should see something like this:

Getting the time from the server... 
1395845174

So we now seem to have acquired a very complicated way of telling unix time!

Also I appear to have committed the deadly sin of 'like here is some code, and it seems to work' without any explanation. So lets dig into this a bit more.

 

All you wanted to know about IB connections but frankly couldn't care less

 

if __name__ == '__main__':
    ##    ## Check that the port is the same as on the Gateway    

    ### ipaddress is 127.0.0.1 if one same machine, clientid is arbitrary
    app = TestApp("127.0.0.1", 4001, 10)

    <snip - more code here>

class TestApp(TestWrapper, TestClient):
    def __init__(self, ipaddress, portid, clientid):
        TestWrapper.__init__(self)
        TestClient.__init__(self, wrapper=self)

        self.connect(ipaddress, portid, clientid)

    <snip - code missing here>


(Pedantic point - Although the code says tws here, it really means gateway. Actually the client doesn't know or care what is on the other end of the connection.)

As I said above you can have any number of IB servers (gateway or TWS session) running; so you need to pass the host (just 'this machine') and port when connecting (here the function is rather dull). As briefly alluded to above each IB server will listen out for clients on a particular port (so these should be unique to the server). 

You can also have numerous (32) clients connecting to the same server, each with their own clientid. For example I have one client picking up prices; another receiving accounting information, another managing orders, various one off clients getting diagnostic information and one doing the washing up.

If a clientid is already in use by a connected session then IB will cry. 

       
Back to the story.

class TestApp(TestWrapper, TestClient):
    def __init__(self, ipaddress, portid, clientid):
        TestWrapper.__init__(self)
        TestClient.__init__(self, wrapper=self)

        self.connect(ipaddress, portid, clientid)

        thread = Thread(target = self.run)
        thread.start()

        setattr(self, "_thread", thread)

<snip>

There are two parts to the IB story: a wrapper and a client. The client asks the IB server (gateway or TWS) to do stuff, and the wrapper receives the messages. The Thread bit kicks off a process inside the client which listens for messages and plumbs them into the right wrapper function. By gluing them together in the App object we can hide all this ugliness from our ultimate trading software.

Writing for the IBApi consists of writing a function in the client which asks the server to do something, and then writing a function in the wrapper to deal with what comes back.

If that sounds like Outer Mongolian to you, don't worry. You shouldn't ever need to mess with this stuff. The point is once we have our client object (instance of IBclient) we can make it do cool things, like tell the time.

Telling the time - the hard way

We now call the speaking clock method of the IBclient:

if __name__ == '__main__':

    app = TestApp("127.0.0.1", 4001, 10)

    current_time = app.speaking_clock()

    <snip>

class TestClient(EClient):
    """    The client method
    We don't override native methods, but instead call them from our own wrappers    
    """    
    def __init__(self, wrapper):
        ## Set up with a wrapper inside        
        EClient.__init__(self, wrapper)

    def speaking_clock(self):
        """        Basic example to tell the time
        :return: unix time, as an int        
        """

        print("Getting the time from the server... ")

        ## Make a place to store the time we're going to return        


        ## This is a queue        
        time_storage=self.wrapper.init_time()

        ## This is the native method in EClient, asks the server to send us the time please        self.reqCurrentTime()

        ## Try and get a valid time        
        MAX_WAIT_SECONDS = 10

        try:
            current_time = time_storage.get(timeout=MAX_WAIT_SECONDS)
        except queue.Empty:
            print("Exceeded maximum wait for wrapper to respond")
            current_time = None
       while self.wrapper.is_error():
          print("Error:")
          print(self.get_error(timeout=5))

return current_time

self.reqCurrentTime() is an example of a classy class EClient functions, from the official IB API manual. These are the functions that ask the server to think about doing something.

 

Things that are interesting here #1: concurrency

The issue we have here is that the IB API is very much set up as an event driven process. So not like normal sequential code like function A calling function B and function A returning the answer back. No instead we have function A just kind of hanging around waiting for function B and then somehow by magic function B just happens.

So what we have to do is make the client-server relationship appear sequential, at least to anything sitting outside the wrapper module. That also means we need to handle the conditions of the thing not finishing in a reasonable time and finishing with an error.

Briefly: time_storage is a queue, which is a bit like a list but it works better in this kind of concurrent process. Once the queue is created the time will be added to it. We are happy to wait for 10 seconds for the time to arrive, but if we wait longer for this to happen then we decide to give up.


Warning: This kind of programming is fairly involved. Based on my experience over the last few years the setup above is fine for trading at frequencies of a few times a day, but more frequent trading would probably require more complicated code which involve rewriting substantial parts of the wrapper and the client. Alternative approaches include this one. 

I've updated this post to include a more robust handling of concurrency. For helping me with this section I'd like to thank Ewald over at groups.io/g/twsapi/, Miguel and Ryan on twitter, and David Beazley whose book I found useful. His curio project also looks interesting, although this is Unix only right now.

Things that are interesting here #2: The contents of self.wrapper

All the things we are pulling out of self.wrapper (which is the thing that is passed in when we initialised the TestClient object) are set somewhere else as if by magic. Actually they get set when the IB server summons the wrapper, calling the appropriate method. In the official IB API manual these are the very classy EWrapper Functions.

There are two basic kinds of classy EWrapper functions / methods: 'native' methods that the server knows about and we override with our own code, and new methods that we use to store, get, or check data that is being passed by the server. In the basic example here there are a bunch of methods to handle the time, and another bunch to deal with errors. Lets deal with the errors first: 

 

Methods that handle errors

class TestWrapper(EWrapper):
    """    The wrapper deals with the action coming back from the IB gateway or TWS instance
    We override methods in EWrapper that will get called when this action happens, like currentTime
    """    

   ## error handling code   
   def init_error(self):
       error_queue=queue.Queue()
       self._my_errors = error_queue

   def get_error(self, timeout=5):
       if self.is_error():
          try:
             return self._my_errors.get(timeout=timeout)
          except queue.Empty:
             ## edge case where another thread has taken the error before us
return None
       return None


   def is_error(self):
      an_error_if=not self._my_errors.empty()
      return an_error_if

   def error(self, id, errorCode, errorString):
      ## Overriden method        

      errormsg = "IB error id %d errorcode %d string %s" % (id, errorCode, errorString)
      self._my_errors.put(errormsg)

Note the two different types of method; init_error, get_error, and is_error are just my own methods to ensure the callback has somewhere to store the error and return it to the users. Wheras error() is mandated by the IB API - if you took it out or renamed it the thing would break.

Methods that actually do something vaguely useful like tell the time

class TestWrapper(EWrapper):
    """    The wrapper deals with the action coming back from the IB gateway or TWS instance
    We override methods in EWrapper that will get called when this action happens, like currentTime
    Extra methods are added as we need to store the results in this objects _my_data store
    These all have the form init_ or is_
    """    

<snip>

## Time telling code
def init_time(self):
    time_queue=queue.Queue()
    self._time_queue = time_queue

    return time_queue

def currentTime(self, time_from_server):
    ## Overriden method    
    self._time_queue.put(time_from_server)

Again there is one mandated method (currentTime), and the other method is added to make getting the time out easier.

Client server interaction for dummies

So lets reiterate what happens here.

    <In the client function>

def speaking_clock(self):

    print("Getting the time from the server... ")

    ## Make a place to store the time we're going to return

    ## This is a queue    

    time_storage=self.wrapper.init_time()

<in the TestWrapper class, of which the client self.wrapper is an instance>

## Time telling codedef init_time(self):
    time_queue=queue.Queue()
    self._time_queue = time_queue

    return time_queue

<back in the client function speaking_clock>

## This is the native method in EClient, asks the server to send us the time please

self.reqCurrentTime()

## Try and get a valid time
MAX_WAIT_SECONDS = 10
try:
    current_time = time_storage.get(timeout=MAX_WAIT_SECONDS)

   
<at some point in this while loop, in the TestWrapper class, of which the client self.wrapper is an instance this will get called at some point ... hopefully.... >

def currentTime(self, time_from_server):
    ## Overriden method    
    self._time_queue.put(time_from_server)

<and back in the client function speaking_clock, in the while loop>

try:
    current_time = time_storage.get(timeout=MAX_WAIT_SECONDS)
except queue.Empty:
    print("Exceeded maximum wait for wrapper to respond")
    current_time = None

while self.wrapper.is_error():
    print(self.get_error())

return current_time

So to summarise now we set _time_queue in the wrapper to accept a value and then ask the server wrapper.reqCurrentTime(); then somewhere in the ether the TestWrapper instance method currentTime gets called by the server with the parameter time_from_server; we put this value into the queue _time_queue in the wrapper instance, and this makes the time_storage.get() work. All this assumes we don't get an error condition, and / or the server falls asleep and the process hits its MAX_WAIT_SECONDS. Pretty much everything else we do with the IB API is a variation on this particular theme so if that makes sense, you are now an expert.

(Note that the parameter names in the wrapper method function definitions don't need to match those in the manual; in the manual it uses time which is already the name of an imported module.)

And we are done

Although this example is very simple, like the author, it does illustrate most of the 'gotchas' from working with Python and the IB API. Subsequent posts will expand on this example to cover the full lifecycle of getting a price, generating an order, getting a fill, finding out what positions we have and working out whether we have made enough money to buy a decent laptop.

This is the first in a series of posts. The next post is on historical data is here.

Appendix: Some technical notes

1) The use of threading here is very basic and more experienced programmers may wish to consult the IB API forum where this topic has had much more technical discussion.

2) There is an even better abstraction of the API (which I plan to use myself in pysystemtrade) to allow you to deal with multiple brokers. First write some generic, non broker specific, classes:

class brokerServer(object):
    """
    Broker server classes are called by the brokers server application (eg IB Gateway)
    We inherit from this and then write hooks from the servers native methods into the methods in this base class
    All the other non native methods from the TestWrapper example could be here as well    

    """
    def __init__(self):

       setattr(self, "_my_data", dict()) 

    def action_to_take_when_time_set(self, time_received):
        print("Do something with time!")

class brokerClient(object):
    """
    Broker server classes are called by the brokers server application (eg IB Gateway)
    We inherit from this for specific brokers and over ride the methods in the base class to ensure a consistent API
    """
    def __init__(self):
        pass
    def speaking_clock(self):
        print("Method needs to be overriden to do anything interesting")

Now we write broker specific classes that inherit from these generic, and also the IBapi classes:

class IBServer(EWrapper, brokerServer):

   def __init__(self):

      brokerClient.__init__(self) ## can't use super because not used in EWrapper

   def currentTime(self, time_from_server):
       ## Overriden method of EWrapper

      ## hook back to the generic brokerClient method
      self.action_to_take_when_time_set(self, time_from_server) 





class IBClient(EClient, brokerClient):
    """    The client method
    """    
    def __init__(self, wrapper):
        ## Set up with a wrapper inside        
        EClient.__init__(self, wrapper)
        brokerClient.__init__(self)

    def speaking_clock(self):
        """
        This is overriding the brokerClient method, so that it presents a consistent API


        Code would be similar to the original example        
        """




I won't be using this structure in the rest of the example posts to avoid confusing readers, but it's here if you wish.