Wednesday 8 March 2017

Interactive brokers native python API


Until quite recently interactive brokers didn't offer a python API for their automated trading software.

Instead you had to put up with various 3rd party solutions, one of which swigibpy I use myself. Swigibpy wrapped around the C++ implementation. I wrote a series of posts on how to use it, starting here.

Although swigiby has been very good to me its always better to use official solutions if they exist. And lo and behold after many people begging them we finally have a native python API, which was released a few weeks ago. I'll also be utilising this API in pysystemtrade when I get round to adding the 'talk to broker' part.

I've decided to reissue the same posts I wrote before to help relatively inexperienced programmers (like myself) get up to speed with using the API. The official documentation for the API is ... well ... tricky ("Our API components are aimed at experienced professional developers willing to enhance the current TWS functionality. And willing to read our minds") [okay, I added the last part myself].

Because I'm lazy the posts are identical, even with the same bad jokes, except where the two implementations differ. So if you're familiar with my earlier series of posts this should be a breeze.

(Assumptions: I am using Python 3.5.2; older or newer versions may break. I will assume you know your way around Python to the extent of being able to create a simple package and modules and run them. My command line examples will be for Linux but similar things ought to be possible. Also that you understand the dangers and risks of trading futures; but you will need to sign a lot of disclaimers before IB let you do this for real so I will let them worry about that.)


Getting a test account, downloading the IB TWS


To play with IB without signing up for real you will need a test account. By the way if you are serious and you get a real IB account you can also request an additional account for simulated trading.

(The test account is not very realistic, eg prices can be total garbage. The simulated account is much better although for some reason you don't always get L1 and L2 data to all the data feeds your real account is signed up to. If you are going to do any systematic trading in the near future I highly recommend signing up to IB and using a proper simulated account. It doesn't cost anything if you don't trade or use any additional data feeds.)

We aren't going to bother downloading the TWS software, which is a rather heavy front end useful for trading yourself; but the much lighter and more stable 'Gateway'. Here is how you do it for unix:

(I advise you to also download the TWS API at some point to have a play, but I don't recommend it for day to day running of a strategy since it seems to be very unstable due to the great lardy weight of fancy ultra bloated GUI that it has to support.)



  1.  Go to https://www.interactivebrokers.com/en/?f=%2Fen%2Fcontrol%2Fsystemstandalone-ibGateway.php%3Fos%3Dunix
  2. Follow the instructions.
Note the link may break; you might have to google or search for the gateway on the IB website. The last section of the instructions asks you to type some gobbledygook to run 'TWS' (they mean gateway). Don't be a fool and create a shell script that does it for you. Then you just have to type something like . runGateway. I have a similar script for TWS.
  1. Select IB API radio button
  2.  Under username put 'edemo' and under password put 'demo123'.
If all goes well you will eventually see a screen with a green bar showing a connected status at the top. This program acts as a server to pass on your instructions from the API connection to the great IB server, wherever that is. You now need to configure your API to accept connections.


  1. Click on the Configure menu. Go to API settings
  2. Socket port - should be 4001.
  3. Trusted IP addresses - should include 127.0.0.1. If it doesn't you will need to add it.
  4. Read only API - uncheck this here.
  5. Go to precautions. You might want to suppress market cap warnings here when you start trading in earnest. Make sure you know what you are doing.
  6. Go to presets. Again check you are happy with the limits shown.
Unlike TWS the gateway is automatically set up to accept connections so this is all you need to do.

(There is nothing special about 4001 so you can change it but be sure to remember the number and only use a radically different number if you realise you might break Linux in the process. Check http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers and make sure your game of Microsoft Ants isn't running. You could run two or more Gateways in parallel each connecting to live and test accounts; and associate a different socket (I use 4001 up to about 4005) with each account and gateway session.

127.0.0.1 is just 'this machine'. If your code is running on the same machine as the Gateway you are fine. Otherwise eg if you are on a network you will have to include the IP address of any other machines that might connect to the Gateway.)




Downloading required python libraries 

Install the python API:


http://interactivebrokers.github.io/#

Install the python libraries:

/IBJts/source/pythonclient $ python3 setup.py install



Note: The test cases, and the documentation refer to a python package called IBApi, but the actual package is called ibapi. Go figure.

You might want to peruse the official IB API manual


Running the example


Run the code in this gist.

If it works properly you should see something like this:

Getting the time from the server... 
1395845174

So we now seem to have acquired a very complicated way of telling unix time!

Also I appear to have committed the deadly sin of 'like here is some code, and it seems to work' without any explanation. So lets dig into this a bit more.

 

All you wanted to know about IB connections but frankly couldn't care less

 

if __name__ == '__main__':
    ##    ## Check that the port is the same as on the Gateway    
    ### ipaddress is 127.0.0.1 if one same machine, clientid is arbitrary
    app = TestApp("127.0.0.1", 4001, 10)
    <snip - more code here>



class TestApp(TestWrapper, TestClient):
    def __init__(self, ipaddress, portid, clientid):
        TestWrapper.__init__(self)
        TestClient.__init__(self, wrapper=self)

        self.connect(ipaddress, portid, clientid)
    <snip - code missing here>


(Pedantic point - Although the code says tws here, it really means gateway. Actually the client doesn't know or care what is on the other end of the connection.)

As I said above you can have any number of IB servers (gateway or TWS session) running; so you need to pass the host (just 'this machine') and port when connecting (here the function is rather dull). As briefly alluded to above each IB server will listen out for clients on a particular port (so these should be unique to the server). 

You can also have numerous (32) clients connecting to the same server, each with their own clientid. For example I have one client picking up prices; another receiving accounting information, another managing orders, various one off clients getting diagnostic information and one doing the washing up.

If a clientid is already in use by a connected session then IB will cry. 

       
Back to the story.

class TestApp(TestWrapper, TestClient):
    def __init__(self, ipaddress, portid, clientid):
        TestWrapper.__init__(self)
        TestClient.__init__(self, wrapper=self)

        self.connect(ipaddress, portid, clientid)

        thread = Thread(target = self.run)
        thread.start()

        setattr(self, "_thread", thread)

<snip>


There are two parts to the IB story: a wrapper and a client. The client asks the IB server (gateway or TWS) to do stuff, and the wrapper receives the messages. The Thread bit kicks off a process inside the client which listens for messages and plumbs them into the right wrapper function. By gluing them together in the App object we can hide all this ugliness from our ultimate trading software.

Writing for the IBApi consists of writing a function in the client which asks the server to do something, and then writing a function in the wrapper to deal with what comes back.

If that sounds like Outer Mongolian to you, don't worry. You shouldn't ever need to mess with this stuff. The point is once we have our client object (instance of IBclient) we can make it do cool things, like tell the time.


Telling the time - the hard way


We now call the speaking clock method of the IBclient:

if __name__ == '__main__':

    app = TestApp("127.0.0.1", 4001, 10)

    current_time = app.speaking_clock()

    <snip>






class TestClient(EClient):
    """    The client method
    We don't override native methods, but instead call them from our own wrappers    
    """    
    def __init__(self, wrapper):
        ## Set up with a wrapper inside        
        EClient.__init__(self, wrapper)

    def speaking_clock(self):
        """        Basic example to tell the time
        :return: unix time, as an int        
        """

        print("Getting the time from the server... ")

        ## Make a place to store the time we're going to return        

        ## This is a queue        
        time_storage=self.wrapper.init_time()

        ## This is the native method in EClient, asks the server to send us the time please        self.reqCurrentTime()

        ## Try and get a valid time        
        MAX_WAIT_SECONDS = 10

        try:
            current_time = time_storage.get(timeout=MAX_WAIT_SECONDS)
        except queue.Empty:
            print("Exceeded maximum wait for wrapper to respond")
            current_time = None
       while self.wrapper.is_error():
          print("Error:")
          print(self.get_error(timeout=5))
return current_time

self.reqCurrentTime() is an example of a classy class EClient functions, from the official IB API manual. These are the functions that ask the server to think about doing something.

 

Things that are interesting here #1: concurrency


The issue we have here is that the IB API is very much set up as an event driven process. So not like normal sequential code like function A calling function B and function A returning the answer back. No instead we have function A just kind of hanging around waiting for function B and then somehow by magic function B just happens.

So what we have to do is make the client-server relationship appear sequential, at least to anything sitting outside the wrapper module. That also means we need to handle the conditions of the thing not finishing in a reasonable time and finishing with an error.

Briefly: time_storage is a queue, which is a bit like a list but it works better in this kind of concurrent process. Once the queue is created the time will be added to it. We are happy to wait for 10 seconds for the time to arrive, but if we wait longer for this to happen then we decide to give up.


Warning: This kind of programming is fairly involved. Based on my experience over the last few years the setup above is fine for trading at frequencies of a few times a day, but more frequent trading would probably require more complicated code which involve rewriting substantial parts of the wrapper and the client. Alternative approaches include this one

I've updated this post to include a more robust handling of concurrency. For helping me with this section I'd like to thank Ewald over at groups.io/g/twsapi/, Miguel and Ryan on twitter, and David Beazley whose book I found useful. His curio project also looks interesting, although this is Unix only right now.



Things that are interesting here #2: The contents of self.wrapper


All the things we are pulling out of self.wrapper (which is the thing that is passed in when we initialised the TestClient object) are set somewhere else as if by magic. Actually they get set when the IB server summons the wrapper, calling the appropriate method. In the official IB API manual these are the very classy EWrapper Functions.

There are two basic kinds of classy EWrapper functions / methods: 'native' methods that the server knows about and we override with our own code, and new methods that we use to store, get, or check data that is being passed by the server. In the basic example here there are a bunch of methods to handle the time, and another bunch to deal with errors. Lets deal with the errors first: 

 

Methods that handle errors

class TestWrapper(EWrapper):
    """    The wrapper deals with the action coming back from the IB gateway or TWS instance
    We override methods in EWrapper that will get called when this action happens, like currentTime
    """    

   ## error handling code   
   def init_error(self):
       error_queue=queue.Queue()
       self._my_errors = error_queue

   def get_error(self, timeout=5):
       if self.is_error():
          try:
             return self._my_errors.get(timeout=timeout)
          except queue.Empty:
             ## edge case where another thread has taken the error before us
return None return None
   def is_error(self):
      an_error_if=not self._my_errors.empty()
      return an_error_if
   def error(self, id, errorCode, errorString):
      ## Overriden method        
      errormsg = "IB error id %d errorcode %d string %s" % (id, errorCode, errorString)
      self._my_errors.put(errormsg)

Note the two different types of method; init_error, get_error, and is_error are just my own methods to ensure the callback has somewhere to store the error and return it to the users. Wheras error() is mandated by the IB API - if you took it out or renamed it the thing would break.


Methods that actually do something vaguely useful like tell the time


class TestWrapper(EWrapper):
    """    The wrapper deals with the action coming back from the IB gateway or TWS instance
    We override methods in EWrapper that will get called when this action happens, like currentTime
    Extra methods are added as we need to store the results in this objects _my_data store
    These all have the form init_ or is_
    """    

<snip>

## Time telling code
def init_time(self):
    time_queue=queue.Queue()
    self._time_queue = time_queue

    return time_queue

def currentTime(self, time_from_server):
    ## Overriden method    
    self._time_queue.put(time_from_server)



Again there is one mandated method (currentTime), and the other method is added to make getting the time out easier.

Client server interaction for dummies


So lets reiterate what happens here.

    <In the client function>


def speaking_clock(self):

    print("Getting the time from the server... ")

    ## Make a place to store the time we're going to return
    
    ## This is a queue    
    time_storage=self.wrapper.init_time()


<in the TestWrapper class, of which the client self.wrapper is an instance>

## Time telling codedef init_time(self):
    time_queue=queue.Queue()
    self._time_queue = time_queue

    return time_queue

<back in the client function speaking_clock>

## This is the native method in EClient, asks the server to send us the time please
self.reqCurrentTime()

## Try and get a valid time
MAX_WAIT_SECONDS = 10
try:
    current_time = time_storage.get(timeout=MAX_WAIT_SECONDS)
   
<at some point in this while loop, in the TestWrapper class, of which the client self.wrapper is an instance this will get called at some point ... hopefully.... >



def currentTime(self, time_from_server):
    ## Overriden method    
    self._time_queue.put(time_from_server)


<and back in the client function speaking_clock, in the while loop>

try:
    current_time = time_storage.get(timeout=MAX_WAIT_SECONDS)
except queue.Empty:
    print("Exceeded maximum wait for wrapper to respond")
    current_time = None

while self.wrapper.is_error():
    print(self.get_error())

return current_time



So to summarise now we set _time_queue in the wrapper to accept a value and then ask the server wrapper.reqCurrentTime(); then somewhere in the ether the TestWrapper instance method currentTime gets called by the server with the parameter time_from_server; we put this value into the queue _time_queue in the wrapper instance, and this makes the time_storage.get() work. All this assumes we don't get an error condition, and / or the server falls asleep and the process hits its MAX_WAIT_SECONDS. Pretty much everything else we do with the IB API is a variation on this particular theme so if that makes sense, you are now an expert.

(Note that the parameter names in the wrapper method function definitions don't need to match those in the manual; in the manual it uses time which is already the name of an imported module.)


And we are done


Although this example is very simple, like the author, it does illustrate most of the 'gotchas' from working with Python and the IB API. Subsequent posts will expand on this example to cover the full lifecycle of getting a price, generating an order, getting a fill, finding out what positions we have and working out whether we have made enough money to buy a decent laptop.

This is the first in a series of posts. The next post is on historical data is here.


Appendix: Some technical notes


1) The use of threading here is very basic and more experienced programmers may wish to consult the IB API forum where this topic has had much more technical discussion.

2) There is an even better abstraction of the API (which I plan to use myself in pysystemtrade) to allow you to deal with multiple brokers. First write some generic, non broker specific, classes:

class brokerServer(object):
    """
    Broker server classes are called by the brokers server application (eg IB Gateway)
    We inherit from this and then write hooks from the servers native methods into the methods in this base class
    All the other non native methods from the TestWrapper example could be here as well    
    """
    def __init__(self):
       setattr(self"_my_data"dict()) 

    def action_to_take_when_time_set(self, time_received):
        print("Do something with time!")


class brokerClient(object):
    """
    Broker server classes are called by the brokers server application (eg IB Gateway)
    We inherit from this for specific brokers and over ride the methods in the base class to ensure a consistent API
    """
    def __init__(self):
        pass
    def speaking_clock(self):
        print("Method needs to be overriden to do anything interesting")


Now we write broker specific classes that inherit from these generic, and also the IBapi classes:


class IBServer(EWrapper, brokerServer):
   def __init__(self):
      brokerClient.__init__(self) ## can't use super because not used in EWrapper


   def currentTime(self, time_from_server):
       ## Overriden method of EWrapper
      ## hook back to the generic brokerClient method
      self.action_to_take_when_time_set(self, time_from_server) 
class IBClient(EClient, brokerClient):
    """    The client method
    """    
    def __init__(self, wrapper):
        ## Set up with a wrapper inside        
        EClient.__init__(self, wrapper)
        brokerClient.__init__(self)

    def speaking_clock(self):
        """
        This is overriding the brokerClient method, so that it presents a consistent API

        Code would be similar to the original example        
        """
I won't be using this structure in the rest of the example posts to avoid confusing readers, but it's here if you wish.

78 comments:

  1. Hi Rob,
    Thank you for the post. Very informative. One thing that I cant get is the Python API itself. I have been trying to install them several times but after installation (uing "TWS API Install 972.18.msi", what I got is a bunch of source code and samples for Cpp, CSharp, Excel, Java in TWS API folder, not Python. Does it mean I chose the wrong installation file? Can you give a hint here? Thanks.

    ReplyDelete
    Replies
    1. You need the latest version 9.73 not the stable version.

      Delete
    2. I think you have to get the API beta version to get the Python example

      Delete
    3. Really useful article, thanks (I am quite new to Python.)
      I have just struggled for the last 4 hours so please if anyone encounters a syntax error on:
      self.async = False
      You are NOT doing anything wrong.
      The API is not yet compatible with python 3.7.
      I rolled back to 3.6 and hey presto.

      Delete
    4. Thank you for posting this comment about self.async = False

      I had this same issue using the IB API with Python 3.7 on a Mac and trying to follow the instructions in this very helpful article.

      Installing Python 3.6.7 and running the example code in 3.6.7 fixed the issue for me also.

      Note that when you install 3.6, you will need to re-install any packages that you want to have access to. So I had to follow the instructions to re-install the ibapi package in 3.6 even though I had previously installed it in 3.7.

      Delete
    5. After posting my previous comment about also experiencing the self.async = False error with Python 3.7 on a Mac, I discovered a different fix. You can make a one line change in one of the ibapi files to make it work with 3.7 instead of needing to revert back to 3.6. Full credit to the ib_insync documentation for mentioning this:

      The Interactive Brokers Python API version 9.73.06 or higher. If using Python 3.7, then comment out line 60 of ibapi/client.py that reads self.async = False.

      Delete
  2. Hi Rob, you are missing a closing parenthesis on line 108 of your code for telling time.

    ReplyDelete
  3. tsk a lot for your advise. I am new in python and have difficulties on install the module

    can you mind to advise how to use pip to install it .

    ReplyDelete
    Replies
    1. Do you mean the IB API python code? You can't use pip as far as I know. Download it from here http://interactivebrokers.github.io/#
      That will probably create a directory called IBJts. From there go to IBJts/source/pythonclient and then type python3 setup.py install (as I did above). This works in linux. I'm afraid I don't know about windows.

      Delete
    2. I'm using anaconda 3 on windows.

      Downloaded to dir under anaconda (IbPy-Master).
      In commandline I go to that dir. Run the setup. Seems OK. I get: (changed myuser)

      `Writing C:\Users\myuser\AppData\Local\Continuum\anaconda3\Lib\site-packages\IbPy2-0.8.0-py3.6.egg-info`

      But in the code I get: ibapi module not found. Help!!!!

      Delete
    3. The egg file is actually just a zip. So rename it as a zip, and copy the ibapi folder to lib/site-packages directory in your Python folder.

      Delete
  4. Hi Rob! I really like your work and it was through your blog that I realized IB finally caved in to demand and added a native Python API which is a wonderful relief.

    I am a newbie to stock and futures trading and I already use another online bank broker. The requirement to trade with IB requires at least 100 real trades (I've only made 25) or 100 simulated trades. I was wondering is the Python API completely usable with a pure DEMO account? In other words can one utilize the Python API in its entirety before even signint up as an official IB client? If so it sounds too good to be true. :-)

    I realize that using a simulated account is better given the more accurate pricing data but given that I don't meet the requirements to sign up with IB at the moment this could be a realistic alternative for me.

    Thanks so much for being an expert on automated trading for the little retail guy! It is much appreciated!

    ReplyDelete
    Replies
    1. In theory you can use the API with a demo account but AFAIK the main problem is that you are sharing that account with other people; so for example if you bought MSFT then polled your account positions you'd find a whole lot more other positions in there as well that other users of the demo account had purchased.

      So you can test some, but not all, of the plumbing.

      Delete
  5. Thanks for your quick feedback! Based on your info I think the wisest course of is to perform 100 real trades on my current brokers account and then sign up for a full account with IB.

    As a side note I am a resident of Canada and as far as I know there are only three brokers in this country that allow one to trade futures. We aren't spoiled with the plethora of options that US retail traders have but given that IB seems to be the most established in the API arena it may be for the best anyways.

    Thanks so much! I'll keep on reading your blog!

    ReplyDelete
    Replies
    1. IB does ask about your trading experience. But they don't seem to verify your answers. YMMV.

      Delete
  6. Hi Rob, may thanks for your IB Python API posts. They're the first I've seen on the web and are very informative.

    There aren't many places yet where to talk about it so I thought about asking you (IB doesn't provide support for the APIs).
    I've tried to install the Python API in a Python 3 virtual environment but it seems hanging soon after the "python3 setup.py install" command.

    Before hanging the Mac Terminal has written:
    running install
    running bdist_egg
    running egg_info
    writing ibapi.egg-info/PKG-INFO
    writing dependency_links to ibapi.egg-info/dependency_links.txt
    writing top-level names to ibapi.egg-info/top_level.txt

    How long did it take to you to complete the install?

    Thanks

    ReplyDelete
    Replies
    1. I'm afraid I don't use any Apple products for ideological reasons (and they cost so much!). The installation (on linux) took literally seconds.

      If you aren't aware of it, this is a good place for this kind of question https://groups.io/g/twsapi

      Delete
    2. Many thanks for the link to the TWS API Users Group, I didn't know that. I was wondering why there wasn't a real group/forum about IB TWS API (https://www.interactivebrokers.com/fluxbb/viewforum.php?id=2 is very much dead) and here you go.
      You have made my day!

      I've actually solved the problem. It's a bug with the python setuptools (http://stackoverflow.com/questions/37868005/python-setup-py-develop-stuck-at-writing-dependency-links/40477249#40477249).

      Here's my solution in case someone else gets-stuck:
      # (Optional, in case you run Python3 from a virtual environment like me) Activate the Python3 virtual environment:
      source activate python_virtenv_name
      # Update python setuptools (option --ignore-installed to force the update):
      pip install --upgrade --ignore-installed setuptools
      # Go in the directory containing the setup.py file from the IB Python API (already unzipped):
      cd /path/to/setup.py
      # Run the python setup command:
      python3 setup.py install

      PS: Regarding Apple products I agree on the cost but that's fine because I don't buy new ones very often (I work on a 2011 Macbook). I've bought the first one out of rejection on the n-th issue with Windows and now I'm fine with it. In the end both OS and shell are Linux like and every now and then I think about migrating but I still stick to it out of stubbornness.

      Delete
    3. @Serge Gardien

      Thank you so much. I switched from using Python in a Jupyter notebook to creating a PyCharm project with a corresponding Conda virtual environment. I couldn't figure out how to add the ibapi modules to the conda virtual env. Rerunning the setup.py in the context of the virtual environment (through the PyCharm terminal) worked for me.

      On the plus-side, I got to learn a lot about Python packages trying to figure out how to add it to the path before coming back to see if anyone else had the same problem.

      Delete
  7. Hi Rob,

    1. Is it possible to write rules based exit? Example; sell within a specific time frame or daily candle bar close lower than yesterday close?

    2. How challenging is it to learn to program it from a newbie perspective? Where should I start?

    Thanks!
    YHS

    ReplyDelete
    Replies
    1. 1. Yes this is possible and I will probably write about this in due course.
      2. I'm afraid I can't answer the first part of your question, and for the second part there are some books here: http://quantocracy.com/books/python/

      Delete
  8. Hello Rob,

    I trued running the code from the example gist and got a BrokenPipeError. The only modifications I made to the code were changing the port number and client id. I am running an older version of python (3.4). Do you have any suggestions on how I might fix this? Thanks.

    BrokenPipeError Traceback (most recent call last)
    /home/wwilliams/ibroPython/example1.py in ()
    106 app = TestApp("127.0.0.1", 4002, 999)
    107
    --> 108 current_time = app.speaking_clock()
    109
    110 print(current_time)

    /home/wwilliams/ibroPython/example1.py in speaking_clock(self)
    68
    69 ## This is the native method in EClient, asks the server to send us the time please
    ---> 70 self.reqCurrentTime()
    71
    72 ## Try and get a valid time

    /usr/lib/python3.4/site-packages/ibapi-9.73.2-py3.4.egg/ibapi/client.py in reqCurrentTime(self)
    264 + make_field(VERSION)
    265
    --> 266 self.sendMsg(msg)
    267
    268

    /usr/lib/python3.4/site-packages/ibapi-9.73.2-py3.4.egg/ibapi/client.py in sendMsg(self, msg)
    74 full_msg = comm.make_msg(msg)
    75 logging.info("%s %s %s", "SENDING", current_fn_name(1), full_msg)
    ---> 76 self.conn.sendMsg(full_msg)
    77
    78

    /usr/lib/python3.4/site-packages/ibapi-9.73.2-py3.4.egg/ibapi/connection.py in sendMsg(self, msg)
    73 logging.debug("acquired lock")
    74 try:
    ---> 75 nSent = self.socket.send(msg)
    76 except socket.error:
    77 logging.debug("exception from sendMsg %s", sys.exc_info())

    ReplyDelete
    Replies
    1. Have you changed the port on both the IB gateway / TWS software? Have you set the gateway / TWS to accept incoming connections; deselected 'read only'.

      The other explanation is that something on your machine / OS is blocking the use of certain ports (firewall?); certainly I've never had this problem with linux using port # in this range, but I've never used mac / windows so I don't know if that would be different.

      Delete
    2. The error that I encountered seems to have been associated with running IB gateway inside of a virtual machine. Tests on Centos and Ubuntu running on virtualbox with a Windows host failed, while testing this script on my Ubuntu laptop worked just fine. Thanks again for putting this tutorial together.

      Delete
  9. Thank you very much for the tutorial! It certainly enlightens me a lot although there are still some (minor) issues. When I executed your code, I got the following output:

    Getting the time from the server...
    IB error id -1 errorcode 2104 string Market data farm connection is OK:hfarm
    IB error id -1 errorcode 2104 string Market data farm connection is OK:jfarm
    IB error id -1 errorcode 2104 string Market data farm connection is OK:eufarm
    IB error id -1 errorcode 2104 string Market data farm connection is OK:usfuture
    IB error id -1 errorcode 2104 string Market data farm connection is OK:cafarm
    IB error id -1 errorcode 2104 string Market data farm connection is OK:cashfarm
    IB error id -1 errorcode 2104 string Market data farm connection is OK:usfarm.us
    IB error id -1 errorcode 2104 string Market data farm connection is OK:usfarm
    IB error id -1 errorcode 2106 string HMDS data farm connection is OK:ilhmds
    IB error id -1 errorcode 2106 string HMDS data farm connection is OK:euhmds
    IB error id -1 errorcode 2106 string HMDS data farm connection is OK:fundfarm
    IB error id -1 errorcode 2106 string HMDS data farm connection is OK:ushmds
    1493513459

    So, I did get the starting message and the unix time but how come there are errors messages in between. I checked the error/warning codes and still does not understand what this error mean.


    Additionally, I also got this error at the very end:

    Exception in thread Thread-8:
    Traceback (most recent call last):
    File "/Users/Chuanquan/anaconda/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
    File "/Users/Chuanquan/.local/lib/python3.5/site-packages/ibapi-9.73.2-py3.5.egg/ibapi/reader.py", line 32, in run
    buf = self.prevBuf + self.conn.recvMsg()
    File "/Users/Chuanquan/.local/lib/python3.5/site-packages/ibapi-9.73.2-py3.5.egg/ibapi/connection.py", line 94, in recvMsg
    buf = self._recvAllMsg()
    File "/Users/Chuanquan/.local/lib/python3.5/site-packages/ibapi-9.73.2-py3.5.egg/ibapi/connection.py", line 113, in _recvAllMsg
    buf = self.socket.recv(4096)
    AttributeError: 'NoneType' object has no attribute 'recv'

    Exception in thread Thread-9:
    Traceback (most recent call last):
    File "/Users/Chuanquan/anaconda/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
    File "/Users/Chuanquan/anaconda/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
    File "/Users/Chuanquan/.local/lib/python3.5/site-packages/ibapi-9.73.2-py3.5.egg/ibapi/client.py", line 245, in run
    self.conn.isConnected(),
    AttributeError: 'NoneType' object has no attribute 'isConnected'


    Thank you again for writing this great post. Any help would be greatly appreciated!

    ReplyDelete
    Replies
    1. Relax - these "errors" are perfectly normal.

      IB error id -1 errorcode 2106 string HMDS data farm connection is OK:ilhmds

      This is literally the system saying "I am okay". The code doesn't distinguish between errors and status messages like this. It can easily be modified to achieve that by adding error code 2104 to a whitelist of ignored messages.

      AttributeError: 'NoneType' object has no attribute 'recv'
      AttributeError: 'NoneType' object has no attribute 'isConnected'

      This is a known bug with the current IB API software. Since you can do everything you want succesfully I'd suggest adding a try: except: clause to the end of your code to deal with the connection problem.

      Delete
    2. the other day I was listening to a webinar from IB about API. There it was mentioned by one of IB's developers that only "IB error id" with a positive value are really error messages. If a negative error id is supplied it means that it is a status message. Thus this error id can be used for filtering.

      Delete
    3. hi Rob,
      you mentioned add try: except: clause to the of the code to fix the "AttributeError: 'NoneType' object has no attribute". I am not sure what I should include in try: except. i added below:
      if __name__ == '__main__':
      ##
      ## Check that the port is the same as on the Gateway
      ## ipaddress is 127.0.0.1 if one same machine, clientid is arbitrary

      app = TestApp("127.0.0.1", 4002, 10)

      current_time = app.speaking_clock()

      print(current_time)

      try:
      app.disconnect()
      except:
      print("bug in IB API to disconnect")

      but still get the same message.
      Thanks

      Delete
    4. Yes try doesn't always work with threaded code.

      Delete
  10. Thank you very much for the blog! It certainly enlightened me. I have 2 lingering issues running your sample code. 1:

    Getting the time from the server...
    IB error id -1 errorcode 2104 string Market data farm connection is OK:hfarm
    IB error id -1 errorcode 2104 string Market data farm connection is OK:jfarm
    IB error id -1 errorcode 2104 string Market data farm connection is OK:eufarm
    IB error id -1 errorcode 2104 string Market data farm connection is OK:usfuture
    IB error id -1 errorcode 2104 string Market data farm connection is OK:cafarm
    IB error id -1 errorcode 2104 string Market data farm connection is OK:cashfarm
    IB error id -1 errorcode 2104 string Market data farm connection is OK:usfarm.us
    IB error id -1 errorcode 2104 string Market data farm connection is OK:usfarm
    IB error id -1 errorcode 2106 string HMDS data farm connection is OK:ilhmds
    IB error id -1 errorcode 2106 string HMDS data farm connection is OK:euhmds
    IB error id -1 errorcode 2106 string HMDS data farm connection is OK:fundfarm
    IB error id -1 errorcode 2106 string HMDS data farm connection is OK:ushmds
    1493513459

    What do these error/warning messages (between the starting message and the desired unix time) mean?

    2:

    Exception in thread Thread-8:
    Traceback (most recent call last):
    File "/Users/Chuanquan/anaconda/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
    File "/Users/Chuanquan/.local/lib/python3.5/site-packages/ibapi-9.73.2-py3.5.egg/ibapi/reader.py", line 32, in run
    buf = self.prevBuf + self.conn.recvMsg()
    File "/Users/Chuanquan/.local/lib/python3.5/site-packages/ibapi-9.73.2-py3.5.egg/ibapi/connection.py", line 94, in recvMsg
    buf = self._recvAllMsg()
    File "/Users/Chuanquan/.local/lib/python3.5/site-packages/ibapi-9.73.2-py3.5.egg/ibapi/connection.py", line 113, in _recvAllMsg
    buf = self.socket.recv(4096)
    AttributeError: 'NoneType' object has no attribute 'recv'

    Exception in thread Thread-9:
    Traceback (most recent call last):
    File "/Users/Chuanquan/anaconda/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
    File "/Users/Chuanquan/anaconda/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
    File "/Users/Chuanquan/.local/lib/python3.5/site-packages/ibapi-9.73.2-py3.5.egg/ibapi/client.py", line 245, in run
    self.conn.isConnected(),
    AttributeError: 'NoneType' object has no attribute 'isConnected'


    I got the above error message at the end of output. What might be causing this. Thank you again for writing this great post. Any help will be much appreciated.

    ReplyDelete
  11. Hi Rob,
    Firstly, thank you for your very interesting post. I have one question regarding the installation of the IB API. I do not understand the significance of executing the setup.py program. As an UBUNTU user, I followed the installation orders on the IB site (which do not include executing setup.py) and have now all the API code in the IBJts directory. Am I missing anything?
    Thanks, Nathan

    ReplyDelete
    Replies
    1. When you come to run python and import the API module you might find that your python cannot see it. One of the things setup.py does is ensure that the module is copied to a directory that is in the python search path (something like /usr/local/lib/python3.5/dist-packages/...). Alternatively you can manually add the IBJts directory to your pythonpath.

      Setup.py also ensure dependencies are installed but that is probably less problematic in this case.

      Delete
  12. Hi Rob,
    Thanks for the nice article.
    I am trying to explore TWS Python API as automated trading tool. I have following concerns that I couldn't figure out from API reference guide or any other web resources. Your thoughts could be very helpful and appreciated. At the moment, I am trying to get 5-min moving average of a set of stocks.

    1. I felt TWS API does not provide any studies. So, I have to calculate those from historical price. How can I get historical price of a set of stocks in regular basis? Should I send historical data request in a loop after certain interval? or there is any way to setup the API server so that it will stream it? What about multiple contract? Should I initiate data request for each securities separately?

    Thanks a lot.
    Wali

    ReplyDelete
  13. Hi Rob,
    Thanks for the nice post.
    I am trying to get intra day 10-min moving average of a set of securities from TWS API. I have following concerns. Your thoughts will be very helpful and appreciated.

    How to get historical data of a set of securities in a continuous stream? Should I issue the historical data request for each securities separately in a loop with a time sleep interval? Or I can setup the server once to get the information continuously until I cancel?

    I felt TWS API doesn't provide any studies. Should we calculate all studies in the TestApp even though it is available in TWS?

    Thanks

    ReplyDelete
  14. Yes, you have to use a repeating loop if you want to place a regular data request for historical data. And yes, you have to place a request for each security separately.

    ReplyDelete
  15. Hi Rob,
    Thanks for your post. I have tried all steps mentioned, but when I run the gist example, it looks like ibapi is not installed.

    File "E:/scratchthis.py", line 21, in
    from ibapi.wrapper import EWrapper

    ModuleNotFoundError: No module named 'ibapi'

    ReplyDelete
    Replies
    1. I don't know what to suggest - if you've run setup.py the module should be visible. Is there an error message when you run setup.py?

      Delete
    2. i think i found the solution. I just copied ibapi folder from /IBJts/source/pythonclient into the working directory. Thanks

      Delete
    3. Hi again Rob,
      I am not sure what the issue is but when I run "IBAPIpythonexample1" I get "Exception in thread Thread-7" error, however when I run "scratchthis" code it runs fine and places the order. Thanks

      Delete
  16. Hi Rob,

    I was thinking of using a 3rd party package like IBridgePy until I found your post - very helpful!

    I tried to run your sample code "IBAPIpythonexample1" but got the following error: AttributeError: 'TestApp' object has no attribute '_my_errors'

    Have you run into anything like this in the past by any chance?

    Thanks a lot,
    Nicolas

    ReplyDelete
    Replies
    1. Can you use the 'contact me' box below. I'll then reply and you can email me the full stack trace.

      Delete
    2. Hi Rob,
      I am having the same issue. Do you have a resolution to this?

      Thanks in advance,
      Ed

      Delete
    3. Swap lines 117 and 124 (as numbered in the gist) and it should give you a more memorable error.
      Nicolas error was because he didn't have the same ports in the API and code.

      Delete
    4. Thanks Rob. I should have investigated more. I really appreciate it!!!!

      Delete
  17. Hi Rob,

    Thank you for your great introduction to IBAPI! As a beginner in python this post really saved me a lot of pain.

    There's one question I wish to ask on the use of threading on the above sample code:

    thread = Thread(target = self.run)
    thread.start()

    setattr(self, "_thread", thread)

    Could I know why's threading needed in this program? I tried deleting the above lines of code and the program returns:
    Getting the time from the server...
    Exceeded maximum wait for wrapper to respond
    None

    Is threading required in general when using the IBAPI?

    Many thanks!!!
    Vincent

    ReplyDelete
    Replies
    1. Yes, threading is needed, because the server communicates on an aysnch basis with the client.

      Delete
    2. Hi Rob,

      Thank you for your quick reply it's really encouraging ...I have been learning Python for two months now and it is my first programming language so please forgive me if I have asked a dumb question.

      On the threading codes you implemented in the program would you share more on the logic behind the codes? I have read a few tutorials online on threading but none of them resembles the codes you wrote... Also, I have read the sample codes from IBAPI as well and it uses the methods app.run() and self.done = true to start and end the thread instead; could I know what's the difference between the two approaches?

      At last, on the line of code:
      time_storage=self.wrapper.init_time()


      I wish to ask why self.wrapper is necessary there? Is it because the data is pulled out from the queue object? I read the next example on fetching historical data and saw that in the code line:
      contract_details_queue = finishableQueue(self.init_contractdetails(reqId))

      self.wrapper was not used. What makes the difference between the two?

      Again thank you for your really kind help on answering my amateur questions!

      Vincent


      Delete
    3. Hi Vincent. It's important for me to say that I'm not an expert on threading but after reading the relevant chapters in this book (http://www.dabeaz.com/cookbook.html) and the messages on the forum https://groups.io/g/twsapi where there are some real experts, I managed to write something which works and is relatively simple.

      You're actually right the .wrapper isn't needed (legacy of the previous IB API I was using): time_storage = self.init_time() should work (try it!)

      But to make things clearer as to what is going on:

      self.wrapper.init_time()
      This creates a queue which the server will dump the result into. This is a 'one time only' queue; we expect to get one result and then we are done.

      For the second example (historical prices) we're going to get more than one result, and we're going to be told by the server when we have all the data we need, so I created this new kind of object finishableQueue to handle that in a more elegant way.

      self.init_historicprices(tickerid) is the actual queue and then the other class sits around it to make it easier to gather multiple items and handle the time out conditions.

      Delete
    4. Hi Rob,

      I would like to know that where is the part of your code that define the init_historicalprices()?

      Me too, try to limit the time my algo take for aoing one action.

      Thanks you,
      Thanasarn

      Delete
    5. Hi again,

      I am sorry for upper comment. I have found that line of code, it is in the TestWrapper.

      Delete
  18. Hi. Found this amazing python package which makes the use of the IB API way easier for less proficient python programmers. IB_INSYNC. check it out. the author is very responsive on GitHub.

    ReplyDelete
    Replies
    1. Thanks Steffen. Heres the link https://github.com/erdewit/ib_insync

      Delete
    2. The author is also active on the TWS API user group at groups.io

      Delete
  19. Can run and get result. But got the following error message:
    Exception in thread Thread-7:
    Traceback (most recent call last):
    File "C:\Program Files\Anaconda3\envs\pyqt5\lib\threading.py", line 914, in _bootstrap_inner
    self.run()
    File "C:\Program Files\Anaconda3\envs\pyqt5\lib\threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
    File "C:\Program Files\Anaconda3\envs\pyqt5\lib\site-packages\ibapi-9.73.2-py3.5.egg\ibapi\client.py", line 248, in run
    self.disconnect()
    File "C:\Program Files\Anaconda3\envs\pyqt5\lib\site-packages\ibapi-9.73.2-py3.5.egg\ibapi\client.py", line 195, in disconnect
    self.conn.disconnect()
    File "C:\Program Files\Anaconda3\envs\pyqt5\lib\site-packages\ibapi-9.73.2-py3.5.egg\ibapi\connection.py", line 55, in disconnect
    self.socket.close()
    AttributeError: 'NoneType' object has no attribute 'close'

    It appears the connection is closed twice. And at the second time, the socket is None and produce this error. I wonder how to properly close the connection when a thread is involved.

    ReplyDelete
    Replies
    1. This is a common problem but it doesn't affect the functionality.

      Delete
  20. If your TWS/Gateway is refusing connections you get an error during the self.connect(...) in TestApp, but the error Queue has not been created yet so you crash. To fix it, move the self.init_error() to the line before self.connect().



    ReplyDelete
  21. This comment has been removed by a blog administrator.

    ReplyDelete
  22. This comment has been removed by a blog administrator.

    ReplyDelete
  23. Hello, thanks for this post, the IB docs are a bit limited and this post is very helpful.

    I don't understand where to get the setup.py and project code, i.e. where is the "/IBJts/source/pythonclient" directory coming from?

    ReplyDelete
  24. Hello all,

    While you guys are using native ibapi for placing order (entry/exit), what kind of tool/lib/platform are you using for algo backtesting in python? Can you share some with me?

    Thanks a lot

    ReplyDelete
  25. Hi Rob,

    What is the right way to reconnect, if the connection will drop while threading is active ?

    ReplyDelete
  26. Wooooow! THANK YOU for this amazing series of posts! I started reading the official documentation and quickly felt out of my depth but your examples are shedding a lot of light, thank you for taking the time to share this.

    I've been using a modified version of your example as I am learning. I am trying to get an account summary but unfortunately for me, it appears that my "callback function" accountSummary def (override) in the wrapper class never gets called... any ideas why? that print statement 'summary callback received' never executes.. here's my code. Thanks a million!!


    from ibapi.wrapper import EWrapper
    from ibapi.client import EClient
    from threading import Thread
    import queue

    class TestWrapper(EWrapper):
    """
    The wrapper deals with the action coming back from the IB gateway or TWS instance
    We override methods in EWrapper that will get called when this action happens, like currentTime
    """

    ## error handling code
    def init_error(self):
    error_queue=queue.Queue()
    self._my_errors = error_queue

    def get_error(self, timeout=5):
    if self.is_error():
    try:
    return self._my_errors.get(timeout=timeout)
    except queue.Empty:
    return None
    return None


    def is_error(self):
    an_error_if=not self._my_errors.empty()
    return an_error_if

    def error(self, id, errorCode, errorString):
    ## Overriden method
    errormsg = "IB error id %d errorcode %d string %s" % (id, errorCode, errorString)
    self._my_errors.put(errormsg)

    ## Time telling code
    def init_time(self):
    time_queue=queue.Queue()
    self._time_queue = time_queue

    return time_queue

    def currentTime(self, time_from_server):
    print(time_from_server)
    ## Overriden method
    self._time_queue.put(time_from_server)

    def accountSummary(self, reqId, account,tag,value, currency ):
    print('summary callback received')
    #print ("AcctSummary. ReqId:", reqId, "AcctNo:", account,"Tag: ", tag, "Value:", value, "Currency:", currency)
    return 'test'



    class TestClient(EClient):
    """
    The client method
    We don't override native methods, but instead call them from our own wrappers
    """
    def __init__(self, wrapper):
    ## Set up with a wrapper inside
    EClient.__init__(self, wrapper)

    def speaking_clock(self):
    """
    Basic example to tell the time
    :return: unix time, as an int
    """

    print("Getting the time from the server... ")

    ## Make a place to store the time we're going to return
    ## This is a queue
    time_storage=self.wrapper.init_time()

    ## This is the native method in EClient, asks the server to send us the time please
    self.reqCurrentTime()

    ## Try and get a valid time
    MAX_WAIT_SECONDS = 10

    try:
    current_time = time_storage.get(timeout=MAX_WAIT_SECONDS)
    except queue.Empty:
    print("Exceeded maximum wait for wrapper to respond")
    current_time = None

    while self.wrapper.is_error():
    print(self.get_error())

    return current_time

    def get_summary(self):
    print('called')
    self.reqAccountSummary(777,"All", "NetLiquidation")


    class TestApp(TestWrapper, TestClient):
    def __init__(self, ipaddress, portid, clientid):
    TestWrapper.__init__(self)
    TestClient.__init__(self, wrapper=self)

    self.connect(ipaddress, portid, clientid)

    thread = Thread(target = self.run)
    thread.start()

    setattr(self, "_thread", thread)

    self.init_error()


    if __name__ == '__main__':
    ##
    ## Check that the port is the same as on the Gateway
    ## ipaddress is 127.0.0.1 if one same machine, clientid is arbitrary

    app = TestApp("127.0.0.1", 7496, 10)

    current_time = app.speaking_clock()
    print(app.get_summary())
    app.disconnect()

    ReplyDelete
    Replies
    1. I don't have time to debug other peoples code. Does the original code work ok?

      Delete
  27. Hi Rob,

    I have question about the implementation. I see you are using Queue quite a lot. Wondering what is the benefits of using Queues? Is it mandatory?

    Thanks,
    Reem

    ReplyDelete
    Replies


    1. Asynchronous programming is a tricky area; I chose Queues as the simplest way of passing messages between different threads.

      Delete
  28. I am confused as to why we are overriding "EWrapper" and "EClient" classes here. Shouldnt this be the point of the ib api, to provide properly implemented classes and functions to connect to the software?

    I guess my concern is; was this done for illustrative purposes or we always need to override and rewrite the wrappers and clients to get a working product ? Thank you.

    ReplyDelete
    Replies
    1. Part of it is to get a nice abstraction, but part of it is that essentially the IB API is only an API, not a complete product. IB have no way of knowing what you will do for example once you receive a price. If they were to provide a more fully featured product that would tie you into doing something specific once a price was received. Plus the API is intended to be relatively lightweight (for minimal support needs) and only for serious programmers who are happy to do plenty of heavy lifting.

      Delete
  29. Thank you so much for sharing this knowledge. Saves me lots of time figure out the cryptic IB API docs. You should tell them to pay you commission on the many trades executed after understanding your explanations.....

    ReplyDelete
  30. Thanks for the code ..It is great way to start for everybody, but I 'm feeling treading and wrapping mechanism are little bit over complicated..Do you use it normally? Do you have other version?

    ReplyDelete
    Replies
    1. The threading is an integral part of how the IB API works.

      Delete
  31. This is really helpful. One thing I'm finding on all the scripts, whether I run it on Linux or Windows, is that I get an unhandled exception in the eReader thread, followed by several exceptions to the message receive functions. The code does what it's supposed to do, but these exceptions seem strange since according to the documentation, the Python API doesn't use the eReader class; it is combined with the queue fuctionality.

    ReplyDelete
  32. Thanks Rob, for going thru the trouble to help beginners like myself. But I am confused by one thing in your "getting the time" sample code (https://gist.github.com/robcarver17/7b4a8e2f1fbfd70b4aaea5d205cb35eb). The code invoked instance methods .run(), .connect() and .disconnect() for class TestApp. But I couldn't see where these methods are defined. EWrapper and EClient classes do not seem to have such methods per IB Api documentation. Sorry for such a newbie question.

    ReplyDelete
  33. Thanks a lot! Just a note: I first tried this wit API version 9.76.1, but received error messages. Then I went to an older version (9.74.01) and got rid of the error messages. See also: https://groups.io/g/twsapi/topic/oserror_winerror_10038_an/33156178?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,33156178

    ReplyDelete
  34. Hi Rob, great as always. I was just looking to get a say 10 years of futures data, do you have any recommendations as to where you get your data from, it is just via the IB api? Happy to pay for it.

    ReplyDelete

Comments are moderated. So there will be a delay before they are published. Don't bother with spam, it wastes your time and mine.