The placing of orders to interactive brokers with swigibpy / python

We are nearly at the end of our journey of simplistic examples of how to get the swigibpy package to mediate between the wonderful world of Python and the dark place that is the Interactive brokers C++ API. Having learned how to get 'snapshot' and streaming prices out of the API we are now ready to actually do some trading- submit orders, check they are active, potentially cancel them, receive fills and get historic execution data.

Interactive brokers now have a native python API. An updated version of this post which uses that API is here: http://qoppac.blogspot.co.uk/2017/03/placing-orders-in-native-python-ib-api.html

Where do I start?

You need to get the code from the following repo.

The example code to run is in test4_IB.py. If you have a live or simulated Gateway / TWS session running (one associated with a real account number) it should work just fine. Note that you can also run this with the edemo account, but the results will be very ugly. This is because you are seeing the orders placed by everyone who is playing around with this account, so you could get all kinds of randomeness.


WARNING I highly recommend that you do not run this code on a real money trading account until / unless you know exactly what you are doing! In particular real trading code should check market liquidity before trading, particularly for market orders.

The output should be as in green below. Of course feel free to change the contract traded and the quantities...

Expiry is 20150908



Getting orderid from IB
Using order id of 1

Placed order, orderid is 1
Getting orderid from IB
Using order id of 2

Placed limit order, orderid is 2
recived fill as follows:

{'orderid': '1', 'account': 'DU15031', 'exchange': 'DTB', 'symbol': 'GBL', 'permid': 193880352, 'execid': '00004468.559df2a0.01.01', 'clientid': '999', 'expiry': '20150908', 'price': 156.37, 'qty': 10, 'side': 'BOT', 'times': '20150709  14:05:11'}

Active orders: (should just be limit order)
{2: {'orderid': 2, 'symbol': 'GBL', 'qty': 10, 'side': 'SELL', 'clientid': 999L, 'expiry': '20150908'}}

Cancelling remaining orders
Cancelling 2
Waiting for cancellation to finish
No active orders now
False

Following executed since midnight this morning:

{'1': {'orderid': '1', 'account': 'DU15031', 'exchange': 'DTB', 'symbol': 'GBL', 'permid': 193880352, 'execid': '00004468.559df2a0.01.01', 'clientid': '999', 'expiry': '20150908', 'price': 156.37, 'qty': 10, 'side': 'BOT', 'times': '20150709  14:05:11'}}

Contract details - what is it?

As usual there is the tedious business of getting a client object and defining a contract. The first proper thing we need to do is get the contract details. This may sound a bit weird, we already have the contract details since we've just defined it? In practice the definition of a contract is very light and the details we get back are much richer.

There is one specific case where you must do this. To see why suppose it is a few months time and you are trading hundreds of futures contracts. A fill comes in for a particular contract instrument code and month/year. However as we shall see the identification for the contract isn't the 'yyyymm' code we use to define the contract, but the full expiry 'yyyymmdd'. You can't just take the first 6 characters of that either since some futures actually expire the month before they should. So to be on the safe side I usually get the expiry from IB before trading to make sure I am applying fills against the correct thingy.

Actually I don't do this immediately before trading but in a separate process that populates a database of contract information...

 

From test4_IB.py
     callback = IBWrapper()
    client=IBclient(callback)
   
    ibcontract = IBcontract()
    ibcontract.secType = "FUT"
    ibcontract.expiry="201509"
    ibcontract.symbol="GBL"
    ibcontract.exchange="DTB"
    cdetails=client.get_contract_details(ibcontract)

WARNING: If you are trading the VIX, which now has weekly expiries, you will need to specify the full expiry date in yyyymmdd format.


I won't go through the client code in detail but there are two 'gotchas' in the server code that appear a few times so lets look at an excerpt.

From wrapper_v4.py, class IBWrapper:
    def contractDetails(self, reqId, contractDetails):       
        contract_details=self.data_contractdetails[reqId]
 
contract_details["contractMonth"]=contractDetails.contractMonth
        ## <SNIP - code removed>
        contract_details["evMultiplier"]=contractDetails.evMultiplier

        contract2 = contractDetails.summary

        contract_details["expiry"]=contract2.expiry
        ## <SNIP - code removed>
 
contract_details is of course a global object that gets picked up by the client object when populated. The first of the two gotchas here is an old one; if you are submitting multiple of these to the same API session watch out to use meaningful request ID's. More subtle is that we don't just do something like contract_details=contractDetails.Instead we actually extract the information into a new object. The reason is that the little jobbys are just weird pointers that will become pointers to something else, usually some weird memory address, by the time you look at them outside of the callback instance. So you have to capture their essence in the wild before they escape. This is the well known quantum pointers effect.

To find out more about contract and contractDetails objects look at the manual (C++ / SocketClient Properties).

Note I use a dicts or dicts of dicts here for contracts, orders and fills but in reality I have little classes for them to keep things neater.

Order placing - can I buy it?

Now the moment we've all been waiting for... we're actually going to buy something. For fun we're going to place two orders, a market order and a limit order.

From test4_IB.py 
     ## Place the order, asking IB to tell us what the next order id is
    ## Note limit price of zero
    orderid1=client.place_new_IB_order(ibcontract, 10, 0.0, "MKT", orderid=None)
   
    print ""
    print "Placed market order, orderid is %d" % orderid1
   
    ## And here is a limit order, unlikely ever to be filled
    ## Note limit price of 100
    orderid2=client.place_new_IB_order(ibcontract, -10, 200.0, "LMT", orderid=None)
    print ""
    print "Placed limit order, orderid is %d" % orderid2


 From wrapper_v4.py, class IBclient:  
    def place_new_IB_order(self, ibcontract, trade, lmtPrice, orderType, orderid=None):

        <SNIP>       
        iborder = IBOrder()
        iborder.action = bs_resolve(trade)
        iborder.lmtPrice = lmtPrice
        iborder.orderType = orderType
        iborder.totalQuantity = abs(trade)
        iborder.tif='DAY'
        iborder.transmit=True
 
        ## We can eithier supply our own ID or ask IB to give us the next valid one
        if orderid is None:
            print "Getting orderid from IB"
            orderid=self.get_next_brokerorderid()
           
        print "Using order id of %d" % orderid
   
         # Place the order
        self.tws.placeOrder(
                orderid,                                    # orderId,
                ibcontract,                                   # contract,
                iborder                                       # order
            )
   
        return orderid

       
Obvious first thing to notice here is the concept of an orderid. This is a number that identifies to IB what the order is; at least temporarily and for today only. Only restriction on order id's is that the next order is higher than the last. This means if you submit an order with an id of 999999 you will lose all the orderids below that. You can also reset the id 'clock' to 1 via an option on the Gateway or TWS API configuration menu.  Safest thing to do is ask IB for the next orderid as done here by supplying None to the calling function.

In practice I generate my own orderid's preferring to reserve them first in my own database. This is fine as long as you are running a single linear process where there is no chance of an 'older' order being submitted before a 'newer' one.

Notice the limit price of 0.0 for the market order won't be used. For more information on more funky order types see the docs:  Reference > Order Types and IBAlgos > Supported Order Types.

 Active order status- have I bought it?

IB can tell us what orders we are working. Unless you ask very quickly (or submit your order outside of trading hours) this is likely only to return limit orders.

From test4_IB.py 
    order_structure=client.get_open_orders()
    print "Active orders: (should just be limit order)"
    print order_structure
    print ""


 From wrapper_v4.py, class IBclient:  

    def get_open_orders(self):
        """
        Returns a list of any open orders
        """
        <SNIP>
        self.tws.reqAllOpenOrders()
        iserror=False
        finished=False
       
        while not finished and not iserror:
            finished=self.cb.flag_order_structure_finished
            iserror=self.cb.flag_iserror
            if (time.time() - start_time) > MAX_WAIT_SECONDS:
                ## You should have thought that IB would teldl you we had finished
                finished=True
            pass
       
        order_structure=self.cb.data_order_structure
        if iserror:
            print self.cb.error_msg
            print "Problem getting open orders"
   
        return order_structure   
 


From wrapper_v4.py, class IBWrapper:

    def init_openorders(self):
        setattr(self, "data_order_structure", {})
        setattr(self, "flag_order_structure_finished", False)

    def add_order_data(self, orderdetails):
        if "data_order_structure" not in dir(self):
            orderdata={}
        else:
            orderdata=self.data_order_structure

        orderid=orderdetails['orderid']
        orderdata[orderid]=orderdetails
                       
        setattr(self, "data_order_structure", orderdata)


    def openOrder(self, orderID, contract, order, orderState):
        """
        Tells us about any orders we are working now
       
        Note these objects are not persistent or interesting so we have to extract what we want
       
       
        """
       
        ## Get a selection of interesting things about the order
        orderdetails=dict(symbol=contract.symbol , expiry=contract.expiry,  qty=int(order.totalQuantity) ,
                       side=order.action , orderid=int(orderID), clientid=order.clientId )
       
        self.add_order_data(orderdetails)

    def openOrderEnd(self):
        """
        Finished getting open orders
        """
        setattr(self, "flag_order_structure_finished", True)

 
In practice I have noticed that the correct end condition for receiving open orders doesn't always trigger so you do need an max waiting time (which is good practice anyway). Here we are just getting a selection of the available information about the order (see C++ > SocketClient Properties > Order and OrderState for more). Again we don't just save the order object itself as this is just a quantum pointer that will change.

Fill data - how much did it cost me?

What happens when an order is filled; completely or partially? Well the following method in the wrapper function is triggered. Notice that the logic is slightly more complicated because this function fulfills two duties. When it's called for a fill then reqId=-1, and action_ib_fill will be called.

From wrapper_v4.py, class IBWrapper:
    def execDetails(self, reqId, contract, execution):
           
    def execDetails(self, reqId, contract, execution):
   
        """
        This is called if
       
        a) we have submitted an order and a fill has come back
        b) We have asked for recent fills to be given to us
       
        We populate the filldata object and also call action_ib_fill in case we need to do something with the
          fill data
       
        See API docs, C++, SocketClient Properties, Contract and Execution for more details
        """
        reqId=int(reqId)
       
        execid=execution.execId
        exectime=execution.time
        thisorderid=int(execution.orderId)
        account=execution.acctNumber
        exchange=execution.exchange
        permid=execution.permId
        avgprice=execution.price
        cumQty=execution.cumQty
        clientid=execution.clientId
        symbol=contract.symbol
        expiry=contract.expiry
        side=execution.side
       
        execdetails=dict(side=str(side), times=str(exectime), orderid=str(thisorderid), qty=int(cumQty), price=float(avgprice), symbol=str(symbol), expiry=str(expiry), clientid=str(clientid), execid=str(execid), account=str(account), exchange=str(exchange), permid=int(permid))

        if reqId==FILL_CODE:
            ## This is a fill from a trade we've just done
            action_ib_fill(execdetails)
           
        else:
            ## This is just execution data we've asked for
            self.add_fill_data(reqId, execdetails)
Again we extract the useful information from the contract and execution objects rather than saving them directly. More on the IB website docs C++ > SocketClient Properties > Execution. To reiterate the expiry here will be the actual yyyymmdd expiry date; not the shorter yyyymm that the original trade was constructed with, hence the need to identify the former before submission.

A brief technical note; the date is the contract expiry not where relevant first or last notice date. This means you should be wary of using this date to tell you when to roll certain kinds of futures contracts eg US bonds.



From IButils:

 def action_ib_fill(execlist):
    print "recived fill as follows:"
    print ""
    print execlist
    print ""

Although this is very boring in practice this would be the function that would update your order status database (as a pro at this stuff, naturally you would have such a thing). It won't be obvious from this simple example unless you can submit a very large order in a thin market but the fills come in as cumulative order updates, not separate fills. Its worth looking at an example. Suppose you try and buy 10 lots, and you get fills of:

• 3 lots @ 100.0
• 6 lots @ 100.2
• 1 lot @ 100.5

 Then the fills that come in will look like this:

• qty: 3 lots, price=100.0
• qty: 9 lots, price=100.13333333
• qty: 10 lots, price=100.17

So if you do care about each partial fill you are going to have to hope that you see every little fill coming in and use a differencing process to see the detail of each.

By the way 'orderid' is only a temporary thing for IB; after tommorrow it won't associate it with this order. Instead you should use 'permid' for your record keeping. 'execid' is different for each part fill so you could use it to make sure you aren't including fill information you already have; in practice this isn't problematic due to the cumulative nature of the information.

Order cancelling - what if I don't want it any more?

Its very easy indeed to cancel an order; we don't even need a call in the client object to do it.

From test4_IB.py
    print "Cancelling the limit order"
    client.tws.cancelOrder(orderid2)

    print "Waiting for cancellation to finish"   
    while client.any_open_orders():
        pass
    print "No active orders now"
    print client.any_open_orders()
    print ""

Past execution data - sorry, repeat that, how much?!

It is clearly very important that fill data is correctly captured by your trading software. One reason being to keep track of what your position is; as we shall see in the next post IB doesn't offer mere mortals a super accurate current position facility. So I generally use my own knowledge of trade history to decide where I am, position wise. Because the fills usually arrive in the wrapper function only once its possible under certain conditions to miss them; eg if your API client dies before you see the fill or just isn't running when one arrives on a previously closed market in the middle of the night. Its generally good practice then to reconcile what IB has for a record of fills versus your own.

This information is only available up to midnight of the day you trade. So I run a reconciliation 3 times a day. If you lose a fill from before today you will need to find it on the IB website account management microsite, and manually enter it into your database.

Here is how we do it.

From test4_IB.py 

    execlist=client.get_executions()
   
    print "Following executed since midnight this morning:"
    print ""
    print execlist


 From wrapper_v4.py, class IBclient:  
    def get_executions(self):
        """
        Returns a list of all executions done today
        """
       
        assert type(reqId) is int
        if reqId==FILL_CODE:
            raise Exception("Can't call get_executions with a reqId of %d as this is reserved for fills %d" % reqId)

        self.cb.init_fill_data()
        self.cb.init_error()
       
        ## We can change ExecutionFilter to subset different orders
       
        self.tws.reqExecutions(reqId, ExecutionFilter())

        iserror=False
        finished=False
       
        start_time=time.time()
       
        while not finished and not iserror:
            finished=self.cb.flag_fill_data_finished
            iserror=self.cb.flag_iserror
            if (time.time() - start_time) > MAX_WAIT_SECONDS:
                finished=True
            pass
   
        if iserror:
            print self.cb.error_msg
            print "Problem getting executions"
       
        execlist=self.cb.data_fill_data[reqId]
       
        return execlist
       
       
We can change ExecutionFilter to subset different orders (C++ > SocketClient Properties > ExecutionFilter).
       


From wrapper_v4.py, class IBWrapper:
    def execDetails(self, reqId, contract, execution):
       
        <SNIP - same code as before>
       
        execdetails=dict(side=str(side), times=str(exectime), orderid=str(thisorderid), qty=int(cumQty), price=float(avgprice), symbol=str(symbol), expiry=str(expiry), clientid=str(clientid), execid=str(execid), account=str(account), exchange=str(exchange), permid=int(permid))

        if reqId==FILL_CODE:
            ## This is a fill from a trade we've just done
            action_ib_fill(execdetails)
           
        else:
            ## This is just execution data we've asked for
            self.add_fill_data(reqId, execdetails)

    def add_fill_data(self, reqId, execdetails):
        if "data_fill_data" not in dir(self):
            filldata={}
        else:
            filldata=self.data_fill_data

        if reqId not in filldata.keys():
            filldata[reqId]={}
           
        execid=execdetails['orderid']
        filldata[reqId][execid]=execdetails
                       
        setattr(self, "data_fill_data", filldata)

           
    def execDetailsEnd(self, reqId):
        """
        No more orders to look at if execution details requested
        """

        setattr(self, "flag_fill_data_finished", True)
 

Its the same hardworking function as before, only this time the reqId will not be -1 (appearing here as FILL_CODE rather than being hardcoded) so we append the fill that is received to a list which is returned to the user.

Are we finished?

 Yes, at least with this post. The last thing I will show you how to do is to get accounting information, so the tedium is nearly over.

This is the fourth in a series of five posts on constructing a simple interface in python to the IB API using swigiby. The first three posts are:

http://qoppac.blogspot.co.uk/2014/03/using-swigibpy-so-that-python-will-play.html

http://qoppac.blogspot.co.uk/2014/04/getting-prices-out-of-ib-api-with.html 

http://qoppac.blogspot.co.uk/2014/04/streaming-prices-from-ib-api-swigibpy.html


The next post is
http://qoppac.blogspot.co.uk/2014/05/getting-accounting-data-out-of.html