Monday, 20 March 2017

Placing orders in the native python IB API

This the fourth in a series of posts on using the native python API for interactive brokers. You should read the first, second, and third, before this one.

It is an updated version of this older post, which used a third party API (swigibpy) which wraps around the C++ API. I've changed the code, but otherwise the post is pretty similar.

We are nearly at the end of our journey of simplistic examples of how to get the swigibpy package to mediate between the wonderful world of Python and the dark place that is the Interactive brokers C++ API. Having learned how to get prices out of the API we are now ready to actually do some trading- submit orders, check they are active, potentially cancel them, receive fills and get historic execution data.


Where do I start?


It's worth reading the relevant part of the documentation.

You need to get the code from the following gist.

If you have a live or simulated Gateway / TWS session running (one associated with a real account number) it should work just fine. Note that you can also run this with the edemo account (password: demo123), but the results might not be reliable. This is because you are seeing the orders placed by everyone who is playing around with this account, so you could get all kinds of randomeness.

WARNING I highly recommend that you do not run this code on a real money trading account until / unless you know exactly what you are doing! In particular real trading code should check market liquidity before trading, particularly for market orders.


Contract details - what is it?


As in the previous post there is the tedious business of creating an object to talk to IB and resolving the contract we want to trade:


app = TestApp("127.0.0.1", 4001, 1)

ibcontract = IBcontract()
ibcontract.secType = "FUT"
ibcontract.lastTradeDateOrContractMonth="201812"
ibcontract.symbol="GE"
ibcontract.exchange="GLOBEX"
## resolve the contract
resolved_ibcontract = app.resolve_ib_contract(ibcontract)




Getting full contract details from the server... 

For the rest of the post anything you see in that nice purple bold font is python output.

WARNING: If you are trading the VIX, which now has weekly expiries, you will need to specify the full expiry date in yyyymmdd format. 

ANOTHER WARNING; the date is the contract expiry not where relevant first or last notice date. This means you should be wary of using this date to tell you when to roll certain kinds of futures contracts eg US bonds.


Order placing - can I buy it?

Now the moment we've all been waiting for... we're actually going to buy something. We build the order object and pass it a client function.


order1=Order()
order1.action="BUY"
order1.orderType="MKT"
order1.totalQuantity=10
order1.transmit = True
orderid1 = app.place_new_IB_order(ibcontract, order1, orderid=None)

print("Placed market order, orderid is %d" % orderid1)

Getting orderid from IB
Using order id of 1
Placed market order, orderid is 1

In the client class:

def place_new_IB_order(self, ibcontract, order, orderid=None):

    ## We can eithier supply our own ID or ask IB to give us the next valid one    
    if orderid is None:
        print("Getting orderid from IB")
        orderid = self.get_next_brokerorderid()

        if orderid is TIME_OUT:
            raise Exception("I couldn't get an orderid from IB, and you didn't provide an orderid")

    print("Using order id of %d" % orderid)

    ## Note: It's possible if you have multiple traidng instances for orderids to be submitted out of sequence    
    ##   in which case IB will break
    # Place the order    
    self.placeOrder(
        orderid,  # orderId,
        ibcontract,  # contract, 
        order  # order    )

    return orderid    


Obvious first thing to notice here is the concept of an orderid. This is a number that identifies to IB what the order is; at least temporarily and for today only. Only restriction on order id's is that the next order is higher than the last. This means if you submit an order with an id of 999999 you will lose all the orderids below that. You can also reset the id 'clock' to 1 via an option on the Gateway or TWS API configuration menu.  Safest thing to do is ask IB for the next orderid as done here by supplying None to the calling function.

I generate my own orderid's preferring to reserve them first in my own database. This is fine as long as you are running a single linear process where there is no chance of an 'older' order being submitted before a 'newer' one.



Fill data - how much did it cost me?

What happens when an order is filled; completely or partially? Well the following method in the wrapper function is triggered. Notice that the logic is slightly more complicated because this function fulfills two duties:


  • A fill has arrived when a trade actually executes, in which case the reqId will be -1 (stored as FILL_CODE here)
  • We can ask for fill information, in which case the reqId will accompany the information. More on this shortly


def execDetails(self, reqId, contract, execution):
    ## overriden method
    execdata = execInformation(execution.execId, contract=contract,
                               ClientId=execution.clientId, OrderId=execution.orderId,
                               time=execution.time, AvgPrice=execution.avgPrice,
                               AcctNumber=execution.acctNumber, Shares=execution.shares,
                               Price = execution.price)

    ## there are some other things in execution you could add    
    ## make sure you add them to the .attributes() field of the execInformation class
    reqId = int(reqId)

    ## We eithier put this into a stream if its just happened, or store it for a specific request    
   if reqId==FILL_CODE:
        self._my_executions_stream.put(execdata)
    else:
        self._my_requested_execution[reqId].put(execdata)

In this case as we haven't yet asked for execution data reqId will be -1, and we'll put the data into my_executions_stream. This is a 'permanent' queue that hosts execution data: I describe it as permanent because we didn't need to run an init_ ... method to set it up in place_new_IB_order like I've shown you in previous posts; the queue is created when the instance of TestWrapper is created.

(Also this would be the function that would update your order status database; as a pro at this stuff, naturally you would have such a thing).

To see the fill data I need to do this:

print("Recent fills")
filldetails = app.recent_fills_and_commissions()
print(filldetails)

There is quite a lot going on under the hood here which is worth understanding so lets examine the client function we're calling.

def recent_fills_and_commissions(self):
    """    Return recent fills, with commissions added in
    :return: dict of execInformation objects, keys are execids    
    """
    recent_fills = self._recent_fills()
    commissions = self._all_commissions() ## we want all commissions
    
    ## glue them together, create a dict, remove duplicates    
    all_data = recent_fills.blended_dict(commissions)

    return all_data

The first thing to note is that this returns information coming from two sources: execDetails (labelled recent_fills here) and commissions. Frustratingly this data is provided by two separate parts of the wrapper function. We've already seen the executions, here it is for commissions:

def commissionReport(self, commreport):

    commdata = execInformation(commreport.execId, Commission=commreport.commission,
                    commission_currency = commreport.currency,
                    realisedpnl = commreport.realizedPNL)


    ## there are some other things in commreport you could add    
    ## make sure you add them to the .attributes() field of the execInformation class
    ## These always go into the 'stream' as could be from a request, 
    ## or a fill thats just happened    

    self._my_commission_stream.put(commdata)

As with execution details this can get called eithier when we've requested execution details (of which more later), or whenever we get an actual fill (as we're doing here). Unlike the execDetails function however we never get a reqId. So all the commissions data is dumped into a 'permanent' _stream queue, rather than one we've initialised already.

Once the executions and commission data has arrived in the 'permanent' _stream queues, we can access it via these functions:

def _recent_fills(self):
    """    Returns any fills since we last called recent_fills
    :return: list of executions as execInformation objects    
    """
    ## we don't set up a queue but access the permanent one    
    fill_queue = self.access_executions_stream()

    list_of_fills=list_of_execInformation()

    while not fill_queue.empty():
        MAX_WAIT_SECONDS = 5        
        try:
            next_fill = fill_queue.get(timeout=MAX_WAIT_SECONDS)
            list_of_fills.append(next_fill)
        except queue.Empty:
            ## corner case where Q emptied since we last checked if empty at top of while loop            
            pass
    ## note this could include duplicates and is a list    

    return list_of_fills

Notice that we clear the queue once we've accessed the data. And _recent_commissions(selfis in a similar vein.

def _recent_commissions(self):

    ## we don't set up a queue, as there is a permanent one    
    comm_queue = self.access_commission_stream()
    list_of_comm=list_of_execInformation()

    while not comm_queue.empty():
        MAX_WAIT_SECONDS = 5        
        try:
            next_comm = comm_queue.get(timeout=MAX_WAIT_SECONDS)
            list_of_comm.append(next_comm)
        except queue.Empty:
            ## corner case where Q emptied since we last checked if empty at top of while loop            pass
    ## note this could include duplicates and is a list    
return list_of_comm

However because all commissions live in the commission_stream queue I actually use another function to get commission data, which returns both the last chunk of commissions, plus any other commission data I've collected:

def _all_commissions(self):
    ## self._commissions is created when the client instance is __init__    
    
    original_commissions = self._commissions
    latest_commissions = self._recent_commissions()

    ## these are just simple lists so we can glue them together
    all_commissions = list_of_execInformation(original_commissions + latest_commissions)

    self._commissions = all_commissions

    # note this could include duplicates and is a list    
    return all_commissions

Finally we have to glue these together. The final line in the function before the return populates the fill data with the relevant commission levels.

def recent_fills_and_commissions(self):    
    
    recent_fills = self._recent_fills()
    commissions = self._all_commissions() ## we want all commissions
    
    ## glue them together, create a dict, remove duplicates    
    all_data = recent_fills.blended_dict(commissions)

    return all_data

Recent fills
{'00004468.58ca0e5f.01.01': Execution - contract: 56825063,GE,FUT,20181217,0.0,,2500,GLOBEX,,USD,GEZ8,GE,False,,combo: ClientId: 1 OrderId: 1 time: 20170316  09:50:31 AvgPrice: 98.055 Price: 98.055 AcctNumber: DU15075 Shares: 10.0 Commission: 24.0 commission_currency: USD realisedpnl: 1.7976931348623157e+308}

Notice this returns a dict; the keyword is the execId. I use some simple objects to gather up and merge together these two streams of data but you can do it differently of course.

Just to reiterate the _recent functions clear the relevant queues as you can see when I try and get the fill data again (although the commission data is saved so self._all_commissions() will still work).

## when I call again should be empty as we've cleared the memory of recent fills
print("Recent fills (should be blank)")
morefilldetails = app.recent_fills_and_commissions()
print(morefilldetails)

Recent fills (should be blank)
{}


IMPORTANT DETAIL: It won't be obvious from this simple example unless you can submit a very large order in a thin market but the fills come in as cumulative order updates, not separate fills. Its worth looking at an example. Suppose you try and buy 10 lots, and you get fills of:

  • 3 lots @ 100.0
  • 6 lots @ 100.2
  • 1 lot @ 100.5
 Then the fills that come in will look like this:

  • qty: 3 lots, price=100.0
  • qty: 9 lots, price=100.13333333
  • qty: 10 lots, price=100.17
So if you do care about each partial fill you are going to have to hope that you see every little fill coming in and use a differencing process to see the detail of each.

By the way 'orderid' is only a temporary thing for IB; after tommorrow it won't associate it with this order. Instead you should use 'permid' for your record keeping. 'execid' is different for each part fill so you could use it to make sure you aren't including fill information you already have; in practice this isn't problematic due to the cumulative nature of the information.



Past execution data - sorry, repeat that, how much?!

It is clearly very important that fill data is correctly captured by your trading software. One reason being to keep track of what your position is; as we shall see in the next post IB doesn't offer mere mortals a super accurate current position facility. So I generally use my own knowledge of trade history to decide where I am, position wise. Because the fills usually arrive in the wrapper function only once its possible under certain conditions to miss them; eg if your API client dies before you see the fill or just isn't running when one arrives on a previously closed market in the middle of the night. Its generally good practice then to reconcile what IB has for a record of fills versus your own.
This information is only available up to midnight of the day you trade. So I run a reconciliation 3 times a day. If you lose a fill from before today you will need to find it on the IB website account management microsite, and manually enter it into your database.
Here is how we do it.
print("Executions today")
execdetails = app.get_executions_and_commissions()
print(execdetails)

def get_executions_and_commissions(self, reqId=DEFAULT_EXEC_TICKER, execution_filter = ExecutionFilter()):

    ## store somewhere    
    execution_queue = finishableQueue(self.init_requested_execution_data(reqId))

    ## We can change ExecutionFilter to subset different orders    
    ## note this will also pull in commissions 
    self.reqExecutions(reqId, execution_filter)

    ## Run until we get a terimination or get bored waiting    
    MAX_WAIT_SECONDS = 10    
    exec_list = list_of_execInformation(execution_queue.get(timeout = MAX_WAIT_SECONDS))

    while self.wrapper.is_error():
        print(self.get_error())

    if execution_queue.timed_out():
        print("Exceeded maximum wait for wrapper to confirm finished whilst getting exec / commissions")
 
<snip>

(Note: We can change ExecutionFilter to subset different orders)     
This is a more familiar pattern: we create a queue for the wrapper to put stuff which will be terminated by a finish event, and then request the IB API to start sending data, some method in the wrapper populates the queue, and we then pull the contents of the queue in. The relevant wrapper function should look familiar:


def execDetails(self, reqId, contract, execution):
    ## overriden method
    execdata = execInformation(execution.execId, contract=contract,
                               ClientId=execution.clientId, OrderId=execution.orderId,
                               time=execution.time, AvgPrice=execution.avgPrice,
                               AcctNumber=execution.acctNumber, Shares=execution.shares,
                               Price = execution.price)

    ## there are some other things in execution you could add    
    ## make sure you add them to the .attributes() field of the execInformation class
    reqId = int(reqId)

    ## We eithier put this into a stream if its just happened, or store it for a specific request    

   if reqId==FILL_CODE:
        self._my_executions_stream.put(execdata)
    else:
        self._my_requested_execution[reqId].put(execdata)
Its the same hardworking function as before, only this time the reqId will not be -1 (appearing here as FILL_CODE rather than being hardcoded) so we append the fill that is received to the requested queue for a given reqId.
The only wrinkle is that the commission data needs to be brought in and merged, as we did before with fills:

def get_executions_and_commissions(self, reqId=DEFAULT_EXEC_TICKER, execution_filter = ExecutionFilter()):

    <snip>
    ## Commissions will arrive seperately. We get all of them, but will only use those relevant for us    
    commissions = self._all_commissions()

    ## glue them together, create a dict, remove duplicates    
    all_data = exec_list.blended_dict(commissions)

    return all_data

   

execdetails = app.get_executions_and_commissions()
print(execdetails)
Executions today
{'00004468.58ca0e5f.01.01': Execution - contract: 56825063,GE,FUT,20181217,0.0,,2500,GLOBEX,,USD,GEZ8,GE,False,,combo: ClientId: 1 OrderId: 1 time: 20170316  09:50:31 AvgPrice: 98.055 Price: 98.055 AcctNumber: DU15075 Shares: 10.0 Commission: 24.0 commission_currency: USD realisedpnl: 1.7976931348623157e+308}
As before the executions are listed in a dict with the execId as the key.

Placing limit orders

No self respecting trader will use market orders (see my post), so how to define limit orders?

order2=Order()
order2.action="SELL"
order2.orderType="LMT"
order2.totalQuantity=12
order2.lmtPrice = 100.0
order2.tif = 'DAY'
order2.transmit = True

orderid2 = app.place_new_IB_order(ibcontract, order2, orderid=None)
print("Placed limit order, orderid is %d" % orderid2)

Getting orderid from IB
Using order id of 2
Placed limit order, orderid is 2


This is just a tiny selection of the available orders, see the docs here.


 Active order status- have I bought it?

IB can tell us what orders we are working. Unless you ask very quickly (or submit your order outside of trading hours) this is likely only to return unfilled limit orders like the one we've just submitted.

open_orders = app.get_open_orders()
print(open_orders)




Client function:


def get_open_orders(self):

    ## store the orders somewhere    
    open_orders_queue = finishableQueue(self.init_open_orders())

    ## You may prefer to use reqOpenOrders() which only retrieves orders for this client    self.reqAllOpenOrders()

    ## Run until we get a terimination or get bored waiting    
    MAX_WAIT_SECONDS = 5    
    open_orders_list = list_of_orderInformation(open_orders_queue.get(timeout = MAX_WAIT_SECONDS))

    while self.wrapper.is_error():
        print(self.get_error())

    if open_orders_queue.timed_out():
        print("Exceeded maximum wait for wrapper to confirm finished whilst getting orders")

    ## open orders queue will be a jumble of order details, turn into a tidy dict with no duplicates    open_orders_dict = open_orders_list.merged_dict()

    return open_orders_dict


Wrapper functions:


def init_open_orders(self):
    open_orders_queue = self._my_open_orders = queue.Queue()

    return open_orders_queue


def orderStatus(self, orderId, status, filled, remaining, avgFillPrice, permid,
                parentId, lastFillPrice, clientId, whyHeld):

    order_details = orderInformation(orderId, status=status, filled=filled,
             avgFillPrice=avgFillPrice, permid=permid,
             parentId=parentId, lastFillPrice=lastFillPrice, clientId=clientId,
                                     whyHeld=whyHeld)

    self._my_open_orders.put(order_details)


def openOrder(self, orderId, contract, order, orderstate):

    order_details = orderInformation(orderId, contract=contract, order=order, 
                                     orderstate = orderstate)
    self._my_open_orders.put(order_details)


def openOrderEnd(self):

    self._my_open_orders.put(FINISHED)

{2: Order - contract: 56825063,GE,FUT,20181217,0.0,?,2500,GLOBEX,,USD,GEZ8,GE,False,,combo: order: 2,1,1017786930: LMT SELL 12@100.000000 DAY orderstate: <ibapi.order_state.OrderState object at 0x7f5e7bcfe860> status: Submitted filled: 0.0 avgFillPrice: 0.0 permid: 1017786930 parentId: 0 lastFillPrice: 0.0 clientId: 1 whyHeld: }

This should be a familiar story now. The only interesting thing is that we get order details from two wrapper functions, orderStatus and openOrder. I don't bother segregating them, instead I add them to the same queue, and then merge them with open_orders_dict = open_orders_list.merged_dict()

In practice I have noticed that the correct end condition for receiving open orders doesn't always trigger so you do need an max waiting time (which is good practice anyway).

As with fills the output is returned as a dict, and the keys are orderids. We just have the single active limit order since the market order (orderid 1) has long since been filled.


Order modification


To modify an existing order we submit a new order, but with an existing orderid.


order3=Order()
order3.action="BUY"
order3.orderType="LMT"
order3.totalQuantity=5
order3.lmtPrice = 10.0
order3.tif = 'DAY'
order3.transmit = True

orderid3 = app.place_new_IB_order(ibcontract, order3, orderid=None)
print("Placed limit order, orderid is %d" % orderid2)

print("Open orders (should be two)")
open_orders = app.get_open_orders()
print(open_orders.keys())

Getting orderid from IB
Using order id of 3
Placed limit order, orderid is 2
Open orders (should be two)

dict_keys([2, 3])


print("Modifying order %d" % orderid3)

order3.lmtPrice = 15.0
print("Limit price was %f will become %f" % (open_orders[orderid3].order.lmtPrice, order3.lmtPrice ))

app.place_new_IB_order(ibcontract, order3, orderid=orderid3)
time.sleep(5)
open_orders = app.get_open_orders()
print("New limit price %f " % open_orders[orderid3].order.lmtPrice)


Modifying order 3
Limit price was 10.000000 will become 15.000000
New limit price 15.000000 


Its advised that you only change the quantity and limit price (where relevant) of an order; not the order type.


Order cancelling - what if I don't want it any more?


Cancelling an order is pretty simple:

print("Cancel order %d " % orderid2)
app.cancel_order(orderid2)
open_orders = app.get_open_orders()

Cancel order 2 
IB error id 2 errorcode 202 string Order Canceled - reason:

My code checks that the order has been cancelled:
def cancel_order(self, orderid):
    self.cancelOrder(orderid)

    ## Wait until order is cancelled    start_time=datetime.datetime.now()
    MAX_WAIT_TIME_SECONDS = 10
    finished = False
    while not finished:
        if orderid not in self.get_open_orders():
            ## finally cancelled            
            finished = True
        if (datetime.datetime.now() - start_time).seconds > MAX_WAIT_TIME_SECONDS:
            print("Wrapper didn't come back with confirmation that order was cancelled!")
            finished = True
    ## return nothing
... but you can also check yourself:
print("Open orders (should just be %d)" % orderid3)
print(open_orders.keys())

Open orders (should just be 3)
dict_keys([3])


Note this will only cancel orders where the clientid (the third number in this call app = TestApp("127.0.0.1", 4001, 1)) is the same as the current client. To cancel all orders, regardless of the client:

print("Cancelling all orders")
app.cancel_all_orders()

print("Any open orders? - should be False")
print(app.any_open_orders())

Cancelling all orders
IB error id 3 errorcode 202 string Order Canceled - reason:
Open orders? - should be False
False


Are we finished?


 Yes, at least with this post. The last thing I will show you how to do is to get accounting information, so the tedium is nearly over.

This is the fourth in a series of five posts on constructing a simple interface in python to the IB API using swigiby. The first three posts are:

http://qoppac.blogspot.co.uk/2017/03/interactive-brokers-native-python-api.html

http://qoppac.blogspot.co.uk/2017/03/historic-data-from-native-ib-pyhon-api.html

http://qoppac.blogspot.co.uk/2017/03/streaming-market-data-from-native.html


The next post will be here.

Friday, 10 March 2017

Streaming market data from native python IB API

This the third in a series of posts on using the native python API for interactive brokers. You should read the first, and the second, before this one.

It is an updated version of this older post, which used a third party API (swigibpy) which wraps around the C++ API. I've changed the code, but not the poor attempts at humour.

In my last post we looked at getting a single snapshot of historical prices. In this one we will look at streamed prices - 'market tick data'. Rather than wait until the historical price feed ends with streaming prices we need to tell the streamer when to stop.

Note: This post has been updated to use a more robust method for dealing with concurrency.


No stream rises higher than its source


Get the source code from this gist.

You'll also need the pandas library.

You may want to read the documentation.


No stream drives anything without being confined

The example code begins in a similar fashion to the historical data example; we make one of these weird client objects containing a server wrapper connection, make one of these slightly less weird contract objects (here it is for December 2018 Eurodollar futures), resolve it into a populated contract object (explained more fully here) and then shove that into a request for market data.


from __main function:

app = TestApp("127.0.0.1", 4001, 1)

ibcontract = IBcontract()
ibcontract.secType = "FUT"
ibcontract.lastTradeDateOrContractMonth="201706"
ibcontract.symbol="GBL"
ibcontract.exchange="DTB"

## resolve the contract
resolved_ibcontract=app.resolve_ib_contract(ibcontract)

tickerid = app.start_getting_IB_market_data(resolved_ibcontract)

time.sleep(30)


Unlike the other functions we've looked at so far there isn't an internal loop here; instead we deliberately hang around whilst some price data comes in.


from TestClient.start_getting_IB_market_data() method:

def start_getting_IB_market_data(self, resolved_ibcontract, 
    tickerid=DEFAULT_MARKET_DATA_ID):
    """
    Kick off market data streaming
    :param resolved_ibcontract: a Contract object    
    :param tickerid: the identifier for the request    

    :return: tickerid    """   
    
    
    self._market_data_q_dict[tickerid] = self.wrapper.init_market_data(tickerid)
    self.reqMktData(tickerid, resolved_ibcontract, "", False, False, [])

    return tickerid

Ah yes its the usual stuff of setting up space within the self.wrapper instance to get the data and then call the tws server request function (strictly speaking its one of those 'EClient' whatdoyoucallits again). However one difference is that we have to store the TestClients pointer to the market data queue (in the dict self._market_data_q), since we'll be returning to it later.



Only dead fish swim with the stream...

 

We now look inside the server wrapper object which gets populated as an instance into self.wrapper. As before there are a few EWrapper functions which get triggered whenever the market data arrives.

There are in fact several methods for 'tickString', 'tickGeneric', 'tickSize' and 'tickPrice'; it seems a bit stochastic (quant speak: english translation completely bloody random and arbitrary) which of these methods gets called when a tick arrives (a tick could be an update to a price or to a quoted size on the top level of the order book). Lets look at the most generic of these:

def tickGeneric(self, tickerid, tickType, value):
    ## overriden method
    this_tick_data=IBtick(self.get_time_stamp(),tickType, value)
    self._my_market_data_dict[tickerid].put(this_tick_data)

All the code does is identify which type of tick it is and then add it to the Queue that lives in the appropriate part of self._my_market_data. You can look at the classes IBtick and tick to see how this is done. I'm using local time as the timestamp here, but again you can change this if you want.


Dipping our toe into the metaphorical market stream


from __main function:
market_data1=app.get_IB_market_data(tickerid)


from TestClient.get_IB_market_data() method:

def get_IB_market_data(self, tickerid):
   ## how long to wait for next item   MAX_WAIT_MARKETDATEITEM = 5   market_data_q = self._market_data_q_dict[tickerid]

   market_data=[]
   finished=False
   while not finished:
      try:
         market_data.append(market_data_q.get(timeout=MAX_WAIT_MARKETDATEITEM))
      except queue.Empty:
         ## no more data         finished=True
   return stream_of_ticks(market_data)

We can see what data we have received so far. This also clears the queue of data that has been transmitted out of the app.wrapper storage. You can write this differently if you like of course.

An individual tick looks like this:

>>> print(market_data1[1])
                            ask_price  ask_size  bid_price  bid_size  \
2017-03-10 11:07:51.564816        NaN       NaN        NaN       NaN   
                           canAutoExecute ignorabletick  last_trade_price  \
2017-03-10 11:07:51.564816           None          None             98.03   
                            last_trade_size pastLimit  
2017-03-10 11:07:51.564816              NaN      None  



In this case the tick was a trade, rather than a quote. The size of the trade arrives in the next tick.


>>> print(market_data1[2])

                            ask_price  ask_size  bid_price  bid_size  \
2017-03-10 11:07:51.564899        NaN       NaN        NaN       NaN   
                           canAutoExecute ignorabletick  last_trade_price  \
2017-03-10 11:07:51.564899           None          None               NaN   
                            last_trade_size pastLimit  
2017-03-10 11:07:51.564899              200      None  


Notice they're shown as a single row of a pandas Data Frame. This is so we can do this:

market_data1_as_df=market_data1.as_pdDataFrame()
print(market_data1_as_df)

The advantage of this approach will be clear later, when I discuss interpretation.


Once in the stream of history you can't get out 

 

If we just let that baby run we'd be receiving streams of prices until the cows came home. So what we do back in the client world is say STOP I've had enough after a preset amount of time (we could also STOP when the N'th tick has arrived, or when there all the slots in the marketdata are tuple are filled, which would be easy enough to code up).

from __main function:
time.sleep(30)
market_data2 = app.stop_getting_IB_market_data(tickerid)


from TestClient.stop_geting_IB_market_data() method:

def stop_getting_IB_market_data(self, tickerid):
    """    
    Stops the stream of market data and returns all the data we've had since we last asked for it
    :param tickerid: identifier for the request    

    :return: market data    
    """

    self.cancelMktData(tickerid)

    ## Sometimes a lag whilst this happens, this prevents 'orphan' ticks appearing 
    time.sleep(5)

    market_data = self.get_IB_market_data(tickerid)
    ## output any errors    while self.wrapper.is_error():
       print(self.get_error())
return market_data


This will also return any data that we haven't yet captured with a previous call to get_IB_market_data. Again feel free to change this.


Making the results meaningful


To understand the results we can use the power of pandas to resample the dataframe. First of all lets glue together the two seperate buckets of data we've captured:

market_data2_as_df=market_data2.as_pdDataFrame()
all_market_data_as_df=pd.concat([market_data1_as_df, market_data2_as_df])

Now to see the bid-ask quoting activity, resolved to a one second resolution:

some_quotes = all_market_data_as_df.resample("1S").last()[["bid_size","bid_price",
                                                    "ask_price",  "ask_size"]]
print(some_quotes.head(10))



                     bid_size  bid_price  ask_price  ask_size
2017-03-10 11:07:51    9952.0     98.030     98.040    3736.0
2017-03-10 11:07:52    2653.0        NaN        NaN     212.0
2017-03-10 11:07:53   17250.0     98.025     98.045    9500.0
2017-03-10 11:07:54    3607.0     98.030     98.040     424.0
2017-03-10 11:07:55   12992.0        NaN        NaN    5920.0
2017-03-10 11:07:56   10073.0        NaN        NaN    3743.0
2017-03-10 11:07:57    9746.0        NaN        NaN    3726.0
2017-03-10 11:07:58    8280.0        NaN        NaN    4110.0
2017-03-10 11:07:59      17.0        NaN        NaN    1723.0
2017-03-10 11:08:00    2920.0        NaN        NaN    3248.0



You could safely forward fill the prices (a nan is shown when there is no updated value).

Or the first few trades, resolved to 10 milliseconds:

some_trades = all_market_data_as_df.resample("10L").last()[["last_trade_price", "last_trade_size"]]
print(some_trades.head(10))

                         last_trade_price  last_trade_size
2017-03-10 11:07:51.560             98.03            200.0
2017-03-10 11:07:51.570               NaN              NaN
2017-03-10 11:07:51.580               NaN              NaN
2017-03-10 11:07:51.590               NaN              NaN

Here I'm using the 'last' method. You could also use an average.

By the way it can be a bit dangerous to average prices too much; for example if you sample prices throughout the day and then take an average as your input into your trading algorithm you will underestimate the actual amount of volatility in the market. Similarly if you are trading high frequency stuff you will be using the active state of the order book and averaging average real time bars is probably not going to be a very wise thing to do. Over this short time period relative to my typical trading speed however its probably okay as mostly all we are going to be removing is a little illusory volatility caused by 'bid-ask' bounce.

Also even with this averaging its still worth running your prices through a 'jump detector' to make sure you don't trade off dirty prices showing spuriously large moves; I see these about once a month for each instrument I trade!

Much much more on this subject in this post


Islands in the stream...


That is it for prices. I use the historical data function whenever I start trading a particular contract but also every day as it gets close prices. This makes my system self recovering since even if it drops for a few days I will end up with daily prices at least being infilled. Also often non actively traded contracts still have close prices, useful if you are using intra contract spreads as a data input. Just be careful how you treat intraday and closing prices if you append them together.

Much much more on this subject in this post

I use market data to get intraday prices where a system requires that, and when I am just about to trade to check the market is liquid enough for what I want to do (or even just to check it is open since I don't bother keeping a holidays calendar for all my markets - I wouldn't want to spend more than 10 minutes of my time a day running this system now would I?). Plus it allows me to dis-aggregate my trading costs into what is coming from the inside spread, the cost of processing / execution delays and having to drop deeper into the order book.

Next on the menu will be placing an order! I will leave the trivial task of building a system which decides what the orders will be to the reader (hint: you might want to use the price in some way).

This is the third in a series of posts. The first two posts are:

http://qoppac.blogspot.co.uk/2017/03/interactive-brokers-native-python-api.html

http://qoppac.blogspot.co.uk/2017/03/historic-data-from-native-ib-pyhon-api.html

The next post on placing orders is:
http://qoppac.blogspot.co.uk/2017/03/placing-orders-in-native-python-ib-api.html


Wednesday, 8 March 2017

Historic data from native IB python API

This is the second in a series of posts on how to use the native python API for interactive brokers. This post is an update of the post I wrote here, which used the 3rd party API swigibpy.

Okay so you have managed to run the time telling code in my last post.

Now we will do something a bit more interesting, get some market prices. Arguably it is still not that interesting, and this stuff will never be as interesting as a decent book or a good bottle of red wine, but we've all got to get the money to buy those books and bottles and this is better than many other options for achieving that.

Note: This post has been updated to use a more robust method for dealing with concurrency.


Let us peruse the IB ABI instructions...

Whenever you are trying something new out with the API you need to identify the EClient (command to ask the server for stuff) and EWrapper (command run by the wrapper client when stuff is presented unto it) functions. So Reading The Frigging Manual is worth doing.

And indeed there is not just one but four places in the weird and mysterious world of EClient where it looks like prices could be found (ignoring whacky methods for things like options):



  • reqMktData
  • reqHistoricalData
  • reqRealTimeBars
  • reqMarketDepth


Market Data: Returns a stream of price ticks (updated quotes and trades). Because it is a stream you have to ask for it, and then ask for it to stop lest you be buried under a huge pile of ticks. It is level 1, eg 'top of the order book' data.

Historical Data: Returns a one off chunk of data, both prices and volumes, with a given look back history and frequency.

Real Time Bars: Returns a stream of averages of the trades (or bid, ask, midpoint) over an interval, currently only 5 seconds. So its kind of like pre-clumped tick data arriving at regular intervals. Its a stream, so you have to ask for it... you get the idea.

Market Depth: Returns a stream of updates to the order book, both prices and size. This requires a requires a level 2 subscription. Its a stream...


Because real time barsmarket data and market depth deliver very similar information I won't discuss them all (and since I am a tightwad I don't have any Level 2 data subscriptions!). In this post we'll go through the 'one shot' historical data; and in the next one we'll look at one kind of streaming data (market data).


Historical Data AKA Those Who Cannot Remember the Past Are Doomed...

First get the code from this gist somewhere on a server far far away...

Also the relevant manual pages are here.


 Getting some price data 

 Just run the file (assuming you still have an IB connection available as per the previous post). You should get something like this:

Getting full contract details from the server... 
Getting historical data from the server... could take 10 seconds to complete 
historic_data
[('20170312', 98.145, 98.16, 98.135, 98.14, 53208), ('20170313', 98.14, 98.16, 98.135, 98.145, 53600), ('20170314', 98.135, 98.16, 98.135, 98.14, 21673)]


"Obviously" these are tuples containing daily prices (date, open, high, low, close, volume) for the September 2018 Eurodollar futures contract. If you get an error you might need to change the expiry date for the contract. I'll show you how to do this later in the post.

(Note before mixing these with any kind of other prices or doing proper backtesting I would personally append a 'fixed' end of date timestamp to them eg 23:59:59 to ensure there is no slight look forward bias. This is discussed more here.)


  The IB contract object or How IB Knows What We Want

Some code:

ibcontract = IBcontract()
ibcontract.secType = "FUT"
ibcontract.lastTradeDateOrContractMonth="201809"
ibcontract.symbol="GE"
ibcontract.exchange="GLOBEX"

(Note if you got an error above here is your chance to change the expiry to the current one)

Here is a real 'gotcha' - the IB contract object (when I say 'gotcha' it can be stuff I've literally spent days banging my metaphorical head against and in some cases my actual head). This is a complete description which identifies a particular thing you can trade. Most of the data you need is available from the https://www.interactivebrokers.com/ product listing pages. Some problems I have (there may be more associated with other asset classes) mainly in not giving enough information to uniquely identify a contract:

  • Getting the exchange name right - it needs to be exactly the same as on the product page. 
  • Having to specify a currency. This seems to be a problem for CAC40 and AEX futures (specify ibcontract.currency="EUR") and silver ("USD").
  • Having to specify a multiplier effectively contract movement in cash terms per price point (For CAC ibcontract.multiplier="10", AEX is 200 and SILVER is 5000)
  • Be careful, some contracts actually have an expiry date in a different
  • You have to specify the exact expiry YYYYMMDD of a VIX contract rather than just YYYYMM because there are now weekly expiries.
(By the way I don't actually use this contract object in the rest of my code. Instead I have another class which contains more useful information, and then a convert_to_ibcontract function. This function handles the horrible special cases above and also translates between meaningless names like  and better names like JohnBoy - my own pet name for the Eurodollar future *).

 (* This is a joke)

The next thing we need to do is get the contract details. This may sound a bit weird, we already have the contract details since we've just defined it? In practice the definition of a contract is very light and the details we get back are much richer.

There is one specific case where you must do this. To see why suppose it is a few months time and you are trading hundreds of futures contracts. A fill comes in for a particular contract instrument code and month/year. However as we shall see the identification for the contract isn't the 'yyyymm' code we use to define the contract, but the full expiry 'yyyymmdd'. You can't just take the first 6 characters of that either since some futures actually expire the month before they should. So to be on the safe side I usually get the expiry from IB before trading to make sure I am applying fills against the correct thingy.

Actually I don't do this immediately before trading but in a separate process that populates a database of contract information...


So we now need to "resolve" this contract into a fully featured contract which properly specifies the entire expiry date:

resolved_ibcontract=app.resolve_ib_contract(ibcontract)

Getting full contract details from the server... 

By the way if you haven't specified the initial contract object precisely enough, so there are multiple possible contracts, then you'll get a warning. You may also get a warning if the IB server doesn't deliver a "I am done with sending you contracts now sir" message by activating the wrapper method contractDetailsEnd()

All this is Fascinating Rob But How do we Actually get the Prices?!


historic_data = app.get_IB_historical_data(resolved_ibcontract)

This calls the following code in the TestClient object:


def get_IB_historical_data(self, ibcontract, durationStr="1 Y", barSizeSetting="1 day",
                           tickerid=DEFAULT_HISTORIC_DATA_ID):

    """    Returns historical prices for a contract, up to today

    ibcontract is a Contract
    :returns list of prices in 4 tuples: Open high low close volume    """

    self.wrapper.init_error()
    print("Getting historical data from the server... ")
    
    ## Make a place to store the data we're going to return    historic_data_queue = finishableQueue(self.init_historicprices(tickerid))
    today = datetime.datetime.now()

    # Request some historical data. Native method in EClient    
    self.reqHistoricalData(
        tickerid, # tickerId,        
        ibcontract,  # contract,        
        today.strftime("%Y%m%d %H:%M:%S %Z"),  # endDateTime,        
        durationStr,  # durationStr,        
        barSizeSetting,  # barSizeSetting,        
        "TRADES",  # whatToShow,        
         1,  # useRTH,        
         1,  # formatDate        
         [] ## chartoptions not used    )

    historic_data_queue = finishableQueue(self.init_historicprices(tickerid))
    ## Wait until we get a completed data, an error, or get bored waiting    MAX_WAIT_SECONDS = 10    historic_data = historic_data_queue.get(timeout = MAX_WAIT_SECONDS)

    while self.wrapper.is_error():
       print(self.get_error())

    if historic_data_queue.timed_out():
       print("Exceeded maximum wait for wrapper to confirm finished - seems to be normal behaviour")

   self.cancelHistoricalData(tickerid)


   return historic_data
The general structure of this function should be obvious to anyone who read the last post. We:

  • call some init functions in the wrapper instance (self.wrapper) so there is somewhere for error messages and our price data to go
  • Ask the IB API server for something with the inherited EClient method self.reqHistoricalData()
  • Wait patiently until the data is received, or until we get bored 
  • Complain about any errors, or return the results as appropriate 

The self.reqHistoricalData() asks the IB server for historical price data. We can set length of time (a year in this case, although because we are using a demo account there isn't that much data available) durationStr, the end time, the bar size (in this case days; not actually the size of the kind of bar you get drinks from) and some other things you can look at the documentation yourself to find out about. Just to note not all the combinations of time period and bar length are permitted - you can't get second by second prices for a year (see this link.) even if your hard drive could store them.

About the only thing that might not be obvious is tickerid. You might be the kind of wild crazy person who requests loads of historical data for different contracts. In which case you would get data coming back randomly and you wouldn't know which contract it was for. So you have to provide some kind of label for the request if you request loads of things at once. Note you can't actually do that because if you request more than 1 set of data about every 10 seconds you get a 'pacing violation'  (basically, don't take the mickey).

This is only one side of the story, as....

Oh Yeah I remember now, we have to have an EWrapper function on the server side... 

From the TestWrapper class, as usual we have a storage method where we put stuff in a Queue:


## Historic data code

def init_historicprices(self, tickerid):
    historic_data_queue = self._my_historic_data_dict[tickerid] = queue.Queue()

    return historic_data_queue

And some methods in EWrapper which have been overriden:


def historicalData(self, tickerid , date:str, open:float, high:float,
                   low:float, close:float, volume:int, barCount:int,
                    WAP:float, hasGaps:int):

    ## Overriden method    ## Note I'm choosing to ignore barCount, WAP and hasGaps but you could use them if you like    bardata=(date, open, high, low, close, volume)

    historic_data_dict=self._my_historic_data_dict

    ## Add on to the current data    if tickerid not in historic_data_dict.keys():
        self.init_historicprices(tickerid)

    historic_data_dict[tickerid].put(bardata)

def historicalDataEnd(self, tickerid, start:str, end:str):
    ## overriden method
    if tickerid not in self._my_historic_data_dict.keys():
        self.init_historicprices(tickerid)

    self._my_historic_data_dict[tickerid].put(FINISHED)


There is no magic in creating these things; to write this stuff you look in the API reference to see what the wrapper function that requires overriding (in this case historicalData) returns. In this case the wrapper function beastie is called repeatedly with the same arguments for each row of data. In theory historicalDataEnd would also be called, however as with getting contract details this might never happen (I've had mixed success with it). 

<snip>

Back in the TestClient object:


    historic_data_queue = finishableQueue(self.init_historicprices(tickerid))
    ## Wait until we get a completed data, an error, or get bored waiting    MAX_WAIT_SECONDS = 10    historic_data = historic_data_queue.get(timeout = MAX_WAIT_SECONDS)


It's now worth understanding what a finishableQueue is; 

## marker for when queue is finishedFINISHED = object()
STARTED = object()
TIME_OUT = object()

class finishableQueue(object):

    def __init__(self, queue_to_finish):

        self._queue = queue_to_finish
        self.status = STARTED

    def get(self, timeout):
        """
        :param timeout: how long to wait before giving up waiting for a FINISHED flag        :return: list of Queue elements        """        contents_of_queue=[]
        finished=False
        while not finished:
            try:
                current_element = self._queue.get(timeout=timeout)
                if current_element is FINISHED:
                    finished = True                    self.status = FINISHED
                else:
                    contents_of_queue.append(current_element)
                    ## keep going and try and get more data
            except queue.Empty:
                ## If we hit a time out it's most probable we're not getting a finished element                ## give up and return what we have                finished = True                self.status = TIME_OUT


        return contents_of_queue

    def timed_out(self):
        return self.status is TIME_OUT


This is just a simple class for keeping track of Queues of multiple items in the wrapper thread where the results have some 'finish' point, which may not come.

And once again, back in the TestClient object:

    historic_data = historic_data_queue.get(timeout = MAX_WAIT_SECONDS)


    while self.wrapper.is_error():
       print(self.get_error())
    if historic_data_queue.timed_out():
       print("Exceeded maximum wait for wrapper to confirm finished - seems to be normal behaviour")
   self.cancelHistoricalData(tickerid)

   return historic_data

So we get this output:

Getting historical data from the server... could take 10 seconds to complete 
historic_data
[('20170312', 98.145, 98.16, 98.135, 98.14, 53208), ('20170313', 98.14, 98.16, 98.135, 98.145, 53600), ('20170314', 98.135, 98.16, 98.135, 98.14, 21673)]

(By the way exactly the same finishableQueue idea is used when we resolve the contract details)

That then is everything you need to know about historical data. I will cover market data, i.e. the wonderful world of streaming data, in the next post.

This is the second in a series of posts. 
The previous post was: qoppac.blogspot.co.uk/2017/03/interactive-brokers-native-python-api.html

The next post is qoppac.blogspot.co.uk/2017/03/streaming-market-data-from-native.html