Getting accounting data out of interactive brokers API into python with swigibpy

This is the final post. Not the final post of the blog; which may be good news or bad. But the final post in my short series on using swigibpy to interact with the C++ API for interactive brokers (IB).

Interactive brokers now have a native python API. An updated version of this post which uses that API is here: http://qoppac.blogspot.co.uk/2017/03/placing-orders-in-native-python-ib-api.html

Having got some prices and submitted some orders we want to know whether we made any money or not. Although the code is rather trivial, interpreting the results requires some understanding.

http://qoppac.blogspot.co.uk/2017/03/getting-position-and-accounting-data.html

So once again you need to get the code from the following repo.

The example code to run is in test5_IB.py and wrapper_v5.py. This will give your results with live or simulated Gateway / TWS session running (it probably won't produce anything of interest with a demo account but you can still follow the code). Note that IB have recently updated their Gateway software so you might need to re-download.


The output from test5_IB.py should be something like what is in green below.

Expiry is 20140908




Portfolio

([('3KTB', '20140617', 12, 106.0, 1272000000.0, 105842090.65714166, 1894912.11, 0.0, 'DU99', 'KRW'), ('BTP', '20140606', 2, 123.2049942, 246409.99, 121025.33335, 4359.32, 0.0, 'DU99', 'EUR'), ('EMG', '', 1100000, 0.91125, 1002374.99, 0.85376155, 63237.28, 0.0, 'DU99', 'GBP')]

account info

[('AccountCode', 'DU99', '', 'DU99'), ('AccountReady', 'true', '', 'DU99'), ('AccountType', 'INDIVIDUAL', '', 'DU99'), 

<SNIP>

, ('WarrantValue', '0', 'USD', 'DU99')]

 

 

Contract details - again?

Note we reuse the contract details code in http://qoppac.blogspot.co.uk/2014/05/the-placing-of-orders-to-interactive.html

The reason we do this is as before to correctly identify our positions we need the yyyymmdd expiry associated with the contract; although the matching up process isn't done here.

 

Positions

We're going to use a single client function to get both accounting and position data. However both of these are going to be looking at streamed accounting updates. This is returned by two different kinds of wrapper function.
 

Since writing this blog IB have introduced an API call to get positions directly - so there are less clunky ways of doing this than I show here. 

 From wrapper_v5.py, class IBclient: 


    def get_IB_account_data(self):

        self.cb.init_portfolio_data()
        self.cb.init_error()
       
        ## Turn on the streaming of accounting information
        self.tws.reqAccountUpdates(True, self.accountid)
       
        start_time=time.time()
        finished=False
        iserror=False

        while not finished and not iserror:
            finished=self.cb.flag_finished_contractdetails
            iserror=self.cb.flag_iserror

            if (time.time() - start_time) > MAX_WAIT_SECONDS:
                finished=True
                print "Didn't get an end for account update, might be missing stuff"
            pass

        ## Turn off the streaming
        ## Note portfolio_structure will also be updated
        self.tws.reqAccountUpdates(False, self.accountid)

        portfolio_data=self.cb.data_portfoliodata
        account_value=self.cb.data_accountvalue

       
       
        if iserror:
            print self.cb.error_msg
            print "Problem getting details"
            return None

        return (account_value, portfolio_data)


From wrapper_v5.py, class IBWrapper:

    def init_portfolio_data(self):
        if "data_portfoliodata" not in dir(self):
            setattr(self, "data_portfoliodata", [])
        if "data_accountvalue" not in dir(self):
            setattr(self, "data_accountvalue", [])
          
      
        setattr(self, "flag_finished_portfolio", False)
      

    def updatePortfolio(self, contract, position, marketPrice, marketValue, averageCost, unrealizedPNL, realizedPNL, accountName):
        """
        Add a row to the portfolio structure
        """

        portfolio_structure=self.data_portfoliodata
              
        portfolio_structure.append((contract.symbol, contract.expiry, position, marketPrice, marketValue, averageCost,
                                    unrealizedPNL, realizedPNL, accountName, contract.currency))

    ## account value
  
    def updateAccountValue(self, key, value, currency, accountName):
        """
        Populates account value dictionary
        """
        account_value=self.data_accountvalue
      
        account_value.append((key, value, currency, accountName))
      

    def accountDownloadEnd(self, accountName):
        """
        Finished can look at portfolio_structure and account_value
        """
        setattr(self, "flag_finished_portfolio", True)
So once we begin streaming both the portfolio_structure and account_value global variables begin updating. Note that with portfolio_structure we have the same 'gotcha' as before, namely that we can't just drop the contract object into the global variable, as all we will get is a pointer that is meaningless outside of the wrapper method. Instead we need to save specific things of interest like contract.symbol.

Portfolio data

The output portfolio_data will be a list of tuples (contract.symbol, contract.expiry, position, marketPrice, marketValue, averageCost, unrealizedPNL, realizedPNL, accountName, contract.currency). Note that the value, cost, PNL will be in local currency. You can extract exchange rates from the account_value information - which I discuss below.

Lets take an example:

('BTP', '20140606', 2, 123.2049942, 246409.99, 121025.33335, 4359.32, 0.0, 'DU99', 'EUR')

This is the June 2014 BTP (Italian 10 year bond) which I am long 2 in account DU99 (not my real account number) which is trading at 123.205 Euros per contract. I should use the contract information from above to match this to the yyyymm contracts my system knows about. My two contracts are worth Euro 246,410 because each contract price point is worth 1000 Euros (I can get this information from the contract details call).

I actually bought each of these things for 121,025 Euros each (so 121.025 in price points). This ill conceived lending of a quarter of a million Euros to the most unstable government in the Euro zone has so far earned me Euros 4359.32 (246,410 minus 2 * 121,025; or 2 * [123.205 - 121.025] * 1000).

Note my realized PnL is zero since I obviously haven't sold any of these futures yet (no quick profit taking for account DU99).

Here is the corresponding exchange rate from the account_value output.


[... ("ExchangeRate", 0.815491, "EUR", "DU99"),....
 ("ExchangeRate", 1.00000, "GBP", "DU99"),... ]

From this we can make out that my base currency is GBP, and 1 Euro is worth 0.815491 GBP. So I can convert my ill gotten gains in monopoly money ECB issued Euros 4359.32 * 0.815491 = £3,555 in real money.

For more information see the online documentation
C++ > Class EWrapper Functions > Account and Portfolio 

Note be careful about using comparisions between these portfolio snapshots from different times to calculate your profits. They won't take trades into account very well, and if you have sold something completely or hadn't bought it for the first snapshot the comparision wont' work.

Finally also be very careful about using these snapshots to work out what positions you have. Firstly it is common to get the position coming back repeated times. So you will need to filter out duplicates. Secondly positions may sometimes not appear at all.

I use my own record of trades to find out what my current position should be. I compare this to the IB snapshot throughout the day. If there are any breaks in the reconciliation I temporarily halt trading. Occasionally this is because IB has sent me a fill I haven't picked up, or because the IB position snapshot is missing some contracts.

Accounting information

The account_value output will be a very long list of tuples. Each one consists of (keyname, value, currency, account number). Currency BASE indicates it is the accounts base currency (GBP for me). Here are some of the more interesting account value entries.

• ExchangeRate: This is what we use in the get positions example above. 
•  FuturesPNL: This is how much we have made trading futures today, by currency. BASE is the total in base currency.
• StockMarketValue: Self explanatory
• NetLiquidation: This is what is my account worth if I close all my positions out (ignoring commissions and bid-ask). Its what I use to ultimately determine the total profit and loss used for rescaling my risk capital.
• CashBalance: Quite self explanatory. Negative means you have borrowed money. BASE is the net of your balances converted into base currency.

When you initially do a futures trade in a non BASE country you will have to borrow money for initial margin. Only if you move into profit beyond this will you have no net borrowing. IB charges you for borrowing money! This is done at LIBOR+spread so is expensive for currencies with higher interest rates (this spread is also why I use IMM's to get direct currency exposure). You can flatten out your exposure by doing a spot FX trade. Personally I try and keep a small positive balance in all currencies, although not excessive as this means I am taking on currency risk. Note you can't trade all possible pairs eg if you find you can't buy AUDGBP then try selling GBPAUD instead. The exception is Korea where you can't hold any speculative currency exposure i.e. not arising from margin requirements in other kinds of trading. All you are allowed to do is flatten your Korean currency position back to zero.

There are many more keywords than shown above. The best advice I have for trying to understand what they all mean is to start with a clean simulated account (a demo account is no good since you are exposed to other peoples random trading, and the account will often be 'empty'). Take a dump of the account value output, then do a trade. Take another dump, then wait for prices to move. By comparing each dump you should be able to see how account value, margin, cash and so on interact. This is quite time consuming but definitely worth it for an insight into how the IB accounts operate for the kind of instruments you want to trade.

Anything missing?

I haven't covered the full gamut of what you can get from the IB API. There are some newer methods put in since I began using it to get positions and account summary information. There is a much richer variety of price and option data that is available. I haven't looked at fundamental data, option data, scanners and managed accounts because I don't use them. However I hope there is enough meat in this series of examples to get you started using IB with swigibpy, and to avoid wasting time trying to understand some of the weirder behaviour.

This is the final of a series of five posts on constructing a simple interface in python to the IB API using swigiby. The first four posts are:
http://qoppac.blogspot.co.uk/2014/03/using-swigibpy-so-that-python-will-play.html

http://qoppac.blogspot.co.uk/2014/04/getting-prices-out-of-ib-api-with.html 

http://qoppac.blogspot.co.uk/2014/04/streaming-prices-from-ib-api-swigibpy.html

http://qoppac.blogspot.co.uk/2014/05/the-placing-of-orders-to-interactive.html


If you've found this series useful then your next steps might be to learn how to design a systematic trading strategy, and to understand the nuts and bolts of creating a fully automated system.

Bitcoin, money, gold and my great unpublished novel

I have been asked by one of the readers of this blog to write about Bitcoin (So the first thing to note is that there is at least one other person reading this, it isn't just you). My first reaction was that I knew nothing about Bitcoin and anything I could say would just be meaningless biased drivel written to satisfy some agenda. Of course that would be no different from nearly any other Bitcoin based commentary. Or the rest of this blog.

Great lost novels of the past #1

My second reaction when I think of Bitcoin is to remember a novel I wrote about 15 years ago over two weeks. As you might expect from the fact it took two weeks to write it wasn't a very good novel. The loss of the only extant copy on a three and a half inch floppy was no great loss to the world. Certainly the disappearance of the floppy disk was a much greater loss to the world. At least to the world of double entendres since only people over 35 will laugh when I remind them that three and half inch floppies were actually stiff (unlike five and a quarter inch versions).

Storage media aside the most interesting part of the novel was the plot. Essentially in the future some bloke had created an alternative underground offshore currency. Another bloke was recruited by M15 to penetrate his organisation allegedly because it was helping to launder money. Actually the governments motive was to reduce tax evasion. Sound familiar?

So when thinking about Bitcoin I get a deep sense of regret that I didn't self publish the awful book at the time. Of course given the shocking writing it would have sold zero copies but subsequently I would have been recognised as a modern day prophet like William Gibson.

A question asked by an eight year old boy

I will now belatedly get to the point. To answer the question 'what is the difference between Bitcoin and money' we have first have to answer the question my son genuinely asked last week 'What is the difference between a cheque and money?'. To answer that question we need to know how what we call 'money' came about. The paper money and coins we use in modern societies are actually just cheques. A ten pound note is just a cheque for ten pounds, payable to bearer and drawn on the Bank of England. It also has a picture of the Queen and Charles Darwin on it which my cheques from failed socialist experiment the Co-op bank don't have. In a sense both are IOU's. One says the Bank of England owes the bearer a tenner. Originally that was ten pounds of gold, but that link has long since been broken. The Co-op cheque says that I owe the named payee ten quid.

The main difference between them is that I can use the ten pound note to buy just about anything. If I had a million ten pound notes I could easily buy a one bedroom flat in a wealthy part of London (unless house price inflation has rendered even that statement untrue by the time you read this). Though there might be some tiresome money laundering forms to fill in. On the other hand if I tried to buy that flat with a cheque for ten million quid there would be a short delay whilst someone ascertained whether that IOU was actually backed by a corresponding amount in my bank account (sadly it wouldn't be). One of the reasons for the picture of the Queen and all the security features is that the immediate acceptability of the humble tenner means that it is worth trying to forge.

Me and My Plumber

My bank account of course just contains electronic versions of the paper ten pound notes. So a cheque is an IOU which can be converted into a paper or electronic IOU. The important point here is we don't have to use money (a specific form of IOU), we could just use any IOU. Suppose my plumber and I agreed to do this. If we can come up with a cool name (carverCash?), agree on some prices (one poor science fiction novel – ten carverCashes. One toilet unblocked – five hundred carverCashes) then we have a currency. We could even set up an electronic exchange mechanism.

So we then have a fully fledged cryptocurrency. Which only two people will accept (and frankly I am not sure about my plumber going for it). This then is the Achilles heel of bitcoin. The other mechanics of bitcoin are not relevant. Its not quite unbreakable security, equivalent of the picture of the Queen and metal strip in the tenner, is not relevant. The cool sounding mining process is not relevant. If hardly anybody wants to use it then it is pointless.

If we think of the world of currencies as a product market then bitcoin is like the small record shop trying to compete against Amazon and Tesco. It might offer something unique and different like the ability to buy rare Pink Floyd white labels or to purchase drugs over the Internet with some anonymity. But most people don't need that so the most realistic best possible outcome is that the shop survives in a small niche; it is very unlikely to end up displacing the giants. The last time a currency 'broke into' the market and became a globally accepted means of exchange was when the US dollar became more important than the British pound about a hundred years ago, because the US displaced Britain as the leader in global trade. The Chinese currency may well overtake the dollar at some point if it becomes freely exchangeable, but that will be for the same reason - because nearly everything we buy is made in China. Bitcoin has no such advantage.

If not money, then what?

So having dismissed the chances of bitcoin becoming a globally accepted currency we are left with is it a possible investment. In my spare time I like to classify assets by their degree of being divorced from reality. If I go and buy a farm from someone that is pretty close to reality. When the farm is owned by a public company and I buy shares in that company we are getting slightly further away. Owning a call option on the companies shares is moving further away again, but underneath it all is the farm – a bad harvest will ultimately reduce the value of my option. Buying an interest rate future is entering a really whacky world since the underlying thing is not real but a price based on a survey of what people think the price should be (at least until the LIBOR market is properly reformed).

The price of anything is driven by supply and demand. Even with assets that are more real there is a good deal of 'animal spirits' in both elements so prices can be moved away from what might be called a fundamental value for quite some time. However in instruments linked more closely to real things there are limits to how crazy things can get. It may take time but speculative bubbles and anti bubbles in asset prices linked to real things always eventually pop at some price ceiling or floor.  This isn't true if you have an asset which doesn't have any connection to underlying reality.

So the value of bitcoins really has no floor or ceiling. If nobody wants them they will be worthless. If everyone wants them, given they will stop making them in the future, they will be priceless. That 'want' will be based purely on subjective perception rather than any underlying real factors. This does not mean they are a bad investment. Rather that one should not invest in them like you would invest in UK equities. Holding a constant long position in UK equities is a reasonable thing to do. For bitcoins it is probably not. We need to actively trade bitcoins.

Bitcoins equal not Money. Gold equal Money? -> Gold equal Bitcoins ?

A better asset to think about when trading bitcoins would be the asset that used to back all money – Gold. Why gold? Gold is not dissimilar to bitcoins in price behaviour. Underlying supply of gold is relatively short run inelastic, so speculative demand shocks will pull the price all over the place. It does have some industrial uses but its value is mostly determined by very subjective demand. If everyone stopped desiring gold tomorrow it would be pretty much worthless and similarly there is no potential ceiling on its future price. There is no income or rights to any underlying cashflow for either (although owning gold futures might earn a 'carry').

Gold does have however a much larger and stickier demand base, bitcoin jewellery being a pure substitute for the shiny yellow stuff. Also it has a long history of being a reasonable inflation hedge. So having a bit of static gold in your portfolio isn't a completely absurd idea.

Its worth noting that I mainly trade gold using trend following signals that are specifically designed to pick up changes in subjective sentiment, like those that drive other less real assets like bitcoins. Using a similar strategy for bitcoins might make sense. If we look at figure 1 we can see that there appear to be some decent trends. Note I have used a log scale so you can see the movements in the past more clearly.

 A simple moving average crossover indicates a short position in figure 2 might make currently sense.  This cursory look suggests going long bitcoins might not be the smartest move. In reality I would want to do much more analysis than this before even thinking about trading bitcoins.

Having done that I would be happy to trade bitcoin futures on a recognised exchange backed by a proper clearing house, if they were liquid enough. After all I trade gold futures. I even trade Eurodollar futures which are even more highly divorced from reality. That isn't possible now and a spot position in the actual asset makes me very nervous since the counterparty risk seems extremely high.

So the short answer is I wouldn't personally touch bitcoin with a bargepole. Not even if I needed to buy some to research my next novel.

The placing of orders to interactive brokers with swigibpy / python

We are nearly at the end of our journey of simplistic examples of how to get the swigibpy package to mediate between the wonderful world of Python and the dark place that is the Interactive brokers C++ API. Having learned how to get 'snapshot' and streaming prices out of the API we are now ready to actually do some trading- submit orders, check they are active, potentially cancel them, receive fills and get historic execution data.

Interactive brokers now have a native python API. An updated version of this post which uses that API is here: http://qoppac.blogspot.co.uk/2017/03/placing-orders-in-native-python-ib-api.html

Where do I start?

You need to get the code from the following repo.

The example code to run is in test4_IB.py. If you have a live or simulated Gateway / TWS session running (one associated with a real account number) it should work just fine. Note that you can also run this with the edemo account, but the results will be very ugly. This is because you are seeing the orders placed by everyone who is playing around with this account, so you could get all kinds of randomeness.


WARNING I highly recommend that you do not run this code on a real money trading account until / unless you know exactly what you are doing! In particular real trading code should check market liquidity before trading, particularly for market orders.

The output should be as in green below. Of course feel free to change the contract traded and the quantities...

Expiry is 20150908



Getting orderid from IB
Using order id of 1

Placed order, orderid is 1
Getting orderid from IB
Using order id of 2

Placed limit order, orderid is 2
recived fill as follows:

{'orderid': '1', 'account': 'DU15031', 'exchange': 'DTB', 'symbol': 'GBL', 'permid': 193880352, 'execid': '00004468.559df2a0.01.01', 'clientid': '999', 'expiry': '20150908', 'price': 156.37, 'qty': 10, 'side': 'BOT', 'times': '20150709  14:05:11'}

Active orders: (should just be limit order)
{2: {'orderid': 2, 'symbol': 'GBL', 'qty': 10, 'side': 'SELL', 'clientid': 999L, 'expiry': '20150908'}}

Cancelling remaining orders
Cancelling 2
Waiting for cancellation to finish
No active orders now
False

Following executed since midnight this morning:

{'1': {'orderid': '1', 'account': 'DU15031', 'exchange': 'DTB', 'symbol': 'GBL', 'permid': 193880352, 'execid': '00004468.559df2a0.01.01', 'clientid': '999', 'expiry': '20150908', 'price': 156.37, 'qty': 10, 'side': 'BOT', 'times': '20150709  14:05:11'}}

Contract details - what is it?

As usual there is the tedious business of getting a client object and defining a contract. The first proper thing we need to do is get the contract details. This may sound a bit weird, we already have the contract details since we've just defined it? In practice the definition of a contract is very light and the details we get back are much richer.

There is one specific case where you must do this. To see why suppose it is a few months time and you are trading hundreds of futures contracts. A fill comes in for a particular contract instrument code and month/year. However as we shall see the identification for the contract isn't the 'yyyymm' code we use to define the contract, but the full expiry 'yyyymmdd'. You can't just take the first 6 characters of that either since some futures actually expire the month before they should. So to be on the safe side I usually get the expiry from IB before trading to make sure I am applying fills against the correct thingy.

Actually I don't do this immediately before trading but in a separate process that populates a database of contract information...

 

From test4_IB.py
     callback = IBWrapper()
    client=IBclient(callback)
   
    ibcontract = IBcontract()
    ibcontract.secType = "FUT"
    ibcontract.expiry="201509"
    ibcontract.symbol="GBL"
    ibcontract.exchange="DTB"
    cdetails=client.get_contract_details(ibcontract)

WARNING: If you are trading the VIX, which now has weekly expiries, you will need to specify the full expiry date in yyyymmdd format.


I won't go through the client code in detail but there are two 'gotchas' in the server code that appear a few times so lets look at an excerpt.

From wrapper_v4.py, class IBWrapper:
    def contractDetails(self, reqId, contractDetails):       
        contract_details=self.data_contractdetails[reqId]
 
contract_details["contractMonth"]=contractDetails.contractMonth
        ## <SNIP - code removed>
        contract_details["evMultiplier"]=contractDetails.evMultiplier

        contract2 = contractDetails.summary

        contract_details["expiry"]=contract2.expiry
        ## <SNIP - code removed>
 
contract_details is of course a global object that gets picked up by the client object when populated. The first of the two gotchas here is an old one; if you are submitting multiple of these to the same API session watch out to use meaningful request ID's. More subtle is that we don't just do something like contract_details=contractDetails.Instead we actually extract the information into a new object. The reason is that the little jobbys are just weird pointers that will become pointers to something else, usually some weird memory address, by the time you look at them outside of the callback instance. So you have to capture their essence in the wild before they escape. This is the well known quantum pointers effect.

To find out more about contract and contractDetails objects look at the manual (C++ / SocketClient Properties).

Note I use a dicts or dicts of dicts here for contracts, orders and fills but in reality I have little classes for them to keep things neater.

Order placing - can I buy it?

Now the moment we've all been waiting for... we're actually going to buy something. For fun we're going to place two orders, a market order and a limit order.

From test4_IB.py 
     ## Place the order, asking IB to tell us what the next order id is
    ## Note limit price of zero
    orderid1=client.place_new_IB_order(ibcontract, 10, 0.0, "MKT", orderid=None)
   
    print ""
    print "Placed market order, orderid is %d" % orderid1
   
    ## And here is a limit order, unlikely ever to be filled
    ## Note limit price of 100
    orderid2=client.place_new_IB_order(ibcontract, -10, 200.0, "LMT", orderid=None)
    print ""
    print "Placed limit order, orderid is %d" % orderid2


 From wrapper_v4.py, class IBclient:  
    def place_new_IB_order(self, ibcontract, trade, lmtPrice, orderType, orderid=None):

        <SNIP>       
        iborder = IBOrder()
        iborder.action = bs_resolve(trade)
        iborder.lmtPrice = lmtPrice
        iborder.orderType = orderType
        iborder.totalQuantity = abs(trade)
        iborder.tif='DAY'
        iborder.transmit=True
 
        ## We can eithier supply our own ID or ask IB to give us the next valid one
        if orderid is None:
            print "Getting orderid from IB"
            orderid=self.get_next_brokerorderid()
           
        print "Using order id of %d" % orderid
   
         # Place the order
        self.tws.placeOrder(
                orderid,                                    # orderId,
                ibcontract,                                   # contract,
                iborder                                       # order
            )
   
        return orderid

       
Obvious first thing to notice here is the concept of an orderid. This is a number that identifies to IB what the order is; at least temporarily and for today only. Only restriction on order id's is that the next order is higher than the last. This means if you submit an order with an id of 999999 you will lose all the orderids below that. You can also reset the id 'clock' to 1 via an option on the Gateway or TWS API configuration menu.  Safest thing to do is ask IB for the next orderid as done here by supplying None to the calling function.

In practice I generate my own orderid's preferring to reserve them first in my own database. This is fine as long as you are running a single linear process where there is no chance of an 'older' order being submitted before a 'newer' one.

Notice the limit price of 0.0 for the market order won't be used. For more information on more funky order types see the docs:  Reference > Order Types and IBAlgos > Supported Order Types.

 Active order status- have I bought it?

IB can tell us what orders we are working. Unless you ask very quickly (or submit your order outside of trading hours) this is likely only to return limit orders.

From test4_IB.py 
    order_structure=client.get_open_orders()
    print "Active orders: (should just be limit order)"
    print order_structure
    print ""


 From wrapper_v4.py, class IBclient:  

    def get_open_orders(self):
        """
        Returns a list of any open orders
        """
        <SNIP>
        self.tws.reqAllOpenOrders()
        iserror=False
        finished=False
       
        while not finished and not iserror:
            finished=self.cb.flag_order_structure_finished
            iserror=self.cb.flag_iserror
            if (time.time() - start_time) > MAX_WAIT_SECONDS:
                ## You should have thought that IB would teldl you we had finished
                finished=True
            pass
       
        order_structure=self.cb.data_order_structure
        if iserror:
            print self.cb.error_msg
            print "Problem getting open orders"
   
        return order_structure   
 


From wrapper_v4.py, class IBWrapper:

    def init_openorders(self):
        setattr(self, "data_order_structure", {})
        setattr(self, "flag_order_structure_finished", False)

    def add_order_data(self, orderdetails):
        if "data_order_structure" not in dir(self):
            orderdata={}
        else:
            orderdata=self.data_order_structure

        orderid=orderdetails['orderid']
        orderdata[orderid]=orderdetails
                       
        setattr(self, "data_order_structure", orderdata)


    def openOrder(self, orderID, contract, order, orderState):
        """
        Tells us about any orders we are working now
       
        Note these objects are not persistent or interesting so we have to extract what we want
       
       
        """
       
        ## Get a selection of interesting things about the order
        orderdetails=dict(symbol=contract.symbol , expiry=contract.expiry,  qty=int(order.totalQuantity) ,
                       side=order.action , orderid=int(orderID), clientid=order.clientId )
       
        self.add_order_data(orderdetails)

    def openOrderEnd(self):
        """
        Finished getting open orders
        """
        setattr(self, "flag_order_structure_finished", True)

 
In practice I have noticed that the correct end condition for receiving open orders doesn't always trigger so you do need an max waiting time (which is good practice anyway). Here we are just getting a selection of the available information about the order (see C++ > SocketClient Properties > Order and OrderState for more). Again we don't just save the order object itself as this is just a quantum pointer that will change.

Fill data - how much did it cost me?

What happens when an order is filled; completely or partially? Well the following method in the wrapper function is triggered. Notice that the logic is slightly more complicated because this function fulfills two duties. When it's called for a fill then reqId=-1, and action_ib_fill will be called.

From wrapper_v4.py, class IBWrapper:
    def execDetails(self, reqId, contract, execution):
           
    def execDetails(self, reqId, contract, execution):
   
        """
        This is called if
       
        a) we have submitted an order and a fill has come back
        b) We have asked for recent fills to be given to us
       
        We populate the filldata object and also call action_ib_fill in case we need to do something with the
          fill data
       
        See API docs, C++, SocketClient Properties, Contract and Execution for more details
        """
        reqId=int(reqId)
       
        execid=execution.execId
        exectime=execution.time
        thisorderid=int(execution.orderId)
        account=execution.acctNumber
        exchange=execution.exchange
        permid=execution.permId
        avgprice=execution.price
        cumQty=execution.cumQty
        clientid=execution.clientId
        symbol=contract.symbol
        expiry=contract.expiry
        side=execution.side
       
        execdetails=dict(side=str(side), times=str(exectime), orderid=str(thisorderid), qty=int(cumQty), price=float(avgprice), symbol=str(symbol), expiry=str(expiry), clientid=str(clientid), execid=str(execid), account=str(account), exchange=str(exchange), permid=int(permid))

        if reqId==FILL_CODE:
            ## This is a fill from a trade we've just done
            action_ib_fill(execdetails)
           
        else:
            ## This is just execution data we've asked for
            self.add_fill_data(reqId, execdetails)
Again we extract the useful information from the contract and execution objects rather than saving them directly. More on the IB website docs C++ > SocketClient Properties > Execution. To reiterate the expiry here will be the actual yyyymmdd expiry date; not the shorter yyyymm that the original trade was constructed with, hence the need to identify the former before submission.

A brief technical note; the date is the contract expiry not where relevant first or last notice date. This means you should be wary of using this date to tell you when to roll certain kinds of futures contracts eg US bonds.



From IButils:

 def action_ib_fill(execlist):
    print "recived fill as follows:"
    print ""
    print execlist
    print ""

Although this is very boring in practice this would be the function that would update your order status database (as a pro at this stuff, naturally you would have such a thing). It won't be obvious from this simple example unless you can submit a very large order in a thin market but the fills come in as cumulative order updates, not separate fills. Its worth looking at an example. Suppose you try and buy 10 lots, and you get fills of:

• 3 lots @ 100.0
• 6 lots @ 100.2
• 1 lot @ 100.5

 Then the fills that come in will look like this:

• qty: 3 lots, price=100.0
• qty: 9 lots, price=100.13333333
• qty: 10 lots, price=100.17

So if you do care about each partial fill you are going to have to hope that you see every little fill coming in and use a differencing process to see the detail of each.

By the way 'orderid' is only a temporary thing for IB; after tommorrow it won't associate it with this order. Instead you should use 'permid' for your record keeping. 'execid' is different for each part fill so you could use it to make sure you aren't including fill information you already have; in practice this isn't problematic due to the cumulative nature of the information.

Order cancelling - what if I don't want it any more?

Its very easy indeed to cancel an order; we don't even need a call in the client object to do it.

From test4_IB.py
    print "Cancelling the limit order"
    client.tws.cancelOrder(orderid2)

    print "Waiting for cancellation to finish"   
    while client.any_open_orders():
        pass
    print "No active orders now"
    print client.any_open_orders()
    print ""

Past execution data - sorry, repeat that, how much?!

It is clearly very important that fill data is correctly captured by your trading software. One reason being to keep track of what your position is; as we shall see in the next post IB doesn't offer mere mortals a super accurate current position facility. So I generally use my own knowledge of trade history to decide where I am, position wise. Because the fills usually arrive in the wrapper function only once its possible under certain conditions to miss them; eg if your API client dies before you see the fill or just isn't running when one arrives on a previously closed market in the middle of the night. Its generally good practice then to reconcile what IB has for a record of fills versus your own.

This information is only available up to midnight of the day you trade. So I run a reconciliation 3 times a day. If you lose a fill from before today you will need to find it on the IB website account management microsite, and manually enter it into your database.

Here is how we do it.

From test4_IB.py 

    execlist=client.get_executions()
   
    print "Following executed since midnight this morning:"
    print ""
    print execlist


 From wrapper_v4.py, class IBclient:  
    def get_executions(self):
        """
        Returns a list of all executions done today
        """
       
        assert type(reqId) is int
        if reqId==FILL_CODE:
            raise Exception("Can't call get_executions with a reqId of %d as this is reserved for fills %d" % reqId)

        self.cb.init_fill_data()
        self.cb.init_error()
       
        ## We can change ExecutionFilter to subset different orders
       
        self.tws.reqExecutions(reqId, ExecutionFilter())

        iserror=False
        finished=False
       
        start_time=time.time()
       
        while not finished and not iserror:
            finished=self.cb.flag_fill_data_finished
            iserror=self.cb.flag_iserror
            if (time.time() - start_time) > MAX_WAIT_SECONDS:
                finished=True
            pass
   
        if iserror:
            print self.cb.error_msg
            print "Problem getting executions"
       
        execlist=self.cb.data_fill_data[reqId]
       
        return execlist
       
       
We can change ExecutionFilter to subset different orders (C++ > SocketClient Properties > ExecutionFilter).
       


From wrapper_v4.py, class IBWrapper:
    def execDetails(self, reqId, contract, execution):
       
        <SNIP - same code as before>
       
        execdetails=dict(side=str(side), times=str(exectime), orderid=str(thisorderid), qty=int(cumQty), price=float(avgprice), symbol=str(symbol), expiry=str(expiry), clientid=str(clientid), execid=str(execid), account=str(account), exchange=str(exchange), permid=int(permid))

        if reqId==FILL_CODE:
            ## This is a fill from a trade we've just done
            action_ib_fill(execdetails)
           
        else:
            ## This is just execution data we've asked for
            self.add_fill_data(reqId, execdetails)

    def add_fill_data(self, reqId, execdetails):
        if "data_fill_data" not in dir(self):
            filldata={}
        else:
            filldata=self.data_fill_data

        if reqId not in filldata.keys():
            filldata[reqId]={}
           
        execid=execdetails['orderid']
        filldata[reqId][execid]=execdetails
                       
        setattr(self, "data_fill_data", filldata)

           
    def execDetailsEnd(self, reqId):
        """
        No more orders to look at if execution details requested
        """

        setattr(self, "flag_fill_data_finished", True)
 

Its the same hardworking function as before, only this time the reqId will not be -1 (appearing here as FILL_CODE rather than being hardcoded) so we append the fill that is received to a list which is returned to the user.

Are we finished?

 Yes, at least with this post. The last thing I will show you how to do is to get accounting information, so the tedium is nearly over.

This is the fourth in a series of five posts on constructing a simple interface in python to the IB API using swigiby. The first three posts are:

http://qoppac.blogspot.co.uk/2014/03/using-swigibpy-so-that-python-will-play.html

http://qoppac.blogspot.co.uk/2014/04/getting-prices-out-of-ib-api-with.html 

http://qoppac.blogspot.co.uk/2014/04/streaming-prices-from-ib-api-swigibpy.html


The next post is
http://qoppac.blogspot.co.uk/2014/05/getting-accounting-data-out-of.html