Getting accounting data out of interactive brokers API into python with swigibpy

This is the final post. Not the final post of the blog; which may be good news or bad. But the final post in my short series on using swigibpy to interact with the C++ API for interactive brokers (IB).

Interactive brokers now have a native python API. An updated version of this post which uses that API is here: http://qoppac.blogspot.co.uk/2017/03/placing-orders-in-native-python-ib-api.html

Having got some prices and submitted some orders we want to know whether we made any money or not. Although the code is rather trivial, interpreting the results requires some understanding.

http://qoppac.blogspot.co.uk/2017/03/getting-position-and-accounting-data.html

So once again you need to get the code from the following repo.

The example code to run is in test5_IB.py and wrapper_v5.py. This will give your results with live or simulated Gateway / TWS session running (it probably won't produce anything of interest with a demo account but you can still follow the code). Note that IB have recently updated their Gateway software so you might need to re-download.


The output from test5_IB.py should be something like what is in green below.

Expiry is 20140908




Portfolio

([('3KTB', '20140617', 12, 106.0, 1272000000.0, 105842090.65714166, 1894912.11, 0.0, 'DU99', 'KRW'), ('BTP', '20140606', 2, 123.2049942, 246409.99, 121025.33335, 4359.32, 0.0, 'DU99', 'EUR'), ('EMG', '', 1100000, 0.91125, 1002374.99, 0.85376155, 63237.28, 0.0, 'DU99', 'GBP')]

account info

[('AccountCode', 'DU99', '', 'DU99'), ('AccountReady', 'true', '', 'DU99'), ('AccountType', 'INDIVIDUAL', '', 'DU99'), 

<SNIP>

, ('WarrantValue', '0', 'USD', 'DU99')]

 

 

Contract details - again?

Note we reuse the contract details code in http://qoppac.blogspot.co.uk/2014/05/the-placing-of-orders-to-interactive.html

The reason we do this is as before to correctly identify our positions we need the yyyymmdd expiry associated with the contract; although the matching up process isn't done here.

 

Positions

We're going to use a single client function to get both accounting and position data. However both of these are going to be looking at streamed accounting updates. This is returned by two different kinds of wrapper function.
 

Since writing this blog IB have introduced an API call to get positions directly - so there are less clunky ways of doing this than I show here. 

 From wrapper_v5.py, class IBclient: 


    def get_IB_account_data(self):

        self.cb.init_portfolio_data()
        self.cb.init_error()
       
        ## Turn on the streaming of accounting information
        self.tws.reqAccountUpdates(True, self.accountid)
       
        start_time=time.time()
        finished=False
        iserror=False

        while not finished and not iserror:
            finished=self.cb.flag_finished_contractdetails
            iserror=self.cb.flag_iserror

            if (time.time() - start_time) > MAX_WAIT_SECONDS:
                finished=True
                print "Didn't get an end for account update, might be missing stuff"
            pass

        ## Turn off the streaming
        ## Note portfolio_structure will also be updated
        self.tws.reqAccountUpdates(False, self.accountid)

        portfolio_data=self.cb.data_portfoliodata
        account_value=self.cb.data_accountvalue

       
       
        if iserror:
            print self.cb.error_msg
            print "Problem getting details"
            return None

        return (account_value, portfolio_data)


From wrapper_v5.py, class IBWrapper:

    def init_portfolio_data(self):
        if "data_portfoliodata" not in dir(self):
            setattr(self, "data_portfoliodata", [])
        if "data_accountvalue" not in dir(self):
            setattr(self, "data_accountvalue", [])
          
      
        setattr(self, "flag_finished_portfolio", False)
      

    def updatePortfolio(self, contract, position, marketPrice, marketValue, averageCost, unrealizedPNL, realizedPNL, accountName):
        """
        Add a row to the portfolio structure
        """

        portfolio_structure=self.data_portfoliodata
              
        portfolio_structure.append((contract.symbol, contract.expiry, position, marketPrice, marketValue, averageCost,
                                    unrealizedPNL, realizedPNL, accountName, contract.currency))

    ## account value
  
    def updateAccountValue(self, key, value, currency, accountName):
        """
        Populates account value dictionary
        """
        account_value=self.data_accountvalue
      
        account_value.append((key, value, currency, accountName))
      

    def accountDownloadEnd(self, accountName):
        """
        Finished can look at portfolio_structure and account_value
        """
        setattr(self, "flag_finished_portfolio", True)
So once we begin streaming both the portfolio_structure and account_value global variables begin updating. Note that with portfolio_structure we have the same 'gotcha' as before, namely that we can't just drop the contract object into the global variable, as all we will get is a pointer that is meaningless outside of the wrapper method. Instead we need to save specific things of interest like contract.symbol.

Portfolio data

The output portfolio_data will be a list of tuples (contract.symbol, contract.expiry, position, marketPrice, marketValue, averageCost, unrealizedPNL, realizedPNL, accountName, contract.currency). Note that the value, cost, PNL will be in local currency. You can extract exchange rates from the account_value information - which I discuss below.

Lets take an example:

('BTP', '20140606', 2, 123.2049942, 246409.99, 121025.33335, 4359.32, 0.0, 'DU99', 'EUR')

This is the June 2014 BTP (Italian 10 year bond) which I am long 2 in account DU99 (not my real account number) which is trading at 123.205 Euros per contract. I should use the contract information from above to match this to the yyyymm contracts my system knows about. My two contracts are worth Euro 246,410 because each contract price point is worth 1000 Euros (I can get this information from the contract details call).

I actually bought each of these things for 121,025 Euros each (so 121.025 in price points). This ill conceived lending of a quarter of a million Euros to the most unstable government in the Euro zone has so far earned me Euros 4359.32 (246,410 minus 2 * 121,025; or 2 * [123.205 - 121.025] * 1000).

Note my realized PnL is zero since I obviously haven't sold any of these futures yet (no quick profit taking for account DU99).

Here is the corresponding exchange rate from the account_value output.


[... ("ExchangeRate", 0.815491, "EUR", "DU99"),....
 ("ExchangeRate", 1.00000, "GBP", "DU99"),... ]

From this we can make out that my base currency is GBP, and 1 Euro is worth 0.815491 GBP. So I can convert my ill gotten gains in monopoly money ECB issued Euros 4359.32 * 0.815491 = £3,555 in real money.

For more information see the online documentation
C++ > Class EWrapper Functions > Account and Portfolio 

Note be careful about using comparisions between these portfolio snapshots from different times to calculate your profits. They won't take trades into account very well, and if you have sold something completely or hadn't bought it for the first snapshot the comparision wont' work.

Finally also be very careful about using these snapshots to work out what positions you have. Firstly it is common to get the position coming back repeated times. So you will need to filter out duplicates. Secondly positions may sometimes not appear at all.

I use my own record of trades to find out what my current position should be. I compare this to the IB snapshot throughout the day. If there are any breaks in the reconciliation I temporarily halt trading. Occasionally this is because IB has sent me a fill I haven't picked up, or because the IB position snapshot is missing some contracts.

Accounting information

The account_value output will be a very long list of tuples. Each one consists of (keyname, value, currency, account number). Currency BASE indicates it is the accounts base currency (GBP for me). Here are some of the more interesting account value entries.

• ExchangeRate: This is what we use in the get positions example above. 
•  FuturesPNL: This is how much we have made trading futures today, by currency. BASE is the total in base currency.
• StockMarketValue: Self explanatory
• NetLiquidation: This is what is my account worth if I close all my positions out (ignoring commissions and bid-ask). Its what I use to ultimately determine the total profit and loss used for rescaling my risk capital.
• CashBalance: Quite self explanatory. Negative means you have borrowed money. BASE is the net of your balances converted into base currency.

When you initially do a futures trade in a non BASE country you will have to borrow money for initial margin. Only if you move into profit beyond this will you have no net borrowing. IB charges you for borrowing money! This is done at LIBOR+spread so is expensive for currencies with higher interest rates (this spread is also why I use IMM's to get direct currency exposure). You can flatten out your exposure by doing a spot FX trade. Personally I try and keep a small positive balance in all currencies, although not excessive as this means I am taking on currency risk. Note you can't trade all possible pairs eg if you find you can't buy AUDGBP then try selling GBPAUD instead. The exception is Korea where you can't hold any speculative currency exposure i.e. not arising from margin requirements in other kinds of trading. All you are allowed to do is flatten your Korean currency position back to zero.

There are many more keywords than shown above. The best advice I have for trying to understand what they all mean is to start with a clean simulated account (a demo account is no good since you are exposed to other peoples random trading, and the account will often be 'empty'). Take a dump of the account value output, then do a trade. Take another dump, then wait for prices to move. By comparing each dump you should be able to see how account value, margin, cash and so on interact. This is quite time consuming but definitely worth it for an insight into how the IB accounts operate for the kind of instruments you want to trade.

Anything missing?

I haven't covered the full gamut of what you can get from the IB API. There are some newer methods put in since I began using it to get positions and account summary information. There is a much richer variety of price and option data that is available. I haven't looked at fundamental data, option data, scanners and managed accounts because I don't use them. However I hope there is enough meat in this series of examples to get you started using IB with swigibpy, and to avoid wasting time trying to understand some of the weirder behaviour.

This is the final of a series of five posts on constructing a simple interface in python to the IB API using swigiby. The first four posts are:
http://qoppac.blogspot.co.uk/2014/03/using-swigibpy-so-that-python-will-play.html

http://qoppac.blogspot.co.uk/2014/04/getting-prices-out-of-ib-api-with.html 

http://qoppac.blogspot.co.uk/2014/04/streaming-prices-from-ib-api-swigibpy.html

http://qoppac.blogspot.co.uk/2014/05/the-placing-of-orders-to-interactive.html


If you've found this series useful then your next steps might be to learn how to design a systematic trading strategy, and to understand the nuts and bolts of creating a fully automated system.

Bitcoin, money, gold and my great unpublished novel

I have been asked by one of the readers of this blog to write about Bitcoin (So the first thing to note is that there is at least one other person reading this, it isn't just you). My first reaction was that I knew nothing about Bitcoin and anything I could say would just be meaningless biased drivel written to satisfy some agenda. Of course that would be no different from nearly any other Bitcoin based commentary. Or the rest of this blog.

Great lost novels of the past #1

My second reaction when I think of Bitcoin is to remember a novel I wrote about 15 years ago over two weeks. As you might expect from the fact it took two weeks to write it wasn't a very good novel. The loss of the only extant copy on a three and a half inch floppy was no great loss to the world. Certainly the disappearance of the floppy disk was a much greater loss to the world. At least to the world of double entendres since only people over 35 will laugh when I remind them that three and half inch floppies were actually stiff (unlike five and a quarter inch versions).

Storage media aside the most interesting part of the novel was the plot. Essentially in the future some bloke had created an alternative underground offshore currency. Another bloke was recruited by M15 to penetrate his organisation allegedly because it was helping to launder money. Actually the governments motive was to reduce tax evasion. Sound familiar?

So when thinking about Bitcoin I get a deep sense of regret that I didn't self publish the awful book at the time. Of course given the shocking writing it would have sold zero copies but subsequently I would have been recognised as a modern day prophet like William Gibson.

A question asked by an eight year old boy

I will now belatedly get to the point. To answer the question 'what is the difference between Bitcoin and money' we have first have to answer the question my son genuinely asked last week 'What is the difference between a cheque and money?'. To answer that question we need to know how what we call 'money' came about. The paper money and coins we use in modern societies are actually just cheques. A ten pound note is just a cheque for ten pounds, payable to bearer and drawn on the Bank of England. It also has a picture of the Queen and Charles Darwin on it which my cheques from failed socialist experiment the Co-op bank don't have. In a sense both are IOU's. One says the Bank of England owes the bearer a tenner. Originally that was ten pounds of gold, but that link has long since been broken. The Co-op cheque says that I owe the named payee ten quid.

The main difference between them is that I can use the ten pound note to buy just about anything. If I had a million ten pound notes I could easily buy a one bedroom flat in a wealthy part of London (unless house price inflation has rendered even that statement untrue by the time you read this). Though there might be some tiresome money laundering forms to fill in. On the other hand if I tried to buy that flat with a cheque for ten million quid there would be a short delay whilst someone ascertained whether that IOU was actually backed by a corresponding amount in my bank account (sadly it wouldn't be). One of the reasons for the picture of the Queen and all the security features is that the immediate acceptability of the humble tenner means that it is worth trying to forge.

Me and My Plumber

My bank account of course just contains electronic versions of the paper ten pound notes. So a cheque is an IOU which can be converted into a paper or electronic IOU. The important point here is we don't have to use money (a specific form of IOU), we could just use any IOU. Suppose my plumber and I agreed to do this. If we can come up with a cool name (carverCash?), agree on some prices (one poor science fiction novel – ten carverCashes. One toilet unblocked – five hundred carverCashes) then we have a currency. We could even set up an electronic exchange mechanism.

So we then have a fully fledged cryptocurrency. Which only two people will accept (and frankly I am not sure about my plumber going for it). This then is the Achilles heel of bitcoin. The other mechanics of bitcoin are not relevant. Its not quite unbreakable security, equivalent of the picture of the Queen and metal strip in the tenner, is not relevant. The cool sounding mining process is not relevant. If hardly anybody wants to use it then it is pointless.

If we think of the world of currencies as a product market then bitcoin is like the small record shop trying to compete against Amazon and Tesco. It might offer something unique and different like the ability to buy rare Pink Floyd white labels or to purchase drugs over the Internet with some anonymity. But most people don't need that so the most realistic best possible outcome is that the shop survives in a small niche; it is very unlikely to end up displacing the giants. The last time a currency 'broke into' the market and became a globally accepted means of exchange was when the US dollar became more important than the British pound about a hundred years ago, because the US displaced Britain as the leader in global trade. The Chinese currency may well overtake the dollar at some point if it becomes freely exchangeable, but that will be for the same reason - because nearly everything we buy is made in China. Bitcoin has no such advantage.

If not money, then what?

So having dismissed the chances of bitcoin becoming a globally accepted currency we are left with is it a possible investment. In my spare time I like to classify assets by their degree of being divorced from reality. If I go and buy a farm from someone that is pretty close to reality. When the farm is owned by a public company and I buy shares in that company we are getting slightly further away. Owning a call option on the companies shares is moving further away again, but underneath it all is the farm – a bad harvest will ultimately reduce the value of my option. Buying an interest rate future is entering a really whacky world since the underlying thing is not real but a price based on a survey of what people think the price should be (at least until the LIBOR market is properly reformed).

The price of anything is driven by supply and demand. Even with assets that are more real there is a good deal of 'animal spirits' in both elements so prices can be moved away from what might be called a fundamental value for quite some time. However in instruments linked more closely to real things there are limits to how crazy things can get. It may take time but speculative bubbles and anti bubbles in asset prices linked to real things always eventually pop at some price ceiling or floor.  This isn't true if you have an asset which doesn't have any connection to underlying reality.

So the value of bitcoins really has no floor or ceiling. If nobody wants them they will be worthless. If everyone wants them, given they will stop making them in the future, they will be priceless. That 'want' will be based purely on subjective perception rather than any underlying real factors. This does not mean they are a bad investment. Rather that one should not invest in them like you would invest in UK equities. Holding a constant long position in UK equities is a reasonable thing to do. For bitcoins it is probably not. We need to actively trade bitcoins.

Bitcoins equal not Money. Gold equal Money? -> Gold equal Bitcoins ?

A better asset to think about when trading bitcoins would be the asset that used to back all money – Gold. Why gold? Gold is not dissimilar to bitcoins in price behaviour. Underlying supply of gold is relatively short run inelastic, so speculative demand shocks will pull the price all over the place. It does have some industrial uses but its value is mostly determined by very subjective demand. If everyone stopped desiring gold tomorrow it would be pretty much worthless and similarly there is no potential ceiling on its future price. There is no income or rights to any underlying cashflow for either (although owning gold futures might earn a 'carry').

Gold does have however a much larger and stickier demand base, bitcoin jewellery being a pure substitute for the shiny yellow stuff. Also it has a long history of being a reasonable inflation hedge. So having a bit of static gold in your portfolio isn't a completely absurd idea.

Its worth noting that I mainly trade gold using trend following signals that are specifically designed to pick up changes in subjective sentiment, like those that drive other less real assets like bitcoins. Using a similar strategy for bitcoins might make sense. If we look at figure 1 we can see that there appear to be some decent trends. Note I have used a log scale so you can see the movements in the past more clearly.

 A simple moving average crossover indicates a short position in figure 2 might make currently sense.  This cursory look suggests going long bitcoins might not be the smartest move. In reality I would want to do much more analysis than this before even thinking about trading bitcoins.

Having done that I would be happy to trade bitcoin futures on a recognised exchange backed by a proper clearing house, if they were liquid enough. After all I trade gold futures. I even trade Eurodollar futures which are even more highly divorced from reality. That isn't possible now and a spot position in the actual asset makes me very nervous since the counterparty risk seems extremely high.

So the short answer is I wouldn't personally touch bitcoin with a bargepole. Not even if I needed to buy some to research my next novel.

The placing of orders to interactive brokers with swigibpy / python

We are nearly at the end of our journey of simplistic examples of how to get the swigibpy package to mediate between the wonderful world of Python and the dark place that is the Interactive brokers C++ API. Having learned how to get 'snapshot' and streaming prices out of the API we are now ready to actually do some trading- submit orders, check they are active, potentially cancel them, receive fills and get historic execution data.

Interactive brokers now have a native python API. An updated version of this post which uses that API is here: http://qoppac.blogspot.co.uk/2017/03/placing-orders-in-native-python-ib-api.html

Where do I start?

You need to get the code from the following repo.

The example code to run is in test4_IB.py. If you have a live or simulated Gateway / TWS session running (one associated with a real account number) it should work just fine. Note that you can also run this with the edemo account, but the results will be very ugly. This is because you are seeing the orders placed by everyone who is playing around with this account, so you could get all kinds of randomeness.


WARNING I highly recommend that you do not run this code on a real money trading account until / unless you know exactly what you are doing! In particular real trading code should check market liquidity before trading, particularly for market orders.

The output should be as in green below. Of course feel free to change the contract traded and the quantities...

Expiry is 20150908



Getting orderid from IB
Using order id of 1

Placed order, orderid is 1
Getting orderid from IB
Using order id of 2

Placed limit order, orderid is 2
recived fill as follows:

{'orderid': '1', 'account': 'DU15031', 'exchange': 'DTB', 'symbol': 'GBL', 'permid': 193880352, 'execid': '00004468.559df2a0.01.01', 'clientid': '999', 'expiry': '20150908', 'price': 156.37, 'qty': 10, 'side': 'BOT', 'times': '20150709  14:05:11'}

Active orders: (should just be limit order)
{2: {'orderid': 2, 'symbol': 'GBL', 'qty': 10, 'side': 'SELL', 'clientid': 999L, 'expiry': '20150908'}}

Cancelling remaining orders
Cancelling 2
Waiting for cancellation to finish
No active orders now
False

Following executed since midnight this morning:

{'1': {'orderid': '1', 'account': 'DU15031', 'exchange': 'DTB', 'symbol': 'GBL', 'permid': 193880352, 'execid': '00004468.559df2a0.01.01', 'clientid': '999', 'expiry': '20150908', 'price': 156.37, 'qty': 10, 'side': 'BOT', 'times': '20150709  14:05:11'}}

Contract details - what is it?

As usual there is the tedious business of getting a client object and defining a contract. The first proper thing we need to do is get the contract details. This may sound a bit weird, we already have the contract details since we've just defined it? In practice the definition of a contract is very light and the details we get back are much richer.

There is one specific case where you must do this. To see why suppose it is a few months time and you are trading hundreds of futures contracts. A fill comes in for a particular contract instrument code and month/year. However as we shall see the identification for the contract isn't the 'yyyymm' code we use to define the contract, but the full expiry 'yyyymmdd'. You can't just take the first 6 characters of that either since some futures actually expire the month before they should. So to be on the safe side I usually get the expiry from IB before trading to make sure I am applying fills against the correct thingy.

Actually I don't do this immediately before trading but in a separate process that populates a database of contract information...

 

From test4_IB.py
     callback = IBWrapper()
    client=IBclient(callback)
   
    ibcontract = IBcontract()
    ibcontract.secType = "FUT"
    ibcontract.expiry="201509"
    ibcontract.symbol="GBL"
    ibcontract.exchange="DTB"
    cdetails=client.get_contract_details(ibcontract)

WARNING: If you are trading the VIX, which now has weekly expiries, you will need to specify the full expiry date in yyyymmdd format.


I won't go through the client code in detail but there are two 'gotchas' in the server code that appear a few times so lets look at an excerpt.

From wrapper_v4.py, class IBWrapper:
    def contractDetails(self, reqId, contractDetails):       
        contract_details=self.data_contractdetails[reqId]
 
contract_details["contractMonth"]=contractDetails.contractMonth
        ## <SNIP - code removed>
        contract_details["evMultiplier"]=contractDetails.evMultiplier

        contract2 = contractDetails.summary

        contract_details["expiry"]=contract2.expiry
        ## <SNIP - code removed>
 
contract_details is of course a global object that gets picked up by the client object when populated. The first of the two gotchas here is an old one; if you are submitting multiple of these to the same API session watch out to use meaningful request ID's. More subtle is that we don't just do something like contract_details=contractDetails.Instead we actually extract the information into a new object. The reason is that the little jobbys are just weird pointers that will become pointers to something else, usually some weird memory address, by the time you look at them outside of the callback instance. So you have to capture their essence in the wild before they escape. This is the well known quantum pointers effect.

To find out more about contract and contractDetails objects look at the manual (C++ / SocketClient Properties).

Note I use a dicts or dicts of dicts here for contracts, orders and fills but in reality I have little classes for them to keep things neater.

Order placing - can I buy it?

Now the moment we've all been waiting for... we're actually going to buy something. For fun we're going to place two orders, a market order and a limit order.

From test4_IB.py 
     ## Place the order, asking IB to tell us what the next order id is
    ## Note limit price of zero
    orderid1=client.place_new_IB_order(ibcontract, 10, 0.0, "MKT", orderid=None)
   
    print ""
    print "Placed market order, orderid is %d" % orderid1
   
    ## And here is a limit order, unlikely ever to be filled
    ## Note limit price of 100
    orderid2=client.place_new_IB_order(ibcontract, -10, 200.0, "LMT", orderid=None)
    print ""
    print "Placed limit order, orderid is %d" % orderid2


 From wrapper_v4.py, class IBclient:  
    def place_new_IB_order(self, ibcontract, trade, lmtPrice, orderType, orderid=None):

        <SNIP>       
        iborder = IBOrder()
        iborder.action = bs_resolve(trade)
        iborder.lmtPrice = lmtPrice
        iborder.orderType = orderType
        iborder.totalQuantity = abs(trade)
        iborder.tif='DAY'
        iborder.transmit=True
 
        ## We can eithier supply our own ID or ask IB to give us the next valid one
        if orderid is None:
            print "Getting orderid from IB"
            orderid=self.get_next_brokerorderid()
           
        print "Using order id of %d" % orderid
   
         # Place the order
        self.tws.placeOrder(
                orderid,                                    # orderId,
                ibcontract,                                   # contract,
                iborder                                       # order
            )
   
        return orderid

       
Obvious first thing to notice here is the concept of an orderid. This is a number that identifies to IB what the order is; at least temporarily and for today only. Only restriction on order id's is that the next order is higher than the last. This means if you submit an order with an id of 999999 you will lose all the orderids below that. You can also reset the id 'clock' to 1 via an option on the Gateway or TWS API configuration menu.  Safest thing to do is ask IB for the next orderid as done here by supplying None to the calling function.

In practice I generate my own orderid's preferring to reserve them first in my own database. This is fine as long as you are running a single linear process where there is no chance of an 'older' order being submitted before a 'newer' one.

Notice the limit price of 0.0 for the market order won't be used. For more information on more funky order types see the docs:  Reference > Order Types and IBAlgos > Supported Order Types.

 Active order status- have I bought it?

IB can tell us what orders we are working. Unless you ask very quickly (or submit your order outside of trading hours) this is likely only to return limit orders.

From test4_IB.py 
    order_structure=client.get_open_orders()
    print "Active orders: (should just be limit order)"
    print order_structure
    print ""


 From wrapper_v4.py, class IBclient:  

    def get_open_orders(self):
        """
        Returns a list of any open orders
        """
        <SNIP>
        self.tws.reqAllOpenOrders()
        iserror=False
        finished=False
       
        while not finished and not iserror:
            finished=self.cb.flag_order_structure_finished
            iserror=self.cb.flag_iserror
            if (time.time() - start_time) > MAX_WAIT_SECONDS:
                ## You should have thought that IB would teldl you we had finished
                finished=True
            pass
       
        order_structure=self.cb.data_order_structure
        if iserror:
            print self.cb.error_msg
            print "Problem getting open orders"
   
        return order_structure   
 


From wrapper_v4.py, class IBWrapper:

    def init_openorders(self):
        setattr(self, "data_order_structure", {})
        setattr(self, "flag_order_structure_finished", False)

    def add_order_data(self, orderdetails):
        if "data_order_structure" not in dir(self):
            orderdata={}
        else:
            orderdata=self.data_order_structure

        orderid=orderdetails['orderid']
        orderdata[orderid]=orderdetails
                       
        setattr(self, "data_order_structure", orderdata)


    def openOrder(self, orderID, contract, order, orderState):
        """
        Tells us about any orders we are working now
       
        Note these objects are not persistent or interesting so we have to extract what we want
       
       
        """
       
        ## Get a selection of interesting things about the order
        orderdetails=dict(symbol=contract.symbol , expiry=contract.expiry,  qty=int(order.totalQuantity) ,
                       side=order.action , orderid=int(orderID), clientid=order.clientId )
       
        self.add_order_data(orderdetails)

    def openOrderEnd(self):
        """
        Finished getting open orders
        """
        setattr(self, "flag_order_structure_finished", True)

 
In practice I have noticed that the correct end condition for receiving open orders doesn't always trigger so you do need an max waiting time (which is good practice anyway). Here we are just getting a selection of the available information about the order (see C++ > SocketClient Properties > Order and OrderState for more). Again we don't just save the order object itself as this is just a quantum pointer that will change.

Fill data - how much did it cost me?

What happens when an order is filled; completely or partially? Well the following method in the wrapper function is triggered. Notice that the logic is slightly more complicated because this function fulfills two duties. When it's called for a fill then reqId=-1, and action_ib_fill will be called.

From wrapper_v4.py, class IBWrapper:
    def execDetails(self, reqId, contract, execution):
           
    def execDetails(self, reqId, contract, execution):
   
        """
        This is called if
       
        a) we have submitted an order and a fill has come back
        b) We have asked for recent fills to be given to us
       
        We populate the filldata object and also call action_ib_fill in case we need to do something with the
          fill data
       
        See API docs, C++, SocketClient Properties, Contract and Execution for more details
        """
        reqId=int(reqId)
       
        execid=execution.execId
        exectime=execution.time
        thisorderid=int(execution.orderId)
        account=execution.acctNumber
        exchange=execution.exchange
        permid=execution.permId
        avgprice=execution.price
        cumQty=execution.cumQty
        clientid=execution.clientId
        symbol=contract.symbol
        expiry=contract.expiry
        side=execution.side
       
        execdetails=dict(side=str(side), times=str(exectime), orderid=str(thisorderid), qty=int(cumQty), price=float(avgprice), symbol=str(symbol), expiry=str(expiry), clientid=str(clientid), execid=str(execid), account=str(account), exchange=str(exchange), permid=int(permid))

        if reqId==FILL_CODE:
            ## This is a fill from a trade we've just done
            action_ib_fill(execdetails)
           
        else:
            ## This is just execution data we've asked for
            self.add_fill_data(reqId, execdetails)
Again we extract the useful information from the contract and execution objects rather than saving them directly. More on the IB website docs C++ > SocketClient Properties > Execution. To reiterate the expiry here will be the actual yyyymmdd expiry date; not the shorter yyyymm that the original trade was constructed with, hence the need to identify the former before submission.

A brief technical note; the date is the contract expiry not where relevant first or last notice date. This means you should be wary of using this date to tell you when to roll certain kinds of futures contracts eg US bonds.



From IButils:

 def action_ib_fill(execlist):
    print "recived fill as follows:"
    print ""
    print execlist
    print ""

Although this is very boring in practice this would be the function that would update your order status database (as a pro at this stuff, naturally you would have such a thing). It won't be obvious from this simple example unless you can submit a very large order in a thin market but the fills come in as cumulative order updates, not separate fills. Its worth looking at an example. Suppose you try and buy 10 lots, and you get fills of:

• 3 lots @ 100.0
• 6 lots @ 100.2
• 1 lot @ 100.5

 Then the fills that come in will look like this:

• qty: 3 lots, price=100.0
• qty: 9 lots, price=100.13333333
• qty: 10 lots, price=100.17

So if you do care about each partial fill you are going to have to hope that you see every little fill coming in and use a differencing process to see the detail of each.

By the way 'orderid' is only a temporary thing for IB; after tommorrow it won't associate it with this order. Instead you should use 'permid' for your record keeping. 'execid' is different for each part fill so you could use it to make sure you aren't including fill information you already have; in practice this isn't problematic due to the cumulative nature of the information.

Order cancelling - what if I don't want it any more?

Its very easy indeed to cancel an order; we don't even need a call in the client object to do it.

From test4_IB.py
    print "Cancelling the limit order"
    client.tws.cancelOrder(orderid2)

    print "Waiting for cancellation to finish"   
    while client.any_open_orders():
        pass
    print "No active orders now"
    print client.any_open_orders()
    print ""

Past execution data - sorry, repeat that, how much?!

It is clearly very important that fill data is correctly captured by your trading software. One reason being to keep track of what your position is; as we shall see in the next post IB doesn't offer mere mortals a super accurate current position facility. So I generally use my own knowledge of trade history to decide where I am, position wise. Because the fills usually arrive in the wrapper function only once its possible under certain conditions to miss them; eg if your API client dies before you see the fill or just isn't running when one arrives on a previously closed market in the middle of the night. Its generally good practice then to reconcile what IB has for a record of fills versus your own.

This information is only available up to midnight of the day you trade. So I run a reconciliation 3 times a day. If you lose a fill from before today you will need to find it on the IB website account management microsite, and manually enter it into your database.

Here is how we do it.

From test4_IB.py 

    execlist=client.get_executions()
   
    print "Following executed since midnight this morning:"
    print ""
    print execlist


 From wrapper_v4.py, class IBclient:  
    def get_executions(self):
        """
        Returns a list of all executions done today
        """
       
        assert type(reqId) is int
        if reqId==FILL_CODE:
            raise Exception("Can't call get_executions with a reqId of %d as this is reserved for fills %d" % reqId)

        self.cb.init_fill_data()
        self.cb.init_error()
       
        ## We can change ExecutionFilter to subset different orders
       
        self.tws.reqExecutions(reqId, ExecutionFilter())

        iserror=False
        finished=False
       
        start_time=time.time()
       
        while not finished and not iserror:
            finished=self.cb.flag_fill_data_finished
            iserror=self.cb.flag_iserror
            if (time.time() - start_time) > MAX_WAIT_SECONDS:
                finished=True
            pass
   
        if iserror:
            print self.cb.error_msg
            print "Problem getting executions"
       
        execlist=self.cb.data_fill_data[reqId]
       
        return execlist
       
       
We can change ExecutionFilter to subset different orders (C++ > SocketClient Properties > ExecutionFilter).
       


From wrapper_v4.py, class IBWrapper:
    def execDetails(self, reqId, contract, execution):
       
        <SNIP - same code as before>
       
        execdetails=dict(side=str(side), times=str(exectime), orderid=str(thisorderid), qty=int(cumQty), price=float(avgprice), symbol=str(symbol), expiry=str(expiry), clientid=str(clientid), execid=str(execid), account=str(account), exchange=str(exchange), permid=int(permid))

        if reqId==FILL_CODE:
            ## This is a fill from a trade we've just done
            action_ib_fill(execdetails)
           
        else:
            ## This is just execution data we've asked for
            self.add_fill_data(reqId, execdetails)

    def add_fill_data(self, reqId, execdetails):
        if "data_fill_data" not in dir(self):
            filldata={}
        else:
            filldata=self.data_fill_data

        if reqId not in filldata.keys():
            filldata[reqId]={}
           
        execid=execdetails['orderid']
        filldata[reqId][execid]=execdetails
                       
        setattr(self, "data_fill_data", filldata)

           
    def execDetailsEnd(self, reqId):
        """
        No more orders to look at if execution details requested
        """

        setattr(self, "flag_fill_data_finished", True)
 

Its the same hardworking function as before, only this time the reqId will not be -1 (appearing here as FILL_CODE rather than being hardcoded) so we append the fill that is received to a list which is returned to the user.

Are we finished?

 Yes, at least with this post. The last thing I will show you how to do is to get accounting information, so the tedium is nearly over.

This is the fourth in a series of five posts on constructing a simple interface in python to the IB API using swigiby. The first three posts are:

http://qoppac.blogspot.co.uk/2014/03/using-swigibpy-so-that-python-will-play.html

http://qoppac.blogspot.co.uk/2014/04/getting-prices-out-of-ib-api-with.html 

http://qoppac.blogspot.co.uk/2014/04/streaming-prices-from-ib-api-swigibpy.html


The next post is
http://qoppac.blogspot.co.uk/2014/05/getting-accounting-data-out-of.html 

Why black box hedge funds should have lazy risk managers

At the time of writing Astra Zeneca shares are up around 15% over a few days on the back of a forthcoming takeover offer. I own a chunk of these and my fingers have been twitching over the 'sell' button with the natural inclination being to take profits as any behavioural finance textbook will tell you.

Indeed I am not the only one http://www.telegraph.co.uk/finance/personalfinance/investing/shares/10794055/AstraZeneca-shares-are-up-14pc-in-a-week.-Should-I-sell.html.

The part of my brain running what Daniel Kahneman in http://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow calls System one is in control. But the same fast instinctive behaviour that made sense when our ancestors were hunting woolly mammoths thousands of years ago isn't a lot of use in this situation. The other part of my brain, System two, knows that logically I should analyse some data and see whether it really does make sense to sell or to hang on. After all analysing data is pretty much what I do.

Enter the black box

I could really do with some kind of rule to tell me whether I should go ahead. That means I would be less likely to succumb to temptation. Three classic trading strategies that would apply in this situation are 'momentum', 'value' and 'merger arbitrage'. The first would say 'price has gone up, buy or hold a long position as it should go up more'. The second would tell me that the dividend yield on these shares has gone done to around the FTSE 100 average so I should probably sell. The latter would say 'Hold the shares until the merger is complete'. Incidentally two out of three these strategies would not sell; one reason they tend to make money in the long run is that they deliberately go against peoples natural (system one) inclinations to take profits early. The second tends to make money by being contrary as well; mainly on the downside people find it hard to buy things that have dropped a lot in price and so represent better value.

Imagine then I have a little black box sitting next to me calculating the correct decisions to make based on one or both of these trading models. Or perhaps more realistically in a separate window on my computer; next to my online brokerage account control panel. Which is resolutely saying as two out of three systems agree: do not sell these shares! But there is nothing stopping me from doing so. I can just move my mouse over to the window and click sell. The black box, assuming it has some kind of data feed to read my positions, will then change its recommendation to a slightly churlish buy Astra Zeneca – unlike me it won't have changed its mind and thinks we should be owning this still British pharma company until the bitter end.

If I then rang a friendly economist (they do exist) he or she would tell me what I need is a commitment mechanism. This is some way of preventing me from pulling the trigger and ignoring my black box. Similar in fact to when a gambling addict requests that he be barred from an online poker site. Yes the irony of the analogy isn't lost on me either. So perhaps I could set up my black box so it automatically submits the trades for me. Note that I no longer just have a systematic strategy for investing, but one which is fully automated. Just to be on the safe side I should also hide my brokerage account password deep within my computer so it takes me a few hours to dig it out.

Borrowing the Black Box

There is a much easier way of achieving this however, which would also free up some spare time to do more socially useful and fun activities. I just need to invest my money with a hedge fund which specialises in merger arbitrage and/or momentum trading. We can ignore for the moment the difficulties faced by small retail investors in trying to open accounts with hedge funds. Also this will not be cheap; I will have to pay at least 2% of my investment every year in fees and probably hand over a fifth of any profits.

But I can do even better by finding a systematic hedge fund, i.e. one which has its own black boxes. This should be cheaper; black boxes are less greedy than shouty blokes in red braces (human traders), even after you have paid some geeks to come up with the models and program them in. In practise it tends not to be cheaper with the resulting higher operating margin going to the funds owners or staff, but that is another story. It is also hard to find systematic merger arbitrage funds but fortunately equity valuation funds are two a penny, and systematic momentum funds are also quite common. Indeed I actually used to work for such a fund. By the way anything in this post isn't necessarily a reference to that particular fund or its employees. It could be about any large systematic hedge fund owned by a publicly traded UK listed company.

One man and his dog (and his computer)

There is a great cliché in the systematic investment industry, which I think was stolen from a famous quote about the space programme. The ideal systematic fund should consist of a computer, a human and a dog. The computer does the trading whilst the human feeds the dog. The dog's job is to bite the human if he goes near the computer. In practise you do need to have humans involved in the running of these things. What sort of things might they be doing?

• Process control. Its almost impossible to design a system which is completely automated. Much easier and safer to make something that requires an occasional knob to be twisted, or rather a particular script to be manually run.
• Data cleaning. Cruddy prices are more common than you think.
• Watch dogging. In case of bugs, incorrect inputs or data, something falling over …. you don't want the computer running amok and trying to trade the entire GDP of Korea in one bond trade. Famously a number of firms trading high frequency strategies have managed to lose their entire capital in a few seconds.
• Improving the strategy, i.e. research.
• Extending the strategy to a wider number of tradeable instruments.
• Reparameterisation; necessary if something changes. If you are running a slow trading strategy this shouldn't be happening very often. High frequency traders need to refit their models frequently. Arguably overlaps with research.
• Exogenous risk control. A good fund will have endogenous risk control, i.e. the risks the models know about are controlled within the models. If a model doesn't know about the risk then you need to exogenously do something about it.

Quite a few of these activities can at least be partially automated, with the exception of research (automated research is just reparameterisation). As with risk control you can view these as endogenising the activity inside the black box. I always try to move any exogenous ad-hoc activity inside the box; first by systematising it and then by automating it. For example suppose you got nervous about executing orders around important non farm payrolls. First you might just turn off your trading system when a big non farm number was coming (ad-hoc). Then you might create a procedure where you turned off your trading for every non farm number (systematised process). Finally you could create a data feed for non farm dates and have the system pull out just before each one (automated process).

However we are still left with the biggest potential source of exogenous risk control, or as I prefer to put it 'I know instinctively better than the trading model I have spent hundreds of man hours developing, because of X' (you can perhaps see where I am going with this). Often in my experience I have found myself or observed others labelling changes as 'research' that were clearly done for risk control reasons. For example 'Our research indicates that we should remove some asymmetrical long bias from our corporate bond bond, corporate bonds feel overvalued so this is clearly a good time to introduce it'. Notice that the researcher speaking here hasn't got a corporate bond valuation model so there is no way of backing up their feelings with data... its System One not System Two speaking here.

We also see the same sort of behavioural biases that the systematic trading model has been designed to avoid coming back in again. So again I have found myself saying 'We should reduce this models risk to take profits because it has been a good year' and mainly heard others saying 'This model hasn't made money for 3 months we should turn it off'. In case you didn't realise 3 months is an statistically pointless length of time to measure performance over for the kinds of models we are talking about.

Gut feeling and behavioural bias is human nature and there is not much we can do about it. Instead you need rigorous peer /management reviews of any proposed change or the critical self examination based on bitter experience if you are on your own. Another good trick is to make the process for making changes to the trading system so bureaucratic, difficult and torturous to do that nobody bothers. Inevitably though the risk manager will complain loudly if their gut calls, sorry rigorous analytical decisions, are too hard to implement. 

Wanted: Risk Manager. Must be wantonly idle

Some more good advice is not to employ anyone with trading experience as a risk manager. Normally this is a good idea, since ex-traders know where the bodies are buried and are excellent supervisors of human traders. But with a systematic system they will just want to start making calls. A cardinal rule of running a systematic system is any risk management decision should reduce the size of a position but never change the sign. Ex-traders find it hard to follow that rule. They are also a bit prone to what Americans like to call Monday morning quarterbacking. Its very easy to see that a particular position shouldn't have been put on, or should have been reduced, if it loses money because something happens.

A better person to employ would be someone who is very smart but extraordinarily lazy, and who can only be bothered to do say one hours work a month. Since they still want to get paid they will make sure that they only take decisions but only when it really matters. You can simulate this by using a variation of Warren Buffets punched card idea. At the start of the year the risk manager gets a card. Every time they make a decision they punch a hole in it. When say 10 holes are in the card you take away their right to make any further decisions. The same rule could apply to any person who has the power to change a trading system parameter.

Two kinds of investors

Unfortunately it isn't just human nature or over enthusiastic risk managers that causes problems with system one thinking. In theory investors in the fund have deliberately handed over their money at least partly to ensure a commitment to the trading model. Certainly for larger institutions that could run their own money in this way this should be one reason, though most investors probably haven't consciously done this. But sure enough when there is poor performance for 3 months, many will put pressure on the manager to do something. Long lock ups, where you cannot redeem your investment for months or years, are unusual in this kind of hedge fund since the underlying assets are very liquid; indeed this is a selling point. However perhaps for systematic managers it makes sense to attract the right kind of investor by imposing a five year lockup on any money put in.

There is one more source of behaviourally inspired interference; the owners of the fund management business – shareholders and their representatives, the senior management of the business. Fund investors might be au fait with the nature of the business – research the model, implement it and then leave it alone unless its behaviour falls outside an expected envelope in a statistically significant way. But if senior managers are from a trading background they may also be prone to the behaviour of the risk managers mentioned above. Again you might be better off with very lazy senior management. If its any help I am available and perfectly happy to work a few hours a month for a CEO type salary.

Shareholders in public companies like predictable businesses with steady cash flows. They attach good price earnings ratios to such businesses. But this kind of business is effectively a non linear leveraged play on the success of the underlying trading strategy. This leads to very lumpy earnings, hard for public shareholders to stomach; they will tend to overvalue the firm when it is doing well and undervalue when vice versa occurs. Ideally then a hedge fund like this shouldn't be publicly owned, but like nearly all hedge funds owned by its employees. They know better than anyone what the business is like, and this is also great for lining up incentives.

But if it must be publicly owned then it is clear what the shareholders need to do. Yes: they need to buy a black box, entrust their hedge fund shares to it and hide their passwords...

To finish its probably worth noting that I only trade futures systematically; thus I really do have no formal trading model at all for Astra Zeneca. If anything my individual share portfolio is definitely value based, so I will probably sell having identified a company with a better dividend yield. When I get round to it that is.

Postscript: I sold my AZN shares on 8th May 2014 and traded them for the higher yielding GSK. Wish me luck!