Friday 24 March 2017

Getting position and accounting data out of IB native python API

This is the final post. Not the final post of the blog; which may be good news or bad. But the final post in my short series on using the new native python API for interactive brokers. Having got some prices and submitted some orders we want to know whether we made any money or not; and what positions we have. Although the code is rather trivial, interpreting the results requires some understanding.

This is an update of the following post where I used the non native swigibpy (which I wrote nearly 3 years ago!).

This the fifth in a series of posts on using the native python API  You should read the firstsecond, third, and fourth. You should also read the documentation.

You need to get the code from this gist.


Getting positions


You will need to run the code in the last post first so you actually have a position to look at (this will submit a market order for Eurodollar futures). 

Now, if you've been following the series so far this kind of code pattern should be easy to follow:

## lets get positions
positions_list = app.get_current_positions()

In the client object:

def get_current_positions(self):

    ## Make a place to store the data we're going to return    
    positions_queue = finishableQueue(self.init_positions())

    ## ask for the data    
    self.reqPositions()
    <SNIP>

In the wrapper object:


def init_positions(self):
    positions_queue = self._my_positions = queue.Queue()

    return positions_queue

def position(self, account, contract, position,
             avgCost):

    ## uses a simple tuple, but you could do other, fancier, things here    
    position_object = (account, contract, position,
             avgCost)

    self._my_positions.put(position_object)

def positionEnd(self):
    ## overriden method
    self._my_positions.put(FINISHED)


Back in the client object:


def get_current_positions(self):

    <SNIP>
    ## poll until we get a termination or die of boredom    
    MAX_WAIT_SECONDS = 10    
    positions_list = positions_queue.get(timeout=MAX_WAIT_SECONDS)

    while self.wrapper.is_error():
        print(self.get_error())

    if positions_queue.timed_out():
        print("Exceeded maximum wait for wrapper to confirm finished whilst getting positions")

    return positions_list


Results:
print(positions_list)

 [
('DU15153', 
140109276672072: 56825063,GE,FUT,20181217,0.0,,2500,,,USD,GEZ8,GE,False,,combo:, 
10.0, 
244889.9)]

The results come back as a 4 tuple: account identifying name, contract, position and realised PNL (this might seem large - but this is a demo account so it will depend on what other people have been trading). I'd advise making a nice class to put these into, although I don't bother doing that here.

Finally be very careful about using these snapshots to work out what positions you have. Firstly it is common to get the position coming back repeated times. So you will need to filter out duplicates. Secondly positions may sometimes not appear at all.

I use my own record of trades to find out what my current position should be. I compare this to the IB snapshot throughout the day. If there are any breaks in the reconciliation I temporarily halt trading in the relevant contract until the break has gone away. Occasionally this is because IB has sent me a fill I haven't picked up, or because the IB position snapshot is missing some contracts.

Getting accounting data


The accounting data is a little more complicated. Firstly, we get three types of accounting data from IB, which need to be distinguished. Secondly, I choose to access the accounting data via a cache in the client object which only refreshes if the data we have is more than 5 minutes old. Of course once you understand the logic feel free to implement this differently yourself.


Accounting values


The first type of accounting data we get are accounting values (returned by updateAccountValue in the wrapper)

## get the account name from the position
## normally you would know your account but this is the demo
nameaccountName = positions_list[0][0]

## and accounting information
accounting_values = app.get_accounting_values(accountName)


In the client object:

def get_accounting_values(self, accountName):

    #All these functions follow the same pattern: check if stale, if not return cache, else update values
    return self._account_cache.get_updated_cache(accountName, ACCOUNT_VALUE_FLAG)

What is this doing under the hood? First we check to see if the cache is up to date (default is to check if we have got an account value in the last 5 minutes). If not then we ask for the accounting data to be updated. We then return the cached value (eithier historical, or just updated)

Let's have a look at the workhorse function that actually updates the data _update_accounting_data:

def _update_accounting_data(self, accountName):
    """    Update the accounting data in the cache
    :param accountName: account we want to get data for    
    :return: nothing    
    """
    ## Make a place to store the data we're going to return    
    accounting_queue = finishableQueue(self.init_accounts(accountName))

    ## ask for the data    
    self.reqAccountUpdates(True, accountName)

    <SNIP>
In the wrapper object:
## get accounting datadef init_accounts(self, accountName):
    accounting_queue = self._my_accounts[accountName] = queue.Queue()

    return accounting_queue


def updateAccountValue(self, key:str, val:str, currency:str,
                        accountName:str):

    ## use this to seperate out different account data    
    data = identifed_as(ACCOUNT_VALUE_FLAG, (key,val, currency))
    self._my_accounts[accountName].put(data)


def updatePortfolio(self, contract, position:float,
                    marketPrice:float, marketValue:float,
                    averageCost:float, unrealizedPNL:float,
                    realizedPNL:float, accountName:str):

    ## use this to seperate out different account data    
    data = identifed_as(ACCOUNT_UPDATE_FLAG, (contract, position, marketPrice, marketValue, averageCost,
                                      unrealizedPNL, realizedPNL))
    self._my_accounts[accountName].put(data)

def updateAccountTime(self, timeStamp:str):

    ## use this to seperate out different account data    
    data = identifed_as(ACCOUNT_TIME_FLAG, timeStamp)
    self._my_accounts[accountName].put(data)


def accountDownloadEnd(self, accountName:str):

    self._my_accounts[accountName].put(FINISHED)

The only novel thing here are these identifed_as(ACCOUNT_VALUE_FLAG, (key,val, currency)) guys. Because we push three different types of data on to the same queue we need to be able to distinguish them from each other. Rather than just push the raw data we're getting from IB (all stored as tuples, except for the single length timeStamp in updateAccountTime) we push an object which comes with a labelling flag to identify where it came from.


Back in the client object:

def _update_accounting_data(self, accountName):

    <SNIP>
    ## poll until we get a termination or die of boredom    
    MAX_WAIT_SECONDS = 10    
    accounting_list = accounting_queue.get(timeout=MAX_WAIT_SECONDS)

    while self.wrapper.is_error():
        print(self.get_error())

    if accounting_queue.timed_out():
        print("Exceeded maximum wait for wrapper to confirm finished whilst getting accounting data")


So far this is boilerplate, but this isn't: 
# seperate things out, because this is one big queue of data with different things in it
    accounting_list = list_of_identified_items(accounting_list)
    seperated_accounting_data = accounting_list.seperate_into_dict()


The queue is made up of three different kinds of data (values, updates, and time) from three different wrapper functions, so we need to split them up. We now have a dictionary, we now use this to update the different parts of the cache:
## update the cache with different elements
self._account_cache.update_cache(accountName, seperated_accounting_data)
We're now returned to the get_accounting_values function that will actually return the updated cache values we want:

def get_accounting_values(self, accountName):

    #All these functions follow the same pattern: check if stale, if not return cache, else update values
    return self._account_cache.get_updated_cache(accountName, ACCOUNT_VALUE_FLAG)

Results:

print(accounting_values)



[('AccountCode', 'DU15153', ''), ('AccountOrGroup', 'DU15153', 'BASE'), ('AccountOrGroup', 'DU15153', 'USD'), ('AccountReady', 'true', ''), .... <SNIP>

The account_value output will be a very long list of three tuples. Each one consists of (keyname, value, currency). Currency BASE indicates it is the accounts base currency (USD for the test account). Here are some of the more interesting account value entries.

  • ExchangeRate: In the form of ('ExchangeRate', '1.00', 'USD') This is a very boring exchange rate as the base currency is USD.
  •  FuturesPNL: This is how much we have made trading futures today, by currency. BASE is the total in base currency.
  • StockMarketValue: Self explanatory
  • NetLiquidation: This is what is my account worth if I close all my positions out (ignoring commissions and bid-ask). Its what I use to ultimately determine the total profit and loss used for rescaling my risk capital.
  • CashBalance: Quite self explanatory. Negative means you have borrowed money. BASE is the net of your balances converted into base currency.

When you initially do a futures trade in a non BASE country you will have to borrow money for initial margin. Only if you move into profit beyond this will you have no net borrowing. IB charges you for borrowing money! This is done at LIBOR+spread so is expensive for currencies with higher interest rates (this spread is also why I use IMM's to get direct currency exposure). You can flatten out your exposure by doing a spot FX trade. Personally I try and keep a small positive balance in all currencies, although not excessive as this means I am taking on currency risk. Note you can't trade all possible pairs eg if you find you can't buy AUDGBP then try selling GBPAUD instead. The exception is Korea where you can't hold any speculative currency exposure i.e. not arising from margin requirements in other kinds of trading. All you are allowed to do is flatten your Korean currency position back to zero.

There are many more keywords than shown above. The best advice I have for trying to understand what they all mean is to start with a clean simulated account (a demo account is no good since you are exposed to other peoples random trading, and the account will often be 'empty'). Take a dump of the account value output, then do a trade. Take another dump, then wait for prices to move. By comparing each dump you should be able to see how account value, margin, cash and so on interact. This is quite time consuming but definitely worth it for an insight into how the IB accounts operate for the kind of instruments you want to trade.


Accounting updates


The next kind of data we get are accounting updates. I won't go through the code in detail, suffice to say it's very similar to accounting values, except that it's returning things populated by the updatePortfolio method in the wrapper; these are labelled in the joint queue with ACCOUNT_UPDATE_FLAG:

accounting_updates = app.get_accounting_updates(accountName)
print(accounting_updates)

[(140109274417416: 56825063,GE,FUT,20181217,0.0,0,2500,,GLOBEX,USD,GEZ8,GE,False,,combo:, 10.0, 97.93250275, 2448312.57, 244889.9, -586.43, 0.0)]

This is a tuple with (contract, position, marketPrice, marketValue, averageCost, unrealizedPNL, realizedPNL) A nice touch is to combine this with the information you get from get positions above: I leave this as an exercise to the reader!

Note: for completeness please note that you can also app.get_accounting_time_from_server(self, accountName) asdfto access the results from the wrapper method updateAccountTime although I don't bother myself. 


Anything missing?


I haven't covered the full gamut of what you can get from the IB API. There is a rich variety of data that is available. I haven't looked at news, fundamental data, option data, scanners and managed accounts because I don't use them. However I hope there is enough meat in this series of examples to get you started using IB with the new python API, and to avoid wasting time trying to understand some of the weirder behaviour.

This is the final of a series of five posts on constructing a simple interface in python to the IB API using swigiby. The first four posts are:


http://qoppac.blogspot.co.uk/2017/03/interactive-brokers-native-python-api.html
http://qoppac.blogspot.co.uk/2017/03/historic-data-from-native-ib-pyhon-api.html
http://qoppac.blogspot.co.uk/2017/03/streaming-market-data-from-native.html
http://qoppac.blogspot.co.uk/2017/03/placing-orders-in-native-python-ib-api.html


If you've found this series useful then your next steps might be to learn how to design a systematic trading strategy, use a python backtester to test your strategy, and to understand the nuts and bolts of creating a fully automated system.

12 comments:

  1. The URLs at the end of the article refer to your old python series. They don't point to the series of articles about the native IB python API.

    ReplyDelete
    Replies
    1. "I will be more careful when I copy and paste. I will be more careful when I copy and paste. I will be more careful..."

      thx!

      Delete
  2. Does position come back as zero for recently closed positions?

    ReplyDelete
    Replies
    1. Usually - but I wouldn't rely on it happening.

      Delete
  3. Hi Rob,

    Great series as usual!

    I have an error at the end of the gist when it tries to call app.disconnect()

    File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/ibapi-9.73.2-py3.5.egg/ibapi/client.py",line 195, in disconnect
    File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/ibapi-9.73.2-py3.5.egg/ibapi/connection.py", line 55, in disconnect
    AttributeError: 'NoneType' object has no attribute 'close'

    I'm not the most proficient python programmer, but is it trying to access a property on another thread? Does it work ok for you

    Hopefully you've had a good year in what has been a tough year generally for CTAs

    Thanks

    Chris

    ReplyDelete
    Replies
    1. This seems to be an ongoing issue with the IB API which doesn't affect functionality (at this point you don't really care if it closes properly)

      Delete
  4. Thank you very much. This blog was (is) a great help for me!

    ReplyDelete
  5. Thanks for all of these great blog posts. Just wanted to point out that updateAccountTime doesn't take accountName as a parameter, it only takes timestamp.

    ReplyDelete
    Replies
    1. As in, there's a bug in this line:
      self._my_accounts[accountName].put(data)
      because we don't have access to accountName in the updateAccountTime function

      Delete
  6. I have a silent reader of this blog since 2 years! I managed to build "something systematic" thanks to your valuable articles and books (among other readings and my own experience/education in banking). Thank you very much Rob.

    ReplyDelete
    Replies
    1. I second that.

      Very informative and useful. And the best thing for me is Rob is based in the UK so it's even more contextual.

      Share your banking experience, Passenger!

      Delete

Comments are moderated. So there will be a delay before they are published. Don't bother with spam, it wastes your time and mine.