Tuesday, 11 April 2017

Investment and trading performance - year three


It's now almost three and a half years since I got my last pay cheque and started on my new career as a speculator (also sometime writer, and itinerant consultant), and exactly three years since I started running my fully automated futures trading system, for which my accounting period is the UK tax year ending on 5th April. Which means it's the third in this trilogy of which the first two instalments were:

http://qoppac.blogspot.co.uk/2015/04/futures-trading-performance-year-one.html
http://qoppac.blogspot.co.uk/2016/04/futures-trading-performance-year-two.html

Sharp eyed readers will notice a subtle difference between this and the previous posts - as well as looking at my futures trading I'll also be considering the performance of my wider portfolios. One of the reasons for doing this is that (spoiler alert) my futures trading hasn't done that well this year - but since the point of a managed futures portfolio is to provide diversification that should be seen in the context of what the wider market has done. Another reason is that I'm currently finalising a new book on building investment portfolios; so I thought it would be useful for you to see how I practice what I preach.

As well as looking at performance I'll also analyse my risk, and explain how I'm going to rebalance the 'investment' parts of my portfolio (the 'trading' part is fully automated).


My investments


In simple terms this is what my portfolio looks like:

  • An interactive brokers account containing:
    • My futures trading positions
    • An equity futures hedge
    • A mixture of UK stocks and equity ETFs against which the hedge is constructed
    • Cash margin
  • Various other brokerage accounts containing:
    • UK stocks
    • ETFs

The reason for all the complexity in my IB account is that when I started trading I didn't have sufficient cash to fund my futures trading; I couldn't sell investments to realise cash as this would have resulted in a lumpy capital gains tax bill; hence I dropped a dollop of stocks and ETFs into the account which I could borrow against for margin and to meet any losses.  But I wanted the performance of my futures account to be as 'pure' as possible, so I decided to hedge out the other assets by shorting equity futures against them. 


Broadly speaking I prefer to group my investments into the following sub-portfolios before I analyse them:


  1. A fully automated futures trading system. This requires cash for margin, and has an absolute return or peer benchmark.
  2. An equity futures hedge, constructed to hedge out the non futures assets I hold in my trading account. This has an absolute return benchmark and is low risk.
  3. A diversified portfolio of ETFs across stocks and bonds (with a small allocation to other alternatives). This is gradually mechanically rebalanced on an annual basis. Given my current allocation an appropriate benchmark would be a classic 60:40 fund like this one.
  4. A portfolio of large and mid cap individual UK stocks; traded (very slowly) using mechanical rules around dividend yield (which you'll find in my new book). Benchmark FTSE 350.
I'm going to look at these in reverse order, so we have the wider investment landscape covered before we discuss my dire futures trading.


UK stocks


My portfolio of UK stocks is chosen to give me broad sector exposure, with a tilt towards higher dividend yielders. This is partly because dividend is a risk factor which should earn an additional risk premium if you don't mind wearing the risk, but also because dividend income is a significant contributor to meeting our household expenses.

In terms of numbers this is what the raw performance looks like:

Dividends earned: 6.8%
Mark to market: 17.9%

Total return: 24.7%

Versus a total return benchmark (the FTSE 350 of large and mid cap stocks) which earned 23.5% in the same period, that's respectable but by no means spectacular. However these numbers understate the true performance, because I took a large chunk of money out of this portfolio last year and (as part of my gradual readjustment of my investments to reduce my exposure to UK equities, which three years ago stood at 69% of all equities! 59% of my entire portfolio; far too high for a properly diversified portfolio). 

To deal with this we need to calculate the internal rate of return (IRR), for which I used the XIRR function in <insert name of favourite spreadsheet package here>.  

Portfolio IRR: 29.2%
FTSE 350 benchmark IRR: 22.7%

Better. Of the 17 stocks I owned at some point during the year, 9 were profitable and 8 were losers. The biggest winners were trucking company Stobart (simple total return, +99%), vampire blood sucking bank HSBC (69%), and manufacturer Vesuvius (24%). Most of the losers were small change losers that I only acquired in January when I did a periodic rebalance, one exception being Braemer shipping. 

Incidentally last year the portfolio returned around plus 8% versus a FTSE 350 loss of 7%. Unfortunately I don't keep records going back further than that.

This is what the UK equity portfolio currently looks like (cash weighting as proportion of this sub portfolio value):

STOB 22.41%
ICP 14.05%
VSVS 9.45%
BKG 9.02%
MARS 7.76%
HSBA 7.63%
LGEN 7.62%
KIE 7.53%
PFC 7.39%
RMG 7.14%


The portfolio isn't quite as textbook as I'd like - Stobart is an obvious overweight even though I sold some during the year, as is the financial sector generally (represented by both ICP and bank HSBA); some of these stocks are still held in taxable accounts on which I have to pay CGT which puts a brake on the rebalancing I can do.


Diversified ETF portfolio


This is a classic bond/equity strategic portfolio, although I do have some other asset classes in tiny amounts (it would be more, but bear in mind a quarter of my risk budget is already in one 'alternative' asset: managed futures). 

I don't target my asset allocation just on this portfolio, but instead on my investments holistically. More on that later.

The ETFs I hold give me exposure in equities and bonds across all the main regions (asia, europe, UK, US and emerging markets) with the exception of Asian bonds where I have no exposure. I also have a small amount in Gold, and commercial property.

Raw performance-
Dividends: 5.0%
Mark to market: 21.6%

Total return: 26.6%

That looks amazing against the benchmark (Vanguard 60:40), but once again this is misleading because as part of my rebalancing I was a net buyer of ETFs during the year. Using the more realistic IRR:

Portfolio IRR: 19.1%
60:40 Benchmark: 18.7%

Roughly in line with the benchmark. Last year the XIRR was 6.2%, and the benchmark earned 0.11%. Incidentally the benchmark and my own portfolio have done so strongly partly because of the depreciation of GBP against most other countries last year. US investors who make up most of the readership of this blog won't have seen such great returns; global equities were up about 14.6% and global bonds were flat, giving a return of around 9% on a 60:40 portfolio.

It might make more sense to consider the performance of my investment portfolio 'in toto' (UK shares plus ETF) since that is what I target my asset allocation against:

Whole investment portfolio IRR: 22.3%
60:40 Benchmark IRR: 17.7%

So far, so good. I won't show the individual ETFs I own, since it's the overall risk in my portfolio which is important - and that will come later.

Futures trading and equity hedge


That was the good news; now for the less good news. Here is the 'money shot'; a graph showing the total value of my trading account (which remember consists of my futures trading, some equities and ETFs which I've already accounted for above, and my futures hedge):



The first plot shows the performance since inception, and the second plot over the last 12 months. In fact I've made almost exactly nothing in the relevant period (the UK tax year 2016-17); in percentage terms just 0.3%.

However that net figure hides a lot of movement; here's the full breakdown:

Stocks & ETF
Mark to market: 22.6%
Dividends 4.9%
Commissions: -0.02% 

Subtotal: 27.6%

Hedge
Mark to market: -13.2%
Commissions: -0.03%

Subtotal: -13.2%

Total for stocks and hedge: 14.4%

Futures
Gross profit: -14.3%
Commissions: -0.73% 
Slippage: -0.56% (Bid ask -0.97%, less execution algo profit 0.41%)
Data fees: -0.03%
FX adjustments: 1.74%
Interest: -0.15% 

Total for futures: -14.0%

Grand total: 0.3%


So broadly speaking I made a bunch of money on my stocks hedge which I promptly lost on my futures trading. Trading costs were an acceptable 1.3% versus a backtested 1.5%. As usual commissions were a little higher than expected (since I don't calculate them for rolls), bid-ask was exactly in line, and algo profit was a bonus since I don't include that in my backtest to be conservative.


Statistics

These are all for my overall account (futures, stocks, and hedges):

Standard deviation of returns (based on weekly, annualised): 22.2% versus long term target 25%
Average drawdown: 9.6%
Max drawdown: -19.4%
Worst day: -5.7%
Best day: +4.8%
Win/Loss ratio: 1.66
% wins: 37%


Benchmarking


I've said before it's difficult to benchmark this portfolio against a pure managed futures fund like AHL, my former employers, because (a) the volatility target I use is a little higher than typical and (b) the inclusion of my stocks plus hedging returns.

Nevertheless, here are the results of my portfolio for the last 3 years:

Hedge:   -1.1%, 16.3%,  14.4%
Futures: 58.2%, 23.2%, -14.0%
Net:     57.2%, 39.6%,   0.3%

And for comparision AHL diversity GBP class with returns multiplied by 2.8 (reflecting my 25% risk target versus the 9% of this AHL fund), of course this is a pure managed futures portfolio:

Futures: 106.9%, -10.6%, -6.2%

[All these figures are based on notional capital; they would be around 4 times higher if calculated as a percentage of margin. AHL figures have fees deducted; my futures figure ignores any cash I'd get on excess margin]

Over the entire 3 years using constant capital (so I can just add up annual returns) the returns of my pure futures portfolio are around 67% versus 77% for AHL; equating to a Sharpe Ratio of just under one for me; exactly in line with my backtest expectations (to which I'd add a large pinch of salt). Not bad for a one man and his laptop operation. Casting a slightly wider net, the SG CTA index that fundseeder use to benchmark my performance had returns of 30%, -3.6% and -7.5% over the relevant three years. 

Overall I think it's fair to say I've probably underperformed the overall CTA world this year based on my pure futures performance, though over two and three years I'm fairly happy with how things have gone.


Performance by instrument

The biggest winners over the year were: VIX +6.4% (contribution), AEX +4.7%, V2X +4.3%, JPY +2.3% and NASDAQ +1.2%

The losers were GAS_US -5%, Corn -2.7%, Eurodollar -2.6%, GBP -2.2%, BTP -2.0%

Only an idiot couldn't have made money out of being short volatility this year; firstly the level slightly fell (from around spot 14.1 to 12.9 over the year) but more importantly there was the usual strong roll return (effectively earning the volatility premium):

AEX is a more interesting picture:

Here the system gradually bled a little money whilst keeping positions small. Then once an uptrend was confirmed in mid december it went strongly long and has benefitted from the continuing rise since then.

JPY (the JPYUSD IMM currency future) shows a classic trend following picture:

There is only one decent trend in the whole year, and the system captures it in classic fashion, giving up a little on the entry and the exit, and bleeding slightly the rest of the time.

What about the losses? Heres GAS:

This is an evil whipsawing with predictable results. Eurodollar is normally a star of trend following portfolios but not this year:

The problem here is that the strong carry in Eurodollar keeps us long even as the trend moves against us after Trump is elected. Another topical market was GBPUSD:

You can probably spot June 23rd! There isn't much one can do about this kind of event, except diversifying; because I have over 40 futures markets the damage done by this multiple sigma event was limited to just 1% of portfolio value, and I actually ended up as a net beneficary of Brexit this year (see here).


Overall performance


Summarising my overall investment portfolio, the numbers look something like this (as a percentage of my total investment assets, including cash held for margin):

Sub-portfolio  / contribution

UK Shares  +8.0%
ETFs +15.8%
Hedge -2.7%

Net investments 21.1%

Futures trading -2.9%

Grand total +18.2%


I think a reasonable benchmark here again is a fully invested 60:40 portfolio I'd hold if I didn't do any trading; and I've narrowly underperformed that (simple return 19.3%). But to reiterate, adding a diversifying asset like managed futures in your portfolio reduces your risk and improves your sharpe ratio; it may also improve your overall return (and this has certainly what has happened in backtesting and also over the last 3 years) but it won't do so every single year. For example, here are the figures for last year (2015-16):

Shares & ETFs  +6.1%
Hedge          +3.8%

Net investments 9.9%

Futures trading +5.4%

Grand total +14.4%

The benchmark here (60:40 again) barely moved with a total return of 0.1%. Clearly the previous year 2014-15 (for which I don't have precise figures) would have been even better since managed futures had a fantastic year.

It's worth considering what might happen in another 2008 event; with equities down 40% and assuming bonds don't bail you out this time round you're looking at a net loss of around 36% in a 60:40 portfolio. If managed futures do as well as they did in 2008 (around 2 times annual vol target, or 50%) then  hypothetically that would give me the following set of returns:

Shares & ETFs  -36%
Hedge          +4%

Net investments -32%

Futures trading +10%

Grand total -22% (versus benchmark 60:40 -36%)

That 10% reduction in losses is the insurance policy that managed futures would hopefully provide in a total market meltdown, and it's worth occasionally underperforming the 60/40 benchmark to get it.


Risk analysis and rebalancing


As part of this annual evaluation I like to look at my risk and evaluate where some rebalancing might be prudent. 

Here are the raw figures. The equity allocation includes the effect of my hedge.

Asia EM Euro UK US Global Total
Bonds 5.06% 3.97% 3.93% 7.17% 1.15% 21%
Equity 6.85% 7.62% 11.32% 21.55% 0.41% 0.07% 48%
Futures 27.55% 28%
Commodities 0.13% 0%
Property 3.22% 3%


As an aside 'property' does not include the value of the house where I live, which is excluded from these calculations. Commodities is a small investment in a gold ETF.

Another way of looking at this is to work out the geographical allocations within bonds and equities, as a proportion of the asset class:


Asia EM Euro UK US Global
Bonds 0.0% 23.8% 18.6% 18.5% 33.7% 5.4%
Equity 14.3% 15.9% 23.7% 45.1% 0.9% 0.2%


My long term strategic risk allocation is: bonds 20%, equities 50%, futures trading 25%, others 5%. I use simple trend following rules (another plug for the new book if you want detail) to vary these weights, which right now are suggesting an overweight to equities with a little less in bonds. After some judicious rebalancing, being careful not to incur a tax liability and minimising transaction costs, my new allocation is: bonds 17%, equities 54%, futures trading 26%, others 3%.

The regional allocation in each asset class is now:

Asia EM Euro UK US Global
Bonds 0.0% 27.3% 21.0% 19.2% 27.0% 5.5%
Equity 14.8% 20.6% 19.9% 37.8% 6.7% 0.1%

Remember the lack of Asian bond exposure is because of a lack of suitable ETFs, not because I hate them.


Summary


Overall I'm perfectly happy with my overall investment strategy. One year isn't long enough to give you statistically significant evidence, and indeed neither is three years, but .

Doing this kind of analysis is quite time consuming, even when you are a spreadsheet ninja, and filling in tax returns also becomes quite onerous. Nevertheless so far at least following an active investment and trading strategy has been a good use of the few days a year I spend on this activity.

Friday, 24 March 2017

Getting position and accounting data out of IB native python API

This is the final post. Not the final post of the blog; which may be good news or bad. But the final post in my short series on using the new native python API for interactive brokers. Having got some prices and submitted some orders we want to know whether we made any money or not; and what positions we have. Although the code is rather trivial, interpreting the results requires some understanding.

This is an update of the following post where I used the non native swigibpy (which I wrote nearly 3 years ago!).

This the fifth in a series of posts on using the native python API  You should read the firstsecond, third, and fourth. You should also read the documentation.

You need to get the code from this gist.


Getting positions


You will need to run the code in the last post first so you actually have a position to look at (this will submit a market order for Eurodollar futures). 

Now, if you've been following the series so far this kind of code pattern should be easy to follow:

## lets get positions
positions_list = app.get_current_positions()

In the client object:

def get_current_positions(self):

    ## Make a place to store the data we're going to return    
    positions_queue = finishableQueue(self.init_positions())

    ## ask for the data    
    self.reqPositions()
    <SNIP>

In the wrapper object:


def init_positions(self):
    positions_queue = self._my_positions = queue.Queue()

    return positions_queue

def position(self, account, contract, position,
             avgCost):

    ## uses a simple tuple, but you could do other, fancier, things here    
    position_object = (account, contract, position,
             avgCost)

    self._my_positions.put(position_object)

def positionEnd(self):
    ## overriden method
    self._my_positions.put(FINISHED)


Back in the client object:


def get_current_positions(self):

    <SNIP>
    ## poll until we get a termination or die of boredom    
    MAX_WAIT_SECONDS = 10    
    positions_list = positions_queue.get(timeout=MAX_WAIT_SECONDS)

    while self.wrapper.is_error():
        print(self.get_error())

    if positions_queue.timed_out():
        print("Exceeded maximum wait for wrapper to confirm finished whilst getting positions")

    return positions_list


Results:
print(positions_list)

 [
('DU15153', 
140109276672072: 56825063,GE,FUT,20181217,0.0,,2500,,,USD,GEZ8,GE,False,,combo:, 
10.0, 
244889.9)]

The results come back as a 4 tuple: account identifying name, contract, position and realised PNL (this might seem large - but this is a demo account so it will depend on what other people have been trading). I'd advise making a nice class to put these into, although I don't bother doing that here.

Finally be very careful about using these snapshots to work out what positions you have. Firstly it is common to get the position coming back repeated times. So you will need to filter out duplicates. Secondly positions may sometimes not appear at all.

I use my own record of trades to find out what my current position should be. I compare this to the IB snapshot throughout the day. If there are any breaks in the reconciliation I temporarily halt trading in the relevant contract until the break has gone away. Occasionally this is because IB has sent me a fill I haven't picked up, or because the IB position snapshot is missing some contracts.

Getting accounting data


The accounting data is a little more complicated. Firstly, we get three types of accounting data from IB, which need to be distinguished. Secondly, I choose to access the accounting data via a cache in the client object which only refreshes if the data we have is more than 5 minutes old. Of course once you understand the logic feel free to implement this differently yourself.


Accounting values


The first type of accounting data we get are accounting values (returned by updateAccountValue in the wrapper)

## get the account name from the position
## normally you would know your account but this is the demo
nameaccountName = positions_list[0][0]

## and accounting information
accounting_values = app.get_accounting_values(accountName)


In the client object:

def get_accounting_values(self, accountName):

    #All these functions follow the same pattern: check if stale, if not return cache, else update values
    return self._account_cache.get_updated_cache(accountName, ACCOUNT_VALUE_FLAG)

What is this doing under the hood? First we check to see if the cache is up to date (default is to check if we have got an account value in the last 5 minutes). If not then we ask for the accounting data to be updated. We then return the cached value (eithier historical, or just updated)

Let's have a look at the workhorse function that actually updates the data _update_accounting_data:

def _update_accounting_data(self, accountName):
    """    Update the accounting data in the cache
    :param accountName: account we want to get data for    
    :return: nothing    
    """
    ## Make a place to store the data we're going to return    
    accounting_queue = finishableQueue(self.init_accounts(accountName))

    ## ask for the data    
    self.reqAccountUpdates(True, accountName)

    <SNIP>
In the wrapper object:
## get accounting datadef init_accounts(self, accountName):
    accounting_queue = self._my_accounts[accountName] = queue.Queue()

    return accounting_queue


def updateAccountValue(self, key:str, val:str, currency:str,
                        accountName:str):

    ## use this to seperate out different account data    
    data = identifed_as(ACCOUNT_VALUE_FLAG, (key,val, currency))
    self._my_accounts[accountName].put(data)


def updatePortfolio(self, contract, position:float,
                    marketPrice:float, marketValue:float,
                    averageCost:float, unrealizedPNL:float,
                    realizedPNL:float, accountName:str):

    ## use this to seperate out different account data    
    data = identifed_as(ACCOUNT_UPDATE_FLAG, (contract, position, marketPrice, marketValue, averageCost,
                                      unrealizedPNL, realizedPNL))
    self._my_accounts[accountName].put(data)

def updateAccountTime(self, timeStamp:str):

    ## use this to seperate out different account data    
    data = identifed_as(ACCOUNT_TIME_FLAG, timeStamp)
    self._my_accounts[accountName].put(data)


def accountDownloadEnd(self, accountName:str):

    self._my_accounts[accountName].put(FINISHED)

The only novel thing here are these identifed_as(ACCOUNT_VALUE_FLAG, (key,val, currency)) guys. Because we push three different types of data on to the same queue we need to be able to distinguish them from each other. Rather than just push the raw data we're getting from IB (all stored as tuples, except for the single length timeStamp in updateAccountTime) we push an object which comes with a labelling flag to identify where it came from.


Back in the client object:

def _update_accounting_data(self, accountName):

    <SNIP>
    ## poll until we get a termination or die of boredom    
    MAX_WAIT_SECONDS = 10    
    accounting_list = accounting_queue.get(timeout=MAX_WAIT_SECONDS)

    while self.wrapper.is_error():
        print(self.get_error())

    if accounting_queue.timed_out():
        print("Exceeded maximum wait for wrapper to confirm finished whilst getting accounting data")


So far this is boilerplate, but this isn't: 
# seperate things out, because this is one big queue of data with different things in it
    accounting_list = list_of_identified_items(accounting_list)
    seperated_accounting_data = accounting_list.seperate_into_dict()


The queue is made up of three different kinds of data (values, updates, and time) from three different wrapper functions, so we need to split them up. We now have a dictionary, we now use this to update the different parts of the cache:
## update the cache with different elements
self._account_cache.update_cache(accountName, seperated_accounting_data)
We're now returned to the get_accounting_values function that will actually return the updated cache values we want:

def get_accounting_values(self, accountName):

    #All these functions follow the same pattern: check if stale, if not return cache, else update values
    return self._account_cache.get_updated_cache(accountName, ACCOUNT_VALUE_FLAG)

Results:

print(accounting_values)



[('AccountCode', 'DU15153', ''), ('AccountOrGroup', 'DU15153', 'BASE'), ('AccountOrGroup', 'DU15153', 'USD'), ('AccountReady', 'true', ''), .... <SNIP>

The account_value output will be a very long list of three tuples. Each one consists of (keyname, value, currency). Currency BASE indicates it is the accounts base currency (USD for the test account). Here are some of the more interesting account value entries.

  • ExchangeRate: In the form of ('ExchangeRate', '1.00', 'USD') This is a very boring exchange rate as the base currency is USD.
  •  FuturesPNL: This is how much we have made trading futures today, by currency. BASE is the total in base currency.
  • StockMarketValue: Self explanatory
  • NetLiquidation: This is what is my account worth if I close all my positions out (ignoring commissions and bid-ask). Its what I use to ultimately determine the total profit and loss used for rescaling my risk capital.
  • CashBalance: Quite self explanatory. Negative means you have borrowed money. BASE is the net of your balances converted into base currency.

When you initially do a futures trade in a non BASE country you will have to borrow money for initial margin. Only if you move into profit beyond this will you have no net borrowing. IB charges you for borrowing money! This is done at LIBOR+spread so is expensive for currencies with higher interest rates (this spread is also why I use IMM's to get direct currency exposure). You can flatten out your exposure by doing a spot FX trade. Personally I try and keep a small positive balance in all currencies, although not excessive as this means I am taking on currency risk. Note you can't trade all possible pairs eg if you find you can't buy AUDGBP then try selling GBPAUD instead. The exception is Korea where you can't hold any speculative currency exposure i.e. not arising from margin requirements in other kinds of trading. All you are allowed to do is flatten your Korean currency position back to zero.

There are many more keywords than shown above. The best advice I have for trying to understand what they all mean is to start with a clean simulated account (a demo account is no good since you are exposed to other peoples random trading, and the account will often be 'empty'). Take a dump of the account value output, then do a trade. Take another dump, then wait for prices to move. By comparing each dump you should be able to see how account value, margin, cash and so on interact. This is quite time consuming but definitely worth it for an insight into how the IB accounts operate for the kind of instruments you want to trade.


Accounting updates


The next kind of data we get are accounting updates. I won't go through the code in detail, suffice to say it's very similar to accounting values, except that it's returning things populated by the updatePortfolio method in the wrapper; these are labelled in the joint queue with ACCOUNT_UPDATE_FLAG:

accounting_updates = app.get_accounting_updates(accountName)
print(accounting_updates)

[(140109274417416: 56825063,GE,FUT,20181217,0.0,0,2500,,GLOBEX,USD,GEZ8,GE,False,,combo:, 10.0, 97.93250275, 2448312.57, 244889.9, -586.43, 0.0)]

This is a tuple with (contract, position, marketPrice, marketValue, averageCost, unrealizedPNL, realizedPNL) A nice touch is to combine this with the information you get from get positions above: I leave this as an exercise to the reader!

Note: for completeness please note that you can also app.get_accounting_time_from_server(self, accountName) asdfto access the results from the wrapper method updateAccountTime although I don't bother myself. 


Anything missing?


I haven't covered the full gamut of what you can get from the IB API. There is a rich variety of data that is available. I haven't looked at news, fundamental data, option data, scanners and managed accounts because I don't use them. However I hope there is enough meat in this series of examples to get you started using IB with the new python API, and to avoid wasting time trying to understand some of the weirder behaviour.

This is the final of a series of five posts on constructing a simple interface in python to the IB API using swigiby. The first four posts are:


http://qoppac.blogspot.co.uk/2017/03/interactive-brokers-native-python-api.html
http://qoppac.blogspot.co.uk/2017/03/historic-data-from-native-ib-pyhon-api.html
http://qoppac.blogspot.co.uk/2017/03/streaming-market-data-from-native.html
http://qoppac.blogspot.co.uk/2017/03/placing-orders-in-native-python-ib-api.html


If you've found this series useful then your next steps might be to learn how to design a systematic trading strategy, use a python backtester to test your strategy, and to understand the nuts and bolts of creating a fully automated system.

Monday, 20 March 2017

Placing orders in the native python IB API

This the fourth in a series of posts on using the native python API for interactive brokers. You should read the first, second, and third, before this one.

It is an updated version of this older post, which used a third party API (swigibpy) which wraps around the C++ API. I've changed the code, but otherwise the post is pretty similar.

We are nearly at the end of our journey of simplistic examples of how to get the swigibpy package to mediate between the wonderful world of Python and the dark place that is the Interactive brokers C++ API. Having learned how to get prices out of the API we are now ready to actually do some trading- submit orders, check they are active, potentially cancel them, receive fills and get historic execution data.


Where do I start?


It's worth reading the relevant part of the documentation.

You need to get the code from the following gist.

If you have a live or simulated Gateway / TWS session running (one associated with a real account number) it should work just fine. Note that you can also run this with the edemo account (password: demo123), but the results might not be reliable. This is because you are seeing the orders placed by everyone who is playing around with this account, so you could get all kinds of randomeness.

WARNING I highly recommend that you do not run this code on a real money trading account until / unless you know exactly what you are doing! In particular real trading code should check market liquidity before trading, particularly for market orders.


Contract details - what is it?


As in the previous post there is the tedious business of creating an object to talk to IB and resolving the contract we want to trade:


app = TestApp("127.0.0.1", 4001, 1)

ibcontract = IBcontract()
ibcontract.secType = "FUT"
ibcontract.lastTradeDateOrContractMonth="201812"
ibcontract.symbol="GE"
ibcontract.exchange="GLOBEX"
## resolve the contract
resolved_ibcontract = app.resolve_ib_contract(ibcontract)




Getting full contract details from the server... 

For the rest of the post anything you see in that nice purple bold font is python output.

WARNING: If you are trading the VIX, which now has weekly expiries, you will need to specify the full expiry date in yyyymmdd format. 

ANOTHER WARNING; the date is the contract expiry not where relevant first or last notice date. This means you should be wary of using this date to tell you when to roll certain kinds of futures contracts eg US bonds.


Order placing - can I buy it?

Now the moment we've all been waiting for... we're actually going to buy something. We build the order object and pass it a client function.


order1=Order()
order1.action="BUY"
order1.orderType="MKT"
order1.totalQuantity=10
order1.transmit = True
orderid1 = app.place_new_IB_order(ibcontract, order1, orderid=None)

print("Placed market order, orderid is %d" % orderid1)

Getting orderid from IB
Using order id of 1
Placed market order, orderid is 1

In the client class:

def place_new_IB_order(self, ibcontract, order, orderid=None):

    ## We can eithier supply our own ID or ask IB to give us the next valid one    
    if orderid is None:
        print("Getting orderid from IB")
        orderid = self.get_next_brokerorderid()

        if orderid is TIME_OUT:
            raise Exception("I couldn't get an orderid from IB, and you didn't provide an orderid")

    print("Using order id of %d" % orderid)

    ## Note: It's possible if you have multiple traidng instances for orderids to be submitted out of sequence    
    ##   in which case IB will break
    # Place the order    
    self.placeOrder(
        orderid,  # orderId,
        ibcontract,  # contract, 
        order  # order    )

    return orderid    


Obvious first thing to notice here is the concept of an orderid. This is a number that identifies to IB what the order is; at least temporarily and for today only. Only restriction on order id's is that the next order is higher than the last. This means if you submit an order with an id of 999999 you will lose all the orderids below that. You can also reset the id 'clock' to 1 via an option on the Gateway or TWS API configuration menu.  Safest thing to do is ask IB for the next orderid as done here by supplying None to the calling function.

I generate my own orderid's preferring to reserve them first in my own database. This is fine as long as you are running a single linear process where there is no chance of an 'older' order being submitted before a 'newer' one.



Fill data - how much did it cost me?

What happens when an order is filled; completely or partially? Well the following method in the wrapper function is triggered. Notice that the logic is slightly more complicated because this function fulfills two duties:


  • A fill has arrived when a trade actually executes, in which case the reqId will be -1 (stored as FILL_CODE here)
  • We can ask for fill information, in which case the reqId will accompany the information. More on this shortly


def execDetails(self, reqId, contract, execution):
    ## overriden method
    execdata = execInformation(execution.execId, contract=contract,
                               ClientId=execution.clientId, OrderId=execution.orderId,
                               time=execution.time, AvgPrice=execution.avgPrice,
                               AcctNumber=execution.acctNumber, Shares=execution.shares,
                               Price = execution.price)

    ## there are some other things in execution you could add    
    ## make sure you add them to the .attributes() field of the execInformation class
    reqId = int(reqId)

    ## We eithier put this into a stream if its just happened, or store it for a specific request    
   if reqId==FILL_CODE:
        self._my_executions_stream.put(execdata)
    else:
        self._my_requested_execution[reqId].put(execdata)

In this case as we haven't yet asked for execution data reqId will be -1, and we'll put the data into my_executions_stream. This is a 'permanent' queue that hosts execution data: I describe it as permanent because we didn't need to run an init_ ... method to set it up in place_new_IB_order like I've shown you in previous posts; the queue is created when the instance of TestWrapper is created.

(Also this would be the function that would update your order status database; as a pro at this stuff, naturally you would have such a thing).

To see the fill data I need to do this:

print("Recent fills")
filldetails = app.recent_fills_and_commissions()
print(filldetails)

There is quite a lot going on under the hood here which is worth understanding so lets examine the client function we're calling.

def recent_fills_and_commissions(self):
    """    Return recent fills, with commissions added in
    :return: dict of execInformation objects, keys are execids    
    """
    recent_fills = self._recent_fills()
    commissions = self._all_commissions() ## we want all commissions
    
    ## glue them together, create a dict, remove duplicates    
    all_data = recent_fills.blended_dict(commissions)

    return all_data

The first thing to note is that this returns information coming from two sources: execDetails (labelled recent_fills here) and commissions. Frustratingly this data is provided by two separate parts of the wrapper function. We've already seen the executions, here it is for commissions:

def commissionReport(self, commreport):

    commdata = execInformation(commreport.execId, Commission=commreport.commission,
                    commission_currency = commreport.currency,
                    realisedpnl = commreport.realizedPNL)


    ## there are some other things in commreport you could add    
    ## make sure you add them to the .attributes() field of the execInformation class
    ## These always go into the 'stream' as could be from a request, 
    ## or a fill thats just happened    

    self._my_commission_stream.put(commdata)

As with execution details this can get called eithier when we've requested execution details (of which more later), or whenever we get an actual fill (as we're doing here). Unlike the execDetails function however we never get a reqId. So all the commissions data is dumped into a 'permanent' _stream queue, rather than one we've initialised already.

Once the executions and commission data has arrived in the 'permanent' _stream queues, we can access it via these functions:

def _recent_fills(self):
    """    Returns any fills since we last called recent_fills
    :return: list of executions as execInformation objects    
    """
    ## we don't set up a queue but access the permanent one    
    fill_queue = self.access_executions_stream()

    list_of_fills=list_of_execInformation()

    while not fill_queue.empty():
        MAX_WAIT_SECONDS = 5        
        try:
            next_fill = fill_queue.get(timeout=MAX_WAIT_SECONDS)
            list_of_fills.append(next_fill)
        except queue.Empty:
            ## corner case where Q emptied since we last checked if empty at top of while loop            
            pass
    ## note this could include duplicates and is a list    

    return list_of_fills

Notice that we clear the queue once we've accessed the data. And _recent_commissions(selfis in a similar vein.

def _recent_commissions(self):

    ## we don't set up a queue, as there is a permanent one    
    comm_queue = self.access_commission_stream()
    list_of_comm=list_of_execInformation()

    while not comm_queue.empty():
        MAX_WAIT_SECONDS = 5        
        try:
            next_comm = comm_queue.get(timeout=MAX_WAIT_SECONDS)
            list_of_comm.append(next_comm)
        except queue.Empty:
            ## corner case where Q emptied since we last checked if empty at top of while loop            pass
    ## note this could include duplicates and is a list    
return list_of_comm

However because all commissions live in the commission_stream queue I actually use another function to get commission data, which returns both the last chunk of commissions, plus any other commission data I've collected:

def _all_commissions(self):
    ## self._commissions is created when the client instance is __init__    
    
    original_commissions = self._commissions
    latest_commissions = self._recent_commissions()

    ## these are just simple lists so we can glue them together
    all_commissions = list_of_execInformation(original_commissions + latest_commissions)

    self._commissions = all_commissions

    # note this could include duplicates and is a list    
    return all_commissions

Finally we have to glue these together. The final line in the function before the return populates the fill data with the relevant commission levels.

def recent_fills_and_commissions(self):    
    
    recent_fills = self._recent_fills()
    commissions = self._all_commissions() ## we want all commissions
    
    ## glue them together, create a dict, remove duplicates    
    all_data = recent_fills.blended_dict(commissions)

    return all_data

Recent fills
{'00004468.58ca0e5f.01.01': Execution - contract: 56825063,GE,FUT,20181217,0.0,,2500,GLOBEX,,USD,GEZ8,GE,False,,combo: ClientId: 1 OrderId: 1 time: 20170316  09:50:31 AvgPrice: 98.055 Price: 98.055 AcctNumber: DU15075 Shares: 10.0 Commission: 24.0 commission_currency: USD realisedpnl: 1.7976931348623157e+308}

Notice this returns a dict; the keyword is the execId. I use some simple objects to gather up and merge together these two streams of data but you can do it differently of course.

Just to reiterate the _recent functions clear the relevant queues as you can see when I try and get the fill data again (although the commission data is saved so self._all_commissions() will still work).

## when I call again should be empty as we've cleared the memory of recent fills
print("Recent fills (should be blank)")
morefilldetails = app.recent_fills_and_commissions()
print(morefilldetails)

Recent fills (should be blank)
{}


IMPORTANT DETAIL: It won't be obvious from this simple example unless you can submit a very large order in a thin market but the fills come in as cumulative order updates, not separate fills. Its worth looking at an example. Suppose you try and buy 10 lots, and you get fills of:

  • 3 lots @ 100.0
  • 6 lots @ 100.2
  • 1 lot @ 100.5
 Then the fills that come in will look like this:

  • qty: 3 lots, price=100.0
  • qty: 9 lots, price=100.13333333
  • qty: 10 lots, price=100.17
So if you do care about each partial fill you are going to have to hope that you see every little fill coming in and use a differencing process to see the detail of each.

By the way 'orderid' is only a temporary thing for IB; after tommorrow it won't associate it with this order. Instead you should use 'permid' for your record keeping. 'execid' is different for each part fill so you could use it to make sure you aren't including fill information you already have; in practice this isn't problematic due to the cumulative nature of the information.



Past execution data - sorry, repeat that, how much?!

It is clearly very important that fill data is correctly captured by your trading software. One reason being to keep track of what your position is; as we shall see in the next post IB doesn't offer mere mortals a super accurate current position facility. So I generally use my own knowledge of trade history to decide where I am, position wise. Because the fills usually arrive in the wrapper function only once its possible under certain conditions to miss them; eg if your API client dies before you see the fill or just isn't running when one arrives on a previously closed market in the middle of the night. Its generally good practice then to reconcile what IB has for a record of fills versus your own.
This information is only available up to midnight of the day you trade. So I run a reconciliation 3 times a day. If you lose a fill from before today you will need to find it on the IB website account management microsite, and manually enter it into your database.
Here is how we do it.
print("Executions today")
execdetails = app.get_executions_and_commissions()
print(execdetails)

def get_executions_and_commissions(self, reqId=DEFAULT_EXEC_TICKER, execution_filter = ExecutionFilter()):

    ## store somewhere    
    execution_queue = finishableQueue(self.init_requested_execution_data(reqId))

    ## We can change ExecutionFilter to subset different orders    
    ## note this will also pull in commissions 
    self.reqExecutions(reqId, execution_filter)

    ## Run until we get a terimination or get bored waiting    
    MAX_WAIT_SECONDS = 10    
    exec_list = list_of_execInformation(execution_queue.get(timeout = MAX_WAIT_SECONDS))

    while self.wrapper.is_error():
        print(self.get_error())

    if execution_queue.timed_out():
        print("Exceeded maximum wait for wrapper to confirm finished whilst getting exec / commissions")
 
<snip>

(Note: We can change ExecutionFilter to subset different orders)     
This is a more familiar pattern: we create a queue for the wrapper to put stuff which will be terminated by a finish event, and then request the IB API to start sending data, some method in the wrapper populates the queue, and we then pull the contents of the queue in. The relevant wrapper function should look familiar:


def execDetails(self, reqId, contract, execution):
    ## overriden method
    execdata = execInformation(execution.execId, contract=contract,
                               ClientId=execution.clientId, OrderId=execution.orderId,
                               time=execution.time, AvgPrice=execution.avgPrice,
                               AcctNumber=execution.acctNumber, Shares=execution.shares,
                               Price = execution.price)

    ## there are some other things in execution you could add    
    ## make sure you add them to the .attributes() field of the execInformation class
    reqId = int(reqId)

    ## We eithier put this into a stream if its just happened, or store it for a specific request    

   if reqId==FILL_CODE:
        self._my_executions_stream.put(execdata)
    else:
        self._my_requested_execution[reqId].put(execdata)
Its the same hardworking function as before, only this time the reqId will not be -1 (appearing here as FILL_CODE rather than being hardcoded) so we append the fill that is received to the requested queue for a given reqId.
The only wrinkle is that the commission data needs to be brought in and merged, as we did before with fills:

def get_executions_and_commissions(self, reqId=DEFAULT_EXEC_TICKER, execution_filter = ExecutionFilter()):

    <snip>
    ## Commissions will arrive seperately. We get all of them, but will only use those relevant for us    
    commissions = self._all_commissions()

    ## glue them together, create a dict, remove duplicates    
    all_data = exec_list.blended_dict(commissions)

    return all_data

   

execdetails = app.get_executions_and_commissions()
print(execdetails)
Executions today
{'00004468.58ca0e5f.01.01': Execution - contract: 56825063,GE,FUT,20181217,0.0,,2500,GLOBEX,,USD,GEZ8,GE,False,,combo: ClientId: 1 OrderId: 1 time: 20170316  09:50:31 AvgPrice: 98.055 Price: 98.055 AcctNumber: DU15075 Shares: 10.0 Commission: 24.0 commission_currency: USD realisedpnl: 1.7976931348623157e+308}
As before the executions are listed in a dict with the execId as the key.

Placing limit orders

No self respecting trader will use market orders (see my post), so how to define limit orders?

order2=Order()
order2.action="SELL"
order2.orderType="LMT"
order2.totalQuantity=12
order2.lmtPrice = 100.0
order2.tif = 'DAY'
order2.transmit = True

orderid2 = app.place_new_IB_order(ibcontract, order2, orderid=None)
print("Placed limit order, orderid is %d" % orderid2)

Getting orderid from IB
Using order id of 2
Placed limit order, orderid is 2


This is just a tiny selection of the available orders, see the docs here.


 Active order status- have I bought it?

IB can tell us what orders we are working. Unless you ask very quickly (or submit your order outside of trading hours) this is likely only to return unfilled limit orders like the one we've just submitted.

open_orders = app.get_open_orders()
print(open_orders)




Client function:


def get_open_orders(self):

    ## store the orders somewhere    
    open_orders_queue = finishableQueue(self.init_open_orders())

    ## You may prefer to use reqOpenOrders() which only retrieves orders for this client    self.reqAllOpenOrders()

    ## Run until we get a terimination or get bored waiting    
    MAX_WAIT_SECONDS = 5    
    open_orders_list = list_of_orderInformation(open_orders_queue.get(timeout = MAX_WAIT_SECONDS))

    while self.wrapper.is_error():
        print(self.get_error())

    if open_orders_queue.timed_out():
        print("Exceeded maximum wait for wrapper to confirm finished whilst getting orders")

    ## open orders queue will be a jumble of order details, turn into a tidy dict with no duplicates    open_orders_dict = open_orders_list.merged_dict()

    return open_orders_dict


Wrapper functions:


def init_open_orders(self):
    open_orders_queue = self._my_open_orders = queue.Queue()

    return open_orders_queue


def orderStatus(self, orderId, status, filled, remaining, avgFillPrice, permid,
                parentId, lastFillPrice, clientId, whyHeld):

    order_details = orderInformation(orderId, status=status, filled=filled,
             avgFillPrice=avgFillPrice, permid=permid,
             parentId=parentId, lastFillPrice=lastFillPrice, clientId=clientId,
                                     whyHeld=whyHeld)

    self._my_open_orders.put(order_details)


def openOrder(self, orderId, contract, order, orderstate):

    order_details = orderInformation(orderId, contract=contract, order=order, 
                                     orderstate = orderstate)
    self._my_open_orders.put(order_details)


def openOrderEnd(self):

    self._my_open_orders.put(FINISHED)

{2: Order - contract: 56825063,GE,FUT,20181217,0.0,?,2500,GLOBEX,,USD,GEZ8,GE,False,,combo: order: 2,1,1017786930: LMT SELL 12@100.000000 DAY orderstate: <ibapi.order_state.OrderState object at 0x7f5e7bcfe860> status: Submitted filled: 0.0 avgFillPrice: 0.0 permid: 1017786930 parentId: 0 lastFillPrice: 0.0 clientId: 1 whyHeld: }

This should be a familiar story now. The only interesting thing is that we get order details from two wrapper functions, orderStatus and openOrder. I don't bother segregating them, instead I add them to the same queue, and then merge them with open_orders_dict = open_orders_list.merged_dict()

In practice I have noticed that the correct end condition for receiving open orders doesn't always trigger so you do need an max waiting time (which is good practice anyway).

As with fills the output is returned as a dict, and the keys are orderids. We just have the single active limit order since the market order (orderid 1) has long since been filled.


Order modification


To modify an existing order we submit a new order, but with an existing orderid.


order3=Order()
order3.action="BUY"
order3.orderType="LMT"
order3.totalQuantity=5
order3.lmtPrice = 10.0
order3.tif = 'DAY'
order3.transmit = True

orderid3 = app.place_new_IB_order(ibcontract, order3, orderid=None)
print("Placed limit order, orderid is %d" % orderid2)

print("Open orders (should be two)")
open_orders = app.get_open_orders()
print(open_orders.keys())

Getting orderid from IB
Using order id of 3
Placed limit order, orderid is 2
Open orders (should be two)

dict_keys([2, 3])


print("Modifying order %d" % orderid3)

order3.lmtPrice = 15.0
print("Limit price was %f will become %f" % (open_orders[orderid3].order.lmtPrice, order3.lmtPrice ))

app.place_new_IB_order(ibcontract, order3, orderid=orderid3)
time.sleep(5)
open_orders = app.get_open_orders()
print("New limit price %f " % open_orders[orderid3].order.lmtPrice)


Modifying order 3
Limit price was 10.000000 will become 15.000000
New limit price 15.000000 


Its advised that you only change the quantity and limit price (where relevant) of an order; not the order type.


Order cancelling - what if I don't want it any more?


Cancelling an order is pretty simple:

print("Cancel order %d " % orderid2)
app.cancel_order(orderid2)
open_orders = app.get_open_orders()

Cancel order 2 
IB error id 2 errorcode 202 string Order Canceled - reason:

My code checks that the order has been cancelled:
def cancel_order(self, orderid):
    self.cancelOrder(orderid)

    ## Wait until order is cancelled    start_time=datetime.datetime.now()
    MAX_WAIT_TIME_SECONDS = 10
    finished = False
    while not finished:
        if orderid not in self.get_open_orders():
            ## finally cancelled            
            finished = True
        if (datetime.datetime.now() - start_time).seconds > MAX_WAIT_TIME_SECONDS:
            print("Wrapper didn't come back with confirmation that order was cancelled!")
            finished = True
    ## return nothing
... but you can also check yourself:
print("Open orders (should just be %d)" % orderid3)
print(open_orders.keys())

Open orders (should just be 3)
dict_keys([3])


Note this will only cancel orders where the clientid (the third number in this call app = TestApp("127.0.0.1", 4001, 1)) is the same as the current client. To cancel all orders, regardless of the client:

print("Cancelling all orders")
app.cancel_all_orders()

print("Any open orders? - should be False")
print(app.any_open_orders())

Cancelling all orders
IB error id 3 errorcode 202 string Order Canceled - reason:
Open orders? - should be False
False


Are we finished?


 Yes, at least with this post. The last thing I will show you how to do is to get accounting information, so the tedium is nearly over.

This is the fourth in a series of five posts on constructing a simple interface in python to the IB API using swigiby. The first three posts are:

http://qoppac.blogspot.co.uk/2017/03/interactive-brokers-native-python-api.html

http://qoppac.blogspot.co.uk/2017/03/historic-data-from-native-ib-pyhon-api.html

http://qoppac.blogspot.co.uk/2017/03/streaming-market-data-from-native.html


The next, and final, post is:
http://qoppac.blogspot.co.uk/2017/03/getting-position-and-accounting-data.html