Historic data from native IB python API

This is the second in a series of posts on how to use the native python API for interactive brokers. This post is an update of the post I wrote here, which used the 3rd party API swigibpy.

Okay so you have managed to run the time telling code in my last post.

Now we will do something a bit more interesting, get some market prices. Arguably it is still not that interesting, and this stuff will never be as interesting as a decent book or a good bottle of red wine, but we've all got to get the money to buy those books and bottles and this is better than many other options for achieving that.

Note: This post has been updated to use a more robust method for dealing with concurrency.

Let us peruse the IB ABI instructions...

Whenever you are trying something new out with the API you need to identify the EClient (command to ask the server for stuff) and EWrapper (command run by the wrapper client when stuff is presented unto it) functions. So Reading The Frigging Manual is worth doing.

And indeed there is not just one but four places in the weird and mysterious world of EClient where it looks like prices could be found (ignoring whacky methods for things like options):

• reqMktData
• reqHistoricalData
• reqRealTimeBars
• reqMarketDepth

Market Data: Returns a stream of price ticks (updated quotes and trades). Because it is a stream you have to ask for it, and then ask for it to stop lest you be buried under a huge pile of ticks. It is level 1, eg 'top of the order book' data.

Historical Data: Returns a one off chunk of data, both prices and volumes, with a given look back history and frequency.

Real Time Bars: Returns a stream of averages of the trades (or bid, ask, midpoint) over an interval, currently only 5 seconds. So its kind of like pre-clumped tick data arriving at regular intervals. Its a stream, so you have to ask for it... you get the idea.

Market Depth: Returns a stream of updates to the order book, both prices and size. This requires a requires a level 2 subscription. Its a stream...


Because real time bars, market data and market depth deliver very similar information I won't discuss them all (and since I am a tightwad I don't have any Level 2 data subscriptions!). In this post we'll go through the 'one shot' historical data; and in the next one we'll look at one kind of streaming data (market data).

Historical Data AKA Those Who Cannot Remember the Past Are Doomed...

First get the code from this gist somewhere on a server far far away...

Also the relevant manual pages are here.

 Getting some price data 

 Just run the file (assuming you still have an IB connection available as per the previous post). You should get something like this:

Getting full contract details from the server... 
Getting historical data from the server... could take 10 seconds to complete 
historic_data
[('20170312', 98.145, 98.16, 98.135, 98.14, 53208), ('20170313', 98.14, 98.16, 98.135, 98.145, 53600), ('20170314', 98.135, 98.16, 98.135, 98.14, 21673)]

"Obviously" these are tuples containing daily prices (date, open, high, low, close, volume) for the September 2018 Eurodollar futures contract. If you get an error you might need to change the expiry date for the contract. I'll show you how to do this later in the post.

(Note before mixing these with any kind of other prices or doing proper backtesting I would personally append a 'fixed' end of date timestamp to them eg 23:59:59 to ensure there is no slight look forward bias. This is discussed more here.)

  The IB contract object or How IB Knows What We Want

Some code:

ibcontract = IBcontract()
ibcontract.secType = "FUT"

ibcontract.lastTradeDateOrContractMonth="201809"

ibcontract.symbol="GE"

ibcontract.exchange="GLOBEX"

(Note if you got an error above here is your chance to change the expiry to the current one)

Here is a real 'gotcha' - the IB contract object (when I say 'gotcha' it can be stuff I've literally spent days banging my metaphorical head against and in some cases my actual head). This is a complete description which identifies a particular thing you can trade. Most of the data you need is available from the https://www.interactivebrokers.com/ product listing pages. Some problems I have (there may be more associated with other asset classes) mainly in not giving enough information to uniquely identify a contract:

• Getting the exchange name right - it needs to be exactly the same as on the product page. 
• Having to specify a currency. This seems to be a problem for CAC40 and AEX futures (specify ibcontract.currency="EUR") and silver ("USD").
• Having to specify a multiplier effectively contract movement in cash terms per price point (For CAC ibcontract.multiplier="10", AEX is 200 and SILVER is 5000)
• Be careful, some contracts actually have an expiry date in a different
• You have to specify the exact expiry YYYYMMDD of a VIX contract rather than just YYYYMM because there are now weekly expiries.

(By the way I don't actually use this contract object in the rest of my code. Instead I have another class which contains more useful information, and then a convert_to_ibcontract function. This function handles the horrible special cases above and also translates between meaningless names like  and better names like JohnBoy - my own pet name for the Eurodollar future *).

 (* This is a joke)

The next thing we need to do is get the contract details. This may sound a bit weird, we already have the contract details since we've just defined it? In practice the definition of a contract is very light and the details we get back are much richer.

There is one specific case where you must do this. To see why suppose it is a few months time and you are trading hundreds of futures contracts. A fill comes in for a particular contract instrument code and month/year. However as we shall see the identification for the contract isn't the 'yyyymm' code we use to define the contract, but the full expiry 'yyyymmdd'. You can't just take the first 6 characters of that either since some futures actually expire the month before they should. So to be on the safe side I usually get the expiry from IB before trading to make sure I am applying fills against the correct thingy.

Actually I don't do this immediately before trading but in a separate process that populates a database of contract information...


So we now need to "resolve" this contract into a fully featured contract which properly specifies the entire expiry date:

resolved_ibcontract=app.resolve_ib_contract(ibcontract)

Getting full contract details from the server... 

By the way if you haven't specified the initial contract object precisely enough, so there are multiple possible contracts, then you'll get a warning. You may also get a warning if the IB server doesn't deliver a "I am done with sending you contracts now sir" message by activating the wrapper method contractDetailsEnd()

All this is Fascinating Rob But How do we Actually get the Prices?!

historic_data = app.get_IB_historical_data(resolved_ibcontract)

This calls the following code in the TestClient object:

def get_IB_historical_data(self, ibcontract, durationStr="1 Y", barSizeSetting="1 day",
                           tickerid=DEFAULT_HISTORIC_DATA_ID):

    """    Returns historical prices for a contract, up to today

    ibcontract is a Contract
    :returns list of prices in 4 tuples: Open high low close volume    """

    self.wrapper.init_error()
    print("Getting historical data from the server... ")
    
    ## Make a place to store the data we're going to return    historic_data_queue = finishableQueue(self.init_historicprices(tickerid))

    today = datetime.datetime.now()

    # Request some historical data. Native method in EClient    

    self.reqHistoricalData(
       tickerid,  # tickerId,       ibcontract,  # contract,       datetime.datetime.today().strftime("%Y%m%d %H:%M:%S %Z"),  # endDateTime,       durationStr,  # durationStr,       barSizeSetting,  # barSizeSetting,       "TRADES",  # whatToShow,       1,  # useRTH,       1,  # formatDate       False,  # KeepUpToDate <<==== added for api 9.73.2       [] ## chartoptions not used     )

    historic_data_queue = finishableQueue(self.init_historicprices(tickerid))

    ## Wait until we get a completed data, an error, or get bored waiting    MAX_WAIT_SECONDS = 10    historic_data = historic_data_queue.get(timeout = MAX_WAIT_SECONDS)

    while self.wrapper.is_error():
       print(self.get_error())

    if historic_data_queue.timed_out():
       print("Exceeded maximum wait for wrapper to confirm finished - seems to be normal behaviour")

   self.cancelHistoricalData(tickerid)


   return historic_data

The general structure of this function should be obvious to anyone who read the last post. We:

• call some init functions in the wrapper instance (self.wrapper) so there is somewhere for error messages and our price data to go
• Ask the IB API server for something with the inherited EClient method self.reqHistoricalData()
• Wait patiently until the data is received, or until we get bored 
• Complain about any errors, or return the results as appropriate 

The self.reqHistoricalData() asks the IB server for historical price data. We can set length of time (a year in this case, although because we are using a demo account there isn't that much data available) durationStr, the end time, the bar size (in this case days; not actually the size of the kind of bar you get drinks from) and some other things you can look at the documentation yourself to find out about. Just to note not all the combinations of time period and bar length are permitted - you can't get second by second prices for a year (see this link.) even if your hard drive could store them.

About the only thing that might not be obvious is tickerid. You might be the kind of wild crazy person who requests loads of historical data for different contracts. In which case you would get data coming back randomly and you wouldn't know which contract it was for. So you have to provide some kind of label for the request if you request loads of things at once. Note you can't actually do that because if you request more than 1 set of data about every 10 seconds you get a 'pacing violation'  (basically, don't take the mickey).

This is only one side of the story, as....

Oh Yeah I remember now, we have to have an EWrapper function on the server side... 

From the TestWrapper class, as usual we have a storage method where we put stuff in a Queue:

## Historic data code



def init_historicprices(self, tickerid):
    historic_data_queue = self._my_historic_data_dict[tickerid] = queue.Queue()

    return historic_data_queue

And some methods in EWrapper which have been overriden:

def historicalData(self, tickerid , bar):

    ## Overriden method    ## Note I'm choosing to ignore barCount, WAP and hasGaps but you could use them if you like    bardata=(bar.date, bar.open, bar.high, bar.low, bar.close, bar.volume)

    historic_data_dict=self._my_historic_data_dict

    ## Add on to the current data    if tickerid not in historic_data_dict.keys():
        self.init_historicprices(tickerid)

    historic_data_dict[tickerid].put(bardata)

def historicalDataEnd(self, tickerid, start:str, end:str):
    ## overriden method
    if tickerid not in self._my_historic_data_dict.keys():
        self.init_historicprices(tickerid)

    self._my_historic_data_dict[tickerid].put(FINISHED)

There is no magic in creating these things; to write this stuff you look in the API reference to see what the wrapper function that requires overriding (in this case historicalData) returns. In this case the wrapper function beastie is called repeatedly with the same arguments for each row of data. In theory historicalDataEnd would also be called, however as with getting contract details this might never happen (I've had mixed success with it). 

<snip>

Back in the TestClient object:

    historic_data_queue = finishableQueue(self.init_historicprices(tickerid))




    ## Wait until we get a completed data, an error, or get bored waiting    MAX_WAIT_SECONDS = 10    historic_data = historic_data_queue.get(timeout = MAX_WAIT_SECONDS)


It's now worth understanding what a finishableQueue is; 


## marker for when queue is finishedFINISHED = object()
STARTED = object()
TIME_OUT = object()

class finishableQueue(object):

    def __init__(self, queue_to_finish):

        self._queue = queue_to_finish
        self.status = STARTED

    def get(self, timeout):
        """
        :param timeout: how long to wait before giving up waiting for a FINISHED flag        :return: list of Queue elements        """        contents_of_queue=[]
        finished=False
        while not finished:
            try:
                current_element = self._queue.get(timeout=timeout)
                if current_element is FINISHED:
                    finished = True                    self.status = FINISHED
                else:
                    contents_of_queue.append(current_element)
                    ## keep going and try and get more data
            except queue.Empty:
                ## If we hit a time out it's most probable we're not getting a finished element                ## give up and return what we have                finished = True                self.status = TIME_OUT


        return contents_of_queue

    def timed_out(self):
        return self.status is TIME_OUT





This is just a simple class for keeping track of Queues of multiple items in the wrapper thread where the results have some 'finish' point, which may not come.


And once again, back in the TestClient object:

    historic_data = historic_data_queue.get(timeout = MAX_WAIT_SECONDS)






    while self.wrapper.is_error():
       print(self.get_error())
    if historic_data_queue.timed_out():
       print("Exceeded maximum wait for wrapper to confirm finished - seems to be normal behaviour")
   self.cancelHistoricalData(tickerid)

   return historic_data

So we get this output:

Getting historical data from the server... could take 10 seconds to complete 
historic_data
[('20170312', 98.145, 98.16, 98.135, 98.14, 53208), ('20170313', 98.14, 98.16, 98.135, 98.145, 53600), ('20170314', 98.135, 98.16, 98.135, 98.14, 21673)]

(By the way exactly the same finishableQueue idea is used when we resolve the contract details)

That then is everything you need to know about historical data. I will cover market data, i.e. the wonderful world of streaming data, in the next post.

This is the second in a series of posts. The previous post was: qoppac.blogspot.co.uk/2017/03/interactive-brokers-native-python-api.html

The next post is qoppac.blogspot.co.uk/2017/03/streaming-market-data-from-native.html