Friday 24 October 2014

The worlds simplest execution algo

As you will know I run a fully automated systematic trading system. As its fully automated, due to my extreme laziness, all the trades are put into the market completely automatically.

When I first started running my system I kept the execution process extremely simple:

  • Check that the best bid (if selling) or best offer (if buying) was large enough to absorb my order
  • Submit a market order

Yes, what a loser. Since I am trading futures, and my broker only has fancy orders for equities, this seemed the easiest option.

I then compounded my misery by creating a nice daily report to tell me how much each trade had cost me. Sure enough most of the time I was paying half the inside spread (the difference between the mid price, and the bid or offer).

After a couple of months of this, and getting fed up with seeing this report add up my losses from trading every day, I decided to bite the bullet and do it properly.

Creating cool execution algorithms (algos) isn't my area of deep expertise, so I had to work from first principles. I also don't have much experience of writing very complicated fast event driven code, and I write in a slowish high level language (python). Finally my orders aren't very large, so there is no need to break them up into smaller slices and track each slice. All this points towards a simple algo being sufficient.

Only one more thing to consider; I get charged for modifying orders. It isn't a big cost, and its worth much less than the saving from smarter execution, but it still means that creating an algo that modifies orders an excessive number of times where this is not necessary probably isn't worth the extra work or cost.

Finally I can't modify a limit order and turn it into a market order. I would have to cancel the order and submit a new one.

What does it do?

A good human trader, wanting to execute a smallish buy order and not worrying about game playing or spoofing etc, will probably do something like this:

  • Submit a limit order, on the same side of the spread they want to trade, joining the current level. So if we are buying we'd submit a buy at the current best bid level. In the jargon this is passive behaviour, waiting for the market to come to us.
  • In an ideal world this initial order would get executed.We'll have gained half the spread in negative execution cost (comparing  the mid versus the best bid).
  • If:
    •  the order isn't being executed after several minutes,
    •  or there are signs the market is about to move against them, and rally
    • or the market has already moved up against them
  • ... then the smart trader would cut their losses and modify their order to pay up and cross the spread. This is aggressive behaviour.
  • The new modified aggressive order would be a buy at the current best offer. In theory this would then be executed, costing half the spread (which if the market has already moved against us, would be more than if we'd just submitted a market order initially).
  • If we're too slow and the market continues to move against us, keep modifying the order to stay on the new best offer, until we're all done

Although that's it in a nutshell there are still a few bells and whistles in getting an algo like this to work, and in such a way that it can deal robustly with anything that gets thrown at it. Below is the detail of the algo. Although this is shown as python code, its not executable since I haven't included many of the relevant subroutines. However it should give you enough of an idea to code something similar up yourself.

Pre trade

It's somewhat dangerous dropping an algo trade into the mix if the market isn't liquid enough; this routine checks that.

pretrademultiplier=4.0
def EasyAlgo_pretrade(ctrade, contract, dbtype,  tws):
    """
    Function easy algo runs before getting a new order
    

    ctrade: The proposed trade, an signed integer
    contract: object indicating what we are trading
    dbtype, tws: handles for which database and tws API server we are dealing with here.


    Returns integer indicating size I am happy with
  
    Zero means market can't support order



   
    """

  
    ## Get market data (a list containing inside spread and size)


    bookdata=get_market_data(dbtype, tws, contract, snapshot=False, maxstaleseconds=5, maxwaitseconds=5)
    

    ## None means the API is not running or the market is closed :-(

    if bookdata is None:
        return (0, bookdata)



    ## Check the market is liquid; the spread and the size have to be within certain limits. We use a multiplier because we are less discerning with limit orders - a wide spread could work in our favour!


    market_liquid=check_is_market_liquid(bookdata, contract.code, multiplier=pretrademultiplier)

    if not market_liquid:
        return (0, bookdata)

    ## If the market is liquid, but maybe the order is large compared to the size on the inside spread, we can cut it down to fit the order book.


    cutctrade=cut_down_trade_to_order_book(bookdata,  ctrade, multiplier=pretrademultiplier)
  
    return (cutctrade, bookdata)

 

New order

Not just one of the best bands in the eighties, also the routine you call when a new order request is issued by the upstream code.


MAX_DELAY=0.03

def EasyAlgo_new_order(order, tws, dbtype, use_orderid, bookdata):

    """
    Function easy algo runs on getting a new order 

    Args:
    order - object of my order type containing the required trade
    tws - connection object to tws API for interactive brokers
    dbtype - database we are accessing
    use_orderid- orderid
    bookdata- list containing best bid and offer, and relevant sizes


    """

    
    ## The s, state, variable is used to ensure that log messages and diagnostics get saved right. Don't worry too much about this



    log=logger()
    diag=diagnostic(dbtype, system="algo",  system3=str(order.orderid))
    s=state_from_sdict(order.orderid, diag, log)   
       

    ## From the order book, and the trade, get the price we would pay if aggressive (sideprice) and the price we pay if we get passive (offsideprice)


    (sideprice, offsideprice)=get_price_sides(bookdata, order.submit_trade)

    if np.isnan(offsideprice) or offsideprice==0:
        log.warning("No offside / limit price in market data so can't issue the order")
        return None
   
    if np.isnan(sideprice) or sideprice==0:
        log.warning("No sideprice in market data so dangerous to issue the order")
        return None


    ## The order object contains the price recorded at the time the order was generated; check to see if a large move since then (should be less than a second, so unlikely unless market data corrupt)









    if not np.isnan(order.submit_price):
        delay=abs((offsideprice/order.submit_price) - 1.0)
        if delay>MAX_DELAY:
            log.warning("Large move since submission - not trading a limit order on that")
            return None
           
    ## We're happy with the order book, so set the limit price to the 'offside' - best offer if selling, best bid if buying




    limitprice=offsideprice
   
    ## We change the order so its now a limit order with the right price


    order.modify(lmtPrice = limitprice)
    order.modify(orderType="LMT")


    ## Need to translate from my object space to the API's native objects









    iborder=from_myorder_to_IBorder(order)
    contract=Contract(code=order.code, contractid=order.contractid)
    ibcontract=make_IB_contract(contract)

    
    ## diagnostic stuff
    ## its important to save this so we can track what happened if orders go squiffy (a technical term)


    s.update(dict(limit_price=limitprice, offside_price=offsideprice, side_price=sideprice,
                  message="StartingPassive", Mode="Passive"))
    timenow=datetime.datetime.now()


    
    ##  The algo memory table is used to store state information for the algo. Key thing here is the Mode which is PASSIVE initially!


    am=algo_memory_table(dbtype)
    am.update_value(order.orderid, "Limit", limitprice)
    am.update_value(order.orderid, "ValidSidePrice", sideprice)
    am.update_value(order.orderid, "ValidOffSidePrice", offsideprice)
    am.update_value(order.orderid, "Trade", order.submit_trade)
    am.update_value(order.orderid, "Started", date_as_float(timenow))
    am.update_value(order.orderid, "Mode", "Passive")
    am.update_value(order.orderid, "LastNotice", date_as_float(timenow))

    am.close()


     ## Place the order
    tws.placeOrder(
            use_orderid,                                    # orderId,
            ibcontract,                                   # contract,
            iborder                                       # order
        )


    ## Return the order upstream, so it can be saved in databases etc. Note if this routine terminates early it returns a None; so the upstream routine knows no order was placed.




   
    return order




Action on tick

A tick comes from the API when any part of the inside order book is updated (best bid or offer, or relevant size).

Within the tws server code I have a routine that keeps marketdata (a list with best bid and  offer, and relevant sizes) up to date as ticks arrive, and then calls the relevant routine.

What does this set of functions do?
  • If we are in a passive state (the initial state, remember!)
    • ... and more than five minutes has elapsed, change to aggressive
    • if buying and the current best bid has moved up from where it started (an adverse price movement), change to aggressive
    • if selling, and the current best offer has moved down from where it started (also adverse) 
    • If there is an unfavourable order imbalance (eg five times as many people selling than buying on the inside spread if we're also selling), change to aggressive.
  • If we are in an aggressive state
    • ... and more than ten minutes has elapsed, cancel the order.
    •  if buying and the current best offer has moved up from where it was last (a further adverse price movement), then update our limit to the new best offer (chase the market up).
    •  if selling and the current best bid has moved down from where it was last (a further adverse price movement), then update our limit to the new best offer

passivetimelimit=5*60 ## max five minutes
totaltimelimit=10*60 ## max another five minute aggressive
maximbalance=5.0 ## amount of imbalance we can copy with
 

def EasyAlgo_on_tick(dbtype, orderid, marketdata, tws, contract):
    """
    Function easy algo runs on getting a tick


    Args: 
    dbtype, tws: handles for database and tws API
    orderid: the orderid that is associated with a tick
    marketdata: summary of the state of current inside spread
    contract: what we are actually trading



    """


    ## diagnostic code
    log=logger()
    diag=diagnostic(dbtype, system="algo",  system3=str(int(orderid)))
    s=state_from_sdict(orderid, diag, log)   



    ## Pull out everything we currently know about this order

    am=algo_memory_table(dbtype)
    trade=am.read_value(orderid, "Trade")
    current_limit=am.read_value(orderid, "Limit")
    Started=am.read_value(orderid, "Started")
    Mode=am.read_value(orderid, "Mode")
    lastsideprice=am.read_value(orderid, "ValidSidePrice")
    lastoffsideprice=am.read_value(orderid, "ValidOffSidePrice")
    LastNotice=am.read_value(orderid, "LastNotice")
 

    ## Can't find this order in our state database!


    if Mode is None or Started is None or current_limit is None or trade is None or LastNotice is None:
        log.critical("Can't get algo memory values for orderid %d CANCELLING" % orderid)
        FinishOrder(dbtype, orderid, marketdata, tws, contract)
       
    Started=float_as_date(Started)
    LastNotice=float_as_date(LastNotice)
    timenow=datetime.datetime.now()

   
    ## If a buy, get the best offer (sideprice) and best bid (offsideprice)

    ## If a sell, get the best bid (sideprice) and best offer (offsideprice)
    (sideprice, offsideprice)=get_price_sides(marketdata, trade)

    s.update(dict(limit_price=current_limit, offside_price=offsideprice, side_price=sideprice,
                  Mode=Mode))

    ## Work out how long we've been trading, and the time since we last 'noticed' the time



    time_trading=(timenow - Started).total_seconds()
    time_since_last=(timenow - LastNotice).seconds


    ## A minute has elapsed since we


    if time_since_last>60:
        s.update(dict(message="One minute since last noticed now %s, total time %d seconds - waiting %d %s %s" % (str(timenow), time_trading, orderid, contract.code, contract.contractid)))
        am.update_value(orderid, "LastNotice", date_as_float(timenow))


    ## We've run out of time - cancel any remaining order

    if time_trading>totaltimelimit:
        s.update(dict(message="Out of time cancelling for %d %s %s" % (orderid, contract.code, contract.contractid)))
        FinishOrder(dbtype, orderid, marketdata, tws, contract)
        return -1

    if not np.isnan(sideprice) and sideprice<>lastsideprice:
        am.update_value(orderid, "ValidSidePrice", sideprice)
   
    if not np.isnan(offsideprice) and offsideprice<>lastoffsideprice:
        am.update_value(orderid, "ValidOffSidePrice", offsideprice)

    am.close()


    if Mode=="Passive":



        ## Out of time (5 minutes) for passive behaviour: panic



        if time_trading>passivetimelimit:
            s.update(dict(message="Out of time moving to aggressive for %d %s %s" % (orderid, contract.code, contract.contractid)))

            SwitchToAggresive(dbtype, orderid, marketdata, tws, contract, trade)
            return -1


        if np.isnan(offsideprice):
            s.update(dict(message="NAN offside price in passive mode - waiting %d %s %s" % (orderid, contract.code, contract.contractid)))
            return -5

        if trade>0:
            ## Buying
            if offsideprice>current_limit:
                ## Since we have put in our limit the price has moved up. We are no longer competitive
               
                s.update(dict(message="Adverse price move moving to aggressive for %d %s %s" % (orderid, contract.code, contract.contractid)))

                SwitchToAggresive(dbtype, orderid, marketdata, tws, contract, trade)
               
                return -1
        elif trade<0:
            ## Selling
            if offsideprice<current_limit:
                ## Since we have put in our limit the price has moved down. We are no longer competitive


                s.update(dict(message="Adverse price move moving to aggressive for %d %s %s" % (orderid, contract.code, contract.contractid)))

                SwitchToAggresive(dbtype, orderid, marketdata, tws, contract, trade)
                return -1

        ## Detect Imbalance (bid size/ask size if we are buying; ask size/bid size if we are selling)




        balancestat=order_imbalance(marketdata, trade)
       
        if balancestat>maximbalance:
                s.update(dict(message="Order book imbalance of %f developed compared to %f, switching to aggressive for %d %s %s" %(balancestat , maximbalance, orderid, contract.code, contract.contractid)))

                SwitchToAggresive(dbtype, orderid, marketdata, tws, contract, trade)
                return -1
           
           

    elif Mode=="Aggressive":

        if np.isnan(sideprice):
            s.update(dict(message="NAN side price in aggressive mode - waiting %d %s %s" % (orderid, contract.code, contract.contractid)))
            return -5

       
        if trade>0:
            ## Buying
            if sideprice>current_limit:
                ## Since we have put in our limit the price has moved up further. Keep up!


                s.update(dict(message="Adverse price move in aggressive mode for %d %s %s" % (orderid, contract.code, contract.contractid)))
                SwitchToAggresive(dbtype, orderid, marketdata, tws, contract, trade)
               

                return -1
        elif trade<0:
            ## Selling
            if sideprice<current_limit:
                ## Since we have put in our limit the price has moved down. Keep up!


                s.update(dict(message="Adverse price move in aggressive mode for %d %s %s" % (orderid, contract.code, contract.contractid)))

                SwitchToAggresive(dbtype, orderid, marketdata, tws, contract, trade)
                return -1
           
    elif Mode=="Finished":
        ## do nothing, still have tick for some reason
        pass

   
    else:
        msg="Mode %s not known for order %d" % (Mode, orderid)
        s.update(dict(message=msg))

        log=logger()
        log.critical(msg)
        raise Exception(msg)


    s.update(dict(message="tick no action %d %s %s" % (orderid, contract.code, contract.contractid)))


    diag.close()



    return 0


def SwitchToAggresive(dbtype, orderid, marketdata, tws, contract, trade):
    """
    What to do... if we want to eithier change our current order to an aggressive limit order, or move an order is already aggressive limit price


    """
    ## diagnostics...
    log=logger()

    diag=diagnostic(dbtype, system="algo",  system3=str(int(orderid)))
    s=state_from_sdict(orderid, diag, log)   



    if tws is None:
        log.info("Switch to aggressive didn't get a tws... can't do anything in orderid %d" % orderid)
        return -1

    
    ## Get the last valid side price (relevant price if crossing the spread) as this will be our new limit order


    am=algo_memory_table(dbtype)

    sideprice=am.read_value(orderid, "ValidSidePrice")

    ordertable=order_table(dbtype)
    order=ordertable.read_order_for_orderid(orderid)
    ordertable.close()
   
    if np.isnan(sideprice):
        s.update(dict(message="To Aggressive: Can't change limit for %d as got nan - will try again" % orderid))
        return -1
    

    ## updating the order

    newlimit=sideprice



    order.modify(lmtPrice = newlimit)
    order.modify(orderType="LMT")

   
    iborder=from_myorder_to_IBorder(order)
    ibcontract=make_IB_contract(contract)


    am.update_value(order.orderid, "Limit", newlimit)
    am.update_value(order.orderid, "Mode", "Aggressive")
    am.close()


     # Update the order
    tws.placeOrder(
            orderid,                                    # orderId,
            ibcontract,                                   # contract,
            iborder                                       # order
        )

    s.update(dict(limit_price=newlimit, side_price=sideprice,
                  message="NowAggressive", Mode="Aggresive"))


    return 0



def FinishOrder(dbtype, orderid, marketdata, tws, contract):
    """
    Algo hasn't worked, lets cancel this order
    """
    diag=diagnostic(dbtype, system="algo",  system3=str(int(orderid)))

    s=state_from_sdict(orderid, diag, log)   
    log=logger()


    if tws is None:
        log.info("Finish order didn't get a tws... can't do anything in orderid %d" % orderid)
        return -1


    log=logger()
    ordertable=order_table(dbtype)

    order=ordertable.read_order_for_orderid(orderid)
   
    log.info("Trying to cancel %d because easy algo failure" % orderid)
    tws.cancelOrder(int(order.brokerorderid))
   
    order.modify(cancelled=True)
    ordertable.update_order(order)

    do_order_completed(dbtype, order)           
   
    EasyAlgo_on_complete(dbtype, order, tws)


    s.update(dict(message="NowCancelling", Mode="Finished"))

    am=algo_memory_table(dbtype)
    am.update_value(order.orderid, "Mode", "Finished")
    am.close()
   
    return -1

Partial or complete fill

Blimey this has actually worked, we've actually got a fill...
                   

def EasyAlgo_on_partial(dbtype, order, tws):
    diag=diagnostic(dbtype, system="algo",  system3=str(int(order.orderid)))

    diag.w(order.filledtrade, system2="filled")
    diag.w(order.filledprice, system2="fillprice")
   
    return 0





def EasyAlgo_on_complete(dbtype, order_filled, tws):
    """
    Function Easy algo runs on completion of trade
    """
   

    diag=diagnostic(dbtype, system="algo",  system3=str(int(order_filled.orderid)))
   
    diag.w("Finished", system2="Mode")
    diag.w(order_filled.filledtrade, system2="filled")
    diag.w(order_filled.filledprice, system2="fillprice")

    am=algo_memory_table(dbtype)
    am.update_value(order_filled.orderid, "Mode", "Finished")
    am.close()

    return 0


And we're done

That's it. Its not perfect and it would be very easy to write high frequency code that would game this kind of strategy. However the proof is in the proverbial traditional English dessert, and my execution costs have reduced by approximately 80% from when I was doing market orders, i.e. I am paying an average of 1/10 of the spread. So it's definitely an improvement, and well worth the day or so it took me to code it up and test it.



 

22 comments:

  1. Hi Rob

    I got your book yesterday, I started reading it, it looks very promising, very rarely you see books that mention portfolio optimization for trade systems ( I think only Perry Kaufman has a couple of pages on that)..anyway
    I would like to setup an automated platform for trading live and back-testing with python...
    for backtesting have you developed your own code? I guess so.I browsed a bit zipline..but I think in the long term it will limit the kind of systems that can be developed...
    how would you recommend that a back-test program should be done in python?

    Many thanks

    ReplyDelete
    Replies
    1. Hi
      I wrote my own stuff

      zipline is interesting as is http://pmorissette.github.io/bt/, and http://gbeced.github.io/pyalgotrade/. But I haven't used them. Here is a list
      http://quant.stackexchange.com/questions/8896/except-zipline-are-there-any-other-pythonic-algorithmic-trading-library-i-can-c

      Of course if you dig around my blog you will find advice on the automated trading side.

      Unfortunately it would take another book to describe how to develop a back test system yourself.

      Mike's book might help you (https://www.quantstart.com/successful-algorithmic-trading-ebook) (tell him I sent you). I haven't read it but it seems to be the only book that tries to attack this.

      At some point I'd like to publish a system that people can use, but that project is some way off.

      Delete
    2. Updated reply: See http://qoppac.blogspot.co.uk/p/pysystemtrade.html

      Delete
  2. You mention that execution costs have come down by 80% versus your earlier market-order strategy. Do you measure these savings relative to the initial mid, or do you update the mid after adverse price movements?
    Eg, suppose the market is 99 - 101 and you want to buy. Your new algo would rest at 99 with original mid at 100. The market now moves to 101 - 103. You update to aggressive and get filled at 103. Do you measure your performance relative to a mid of 100 or 102?
    80% reduction seems extremely good if measured versus the original mid in futures markets that typically trade at min tick size. Any intuition? Thanks!

    ReplyDelete
    Replies
    1. It's versus the initial mid of 100; on the assumption that I'd get filled at 101 if I'd just submitted a market order at that point. So I'd pay 103 versus 100; 3 ticks slippage, a market order would have cost me 1 tick slippage.

      Delete
  3. Hi Robert, I loved your book. Am in its detailed second read now. But n3ed your help: Sharpe or Sortino! Am confused...as Sortino seems obvious given the leeway it has for upside volatility. Yet, everyone seems to prefer, at least publicly, Sharpe; even you. Its easier to calculate can be one--albeit--minor advantage I can see. Kindly show me the light on this one. Thanks!

    ReplyDelete
    Replies
    1. Yes it's easier to calculate Sharpe. All performance measures have disadvantages; ideally you should use several.

      Delete
  4. This comment has been removed by the author.

    ReplyDelete
  5. Thanks Rob! 'Use the average' is my new ideology!

    ReplyDelete
  6. I've found your blog unfortunately today, cause you really do a great job sharing intelligent and fundamental principals in this challenge area. Keep your work up

    ReplyDelete
  7. Hi Rob,
    It's been a while since you wrote this blog post, but it is as relevant as it ever was. Thank you!

    There is something I don't understand, and hope you could explain:

    When you only used market orders, you always paid half the inside spread. With the new algo, you start passive, hoping that the market will go in your direction. if it did, you gained half the spread. I figure it is safe to assume that the market is as likely to go in your direction as it is in the opposite direction. Now if it goes against you, then you change to aggressive, and will most likely pay 1.5 of the original spread. Doesn't this leave you with an average execution cost of half the spread, as you paid with market orders?

    ReplyDelete
    Replies
    1. That would be true if (a) there is a 50% chance of being filled before you change to aggressive and (b) you always get filled immediately with just a one tick move. In practice the average loss is higher than 1.5 times the spread; since sometimes the market keeps moving away and the algo chases it. And fortunately you tend to get passively filled more often than not - I haven't analysed it but perhaps 2/3 of the time.

      Delete
  8. Hi Rob,

    Hope you are well, I am still enjoying very much (re)reading your many posts. I love your execution algo which I think I finally got my head around. A couple of things I don't quite get though:
    1. Once you change from passive to aggressive and you are forced to chase the market, wouldn't a marketorder achieve more or less the same outcome?
    2. Why is the algo averse to trading when there is no size at your side. If there are bids, no asks and you come in to sell, wouldn't you be ok to place a limit order at the ask?

    ReplyDelete
    Replies
    1. Sorry for the delay in responding.

      1. Yes, but it's not possible to update an order type only the price, so I'd need to cancel the existing order and submit a new market order. This would cost something since I get charged for cancelling orders. But to be fair this might work out cheaper than chasing the market. Definitely something to consider for algo version 2.

      2. Yes I guess if the market is 109.00 bid - nothing; then it would make sense to offer 109.01 (or even a cheeky 109.02... or higher). I do store the typical bid-ask spread in my database so in theory this is something I could do.

      Delete
    2. Thanks v much for getting back to me Rob.

      Delete
  9. Rob, As a follow up to to my previous questions, I think I now see one of the potential risks of placing an bid when, say there is only an offer in the market. Someone could dangle a minimum sized offer as bait, cross the spread to take your trade and then massively move away from that original offer on the assumption that you or your algo will be forced to chase the market up (which is what your algo might well do if it gets an initial fill). I have almost zero knowledge of execution best practices so apologies for the noob question.

    ReplyDelete
    Replies
    1. It's always possible for people to game you if they know how your algo works. The more complex and weird your algo is, the more likely this would happen. So I'm not interested in getting into a war with HFT traders as I don't have the weaponry; just doing sensible things and trying not to get picked off.

      Delete
  10. Hi Rob - Thanks for sharing the passive-aggressive idea.

    I am wondering how this 80% performance compares to what we'd get using other loosely similar fancy IB orders, e.g. a trail with amount being half the spread?
    https://www.interactivebrokers.com.hk/en/index.php?f=605

    ReplyDelete
    Replies
    1. No idea as I have never used that order type, not even sure if you can use it in the API. If you can, then I can experiment to check

      Delete
  11. Seems in theory you can: https://interactivebrokers.github.io/tws-api/basic_orders.html
    But I haven't tried to be honest, that's a bit tricky to backtest.

    ReplyDelete
  12. Hi Rob, Is there any books you suggest for execution algorithm from engineering perspective? what kind of tips and trick you do as you mix the periodically pull and event driven programming model.

    Any suggestion?

    ReplyDelete
    Replies
    1. No I don't know of any, maybe ask on elitetrader.com or nuclearphynance

      Delete

Comments are moderated. So there will be a delay before they are published. Don't bother with spam, it wastes your time and mine.