Friday, 24 October 2014

The worlds simplest execution algo

As you will know I run a fully automated systematic trading system. As its fully automated, due to my extreme laziness, all the trades are put into the market completely automatically.

When I first started running my system I kept the execution process extremely simple:

  • Check that the best bid (if selling) or best offer (if buying) was large enough to absorb my order
  • Submit a market order

Yes, what a loser. Since I am trading futures, and my broker only has fancy orders for equities, this seemed the easiest option.

I then compounded my misery by creating a nice daily report to tell me how much each trade had cost me. Sure enough most of the time I was paying half the inside spread (the difference between the mid price, and the bid or offer).

After a couple of months of this, and getting fed up with seeing this report add up my losses from trading every day, I decided to bite the bullet and do it properly.

Creating cool execution algorithms (algos) isn't my area of deep expertise, so I had to work from first principles. I also don't have much experience of writing very complicated fast event driven code, and I write in a slowish high level language (python). Finally my orders aren't very large, so there is no need to break them up into smaller slices and track each slice. All this points towards a simple algo being sufficient.

Only one more thing to consider; I get charged for modifying orders. It isn't a big cost, and its worth much less than the saving from smarter execution, but it still means that creating an algo that modifies orders an excessive number of times where this is not necessary probably isn't worth the extra work or cost.

Finally I can't modify a limit order and turn it into a market order. I would have to cancel the order and submit a new one.

What does it do?

A good human trader, wanting to execute a smallish buy order and not worrying about game playing or spoofing etc, will probably do something like this:

  • Submit a limit order, on the same side of the spread they want to trade, joining the current level. So if we are buying we'd submit a buy at the current best bid level. In the jargon this is passive behaviour, waiting for the market to come to us.
  • In an ideal world this initial order would get executed.We'll have gained half the spread in negative execution cost (comparing  the mid versus the best bid).
  • If:
    •  the order isn't being executed after several minutes,
    •  or there are signs the market is about to move against them, and rally
    • or the market has already moved up against them
  • ... then the smart trader would cut their losses and modify their order to pay up and cross the spread. This is aggressive behaviour.
  • The new modified aggressive order would be a buy at the current best offer. In theory this would then be executed, costing half the spread (which if the market has already moved against us, would be more than if we'd just submitted a market order initially).
  • If we're too slow and the market continues to move against us, keep modifying the order to stay on the new best offer, until we're all done

Although that's it in a nutshell there are still a few bells and whistles in getting an algo like this to work, and in such a way that it can deal robustly with anything that gets thrown at it. Below is the detail of the algo. Although this is shown as python code, its not executable since I haven't included many of the relevant subroutines. However it should give you enough of an idea to code something similar up yourself.

Pre trade

It's somewhat dangerous dropping an algo trade into the mix if the market isn't liquid enough; this routine checks that.

pretrademultiplier=4.0
def EasyAlgo_pretrade(ctrade, contract, dbtype,  tws):
    """
    Function easy algo runs before getting a new order
    

    ctrade: The proposed trade, an signed integer
    contract: object indicating what we are trading
    dbtype, tws: handles for which database and tws API server we are dealing with here.


    Returns integer indicating size I am happy with
  
    Zero means market can't support order



   
    """

  
    ## Get market data (a list containing inside spread and size)


    bookdata=get_market_data(dbtype, tws, contract, snapshot=False, maxstaleseconds=5, maxwaitseconds=5)
    

    ## None means the API is not running or the market is closed :-(

    if bookdata is None:
        return (0, bookdata)



    ## Check the market is liquid; the spread and the size have to be within certain limits. We use a multiplier because we are less discerning with limit orders - a wide spread could work in our favour!


    market_liquid=check_is_market_liquid(bookdata, contract.code, multiplier=pretrademultiplier)

    if not market_liquid:
        return (0, bookdata)

    ## If the market is liquid, but maybe the order is large compared to the size on the inside spread, we can cut it down to fit the order book.


    cutctrade=cut_down_trade_to_order_book(bookdata,  ctrade, multiplier=pretrademultiplier)
  
    return (cutctrade, bookdata)

 

New order

Not just one of the best bands in the eighties, also the routine you call when a new order request is issued by the upstream code.


MAX_DELAY=0.03

def EasyAlgo_new_order(order, tws, dbtype, use_orderid, bookdata):

    """
    Function easy algo runs on getting a new order 

    Args:
    order - object of my order type containing the required trade
    tws - connection object to tws API for interactive brokers
    dbtype - database we are accessing
    use_orderid- orderid
    bookdata- list containing best bid and offer, and relevant sizes


    """

    
    ## The s, state, variable is used to ensure that log messages and diagnostics get saved right. Don't worry too much about this



    log=logger()
    diag=diagnostic(dbtype, system="algo",  system3=str(order.orderid))
    s=state_from_sdict(order.orderid, diag, log)   
       

    ## From the order book, and the trade, get the price we would pay if aggressive (sideprice) and the price we pay if we get passive (offsideprice)


    (sideprice, offsideprice)=get_price_sides(bookdata, order.submit_trade)

    if np.isnan(offsideprice) or offsideprice==0:
        log.warning("No offside / limit price in market data so can't issue the order")
        return None
   
    if np.isnan(sideprice) or sideprice==0:
        log.warning("No sideprice in market data so dangerous to issue the order")
        return None


    ## The order object contains the price recorded at the time the order was generated; check to see if a large move since then (should be less than a second, so unlikely unless market data corrupt)









    if not np.isnan(order.submit_price):
        delay=abs((offsideprice/order.submit_price) - 1.0)
        if delay>MAX_DELAY:
            log.warning("Large move since submission - not trading a limit order on that")
            return None
           
    ## We're happy with the order book, so set the limit price to the 'offside' - best offer if selling, best bid if buying




    limitprice=offsideprice
   
    ## We change the order so its now a limit order with the right price


    order.modify(lmtPrice = limitprice)
    order.modify(orderType="LMT")


    ## Need to translate from my object space to the API's native objects









    iborder=from_myorder_to_IBorder(order)
    contract=Contract(code=order.code, contractid=order.contractid)
    ibcontract=make_IB_contract(contract)

    
    ## diagnostic stuff
    ## its important to save this so we can track what happened if orders go squiffy (a technical term)


    s.update(dict(limit_price=limitprice, offside_price=offsideprice, side_price=sideprice,
                  message="StartingPassive", Mode="Passive"))
    timenow=datetime.datetime.now()


    
    ##  The algo memory table is used to store state information for the algo. Key thing here is the Mode which is PASSIVE initially!


    am=algo_memory_table(dbtype)
    am.update_value(order.orderid, "Limit", limitprice)
    am.update_value(order.orderid, "ValidSidePrice", sideprice)
    am.update_value(order.orderid, "ValidOffSidePrice", offsideprice)
    am.update_value(order.orderid, "Trade", order.submit_trade)
    am.update_value(order.orderid, "Started", date_as_float(timenow))
    am.update_value(order.orderid, "Mode", "Passive")
    am.update_value(order.orderid, "LastNotice", date_as_float(timenow))

    am.close()


     ## Place the order
    tws.placeOrder(
            use_orderid,                                    # orderId,
            ibcontract,                                   # contract,
            iborder                                       # order
        )


    ## Return the order upstream, so it can be saved in databases etc. Note if this routine terminates early it returns a None; so the upstream routine knows no order was placed.




   
    return order




Action on tick

A tick comes from the API when any part of the inside order book is updated (best bid or offer, or relevant size).

Within the tws server code I have a routine that keeps marketdata (a list with best bid and  offer, and relevant sizes) up to date as ticks arrive, and then calls the relevant routine.

What does this set of functions do?
  • If we are in a passive state (the initial state, remember!)
    • ... and more than five minutes has elapsed, change to aggressive
    • if buying and the current best bid has moved up from where it started (an adverse price movement), change to aggressive
    • if selling, and the current best offer has moved down from where it started (also adverse) 
    • If there is an unfavourable order imbalance (eg five times as many people selling than buying on the inside spread if we're also selling), change to aggressive.
  • If we are in an aggressive state
    • ... and more than ten minutes has elapsed, cancel the order.
    •  if buying and the current best offer has moved up from where it was last (a further adverse price movement), then update our limit to the new best offer (chase the market up).
    •  if selling and the current best bid has moved down from where it was last (a further adverse price movement), then update our limit to the new best offer

passivetimelimit=5*60 ## max five minutes
totaltimelimit=10*60 ## max another five minute aggressive
maximbalance=5.0 ## amount of imbalance we can copy with
 

def EasyAlgo_on_tick(dbtype, orderid, marketdata, tws, contract):
    """
    Function easy algo runs on getting a tick


    Args: 
    dbtype, tws: handles for database and tws API
    orderid: the orderid that is associated with a tick
    marketdata: summary of the state of current inside spread
    contract: what we are actually trading



    """


    ## diagnostic code
    log=logger()
    diag=diagnostic(dbtype, system="algo",  system3=str(int(orderid)))
    s=state_from_sdict(orderid, diag, log)   



    ## Pull out everything we currently know about this order

    am=algo_memory_table(dbtype)
    trade=am.read_value(orderid, "Trade")
    current_limit=am.read_value(orderid, "Limit")
    Started=am.read_value(orderid, "Started")
    Mode=am.read_value(orderid, "Mode")
    lastsideprice=am.read_value(orderid, "ValidSidePrice")
    lastoffsideprice=am.read_value(orderid, "ValidOffSidePrice")
    LastNotice=am.read_value(orderid, "LastNotice")
 

    ## Can't find this order in our state database!


    if Mode is None or Started is None or current_limit is None or trade is None or LastNotice is None:
        log.critical("Can't get algo memory values for orderid %d CANCELLING" % orderid)
        FinishOrder(dbtype, orderid, marketdata, tws, contract)
       
    Started=float_as_date(Started)
    LastNotice=float_as_date(LastNotice)
    timenow=datetime.datetime.now()

   
    ## If a buy, get the best offer (sideprice) and best bid (offsideprice)

    ## If a sell, get the best bid (sideprice) and best offer (offsideprice)
    (sideprice, offsideprice)=get_price_sides(marketdata, trade)

    s.update(dict(limit_price=current_limit, offside_price=offsideprice, side_price=sideprice,
                  Mode=Mode))

    ## Work out how long we've been trading, and the time since we last 'noticed' the time



    time_trading=(timenow - Started).total_seconds()
    time_since_last=(timenow - LastNotice).seconds


    ## A minute has elapsed since we


    if time_since_last>60:
        s.update(dict(message="One minute since last noticed now %s, total time %d seconds - waiting %d %s %s" % (str(timenow), time_trading, orderid, contract.code, contract.contractid)))
        am.update_value(orderid, "LastNotice", date_as_float(timenow))


    ## We've run out of time - cancel any remaining order

    if time_trading>totaltimelimit:
        s.update(dict(message="Out of time cancelling for %d %s %s" % (orderid, contract.code, contract.contractid)))
        FinishOrder(dbtype, orderid, marketdata, tws, contract)
        return -1

    if not np.isnan(sideprice) and sideprice<>lastsideprice:
        am.update_value(orderid, "ValidSidePrice", sideprice)
   
    if not np.isnan(offsideprice) and offsideprice<>lastoffsideprice:
        am.update_value(orderid, "ValidOffSidePrice", offsideprice)

    am.close()


    if Mode=="Passive":



        ## Out of time (5 minutes) for passive behaviour: panic



        if time_trading>passivetimelimit:
            s.update(dict(message="Out of time moving to aggressive for %d %s %s" % (orderid, contract.code, contract.contractid)))

            SwitchToAggresive(dbtype, orderid, marketdata, tws, contract, trade)
            return -1


        if np.isnan(offsideprice):
            s.update(dict(message="NAN offside price in passive mode - waiting %d %s %s" % (orderid, contract.code, contract.contractid)))
            return -5

        if trade>0:
            ## Buying
            if offsideprice>current_limit:
                ## Since we have put in our limit the price has moved up. We are no longer competitive
               
                s.update(dict(message="Adverse price move moving to aggressive for %d %s %s" % (orderid, contract.code, contract.contractid)))

                SwitchToAggresive(dbtype, orderid, marketdata, tws, contract, trade)
               
                return -1
        elif trade<0:
            ## Selling
            if offsideprice<current_limit:
                ## Since we have put in our limit the price has moved down. We are no longer competitive


                s.update(dict(message="Adverse price move moving to aggressive for %d %s %s" % (orderid, contract.code, contract.contractid)))

                SwitchToAggresive(dbtype, orderid, marketdata, tws, contract, trade)
                return -1

        ## Detect Imbalance (bid size/ask size if we are buying; ask size/bid size if we are selling)




        balancestat=order_imbalance(marketdata, trade)
       
        if balancestat>maximbalance:
                s.update(dict(message="Order book imbalance of %f developed compared to %f, switching to aggressive for %d %s %s" %(balancestat , maximbalance, orderid, contract.code, contract.contractid)))

                SwitchToAggresive(dbtype, orderid, marketdata, tws, contract, trade)
                return -1
           
           

    elif Mode=="Aggressive":

        if np.isnan(sideprice):
            s.update(dict(message="NAN side price in aggressive mode - waiting %d %s %s" % (orderid, contract.code, contract.contractid)))
            return -5

       
        if trade>0:
            ## Buying
            if sideprice>current_limit:
                ## Since we have put in our limit the price has moved up further. Keep up!


                s.update(dict(message="Adverse price move in aggressive mode for %d %s %s" % (orderid, contract.code, contract.contractid)))
                SwitchToAggresive(dbtype, orderid, marketdata, tws, contract, trade)
               

                return -1
        elif trade<0:
            ## Selling
            if sideprice<current_limit:
                ## Since we have put in our limit the price has moved down. Keep up!


                s.update(dict(message="Adverse price move in aggressive mode for %d %s %s" % (orderid, contract.code, contract.contractid)))

                SwitchToAggresive(dbtype, orderid, marketdata, tws, contract, trade)
                return -1
           
    elif Mode=="Finished":
        ## do nothing, still have tick for some reason
        pass

   
    else:
        msg="Mode %s not known for order %d" % (Mode, orderid)
        s.update(dict(message=msg))

        log=logger()
        log.critical(msg)
        raise Exception(msg)


    s.update(dict(message="tick no action %d %s %s" % (orderid, contract.code, contract.contractid)))


    diag.close()



    return 0


def SwitchToAggresive(dbtype, orderid, marketdata, tws, contract, trade):
    """
    What to do... if we want to eithier change our current order to an aggressive limit order, or move an order is already aggressive limit price


    """
    ## diagnostics...
    log=logger()

    diag=diagnostic(dbtype, system="algo",  system3=str(int(orderid)))
    s=state_from_sdict(orderid, diag, log)   



    if tws is None:
        log.info("Switch to aggressive didn't get a tws... can't do anything in orderid %d" % orderid)
        return -1

    
    ## Get the last valid side price (relevant price if crossing the spread) as this will be our new limit order


    am=algo_memory_table(dbtype)

    sideprice=am.read_value(orderid, "ValidSidePrice")

    ordertable=order_table(dbtype)
    order=ordertable.read_order_for_orderid(orderid)
    ordertable.close()
   
    if np.isnan(sideprice):
        s.update(dict(message="To Aggressive: Can't change limit for %d as got nan - will try again" % orderid))
        return -1
    

    ## updating the order

    newlimit=sideprice



    order.modify(lmtPrice = newlimit)
    order.modify(orderType="LMT")

   
    iborder=from_myorder_to_IBorder(order)
    ibcontract=make_IB_contract(contract)


    am.update_value(order.orderid, "Limit", newlimit)
    am.update_value(order.orderid, "Mode", "Aggressive")
    am.close()


     # Update the order
    tws.placeOrder(
            orderid,                                    # orderId,
            ibcontract,                                   # contract,
            iborder                                       # order
        )

    s.update(dict(limit_price=newlimit, side_price=sideprice,
                  message="NowAggressive", Mode="Aggresive"))


    return 0



def FinishOrder(dbtype, orderid, marketdata, tws, contract):
    """
    Algo hasn't worked, lets cancel this order
    """
    diag=diagnostic(dbtype, system="algo",  system3=str(int(orderid)))

    s=state_from_sdict(orderid, diag, log)   
    log=logger()


    if tws is None:
        log.info("Finish order didn't get a tws... can't do anything in orderid %d" % orderid)
        return -1


    log=logger()
    ordertable=order_table(dbtype)

    order=ordertable.read_order_for_orderid(orderid)
   
    log.info("Trying to cancel %d because easy algo failure" % orderid)
    tws.cancelOrder(int(order.brokerorderid))
   
    order.modify(cancelled=True)
    ordertable.update_order(order)

    do_order_completed(dbtype, order)           
   
    EasyAlgo_on_complete(dbtype, order, tws)


    s.update(dict(message="NowCancelling", Mode="Finished"))

    am=algo_memory_table(dbtype)
    am.update_value(order.orderid, "Mode", "Finished")
    am.close()
   
    return -1

Partial or complete fill

Blimey this has actually worked, we've actually got a fill...
                   

def EasyAlgo_on_partial(dbtype, order, tws):
    diag=diagnostic(dbtype, system="algo",  system3=str(int(order.orderid)))

    diag.w(order.filledtrade, system2="filled")
    diag.w(order.filledprice, system2="fillprice")
   
    return 0





def EasyAlgo_on_complete(dbtype, order_filled, tws):
    """
    Function Easy algo runs on completion of trade
    """
   

    diag=diagnostic(dbtype, system="algo",  system3=str(int(order_filled.orderid)))
   
    diag.w("Finished", system2="Mode")
    diag.w(order_filled.filledtrade, system2="filled")
    diag.w(order_filled.filledprice, system2="fillprice")

    am=algo_memory_table(dbtype)
    am.update_value(order_filled.orderid, "Mode", "Finished")
    am.close()

    return 0


And we're done

That's it. Its not perfect and it would be very easy to write high frequency code that would game this kind of strategy. However the proof is in the proverbial traditional English dessert, and my execution costs have reduced by approximately 80% from when I was doing market orders, i.e. I am paying an average of 1/10 of the spread. So it's definitely an improvement, and well worth the day or so it took me to code it up and test it.



 

11 comments:

  1. I really dont like to read blogs. But when I found your post, I started to like it. I found it interesting and informative. Thank you for the effort in posting this kind of information.

    Say
    www.imarksweb.org

    ReplyDelete
  2. Hi Rob

    I got your book yesterday, I started reading it, it looks very promising, very rarely you see books that mention portfolio optimization for trade systems ( I think only Perry Kaufman has a couple of pages on that)..anyway
    I would like to setup an automated platform for trading live and back-testing with python...
    for backtesting have you developed your own code? I guess so.I browsed a bit zipline..but I think in the long term it will limit the kind of systems that can be developed...
    how would you recommend that a back-test program should be done in python?

    Many thanks

    ReplyDelete
    Replies
    1. Hi
      I wrote my own stuff

      zipline is interesting as is http://pmorissette.github.io/bt/, and http://gbeced.github.io/pyalgotrade/. But I haven't used them. Here is a list
      http://quant.stackexchange.com/questions/8896/except-zipline-are-there-any-other-pythonic-algorithmic-trading-library-i-can-c

      Of course if you dig around my blog you will find advice on the automated trading side.

      Unfortunately it would take another book to describe how to develop a back test system yourself.

      Mike's book might help you (https://www.quantstart.com/successful-algorithmic-trading-ebook) (tell him I sent you). I haven't read it but it seems to be the only book that tries to attack this.

      At some point I'd like to publish a system that people can use, but that project is some way off.

      Delete
    2. Updated reply: See http://qoppac.blogspot.co.uk/p/pysystemtrade.html

      Delete
  3. You mention that execution costs have come down by 80% versus your earlier market-order strategy. Do you measure these savings relative to the initial mid, or do you update the mid after adverse price movements?
    Eg, suppose the market is 99 - 101 and you want to buy. Your new algo would rest at 99 with original mid at 100. The market now moves to 101 - 103. You update to aggressive and get filled at 103. Do you measure your performance relative to a mid of 100 or 102?
    80% reduction seems extremely good if measured versus the original mid in futures markets that typically trade at min tick size. Any intuition? Thanks!

    ReplyDelete
    Replies
    1. It's versus the initial mid of 100; on the assumption that I'd get filled at 101 if I'd just submitted a market order at that point. So I'd pay 103 versus 100; 3 ticks slippage, a market order would have cost me 1 tick slippage.

      Delete
  4. Hi Robert, I loved your book. Am in its detailed second read now. But n3ed your help: Sharpe or Sortino! Am confused...as Sortino seems obvious given the leeway it has for upside volatility. Yet, everyone seems to prefer, at least publicly, Sharpe; even you. Its easier to calculate can be one--albeit--minor advantage I can see. Kindly show me the light on this one. Thanks!

    ReplyDelete
    Replies
    1. Yes it's easier to calculate Sharpe. All performance measures have disadvantages; ideally you should use several.

      Delete
  5. This comment has been removed by the author.

    ReplyDelete
  6. Thanks Rob! 'Use the average' is my new ideology!

    ReplyDelete
  7. I've found your blog unfortunately today, cause you really do a great job sharing intelligent and fundamental principals in this challenge area. Keep your work up

    ReplyDelete