tag:blogger.com,1999:blog-261139923818144971.post1042630594194607814..comments2017-06-25T11:50:13.129+01:00Comments on Investment Idiocy: Correlations, Weights, Multipliers.... (pysystemtrade)Rob Carvernoreply@blogger.comBlogger98125tag:blogger.com,1999:blog-261139923818144971.post-53775798141350342972017-05-03T16:21:17.640+01:002017-05-03T16:21:17.640+01:00Yes, I agree. That was also my initial intuition a...Yes, I agree. That was also my initial intuition as well. I compared the weekly non-overlapping approach with the overlapping 3-day approach over the same time-frame and the markets that are synchronous the correlation estimates are very similar. More importantly, for example when I ran it on the Hang Seng the rolling 3-day approach was quite close to the weekly approach. So obviously some slight differences but, as you say, the approach doesn't seem like something crazy. <br /><br />Your response is much appreciated.OTNY33http://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-59024878569757776382017-05-03T16:02:21.190+01:002017-05-03T16:02:21.190+01:00The guys at AQR know what they are doing and almos...The guys at AQR know what they are doing and almost without exception are smarter than me. So it would be surprising if they'd done something crazy.<br /><br />Using overlapping returns is generally frowned upon (eg in OLS regression, essentially this is a bias versus variance problem). Using overlapping returns artifically increases the data you have (you only really have completely new observations every 3 days) and that benefit must come at a cost.<br /><br />For correlations it *might* be okay; I rarely work overlapping returns so I don't know their properties well enough to say whether they are fine or not. My intuition and a few minutes of playing with excel suggests they will be fine with zero autocorrelation, but maybe not if autocorrelation comes in.<br /><br />But I don't see the point in using two types of returns - one to calculate vol, one to calculate correlation (in most software you work out a single covariance matrix which embodies both things).Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-69907677826716621052017-05-03T15:44:42.085+01:002017-05-03T15:44:42.085+01:00Thank you! The idea actually came from the Betting...Thank you! The idea actually came from the Betting Against Beta paper by the guys at AQR. They say they use overlapping(or rolling) 3-day log returns to calculate correlations to control for non-synchronous trading over 120 trading days. <br /><br />I think it is safe to say your disagreeing with their approach? <br />OTNY33http://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-70673978724367075392017-05-02T09:37:47.017+01:002017-05-02T09:37:47.017+01:00If you're using daily closing prices to calcul...If you're using daily closing prices to calculate returns across different markets then nonsynchrous is clearly an issue. <br /><br />BUT NEVER, EVER, EVERY USE ROLLING RETURNS!!!!! They will result in your volatility estimate being understated.<br /><br />Instead use non overlapping 3 day returns eg P_3 - P_0, P_6 - P_3, P_9 - P_6 where P_t is the price on day t.<br /><br />As for wether 3 days is enough, well even 2 days would help a bit with nonsynchrous data, although 3 days is better, and 5 days (which is what I use, eg weekly returns if you're in business day space) is better still.<br /><br />On a system trading at the kind of speed I trade at using 60 days worth of correlations probably is too short a period, since it isn't long enough to give you a stable estimate. It's also only 20 observations if you're using 3 day non overlapping returns (it's even worse for weekly returns of course, only 12 observations). Your estimate will be very noisy.<br /><br />Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-16848503328442688822017-04-29T22:43:35.474+01:002017-04-29T22:43:35.474+01:00Rob,
Is one way to estimate correlations with non...Rob,<br /><br />Is one way to estimate correlations with nonsynchronous trading to run correlations on rolling 3-day returns over a lookback of 60-days?(which I know is much shorter than yours)OTNY33http://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-45737020909482062362017-04-21T20:19:26.286+01:002017-04-21T20:19:26.286+01:00Any resources you may have or know of on the subje...Any resources you may have or know of on the subject would be greatly appreciated though!!ESHKDhttp://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-60012069341468567992017-04-21T20:16:51.539+01:002017-04-21T20:16:51.539+01:00Gotcha. It still appears more right to me to set t...Gotcha. It still appears more right to me to set the buffer around % allocations(as opposed to contracts). So obviously I'm missing something. I don't want to exhaust you so thank you for your response!ESHKDhttp://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-24908403691190798482017-04-21T06:34:44.724+01:002017-04-21T06:34:44.724+01:00ESHKD
"Did you estimate the buffer purely on ...ESHKD<br />"Did you estimate the buffer purely on a backtest? eg tried a bunch of different buffers for a give set of costs and settled on 10%?"<br /><br />In fact there are well known analytical methods for deriving the optimal buffer size (we should always avoid fitting when we can), and 10% is a pretty conservative value which I chose to avoid the complexity of including yet another table in my book (it's what I use in my own system, regardless of cost, so this is a simplification I'm personally fine with).<br /><br />"Using this logic then, if I have 1 contract, the upper buffer is 1.1 contracts. So if my optimal number of contracts goes above 1.1(or 1.5 pretty much) then I make a trade. So if my optimal number is 1.6 I just trade one contract?" <br /><br />Correct.<br /><br /><br />"Also, in the example I just gave, isnt the impact different for contracts of varying values. So 1 more contract in JGB is $1m vs 1 more contract in TYA being $125k?"<br /><br />Yes, but (a) it's contract risk that is important not notional size, and (b) there clearly isn't anything we can do about this!Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-3136379151374312122017-04-21T00:50:14.869+01:002017-04-21T00:50:14.869+01:00Also I apologize I should have referenced your boo...Also I apologize I should have referenced your book first. I wasn't fully aware that you calculate the buffer using actual contracts, not percents as I thought. <br /><br />The example in your book is: the upper buffer is 133.52 * 1.1 = 147 contracts. <br /><br />Using this logic then, if I have 1 contract, the upper buffer is 1.1 contracts. So if my optimal number of contracts goes above 1.1(or 1.5 pretty much) then I make a trade. So if my optimal number is 1.6 I just trade one contract? <br /><br />Also, in the example I just gave, isnt the impact different for contracts of varying values. So 1 more contract in JGB is $1m vs 1 more contract in TYA being $125k?<br /><br />I find this aspect interesting, greatly appreciate your thoughts.<br />ESHKDhttp://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-70275555668054683122017-04-20T23:55:01.588+01:002017-04-20T23:55:01.588+01:00Did you estimate the buffer purely on a backtest? ...Did you estimate the buffer purely on a backtest? eg tried a bunch of different buffers for a give set of costs and settled on 10%?ESHKDhttp://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-57023123267640944902017-04-18T06:55:17.289+01:002017-04-18T06:55:17.289+01:00Theoretically the optimal buffer size depends only...Theoretically the optimal buffer size depends only on costs. Higher costs means a larger buffer. Lower costs means a smaller buffer. I use a constant buffer size purely to make life simpler.Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-23551738590596262062017-04-17T18:43:55.343+01:002017-04-17T18:43:55.343+01:00This is an interesting question. What is the reaso...This is an interesting question. What is the reasoning for a 10% buffer(like why not 5% or 15%)?. Two scenarios pop into my head.<br /><br />a.) Lets say you have 10 commodities and each are 9% above their optimal weights. That would leave you 90% more exposed to commodities than your model would call for. Obviously your willing to accept this risk(unless you have some other limits in place). Or say all the 10 commodities have optimal weights of -5% and your current position in all commodities is 4%. You should be -50% short commodities but instead your 40% long commodities. <br /><br />b.) With the 10% buffer there is room for path dependency. Taking the example above, if you establish positions in 10 commodities in January and the signals give them 4% weights each and say commodities dont move around much for, say 3-months, you end up being long 40% commodities for those three months. On the other hand, say you establish positions in February and the signals for all commodities is -5% and don't move around a lot for 3 months. You are now -50% short for a few months(2 overlapping with the January scenario). Certainly you can say well they didn't move much so overall the difference might not be that important. But in real time, say in March, we obviously don't know the next few months will be less volatile we just know we're either 40% long commodities or -50% short commodities.<br /><br />These are just a few thoughts. Obviously you can mitigate some of the problem by setting exposure limits and such. But the path dependency scenario would still be there(especially with larger buffers).<br /><br />Obviously I'm biased towards a smaller buffer. By how much I'm not sure that's why I'd love to get your thoughts on the matter! <br /><br />Or say you have 11 strategies equally weighted trading one market. Presumably each strategy is profitable after some conservative measure of costs without any buffering. If 10 of the strategies show now changes in weights and 1 strategy is now 100% long(9% increase in position) you'd be ignoring that valuable signal.<br /><br />Would love your thoughts!ESHKDhttp://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-33241997813596701432017-03-21T06:24:19.942+00:002017-03-21T06:24:19.942+00:00To get the performance of a trading rule you run t...To get the performance of a trading rule you run through the position sizing method in the book allocating 100% to a given trading rule.<br /><br />1) No, that isn't how the system works at all. Read the rest of the book before asking any more questions.<br />2) yes - again this in discussed later in the book<br />3) No, I use continous positions. You need to read chapter 7 again as you don't seem to have quite got the gist.<br /><br /> f*w - lemada*w*sigma*w<br /><br />I don't think I've ever used this formula in my book, or on my blog, so I can't really explain it.Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-29483813399991305832017-03-20T22:52:32.600+00:002017-03-20T22:52:32.600+00:00Hi Rob,
Thank you so much for your book. It it ve...Hi Rob,<br /><br />Thank you so much for your book. It it very educative. I was trying to understand more about trading rules correlations in "Chapter 8: Combined Forecasts". You mentioned that back-testing the performance of trading rules to get correlation.<br /><br />Could you share a bit more insights on how you get the performance of trading rules, please?<br />(1) Do you set buy/sell threshold at +/- 10? meaning that no position held when signal is [-10,10], only 1 position held when signal is [10,20] and [-20,-10] and 2 positions held when signal is at -20/+20?<br />(2) Trading cost is considered? (I think the answer is yes.)<br />(3) You entry a buy trade, say at signal=10. When do you signal to exit the trade? when signal<10 or signal=0?<br /><br />or you use dynamic positions, meaning the position varies with signal all the time.<br /><br />Another question regarding optimisation:<br />In the formula: f*w - lemada*w*sigma*w' to estimate weights<br />(1) f is rules' sharpe ratio calculated using the rules' historical performance pooled from all instruments or just the sharpe of the rule from the instrument we look at?<br />(2) how do you define lemada? =0.0001? if so, is it always 0.0001?<br /><br />Sorry if those two questions had been asked before.<br /><br />Thanks,<br />DeanoGu langyuhttp://www.blogger.com/profile/16054821256101608361noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-15858215407372944062017-03-13T16:17:06.439+00:002017-03-13T16:17:06.439+00:00Yes. You should use a nearer month for carry if yo...Yes. You should use a nearer month for carry if you can, and trade further out, but this isn't possible in bonds, equities or FX. See appendix B.Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-90681759521581606682017-03-13T16:10:52.858+00:002017-03-13T16:10:52.858+00:00Rob, in your legacy.csv modules, some specific fut...Rob, in your legacy.csv modules, some specific futures have the "price contract" as the "front month"(closest contract) like Bund, US20 & US10, etc. meanwhile, others such as Wheat, , gas, crude, etc have the "carry contract" as the front month. is this by design?Dolphhttp://www.blogger.com/profile/15588741885676289695noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-89901957483022988252017-03-13T06:38:45.613+00:002017-03-13T06:38:45.613+00:00Well 92% of the weight on the correlations will be...Well 92% of the weight on the correlations will be coming from the last 3 years. So yes you could speed this up by using a rolling out of sample although the results will be slightly different. 5 years would be better as this gets you up to 99%.Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-51660349476836649322017-03-11T10:44:26.606+00:002017-03-11T10:44:26.606+00:00OK, but in your simulations you work with an expan...OK, but in your simulations you work with an expanding window and do calculations yearly based on weekly data. If we use EWM-span of 125 it means the rolling correlations go back roughly about 3 years (125*5 days). So if for example the total period is from 1990-2016, is the last element of last calculation (1990-2016) then a correct estimate of the correlation of the whole period, because data before 2012 is 'ignored' ?<br /><br />Maybe it's then faster to work with a rolling out-of-sample frame to do this calculations ?<br /><br />Or is my idea on this not correct ?<br /><br />Kris Krishttp://www.blogger.com/profile/17919667101415157462noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-28287407067770538462017-03-10T06:15:48.693+00:002017-03-10T06:15:48.693+00:00ewm.corr returns rolling correlations; each elemen...ewm.corr returns rolling correlations; each element in the list is already an exponentially weighted average of correlations. Since I'm doing the rolling through time process myself I only need the last of these elements.Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-33198178595926293882017-03-09T21:40:27.524+00:002017-03-09T21:40:27.524+00:00I see that the ewm.corr-function return a list of ...I see that the ewm.corr-function return a list of correlations for each date and not a correlation matrix.<br />For the classic corr-function the result a matrix of correlation coëfficients.<br /><br />In your code (https://github.com/robcarver17/pysystemtrade/blob/ba7fe7782837b0df0dea83631da19d98a1d8c84f/syscore/correlations.py#L173) I see you only takes the latest value for each year of the ewm.corr function. <br />I should expect that we must take a kind of average of all correlation values from a pair to calculate the correlation coëfficient for each pair. Can you clarify this, thanks.<br /><br />KrisKrishttp://www.blogger.com/profile/17919667101415157462noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-76280141570939336092017-03-09T06:20:46.122+00:002017-03-09T06:20:46.122+00:00Oh sorry I misunderstood. You are using WEEKLY RET...Oh sorry I misunderstood. You are using WEEKLY RETURNS to estimate instrument weights: that's fine. I thought you were actually redoing the bootstrapping every week.Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-6611054274989205442017-03-08T20:39:06.376+00:002017-03-08T20:39:06.376+00:00Thanks for the reply. I had applied the same metho...Thanks for the reply. I had applied the same method for the instrument weights as for the forecast weights. You'd mentioned above:<br />"Also the default is to use weekly returns for optimisation. This has two advantages; firstly it's faster. Secondly correlations of daily returns tend to be unrealistically low (because for example of different market closes when working across instruments)."<br />Why would the default for forecast weights be weekly but not for instrument weights?<br />Thanks!AvantGardehttp://www.blogger.com/profile/13449856505135293636noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-38105353005991843832017-03-08T12:10:02.100+00:002017-03-08T12:10:02.100+00:00"I am using weekly bootstrapping to estimate ..."I am using weekly bootstrapping to estimate instrument weights...." I think this is a little... well if I'm being honest I think its insane.<br /><br />Okay so to answer the question for backtesting I use one config, and then for my live trading system I use another config which has fixed weights. Personally I run these seperately for different reasons, the backtest to get an idea of historical performance, the 'as live' backtest with fixed weights to compare against what I'm currently doing and for actual trading.<br /><br />There is no configurable way of mixing these, so you'd need to write some code that takes the estimate bootstrapped weights and then replaces them with fixed weights after a certain date. <br />Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-55704908207667524672017-03-08T09:27:17.775+00:002017-03-08T09:27:17.775+00:00Hi Rob,
I’ve not been clear. I understand how EWMA...Hi Rob,<br />I’ve not been clear. I understand how EWMA works and the process of smoothing.<br />The problem I am having is that I am using weekly bootstrapping to estimate instrument weights. However, each day when I run pysystemtrade the calculated instrument weights can vary significantly day to day due to the nature of bootstrapping. This leads to situations where e.g., pysystemtrade would have generated a trade yesterday when I was running it (which I would have executed), but when I run it today the instrument weight estimates may have changed enough due to the bootstrapping so that the trade that was generated and executed yesterday does not show up as a trade that was generated yesterday today. This makes me less trusting of the backtested performance, as the majority of trades that were historically generated but excluded after resampling are losing trades.<br />I only sample the market once a day generally (so that repeated sampling of the market overwriting the current day’s mark is not an issue).<br />I would like to use the bootstrapping to estimate the weights ANNUALLY and apply the smooth to adjust between last year’s calculated weight, and today’s. But if I am using fixed weights (after having estimated via bootstrapping) by setting them as fixed in the config, there are no longer two data points to smooth between as I have only one fixed estimate in the config.<br />How can I insert an historical weight for last year and a new fixed weight this year (by fixing it in the config) and smooth between them? <br />AvantGardehttp://www.blogger.com/profile/13449856505135293636noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-38009321469878379472017-02-28T19:59:49.741+00:002017-02-28T19:59:49.741+00:00Thanks for this tip!
I always try to write my ow...Thanks for this tip! <br /><br />I always try to write my own code (don't like dependency of others code) and also I don't see how I can use the pandas libraries into vb.net. <br /><br />But I've found the functions here :<br />https://github.com/pandas-dev/pandas/blob/master/pandas/window.pyx --> EWMCOV <br /><br />and here :<br />https://github.com/pandas-dev/pandas/blob/v0.19.2/pandas/core/window.py#L1576-L1596 --> corr<br />So I can analyse how they to the stuff and can write it in VB.NET<br /><br />KrisKrishttp://www.blogger.com/profile/17919667101415157462noreply@blogger.com