tag:blogger.com,1999:blog-261139923818144971.post1042630594194607814..comments2017-04-30T10:46:36.819+01:00Comments on Investment Idiocy: Correlations, Weights, Multipliers.... (pysystemtrade)Rob Carvernoreply@blogger.comBlogger92125tag:blogger.com,1999:blog-261139923818144971.post-60012069341468567992017-04-21T20:16:51.539+01:002017-04-21T20:16:51.539+01:00Gotcha. It still appears more right to me to set t...Gotcha. It still appears more right to me to set the buffer around % allocations(as opposed to contracts). So obviously I'm missing something. I don't want to exhaust you so thank you for your response!ESHKDhttp://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-24908403691190798482017-04-21T06:34:44.724+01:002017-04-21T06:34:44.724+01:00ESHKD
"Did you estimate the buffer purely on ...ESHKD<br />"Did you estimate the buffer purely on a backtest? eg tried a bunch of different buffers for a give set of costs and settled on 10%?"<br /><br />In fact there are well known analytical methods for deriving the optimal buffer size (we should always avoid fitting when we can), and 10% is a pretty conservative value which I chose to avoid the complexity of including yet another table in my book (it's what I use in my own system, regardless of cost, so this is a simplification I'm personally fine with).<br /><br />"Using this logic then, if I have 1 contract, the upper buffer is 1.1 contracts. So if my optimal number of contracts goes above 1.1(or 1.5 pretty much) then I make a trade. So if my optimal number is 1.6 I just trade one contract?" <br /><br />Correct.<br /><br /><br />"Also, in the example I just gave, isnt the impact different for contracts of varying values. So 1 more contract in JGB is $1m vs 1 more contract in TYA being $125k?"<br /><br />Yes, but (a) it's contract risk that is important not notional size, and (b) there clearly isn't anything we can do about this!Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-3136379151374312122017-04-21T00:50:14.869+01:002017-04-21T00:50:14.869+01:00Also I apologize I should have referenced your boo...Also I apologize I should have referenced your book first. I wasn't fully aware that you calculate the buffer using actual contracts, not percents as I thought. <br /><br />The example in your book is: the upper buffer is 133.52 * 1.1 = 147 contracts. <br /><br />Using this logic then, if I have 1 contract, the upper buffer is 1.1 contracts. So if my optimal number of contracts goes above 1.1(or 1.5 pretty much) then I make a trade. So if my optimal number is 1.6 I just trade one contract? <br /><br />Also, in the example I just gave, isnt the impact different for contracts of varying values. So 1 more contract in JGB is $1m vs 1 more contract in TYA being $125k?<br /><br />I find this aspect interesting, greatly appreciate your thoughts.<br />ESHKDhttp://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-70275555668054683122017-04-20T23:55:01.588+01:002017-04-20T23:55:01.588+01:00Did you estimate the buffer purely on a backtest? ...Did you estimate the buffer purely on a backtest? eg tried a bunch of different buffers for a give set of costs and settled on 10%?ESHKDhttp://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-57023123267640944902017-04-18T06:55:17.289+01:002017-04-18T06:55:17.289+01:00Theoretically the optimal buffer size depends only...Theoretically the optimal buffer size depends only on costs. Higher costs means a larger buffer. Lower costs means a smaller buffer. I use a constant buffer size purely to make life simpler.Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-23551738590596262062017-04-17T18:43:55.343+01:002017-04-17T18:43:55.343+01:00This is an interesting question. What is the reaso...This is an interesting question. What is the reasoning for a 10% buffer(like why not 5% or 15%)?. Two scenarios pop into my head.<br /><br />a.) Lets say you have 10 commodities and each are 9% above their optimal weights. That would leave you 90% more exposed to commodities than your model would call for. Obviously your willing to accept this risk(unless you have some other limits in place). Or say all the 10 commodities have optimal weights of -5% and your current position in all commodities is 4%. You should be -50% short commodities but instead your 40% long commodities. <br /><br />b.) With the 10% buffer there is room for path dependency. Taking the example above, if you establish positions in 10 commodities in January and the signals give them 4% weights each and say commodities dont move around much for, say 3-months, you end up being long 40% commodities for those three months. On the other hand, say you establish positions in February and the signals for all commodities is -5% and don't move around a lot for 3 months. You are now -50% short for a few months(2 overlapping with the January scenario). Certainly you can say well they didn't move much so overall the difference might not be that important. But in real time, say in March, we obviously don't know the next few months will be less volatile we just know we're either 40% long commodities or -50% short commodities.<br /><br />These are just a few thoughts. Obviously you can mitigate some of the problem by setting exposure limits and such. But the path dependency scenario would still be there(especially with larger buffers).<br /><br />Obviously I'm biased towards a smaller buffer. By how much I'm not sure that's why I'd love to get your thoughts on the matter! <br /><br />Or say you have 11 strategies equally weighted trading one market. Presumably each strategy is profitable after some conservative measure of costs without any buffering. If 10 of the strategies show now changes in weights and 1 strategy is now 100% long(9% increase in position) you'd be ignoring that valuable signal.<br /><br />Would love your thoughts!ESHKDhttp://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-33241997813596701432017-03-21T06:24:19.942+00:002017-03-21T06:24:19.942+00:00To get the performance of a trading rule you run t...To get the performance of a trading rule you run through the position sizing method in the book allocating 100% to a given trading rule.<br /><br />1) No, that isn't how the system works at all. Read the rest of the book before asking any more questions.<br />2) yes - again this in discussed later in the book<br />3) No, I use continous positions. You need to read chapter 7 again as you don't seem to have quite got the gist.<br /><br /> f*w - lemada*w*sigma*w<br /><br />I don't think I've ever used this formula in my book, or on my blog, so I can't really explain it.Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-29483813399991305832017-03-20T22:52:32.600+00:002017-03-20T22:52:32.600+00:00Hi Rob,
Thank you so much for your book. It it ve...Hi Rob,<br /><br />Thank you so much for your book. It it very educative. I was trying to understand more about trading rules correlations in "Chapter 8: Combined Forecasts". You mentioned that back-testing the performance of trading rules to get correlation.<br /><br />Could you share a bit more insights on how you get the performance of trading rules, please?<br />(1) Do you set buy/sell threshold at +/- 10? meaning that no position held when signal is [-10,10], only 1 position held when signal is [10,20] and [-20,-10] and 2 positions held when signal is at -20/+20?<br />(2) Trading cost is considered? (I think the answer is yes.)<br />(3) You entry a buy trade, say at signal=10. When do you signal to exit the trade? when signal<10 or signal=0?<br /><br />or you use dynamic positions, meaning the position varies with signal all the time.<br /><br />Another question regarding optimisation:<br />In the formula: f*w - lemada*w*sigma*w' to estimate weights<br />(1) f is rules' sharpe ratio calculated using the rules' historical performance pooled from all instruments or just the sharpe of the rule from the instrument we look at?<br />(2) how do you define lemada? =0.0001? if so, is it always 0.0001?<br /><br />Sorry if those two questions had been asked before.<br /><br />Thanks,<br />DeanoGu langyuhttp://www.blogger.com/profile/16054821256101608361noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-15858215407372944062017-03-13T16:17:06.439+00:002017-03-13T16:17:06.439+00:00Yes. You should use a nearer month for carry if yo...Yes. You should use a nearer month for carry if you can, and trade further out, but this isn't possible in bonds, equities or FX. See appendix B.Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-90681759521581606682017-03-13T16:10:52.858+00:002017-03-13T16:10:52.858+00:00Rob, in your legacy.csv modules, some specific fut...Rob, in your legacy.csv modules, some specific futures have the "price contract" as the "front month"(closest contract) like Bund, US20 & US10, etc. meanwhile, others such as Wheat, , gas, crude, etc have the "carry contract" as the front month. is this by design?Dolphhttp://www.blogger.com/profile/15588741885676289695noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-89901957483022988252017-03-13T06:38:45.613+00:002017-03-13T06:38:45.613+00:00Well 92% of the weight on the correlations will be...Well 92% of the weight on the correlations will be coming from the last 3 years. So yes you could speed this up by using a rolling out of sample although the results will be slightly different. 5 years would be better as this gets you up to 99%.Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-51660349476836649322017-03-11T10:44:26.606+00:002017-03-11T10:44:26.606+00:00OK, but in your simulations you work with an expan...OK, but in your simulations you work with an expanding window and do calculations yearly based on weekly data. If we use EWM-span of 125 it means the rolling correlations go back roughly about 3 years (125*5 days). So if for example the total period is from 1990-2016, is the last element of last calculation (1990-2016) then a correct estimate of the correlation of the whole period, because data before 2012 is 'ignored' ?<br /><br />Maybe it's then faster to work with a rolling out-of-sample frame to do this calculations ?<br /><br />Or is my idea on this not correct ?<br /><br />Kris Krishttp://www.blogger.com/profile/17919667101415157462noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-28287407067770538462017-03-10T06:15:48.693+00:002017-03-10T06:15:48.693+00:00ewm.corr returns rolling correlations; each elemen...ewm.corr returns rolling correlations; each element in the list is already an exponentially weighted average of correlations. Since I'm doing the rolling through time process myself I only need the last of these elements.Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-33198178595926293882017-03-09T21:40:27.524+00:002017-03-09T21:40:27.524+00:00I see that the ewm.corr-function return a list of ...I see that the ewm.corr-function return a list of correlations for each date and not a correlation matrix.<br />For the classic corr-function the result a matrix of correlation coëfficients.<br /><br />In your code (https://github.com/robcarver17/pysystemtrade/blob/ba7fe7782837b0df0dea83631da19d98a1d8c84f/syscore/correlations.py#L173) I see you only takes the latest value for each year of the ewm.corr function. <br />I should expect that we must take a kind of average of all correlation values from a pair to calculate the correlation coëfficient for each pair. Can you clarify this, thanks.<br /><br />KrisKrishttp://www.blogger.com/profile/17919667101415157462noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-76280141570939336092017-03-09T06:20:46.122+00:002017-03-09T06:20:46.122+00:00Oh sorry I misunderstood. You are using WEEKLY RET...Oh sorry I misunderstood. You are using WEEKLY RETURNS to estimate instrument weights: that's fine. I thought you were actually redoing the bootstrapping every week.Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-6611054274989205442017-03-08T20:39:06.376+00:002017-03-08T20:39:06.376+00:00Thanks for the reply. I had applied the same metho...Thanks for the reply. I had applied the same method for the instrument weights as for the forecast weights. You'd mentioned above:<br />"Also the default is to use weekly returns for optimisation. This has two advantages; firstly it's faster. Secondly correlations of daily returns tend to be unrealistically low (because for example of different market closes when working across instruments)."<br />Why would the default for forecast weights be weekly but not for instrument weights?<br />Thanks!AvantGardehttp://www.blogger.com/profile/13449856505135293636noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-38105353005991843832017-03-08T12:10:02.100+00:002017-03-08T12:10:02.100+00:00"I am using weekly bootstrapping to estimate ..."I am using weekly bootstrapping to estimate instrument weights...." I think this is a little... well if I'm being honest I think its insane.<br /><br />Okay so to answer the question for backtesting I use one config, and then for my live trading system I use another config which has fixed weights. Personally I run these seperately for different reasons, the backtest to get an idea of historical performance, the 'as live' backtest with fixed weights to compare against what I'm currently doing and for actual trading.<br /><br />There is no configurable way of mixing these, so you'd need to write some code that takes the estimate bootstrapped weights and then replaces them with fixed weights after a certain date. <br />Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-55704908207667524672017-03-08T09:27:17.775+00:002017-03-08T09:27:17.775+00:00Hi Rob,
I’ve not been clear. I understand how EWMA...Hi Rob,<br />I’ve not been clear. I understand how EWMA works and the process of smoothing.<br />The problem I am having is that I am using weekly bootstrapping to estimate instrument weights. However, each day when I run pysystemtrade the calculated instrument weights can vary significantly day to day due to the nature of bootstrapping. This leads to situations where e.g., pysystemtrade would have generated a trade yesterday when I was running it (which I would have executed), but when I run it today the instrument weight estimates may have changed enough due to the bootstrapping so that the trade that was generated and executed yesterday does not show up as a trade that was generated yesterday today. This makes me less trusting of the backtested performance, as the majority of trades that were historically generated but excluded after resampling are losing trades.<br />I only sample the market once a day generally (so that repeated sampling of the market overwriting the current day’s mark is not an issue).<br />I would like to use the bootstrapping to estimate the weights ANNUALLY and apply the smooth to adjust between last year’s calculated weight, and today’s. But if I am using fixed weights (after having estimated via bootstrapping) by setting them as fixed in the config, there are no longer two data points to smooth between as I have only one fixed estimate in the config.<br />How can I insert an historical weight for last year and a new fixed weight this year (by fixing it in the config) and smooth between them? <br />AvantGardehttp://www.blogger.com/profile/13449856505135293636noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-38009321469878379472017-02-28T19:59:49.741+00:002017-02-28T19:59:49.741+00:00Thanks for this tip!
I always try to write my ow...Thanks for this tip! <br /><br />I always try to write my own code (don't like dependency of others code) and also I don't see how I can use the pandas libraries into vb.net. <br /><br />But I've found the functions here :<br />https://github.com/pandas-dev/pandas/blob/master/pandas/window.pyx --> EWMCOV <br /><br />and here :<br />https://github.com/pandas-dev/pandas/blob/v0.19.2/pandas/core/window.py#L1576-L1596 --> corr<br />So I can analyse how they to the stuff and can write it in VB.NET<br /><br />KrisKrishttp://www.blogger.com/profile/17919667101415157462noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-47953185179922868092017-02-28T06:22:55.730+00:002017-02-28T06:22:55.730+00:00Sorry, yes I use exponential weighting a lot. With...Sorry, yes I use exponential weighting a lot. With respect to the first, yes I calculate correlations using an exponential estimator: http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.ewmcorr.htmlRob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-41022012447778854762017-02-27T21:41:29.472+00:002017-02-27T21:41:29.472+00:00OK,but in this article I found 2 different paramet...OK,but in this article I found 2 different parameters refering to exponantional weighting :<br /><br />- under 'Forecast Diversification Multiplier' --> 'correlation' : I found "using_exponent: True # use an exponentially weighted correlation, or all the values equally"<br /><br />- under 'Forecast Diversification Multiplier' --> 'Smoothing again' : I found "ewma_span: 125 ## smooth to apply"<br /><br />I am a little bit confused about the 2 parameters. I understand that the second parameter (smoothing again) is to smooth the jump on the 1st January each year. <br /><br />But what about the first parameter (correlation) ? I thought that you use some kind of exponantial weighting for calculating the correlations, but maybe I'm wrong ? Sorry, but it is not so clear for me.<br /><br />Kris<br /><br />Krishttp://www.blogger.com/profile/17919667101415157462noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-56105157997830032492017-02-27T20:03:20.260+00:002017-02-27T20:03:20.260+00:00No, on the actual multiplier. It's calculated ...No, on the actual multiplier. It's calculated from correlations, updated annually. Without a smooth it would be jumpy on the 1st January each year. Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-61139107661532640002017-02-27T19:13:13.407+00:002017-02-27T19:13:13.407+00:00Hi Robert,
For the diversification multiplier you...Hi Robert,<br /><br />For the diversification multiplier you mention to use exponential weighting. Where or how you implement this? On the returns or on the deviations of the returns from the expected returns (so just before the calculation of the covariances)? Or maybe at an other place ? <br /><br />Can you give me some direction?<br /><br />Thanks<br /><br />KrisKrishttp://www.blogger.com/profile/17919667101415157462noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-5122689131053127692017-02-20T18:36:04.030+00:002017-02-20T18:36:04.030+00:00(goes to the blackboard to write "I will not ...(goes to the blackboard to write "I will not overfit" 50 times)...sorry, I've read your statements on overfitting more than once, but had a lapse in memory when this question popped into my thick skull. Thanks for your response.Chad Bhttp://www.blogger.com/profile/13026562498196984544noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-2130180258649559192017-02-20T06:16:59.523+00:002017-02-20T06:16:59.523+00:00http://pandas.pydata.org/pandas-docs/version/0.17....http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.ewma.htmlRob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.com