tag:blogger.com,1999:blog-261139923818144971.post8960556854419287713..comments2018-03-20T08:13:05.111+00:00Comments on Investment Idiocy: Optimising weights with costs (pysystemtrade)Rob Carvernoreply@blogger.comBlogger44125tag:blogger.com,1999:blog-261139923818144971.post-43687890964853950042018-02-20T09:17:03.686+00:002018-02-20T09:17:03.686+00:00Thanks and understood. I agree it can be time cons...Thanks and understood. I agree it can be time consuming. My initial starting point was that volscales could vary enormously depending on the market (0.02 is about right for a market whose trend amplitude shows doubling and halving which some commodity markets can display), whereas 0.2 is right for markets with usual shallower trends. This in turn seems to result in wide turnover ranges. I will see if I can come with an alternative formulation of trend patterns and see if that makes the outcomes more stable.<br /><br />P.S. To quote Gene Hackman in the movie 'Heist': 'I am burnt..', but somehow the movie survives with the fiction that he is not, and I shall try to continue with that fiction, if that's OK)Patrickhttps://www.blogger.com/profile/01034323689613977187noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-40348956829999849842018-02-20T08:39:44.609+00:002018-02-20T08:39:44.609+00:00This comment has been removed by the author.Kamal Murarihttps://www.blogger.com/profile/12326045785649560867noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-40450649541895094212018-02-16T10:47:04.612+00:002018-02-16T10:47:04.612+00:00It does sound weird. I have done a similar exercis...It does sound weird. I have done a similar exercise but I can't remember what range of volscale I used. I didn't do it very scientifically but if I toned the vol scale down too much then I stopped seeing 'realistic' looking price series (and also the Sharpe Ratio of the ewmac pairs became unrealistically high). I haven't really got time to dig into this, but my approach was basically to generate some results with simulated data and then check they were roughly in the right ball park with real price data.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-27707517875371184712018-02-15T20:06:48.145+00:002018-02-15T20:06:48.145+00:00Well, this is interesting, if I have done things c...Well, this is interesting, if I have done things correctly. Using simulated data, if I use a volscale of ~ 0.16 (to amplitude) and take out the fastest and slowest rules plus focus only on trend frequencies of 1-6 pa, turnover doesn't vary that much (still varies more than I would expect). However when I lower the volscale to 0.02 and look at wider ranges, turnover varies quite a bit. Taking the ewmac2,8 as an example and looking at a trend frequency from once every three years to once a month, the turnover falls from 57 to 18. Its also slightly counterintuitive to me that turnover would fall if trend frequency goes up. Could this be correct? I could have done something fatally wrong, but if not, would it be correct to think that simulated turnover has some limitations (given that I have no idea what the correct 'harmonic frequency', as you put it, will be). Happy to send across a small table of results by email or through whatever channel which might allow formatted tables to be read easily.Patrickhttps://www.blogger.com/profile/01034323689613977187noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-38260030050088080412018-02-15T15:17:41.032+00:002018-02-15T15:17:41.032+00:00This comment has been removed by the author.Kamal Murarihttps://www.blogger.com/profile/12326045785649560867noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-5289114982384222392018-02-14T16:08:01.299+00:002018-02-14T16:08:01.299+00:00I think the spreadsheet is correct - the graphs lo...I think the spreadsheet is correct - the graphs look more realistic.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-80620133014022387402018-02-14T14:53:16.113+00:002018-02-14T14:53:16.113+00:00Thanks Rob, I have looked at your spreadsheet here...Thanks Rob, I have looked at your spreadsheet here https://tinyurl.com/ycs8lmp3 and the github code and as far as I can tell they seem to be generating different price processes. The code produce a price path with gaussian 'error' overlayed, whereas the spreadsheet produces a price path along the lines what I think I stated above which is the cumulative(drift + error). I don't know if the code needs correcting or the spreadsheet. Patrickhttps://www.blogger.com/profile/01034323689613977187noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-28309451446416739262018-02-14T11:38:10.422+00:002018-02-14T11:38:10.422+00:00https://github.com/robcarver17/systematictradingex...https://github.com/robcarver17/systematictradingexamples/blob/master/commonrandom.pyRob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-12064411668954563352018-02-14T11:03:35.855+00:002018-02-14T11:03:35.855+00:00Fantastic presentation, thanks. So I am now trying...Fantastic presentation, thanks. So I am now trying to recreate trending markets at different frequencies. Question I have is how to superimpose noise on the sawtooth. The sawtooth is the cumsum of all drift terms, so noise should also be the cumsum of noise?Patrickhttps://www.blogger.com/profile/01034323689613977187noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-63423230543080510672018-02-13T13:26:06.326+00:002018-02-13T13:26:06.326+00:00Using random data to estimate turnover is an excel...Using random data to estimate turnover is an excellent idea, and one I have talked about (i.e https://www.youtube.com/watch?v=s2NSlxHq6Bc ) and blogged about (https://qoppac.blogspot.co.uk/2015/11/using-random-data.html).Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-85970137660464402992018-02-13T12:03:46.479+00:002018-02-13T12:03:46.479+00:00Thank you Rob. With regard to estimating expected ...Thank you Rob. With regard to estimating expected turnover for a each rule variation, rather than using real data, I wonder if it is best to fix these simply with the help of simulated gaussian returns? Otherwise, if we are removing rules based on speed thresholds, and if estimated speeds vary as we change our window of observations, it is possible to end up with changing rule sets for an instrument over time and in different backtests. Could be done but it seems overkill and also I personally can't see why expected turnover should vary over time for a given rule. I am probably preaching to the converted in CH3, I get the impression this is exactly what you recommend doing? <br /><br />As you can see I am still thoroughly enjoying (re) reading your excellent book. Patrickhttps://www.blogger.com/profile/01034323689613977187noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-46049587451208304422018-02-12T10:33:57.721+00:002018-02-12T10:33:57.721+00:00Yes you're right about this being an approxima...Yes you're right about this being an approximation.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-36454817713476063152018-02-11T12:15:47.597+00:002018-02-11T12:15:47.597+00:00Hi Rob,
I saw that on page 196 of Systematic Trad...Hi Rob, <br />I saw that on page 196 of Systematic Trading, third para, you indicate that we can approximate turnover of combined forecasts by taking the weighted average turnover of the individual forecasts. I assume this is in fact intended as a means to get a quick approximation as I would think the 'turnover of the weighted average forecasts' to be lower than the 'weighted average of the individual turnovers' (due to imperfect correlation)? As a crosscheck, tentatively, I think I managed to verify this via a quick comparison using random returns data. Patrickhttps://www.blogger.com/profile/01034323689613977187noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-83850115283685383142017-06-05T11:09:38.601+01:002017-06-05T11:09:38.601+01:00No, it would be better to slow down your trading c...No, it would be better to slow down your trading considerably :-)Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-84762985520257710722017-06-02T09:02:53.507+01:002017-06-02T09:02:53.507+01:00Rob,
I have noted that when using pysystem on col...Rob, <br />I have noted that when using pysystem on collections of shares such as ETF's and particularly unit trusts that the cost ceiling of SR-turnover of 0.13 is very restrictive on account of the lower volatility of these grouped instruments. <br />If one wanted to use these vehicles, would it be better to increase the cost ceiling to say 1.5 or set apply_cost_weight: False and cost_multiplier: 1?<br /><br />Thanks<br /><br />ChrisCJhttps://www.blogger.com/profile/06817263570026256611noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-85606531986313979572017-05-14T18:10:46.140+01:002017-05-14T18:10:46.140+01:00That's correct, and indeed it's something ...That's correct, and indeed it's something I note in my book - if you eliminate the high cost rules it's not worth bothering with adjustment.<br /><br />The adjustment factors are theoretically "correct" in the sense that they properly account for the appropriate adjustment given a situation in which you know the sharpe ratio difference with something close to certainty. Experimenting with them would be akin to fitting yet another parameter, in sample, and so a horrendously bad thing.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-15515999364303376972017-05-14T16:31:57.306+01:002017-05-14T16:31:57.306+01:00Hi Rob
In your code you calculate raw forecast we...Hi Rob<br /><br />In your code you calculate raw forecast weights and afterwards you use an adjustment factor based on cost to change the weights. Also descriped in this article under 'Calculate weights using gross returns, and adjust subsequently for cost levels'<br /><br />I've did some tests on this but my conclusion is that the adjustment factor is always low. This is normal because we eliminate rule variation with SR cost>0.13. This results in a very small adjustment on the raw weights. <br />If I recalculate the raw weights I see they vary for each calculation (which is normal I think) but I should expect that the adjustment factor should have a greater impact on the weights than recalculations.<br /><br />Did you have experimented with bigger adjustment factors, so costs has a bigger impact on the raw weights ?<br /><br />Kris Krishttps://www.blogger.com/profile/17919667101415157462noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-20898408222719423822017-05-03T09:01:19.024+01:002017-05-03T09:01:19.024+01:00"- Why won't I need to know deal with the..."- Why won't I need to know deal with the scaling factor to calculate forecast turnover, but required to deal with a scaling factor (average annual position holding) is required for position turnover?"<br /><br />Because forecasts are automatically scaled (average absolute value of 10), whereas the average absolute position value could be anything.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-62645329084925880712017-05-03T07:09:53.527+01:002017-05-03T07:09:53.527+01:00Thank you for your reply. The future rolling post ...Thank you for your reply. The future rolling post is helpful. I had always been thinking the future liquidity is sole a monotonic (decreasing) function of the distance of maturity date.<br /><br />Again, this is another reason to normalise cost. If you do this then you can just use the forecast turnover to calculate expected costs (though there may be further costs from changes in volatility affecting position size) multiplied by the SR of costs; and you don't need to worry about scaling factors.<br /><br />- Why won't I need to know deal with the scaling factor to calculate forecast turnover, but required to deal with a scaling factor (average annual position holding) is required for position turnover?<br /><br />Thanks in advanceYThttps://www.blogger.com/profile/06542245861917242332noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-12406938628241094342017-05-02T09:28:02.485+01:002017-05-02T09:28:02.485+01:00"Firstly, I do not understand why normalisati..."Firstly, I do not understand why normalisation is required "<br /><br />It's better to normalise and use SR of costs rather than actual $ figures or even percentages; because then you can easily compare costs across instruments and across time. In the absence of accurate figures for historic values of the bid-ask spread using vol normalised costs gives you a better approximation than assuming that current bid-ask spreads were realisable several decades ago.<br /><br /><br />"Furthermore, if I look into what is implemented in your python library, it seems that 2 * factor is missing" I'll look into this, issue is https://github.com/robcarver17/pysystemtrade/issues/61<br /><br />Thirdly, it is not clear how to include turnover estimation at forecast level as opposed to position level? <br /><br />Again, this is another reason to normalise cost. If you do this then you can just use the forecast turnover to calculate expected costs (though there may be further costs from changes in volatility affecting position size) multiplied by the SR of costs; and you don't need to worry about scaling factors.<br /><br />4) yet this is not true for commodity where seasonality could play a big role,<br /><br />See http://qoppac.blogspot.co.uk/2015/05/systems-building-futures-rolling.html<br />Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-567062114699812502017-04-26T21:53:55.828+01:002017-04-26T21:53:55.828+01:00Rob, I bought your book and in the process of repl...Rob, I bought your book and in the process of replicating the logics to confirm my understanding. I have some questions regarding to cost estimation. I can grasp the standardised cost, i.e., SR for a round trip, yet I find it is confusing on the estimation of the turnover (for forecast level, instrument level and subsystem level)<br /><br />In your book, you used position level as example, and provided a formula as: average position change / 2 * average holding per year. Firstly, I do not understand why normalisation is required for this. If a round trip for a block value for an instrument costs 0.01 SR unit, and the number of block values changes says 10 round trip, doesn't this imply 0.01 * 10 = 0.10 SR unit, which should be subtracted from the expected return in various optimisation stage?<br /><br />Furthermore, if I look into what is implemented in your python library, it seems that 2 * factor is missing (at least on the forecast level). Could you clarify this?<br /><br />Thirdly, it is not clear how to include turnover estimation at forecast level as opposed to position level? Forecast generated is only proportional, not equal to estimated SR for a rule variation, wouldn't the cost estimation in the way you described in the book require to be scaled by an unknown factor?<br /><br />As a separate note, on the carry strategy, for various future strategies, the front and back (2nd front) future are sufficient to use as proxy of carry, yet this is not true for commodity where seasonality could play a big role, I wonder how do you pick the right pair of future systematically? Moreover, is there a point to look at the how term structure?YThttps://www.blogger.com/profile/06542245861917242332noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-35389651792305363822017-04-25T18:14:12.502+01:002017-04-25T18:14:12.502+01:00ThanksThanksCJhttps://www.blogger.com/profile/06817263570026256611noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-81279693055227347142017-04-25T14:56:48.848+01:002017-04-25T14:56:48.848+01:00i: optimisation is done without costs (pure pre-co...i: optimisation is done without costs (pure pre-cost returns), and then the weights that come out are adjusted with more expensive assets seeing their weights reduced.<br /><br />ii: optimisation is done with post cost returns, but then weights aren't adjusted afterwards<br /><br />Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-13335908868310466592017-04-25T14:35:46.333+01:002017-04-25T14:35:46.333+01:00Rob,
Just trying to get my head around the differ...Rob,<br /><br />Just trying to get my head around the difference between<br />i) Calculate weights using gross returns, and adjust subsequently for cost levels<br />apply_cost_weight: True<br />cost_multiplier: 0<br /><br />ii) using half-pooling<br />apply_cost_weight: False<br />cost_multiplier: 1.0<br /><br />I only vaguely understand the difference and would appreciate more clarity!<br /><br />Many thanks<br /><br />ChrisCJhttps://www.blogger.com/profile/06817263570026256611noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-59729354372798372632017-04-18T21:30:23.034+01:002017-04-18T21:30:23.034+01:00Yes this make sense and it's an interesting id...Yes this make sense and it's an interesting idea. I will try this out.Krishttps://www.blogger.com/profile/17919667101415157462noreply@blogger.com