tag:blogger.com,1999:blog-261139923818144971.post1042630594194607814..comments2024-03-18T14:17:52.095+00:00Comments on This Blog is Systematic: Correlations, Weights, Multipliers.... (pysystemtrade)Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comBlogger153125tag:blogger.com,1999:blog-261139923818144971.post-71284101204480255262023-11-04T12:04:44.374+00:002023-11-04T12:04:44.374+00:00It would be much easier for me if you joined my ET...It would be much easier for me if you joined my ET thread, here https://www.elitetrader.com/et/threads/fully-automated-futures-trading.289589/ It's rather clunky to comment here and there is a community of people who can answer your questions if I'm busy. Anyway, the answers are (1) No, you should use the position sizing and forecasts, (2) no or expanding window bootstrapping, I'd actually pick 256 days with replacement after one year and 512 after two years and son on.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-87334900744412148412023-11-03T20:59:34.195+00:002023-11-03T20:59:34.195+00:00Thanks a lot for your reply and I'm looking fo...Thanks a lot for your reply and I'm looking forward to buying your new book once you release it. <br />I am planning to read the Advanced Futures Trading Strategies as soon as I get a good grasp on Systematic Trading.<br /><br />Just to double check I understand the logic correctly:<br />i) When I use p&l for each trading rule to calculate forecast weights, I don't use the forecasts(and position sizing) of trading rules and just use fixed position based on entry and exit signals of each rule. Is that correct?<br />ii) In bootsrapping with expanding window method, if the bootstrap length=100, do you pick random 100 continuous days from year 1 of your data in the first run and then another 100 continious random days from year 1 + 2 from your data on the next run? Is that logic is correct?<br /><br />Many thanks again Rob, I really appreciate your help.Umuthttps://www.blogger.com/profile/02341308088558711070noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-79056507011693546972023-11-03T08:57:11.365+00:002023-11-03T08:57:11.365+00:00i) yes
ii) You could do that (A), or you could boo...i) yes<br />ii) You could do that (A), or you could bootstrap the weights for each instrument and then take an average (B), or you could do (C) a mixture this: a 'run' is a single set of returns for a single instrument, and you take the average of those runs so you're basically selecting both instruments and data each time. Different methods have pros and cons, they will or will not give more weight to instruments with more data, they are more or less robust. I can't fit a full discussion here, there's another book in the works in a few years that might cover it... but I use CRob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-59242279977143889872023-11-02T22:45:25.032+00:002023-11-02T22:45:25.032+00:00Hi Rob,
I'm not sure if you are still monitor...Hi Rob,<br /><br />I'm not sure if you are still monitoring this space but regardless, I really appreciate your sharing your wisdom with us here. I've been reading your book, Systematic Trading, for the last couple of weeks with great admiration for your work. <br /><br />I'm struggling with calculating the forecast weights with bootstrapping. I've managed to generate scaled forecasts for my trading rules. now I'm at the stage of combining forecasts.<br /><br />My question is:<br />i) Do I use p&l for each trading rule and try to optimise the profit in order to calculate forecast weights? <br />ii)If I use bootstrapping with pooling, for each run, do I calculate different set of weights per instrument that I'm pooling and get the average of these weights on each run and get a final average in the end to calculate the final weights? <br /><br />Your help would be much appreciated.<br />Many thanks.Umuthttps://www.blogger.com/profile/02341308088558711070noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-55423030485352519702022-07-13T14:57:53.402+01:002022-07-13T14:57:53.402+01:00Got it. Thanks!Got it. Thanks!Sam Fisherhttps://www.blogger.com/profile/08714568798621251488noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-81193995020510207302022-07-13T14:38:07.286+01:002022-07-13T14:38:07.286+01:00Yes, YesYes, YesRob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-86931508403567839012022-07-13T13:50:54.536+01:002022-07-13T13:50:54.536+01:00Hi Rob, I really appreciate that you're still ...Hi Rob, I really appreciate that you're still replying to a blog written six years ago. So if you can see my comment, here is one question:<br /><br />Am I understanding correctly that in the combined forecast stage you are using two different correlations?:<br />1) For the forecast weights calculation, be it bootstrapping or shrinkage, you use the correlation matrix calculated from the PERFORMANCE curves; and<br />2) For the fdm calculation, you use the correlation matrix calculated from the FORECAST values.<br />(the capitalization is supposed to be highlighting, instead of yelling, sorry about that)<br /><br />Thank you very much!Sam Fisherhttps://www.blogger.com/profile/08714568798621251488noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-74444007191960941012022-04-26T13:19:13.139+01:002022-04-26T13:19:13.139+01:00I should have checked your code and blog before I ...I should have checked your code and blog before I asked.<br />I check the function get_scaled_forecast and found that forecast_scalar is a time series ,we only use the scalar that was available at that time,avoid forward looking.<br />Thanks for your reply!Wengwghttps://www.blogger.com/profile/18304759125820033277noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-88816403933726711052022-04-26T08:51:25.454+01:002022-04-26T08:51:25.454+01:00I don't bootstrap forecast scalars, I just est...I don't bootstrap forecast scalars, I just estimate them using an expanding window of all historic data.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-61314603797675921012022-04-26T06:09:13.236+01:002022-04-26T06:09:13.236+01:00Hi Rob,I have a question about estimating forecast...Hi Rob,I have a question about estimating forecast weight with bootstrapping.We need expanding window performance curve for each rule variation for each instrument, right? (with some abstract notional capital and volatility target, etc). And first,we calculate forecasts then scale and limit them to -20 ~ +20.<br />My question is,how we get forecast scalars? If we use all sample data to estimate forecast scalars,then make backtest on these sample,It seems like we are using something from future,because we wouldn't actually know these "latest" forecast scalars at the beginning of the backtest.<br />Because we limit forecast to -20 ~ +20,performance curve would change if we use different forecast scalar.I know that forecast scalars seems change little over years,maybe it's not a big problem?<br />Apologies if I misunderstood.Thank you!Wengwghttps://www.blogger.com/profile/18304759125820033277noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-10337560375120008852021-03-03T10:57:48.284+00:002021-03-03T10:57:48.284+00:00Hi Michael. The hump effect you describe is spot o...Hi Michael. The hump effect you describe is spot on; and is also confused somewhat by costs which make the faster lookbacks look worse. In the plots I did indeed include the mean as well as correlations. However my thinking on this subject has developed some more, so it might be worth reading the series of posts:<br />https://qoppac.blogspot.com/2018/12/portfolio-construction-through.html (first post, read the series)<br />Follow ups:<br />https://qoppac.blogspot.com/2019/12/new-and-improved-sharpe-ratio.html<br /><br />https://qoppac.blogspot.com/2020/11/improving-use-of-correlations-in.html<br />Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-23294793216398938632021-03-02T19:29:10.138+00:002021-03-02T19:29:10.138+00:00Above, when you are weighting the different signal...Above, when you are weighting the different signals (carry and lookbacks), are you accounting for both Sharpe and correlation or just correlation?<br />It would seem that the correlations would favor the shortest and longest lookback since they are more diversifying but in theory the middle lookback should have a higher Sharpe because that should be the hump of a curve with Sharpe falling off as you approach mean reversion as the lookback gets too short or too long.Michael Newtonhttps://www.blogger.com/profile/15187744343817723912noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-40256663874388174932021-01-13T14:54:23.451+00:002021-01-13T14:54:23.451+00:00Hi Chad. I've played with stuff like this http...Hi Chad. I've played with stuff like this https://core.ac.uk/download/pdf/162459222.pdf but not used it in anger. I doubt it would be much use in my standard framework, where the correlation matrix is the returns of trading strategies, which are relatively stable and pretty linear. It will probably fare better if the correlation matrix is of underlying returns, as you'd use in a classical portfolio optimisation. Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-72793534491311213372021-01-13T14:50:32.343+00:002021-01-13T14:50:32.343+00:00Hi Rob, have you experimented with conditional pro...Hi Rob, have you experimented with conditional probability (e.g. Bayesian) as a substitute for standard correlation measures?<br /><br />I've read elsewhere that "Correlation causes many conceptual misinterpretations, especially related to causal structures."Chad Bhttps://www.blogger.com/profile/13026562498196984544noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-11135796554220273602020-09-11T10:38:41.664+01:002020-09-11T10:38:41.664+01:00You need to have some weights in the backtest, and...You need to have some weights in the backtest, and the only honest way of doing this is to do a backward looking window. Makes sense to update this in the backtest, as there isn't much data to begin with. But at some point the weights are pretty stable and there is almost no value in changing them. So that's why I don't bother in live trading.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-38349632524945180872020-09-11T04:37:15.611+01:002020-09-11T04:37:15.611+01:00Thanks Rob. Interesting, so that you don't do ...Thanks Rob. Interesting, so that you don't do bootstrap even after having, say, a whole new year data on Live trading? Then what's the point of rolling-window backtest....The optimizing process in live trading doesn't take new data and giving up out of date market conditions.I am bit lost here....SmileIChttps://www.blogger.com/profile/00748503144341748295noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-51911628158832701492020-08-20T13:11:27.485+01:002020-08-20T13:11:27.485+01:00I don't update my weights at all in live tradi...I don't update my weights at all in live trading, but it's more appropriate to do so in simulation since there are markets coming into the system, also to begin with there isn't much data to work off. The smoothing is there so that there aren't massive transaction costs when the weights change.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-69209359046170796882020-08-14T08:52:55.485+01:002020-08-14T08:52:55.485+01:00Hi Rob,
just on top of this interesting post, 1. d...Hi Rob,<br />just on top of this interesting post, 1. do you have any insights on what's the best frequency to update forecast and subsystem weights using bootstrap? 2.if i understand correctly, in live trading, the bootstrap weights(both forecast and subsystem) are updated in discrete way, then why there is smooth method applied in the backtest which might not be the best estimation of live trading?SmileIChttps://www.blogger.com/profile/00748503144341748295noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-67047648177315311512020-06-30T09:27:29.031+01:002020-06-30T09:27:29.031+01:00"https://www.barchart.com/futures/quotes/FXU2..."https://www.barchart.com/futures/quotes/FXU20/overview" it says '10 times index'. Basically if the index moves by 1 point (eg from 3370 to 3371) you will earn or lose 10 euros. Each future has a different multiplier and they are (nearly always) constant. However the value of a 1% move in the index (which will be point value * price * 1%) will change as the price changes.<br /><br />Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-63914399621253166582020-06-29T22:41:41.087+01:002020-06-29T22:41:41.087+01:00Hi Rob! Just have a question about the chapter 15....Hi Rob! Just have a question about the chapter 15. I am using your spreadsheets (https://www.systematicmoney.org/systematic-trading-resources) but I have one big question. Maybe is something silly but I can't figure it out.<br /><br />When you talk about "Each point is worth (c)" in the trading dairy I don't know where does this come from. In your book it just says: <**For Euro Stoxx a point move in the futures price cost 10 Euros>.<br />But 1% of a future worth 3370 is 33.70...<br />And then I donĀ“t understand why is the point (c) -in the trading diary- constant- on October 2014 and on December 2014 - in the spreadsheet as the price (b) changes. <br /><br />Thankyour very muchAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-85457142612695635492020-06-24T15:57:06.750+01:002020-06-24T15:57:06.750+01:00Just bought your book about a week ago. Really use...Just bought your book about a week ago. Really useful stuff Rob. Really apreciate it. Just getting started with trading. Hopefully will make some things work out.<br /><br />Thanks!!!<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-21926470030953539122020-03-16T08:38:56.774+00:002020-03-16T08:38:56.774+00:00Unequal length: yes, you have to truncate
Asynchro...Unequal length: yes, you have to truncate<br />Asynchrous: Use a lower frequency. I use weekly returns.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-64012871048231935562020-03-15T18:00:38.481+00:002020-03-15T18:00:38.481+00:00Hi Rob,
What might be a legitimate way to deal wi...Hi Rob,<br /> What might be a legitimate way to deal with unequal-length and/or asynchronous time series, specifically for correlation calculations? My brief search yielded the notion of truncating the longer series to match the shorter one.<br /><br />This could be applicable to cross-correlations for say NQ-100 versus DAX or Hang Seng Index futs, but also for markets on the same continent but different session hours & durations (e.g. ICE Coffee vs WTI Crude).Chad Bhttps://www.blogger.com/profile/13026562498196984544noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-21911213999989529942019-11-26T13:18:55.467+00:002019-11-26T13:18:55.467+00:00These are money terms not %
system.accounts.portf...These are money terms not %<br /><br />system.accounts.portfolio().percent().stats()<br /><br />.... will give you more reasonable answers.<br /><br />Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-72116556840185435242019-11-25T23:44:21.820+00:002019-11-25T23:44:21.820+00:00Robert, I bought your book recently and it's s...Robert, I bought your book recently and it's so good I can't put it down.<br /><br />Hope I'm not too late to the dance for this blog post.<br /><br />At present I'm trying to replicate the code to produce the final graph showing the Sharpe returns.<br /><br />I get a similar graphing pattern: https://ibb.co/QHYyMqt<br /><br />But notice the x-axis has 1e6. Large numbers like this also exist in the code output:<br /><br />[[('min', '-9.796e+04'), ('max', '2.721e+04'), ('median', '86.45'), ('mean', '134.6'), ('std', '3899'), ('skew', '-2.784'), ('ann_mean', '3.445e+04'), ('ann_std', '6.239e+04'), ('sharpe', '0.5522'), ('sortino', '0.6348'), ('avg_drawdown', '-5.859e+04'), ('time_in_drawdown', '0.9573'), ('calmar', '0.1773'), ('avg_return_to_drawdown', '0.588'), ('avg_loss', '-2465'), ('avg_gain', '2566'), ('gaintolossratio', '1.041'), ('profitfactor', '1.113'), ('hitrate', '0.5167'), ('t_stat', '3.439'), ('p_value', '0.000586')], ('You can also plot / print:', ['rolling_ann_std', 'drawdown', 'curve', 'percent', 'cumulative'])]<br /><br />Notice Sharpe ratio looks right but ann_std and other values are massive numbers.<br /><br />1. Is there a full code example that shows this particular Sharpe graph and I've been stupid and missed it?<br />2. If not, is this likely due to me being unable to configure the system.config.forecast_correlation_estimate["func"] setting? [I get error AttributeError: module 'syscore' has no attribute 'correlations' if I uncomment that config line]<br />3. If not, Is it something else I'm missing from my code?<br /><br />I know you're a very busy man, so I won't hold it against you if don't have time to respond. <br /><br />But I'm so very close that any help, even a sentence or two response that points me in the right direction would be great.<br /><br />Here is my code:<br /><br />from matplotlib.pyplot import show, title<br />from systems.provided.futures_chapter15.estimatedsystem import futures_system<br />import syscore<br /><br />system=futures_system()<br /><br />system.config.forecast_weight_estimate["pool_instruments"]=True<br />system.config.forecast_weight_estimate["method"]="bootstrap"<br />system.config.forecast_weight_estimate["equalise_means"]=False<br />system.config.forecast_weight_estimate["monte_runs"]=200<br />system.config.forecast_weight_estimate["bootstrap_length"]=104<br />system.config.forecast_weight_estimate["ewma_span"]=125<br />system.config.forecast_weight_estimate["cleaning"]=True<br /><br />system.config.forecast_correlation_estimate["pool_instruments"]=True<br /># system.config.forecast_correlation_estimate["func"]=syscore.correlations.CorrelationEstimator<br />system.config.forecast_correlation_estimate["frequency"]="W"<br />system.config.forecast_correlation_estimate["date_method"]="expanding"<br />system.config.forecast_correlation_estimate["using_exponent"]=True<br />system.config.forecast_correlation_estimate["ew_lookback"]=250<br />system.config.forecast_correlation_estimate["min_periods"]=20<br /><br />system.config.forecast_div_mult_estimate["ewma_span"]=125<br /># system.config.forecast_div_mult_estimate["floor_at_zero"]=True<br /><br />system=futures_system(config=system.config)<br /><br />print(system.accounts.portfolio().stats())<br /><br />system.accounts.portfolio().cumsum().plot()<br /><br />show()<br />Anonymoushttps://www.blogger.com/profile/14176815151771158773noreply@blogger.com