tag:blogger.com,1999:blog-261139923818144971.post1042630594194607814..comments2020-08-08T07:51:50.543+01:00Comments on This Blog is Systematic.: Correlations, Weights, Multipliers.... (pysystemtrade)Rob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comBlogger136125tag:blogger.com,1999:blog-261139923818144971.post-67047648177315311512020-06-30T09:27:29.031+01:002020-06-30T09:27:29.031+01:00"https://www.barchart.com/futures/quotes/FXU2..."https://www.barchart.com/futures/quotes/FXU20/overview" it says '10 times index'. Basically if the index moves by 1 point (eg from 3370 to 3371) you will earn or lose 10 euros. Each future has a different multiplier and they are (nearly always) constant. However the value of a 1% move in the index (which will be point value * price * 1%) will change as the price changes.<br /><br />Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-63914399621253166582020-06-29T22:41:41.087+01:002020-06-29T22:41:41.087+01:00Hi Rob! Just have a question about the chapter 15....Hi Rob! Just have a question about the chapter 15. I am using your spreadsheets (https://www.systematicmoney.org/systematic-trading-resources) but I have one big question. Maybe is something silly but I can't figure it out.<br /><br />When you talk about "Each point is worth (c)" in the trading dairy I don't know where does this come from. In your book it just says: <**For Euro Stoxx a point move in the futures price cost 10 Euros>.<br />But 1% of a future worth 3370 is 33.70...<br />And then I donĀ“t understand why is the point (c) -in the trading diary- constant- on October 2014 and on December 2014 - in the spreadsheet as the price (b) changes. <br /><br />Thankyour very muchAnonymoushttps://www.blogger.com/profile/10308018794377992891noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-85457142612695635492020-06-24T15:57:06.750+01:002020-06-24T15:57:06.750+01:00Just bought your book about a week ago. Really use...Just bought your book about a week ago. Really useful stuff Rob. Really apreciate it. Just getting started with trading. Hopefully will make some things work out.<br /><br />Thanks!!!<br />Anonymoushttps://www.blogger.com/profile/10308018794377992891noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-21926470030953539122020-03-16T08:38:56.774+00:002020-03-16T08:38:56.774+00:00Unequal length: yes, you have to truncate
Asynchro...Unequal length: yes, you have to truncate<br />Asynchrous: Use a lower frequency. I use weekly returns.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-64012871048231935562020-03-15T18:00:38.481+00:002020-03-15T18:00:38.481+00:00Hi Rob,
What might be a legitimate way to deal wi...Hi Rob,<br /> What might be a legitimate way to deal with unequal-length and/or asynchronous time series, specifically for correlation calculations? My brief search yielded the notion of truncating the longer series to match the shorter one.<br /><br />This could be applicable to cross-correlations for say NQ-100 versus DAX or Hang Seng Index futs, but also for markets on the same continent but different session hours & durations (e.g. ICE Coffee vs WTI Crude).Chad Bhttps://www.blogger.com/profile/13026562498196984544noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-21911213999989529942019-11-26T13:18:55.467+00:002019-11-26T13:18:55.467+00:00These are money terms not %
system.accounts.portf...These are money terms not %<br /><br />system.accounts.portfolio().percent().stats()<br /><br />.... will give you more reasonable answers.<br /><br />Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-72116556840185435242019-11-25T23:44:21.820+00:002019-11-25T23:44:21.820+00:00Robert, I bought your book recently and it's s...Robert, I bought your book recently and it's so good I can't put it down.<br /><br />Hope I'm not too late to the dance for this blog post.<br /><br />At present I'm trying to replicate the code to produce the final graph showing the Sharpe returns.<br /><br />I get a similar graphing pattern: https://ibb.co/QHYyMqt<br /><br />But notice the x-axis has 1e6. Large numbers like this also exist in the code output:<br /><br />[[('min', '-9.796e+04'), ('max', '2.721e+04'), ('median', '86.45'), ('mean', '134.6'), ('std', '3899'), ('skew', '-2.784'), ('ann_mean', '3.445e+04'), ('ann_std', '6.239e+04'), ('sharpe', '0.5522'), ('sortino', '0.6348'), ('avg_drawdown', '-5.859e+04'), ('time_in_drawdown', '0.9573'), ('calmar', '0.1773'), ('avg_return_to_drawdown', '0.588'), ('avg_loss', '-2465'), ('avg_gain', '2566'), ('gaintolossratio', '1.041'), ('profitfactor', '1.113'), ('hitrate', '0.5167'), ('t_stat', '3.439'), ('p_value', '0.000586')], ('You can also plot / print:', ['rolling_ann_std', 'drawdown', 'curve', 'percent', 'cumulative'])]<br /><br />Notice Sharpe ratio looks right but ann_std and other values are massive numbers.<br /><br />1. Is there a full code example that shows this particular Sharpe graph and I've been stupid and missed it?<br />2. If not, is this likely due to me being unable to configure the system.config.forecast_correlation_estimate["func"] setting? [I get error AttributeError: module 'syscore' has no attribute 'correlations' if I uncomment that config line]<br />3. If not, Is it something else I'm missing from my code?<br /><br />I know you're a very busy man, so I won't hold it against you if don't have time to respond. <br /><br />But I'm so very close that any help, even a sentence or two response that points me in the right direction would be great.<br /><br />Here is my code:<br /><br />from matplotlib.pyplot import show, title<br />from systems.provided.futures_chapter15.estimatedsystem import futures_system<br />import syscore<br /><br />system=futures_system()<br /><br />system.config.forecast_weight_estimate["pool_instruments"]=True<br />system.config.forecast_weight_estimate["method"]="bootstrap"<br />system.config.forecast_weight_estimate["equalise_means"]=False<br />system.config.forecast_weight_estimate["monte_runs"]=200<br />system.config.forecast_weight_estimate["bootstrap_length"]=104<br />system.config.forecast_weight_estimate["ewma_span"]=125<br />system.config.forecast_weight_estimate["cleaning"]=True<br /><br />system.config.forecast_correlation_estimate["pool_instruments"]=True<br /># system.config.forecast_correlation_estimate["func"]=syscore.correlations.CorrelationEstimator<br />system.config.forecast_correlation_estimate["frequency"]="W"<br />system.config.forecast_correlation_estimate["date_method"]="expanding"<br />system.config.forecast_correlation_estimate["using_exponent"]=True<br />system.config.forecast_correlation_estimate["ew_lookback"]=250<br />system.config.forecast_correlation_estimate["min_periods"]=20<br /><br />system.config.forecast_div_mult_estimate["ewma_span"]=125<br /># system.config.forecast_div_mult_estimate["floor_at_zero"]=True<br /><br />system=futures_system(config=system.config)<br /><br />print(system.accounts.portfolio().stats())<br /><br />system.accounts.portfolio().cumsum().plot()<br /><br />show()<br />Anonymoushttps://www.blogger.com/profile/14176815151771158773noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-32981823584104498332018-02-28T08:57:52.974+00:002018-02-28T08:57:52.974+00:00I guess you could try using goal seek instead of M...I guess you could try using goal seek instead of MINVERSE but not sure if it works with constraints. Sorry optimisation in excel is a pretty ugly thing that I have always tried to avoid doing.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-1679854388917102392018-02-27T21:22:53.437+00:002018-02-27T21:22:53.437+00:00I did it in Excel by inverting the correlation mat...I did it in Excel by inverting the correlation matrix using MINVERSE. It seems strange, but the correlations were all positive, yet some of the weights were negative. For example, I think with three assets, all with standard deviation of 0.1, and correlations of AvB=0.85, AvC=0.25 and BvC=0.5 I calculate that A & C have positive weights (69% and 57%) and B has a negative weight (-26%). I have the constraint that the sum of weights = 100%, but I haven't worked out yet if I can incorporate minimum weights into the matrix approach.Simonhttps://www.blogger.com/profile/02630148937587886911noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-29021020609216272852018-02-27T18:34:44.592+00:002018-02-27T18:34:44.592+00:00Depends on what optimiser you are using, but most ...Depends on what optimiser you are using, but most should allow you to set positive weights as a constraint. If you're assuming same vol and same sharpe ratio then I think (correct me if I am wrong) you will only get negative weights if you have negative correlations, so you could also try flooring these at zero.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-86291996733369908172018-02-27T17:57:29.993+00:002018-02-27T17:57:29.993+00:00This comment has been removed by a blog administrator.Simonhttps://www.blogger.com/profile/02630148937587886911noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-39469107318262005002018-02-27T17:56:32.192+00:002018-02-27T17:56:32.192+00:00Dear Rob, Thanks for sharing your work in your boo...Dear Rob, Thanks for sharing your work in your books and via the website. I have a question regarding the weights of the trading rules. I am interested in seeing how closely the handcrafted weights would compare to the minimum variance portfolio weights using correlations (and assuming the same volatility for each rule). I have been trying to take some examples of correlation tables (representing a possible set of trading rules) where I have assigned correlations ranging between 0 and 1, typically 0.5 to 0.75. I then calculate the minimum variance weights by inverting the covariance matrix (or correlation matrix as vols are the same). When I calculate the weights I find several have a negative weight. This doesn't make sense in the context of rules as we would not 'short' a trading rule - instead we would discard it or reverse it. Is there an easy way to adjust for the constraint of needing all weights >+ zero? Many thanks, SimonSimonhttps://www.blogger.com/profile/02630148937587886911noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-43524637914424347372018-02-20T09:05:30.820+00:002018-02-20T09:05:30.820+00:00It is appealing, but it depends on what your prior...It is appealing, but it depends on what your priority is. I am interested in expected risk being close to realised risk. Measuring realised risk with something that only uses the tails is subject to more parameter uncertainty so it's harder to know if you're doing it right. But if you're trading something highly non normal (like XIV) with too much leverage (like ... XIV) then it's worth doingRob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-36020253117107178502018-02-19T20:50:24.074+00:002018-02-19T20:50:24.074+00:00Hi Rob, do you have any perspective of the (non)us...Hi Rob, do you have any perspective of the (non)usefulness of using a different volatility measure than stddev for Sharpe (e.g. CVaR), consistent with a 95% CVaR as part of a 'Modified Sharpe'? Seems this would have more appeal when one is more concerned about the tails of the non-normal returns, esp with the volatility products like VIX.Chad Bhttps://www.blogger.com/profile/13026562498196984544noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-32886975353276565422017-09-04T10:34:21.507+01:002017-09-04T10:34:21.507+01:00Understood. And thank you again, Rob.Understood. And thank you again, Rob.Patrickhttps://www.blogger.com/profile/01034323689613977187noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-84865759157228580122017-09-04T09:28:58.609+01:002017-09-04T09:28:58.609+01:00Patrick.
Essentially the question is what is the...Patrick. <br /><br />Essentially the question is what is the difference between the correlation of forecast RETURNS rather than forecast VALUES. If returns are more correlated than values, the FDM will be too high (producing a trading rule forecast that has too much volatility). And vice versa. In fact the answer isn't obvious and depends very much on the properties of the signal and the underlying market.<br /><br />At the end of the day I think it's more intuitive to state the FDM in terms that it's ensuring the final combined forecast VALUE has the correct scaling properties. FDM would have to be way off for me to drop this point of view. And it usually is pretty close - in fact any effect is dominated by the problem that volatility and correlations aren't stable enough, which means that trying to hit a volatility target is an imprecise science at best. <br /><br />"Would I pool all instrument data if they didn't share the same rule variations" - ideally yes, but it's very complicated to do this - essentially the problem being to average a series of correlation matrices with some overlapping and non overlapping elements, and produce a well defined matrix at the end. There are techniques to do this, but it strikes me as overkill to bother...Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-14862211563532115332017-09-03T17:31:43.092+01:002017-09-03T17:31:43.092+01:00Many thanks Rob. The more I dig into your system, ...Many thanks Rob. The more I dig into your system, the more I appreciate the serious thought which has gone into engineering it. A couple of follow-up observations. From my comparatively low vantage point, I also see that from a practical point of view certain simplifications are desirable (at very little cost to robustness). Would you say that is the case with the FDM? Strictly, if it made a big difference, should we be adjusting for reduced portfolio volatility of forecast returns rather than forecasts themselves? Also, wrt FDMs would you pool all instrument data whether or not they share the same rule variations after costs? Patrickhttps://www.blogger.com/profile/01034323689613977187noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-27756858460903760172017-09-02T14:05:24.023+01:002017-09-02T14:05:24.023+01:00You've got it. Couple of interesting points fr...You've got it. Couple of interesting points from what you've said though:<br /><br />- the fact that a forecast for carry comes out as mu/sigma is a nice property, but in general we only require that raw forecasts are PROPORTIONAL to mu/sigma. So some further correcting factor may be necessary (the forecast scalar)<br /><br />- in terms of multiple rules; obviously if weights added up to 1 and rules were 100% correlated you'd end up with a joint forecast with exactly the same properties as the individual forecasts. In theory the forecast diversification multiplier deals with this problem; with the caveat that it's backward looking and based on historic correlations so it only works in expectation.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-702370039248608332017-09-02T13:47:42.422+01:002017-09-02T13:47:42.422+01:00from systems.provided.futures_chapter15.basesystem...from systems.provided.futures_chapter15.basesystem import futures_system<br />from matplotlib.pyplot import show<br /><br />system = futures_system(log_level="on")<br />a=system.combForecast.get_all_forecasts("EUROSTX")<br />b=system.combForecast.get_forecast_weights("EUROSTX")<br />c=a*b<br />c.plot()<br />https://drive.google.com/file/d/0B2xHDlIRSeeXZXZjenU1QlRaQkk/view?usp=sharingRob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-35431593010455014332017-08-31T13:08:15.964+01:002017-08-31T13:08:15.964+01:00Hi Rob,
Do you have the breakdown of subsystem si...Hi Rob,<br /><br />Do you have the breakdown of subsystem signals for the Eurostoxx? You never get short in 2015, only less long? It looks like the market heads down quite a bit. Is this because of the carry signal dwarfing the trend signal? Optically, I can't line up the forecast weights with the chart.<br /><br />Thanks!Matthttps://www.blogger.com/profile/16122419489436306940noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-55295740347327988002017-08-30T19:35:55.430+01:002017-08-30T19:35:55.430+01:00OK I think get it. Really had to have a think and ...OK I think get it. Really had to have a think and run some simulations but as far as I can tell there seem to be two effects in play here. The arithmetic returns from applying a rule on an instrument is the product of two rvs: instrument returns and forecasts. Assuming independence between these rvs, the variance can be shown to be function of their first two moments. Over sufficiently long periods, these moments across different instruments are equal (asymptotically converge). However over shorter periods there may be divergence (i.e. different averages, different vols) which will violate the assumption of equal vols required to be able to run the optimiser using correlations only. As far as I can tell there also is a more subtle effect, and that arises from the fact that forecasts and instrument returns are not independent (EWMAC 2,8 and daily returns when using random gaussian data have a correlation of 45%). This inconveniently introduces covariance terms in the calc for vol. However, in the cross section of a single rule applied across different instruments over sufficiently long periods of time, the covar terms should have an equal effect across all instruments. Again over short periods there may be divergence. This divergence in small samples from the assumed distribution of the population is presumably why it is sensible to equalise vols before optimising. Am I on the right track?<br /><br />BTW please feel free to delete my earlier comment.Patrickhttps://www.blogger.com/profile/01034323689613977187noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-63334568998745555252017-08-28T13:08:36.138+01:002017-08-28T13:08:36.138+01:00Hi Rob, it's possible I worded my original que...Hi Rob, it's possible I worded my original question poorly, <br /><br />I will try to be more systematic, so please bear with me. To recap on my understanding:<br /><br />1.'vol normalisation' is what you do when standardising forecasts. This is typically done by dividing by rolling 36 day EWMA * current price<br />2. 'vol standardisation' is what you do when standardising subsystems. I would again use rolling 36 day EWM vol (times block value, etc) for this<br />3. 'vol equalisation' is what you do prior to optimisation to scale the returns over the entire (expanding) window so over this window they have the same volatility<br /><br />4. Assuming the above is correct, a subsystem position for carry and EWMAC variation is proportional to exp return/vol^2 (which co-incidentally seems to be proportional to optimal kelly scale - although not for the breakout rule).<br /><br />5. When I said originally 'Now when using pooled data for forecasts, my thinking is fuzzier: is it advisable not to equalise means or vol?', to be clearer I was trying to ask whether it makes sense to equalise vols prior to optimising forecast weights when pooling (not whether to equalise vols when optimising subsystem weights, if we had pooled forecasts previously). In a previous post ('a little demonstration of portfolio optmisation') you do an asset vol 'normalisation' which I believe is the same as 'equalisation' discussed here (scale the whole window, although not done on an expanding window) but I got the impression for forecasts that the normalisation is handled as above and this took care of the need for further equalisation (for forecasts at least).<br /><br />I must admit, I had always thought that if you want to use only correlations and means to optimise then intuitively you should equalise vols in the window being sampled (because to quote you this reduces the covar matrix to a correlation matrix). However I had somehow accepted the fact that normalising forecasts by recent vol ended up doing something similar (also from reading some comments by you about not strictly needing to equalise vols for forecasts, etc). But I guess a different issue arises when pooling short histories?<br /><br />In summary, assuming you deciphered my original question correctly, are you saying it is still important to equalise vols of forecast as the 'realised' variance of forecast returns are proportional to the the level of the forecast (so a forecast of 10 would have twice the variance of a forecast of 5), causing the optimiser to downweight elevated forecasts, which is a problem when pooling short data histories? By equalising vols over the entire window being used for optimisation, we end up removing this effect? If that is what you are saying then I promise to go away and think about this much more deeply.<br /><br />Thanks again for taking the time.Patrickhttps://www.blogger.com/profile/01034323689613977187noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-56805964771270184932017-08-27T12:46:14.613+01:002017-08-27T12:46:14.613+01:00There is rolling historical 36d EWMA vol used for ...<br />There is rolling historical 36d EWMA vol used for position sizing. Realised vol is what actually happens in the future. <br /><br />When doing optimisation we use a different estimated vol - the vol of returns over a given period (the entire backtest if using an expanding window).<br /><br />The vol used for vol standardisation is the estimated vol of the p&l of the instrument subsystem returns. If all forecasts were identical, and we could predict vol perfectly, then this would automatically be the same for all instruments (because when we scale positions for instrument subsystems we target the same volatility per unit forecast). The fact that estimated vol isn't a perfect forecast adds some randomness to this; a randomness that will be problematic for very short data histories. More problematic again is that for instruments with short histories they will have different forecasts. An instrument with an average forecast of 10 over a year compared to another with an average of 5 will have twice the estimated volatility over that backtest period. But that is an artifact of the different forecast level - it doesn't mean we should actively downweight the higher forecast instrument.<br /><br />I agree I could have been stricter with the terminology I use about equalisation, normalisation and standardisation. There are at least three things going on, which are subtly different:<br /><br />a) many forecasts involve dividing something by estimated vol to get a forecast that is proportional to sharpe ratio (normalisation)<br />b) position scaling involves using estimated vol and assuming it is a good forecast of future vol <br />c) equalisation of standard deviation estimates when doing optimisation for all the reasons I've discussed.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-19274349934839063982017-08-25T15:08:28.178+01:002017-08-25T15:08:28.178+01:00Hi Rob thanks for this. Just so I can get my head ...Hi Rob thanks for this. Just so I can get my head around your answer a bit better, a question on terminology: are 'estimated' vol and 'realised' vol the same thing and equal to the vol used for standardisation (i.e. rolling historic 36d EWM vol)? As I understand it the two inputs into the the optimisation you do are correlation and mean returns. So are you saying that if we relied merely on vol standardisation (using recent realised vol) then a period of high vol for an instrument with a short data history but high forecasts would lower the forecasts and their corresponding weights? I am failing to make the connection between high forecasts and high price vol which is used for standardisation. I am sorry if I have completely missed the point.<br /><br />On a related point, and I should have asked this earlier: on page 289 of your book you recommend that prior to optimisation we should ensure 'returns have been vol normalised' I assume this is the same as 'equalisation' that you refer to in this post and not the same as standardisation (btw the term volatility normalised is in bold so perhaps your publishers might consider putting a reference in the glossary for future editions before your book becomes compulsory reading for our grandkids).<br /> Patrickhttps://www.blogger.com/profile/01034323689613977187noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-21908696567736658772017-08-25T09:34:33.162+01:002017-08-25T09:34:33.162+01:00You're correct that the solution relies entire...You're correct that the solution relies entirely on correlations in this case (in fact it's the optimal Sharpe, maximum return and the minimum variance solution).<br /><br />"Now when using pooled data for forecasts, my thinking is fuzzier: is it advisable not to equalise means or vol?"<br /><br />No, you should still equalise them. Basically the logic in all cases is this. Vol targeting equalises expected vol, but not realised vol which will still be different. If realised vol goes into the optimiser then it will have an effect on the weights, which we don't want. To take an extreme example if you have an instrument with a very short data history which happens to have a very strong forecast in that period then it's estimated vol will be unrealistically high and it will see it's weights downgraded unless we equalise vols.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.com