tag:blogger.com,1999:blog-261139923818144971.post1042630594194607814..comments2017-08-23T14:05:58.181+01:00Comments on Investment Idiocy: Correlations, Weights, Multipliers.... (pysystemtrade)Rob Carvernoreply@blogger.comBlogger110125tag:blogger.com,1999:blog-261139923818144971.post-26631427478077085002017-08-16T03:18:50.738+01:002017-08-16T03:18:50.738+01:00I have read chapter 14 again and it was helpful.
...I have read chapter 14 again and it was helpful.<br /><br />I should have checked the book before I asked.<br /><br />Thanks for reply.Minkyu Kimhttps://www.blogger.com/profile/12721284558759239431noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-48856436433494955842017-08-14T11:37:58.946+01:002017-08-14T11:37:58.946+01:00Solution 1 is correct. Chapter 14 actually discuss...Solution 1 is correct. Chapter 14 actually discusses exactly this problem so it might be worth (re) reading it.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-81444072621210162672017-07-30T17:56:08.654+01:002017-07-30T17:56:08.654+01:00Thank you, Gainz! I see that you have an additiona...Thank you, Gainz! I see that you have an additional folder in 'sysdata' and the 'price' CSVs are 15-minute entries. Can you comment on the adaptation scheme in the stages of the workflow? For example, looking at the /systems/rawdata.py script, only daily prices are mentioned. Did you go that far with changing the code to recognizing intraday, but left the names as 'daily', or is the actual adaptation still needed?Chad Bhttps://www.blogger.com/profile/13026562498196984544noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-4094523543436383862017-07-26T19:33:38.691+01:002017-07-26T19:33:38.691+01:00Just briefly looking at the intra-day posts in thi...Just briefly looking at the intra-day posts in this thread, I can say I spent a considerable amount of time last year making the program intra-day compatible (I was using tradestation data). It's been some time since I looked at this project, but I can refer you to where I left off (I believe it works, but I haven't had anybody review my work): https://github.com/anthonywise/pysystemtrade/tree/tscompare<br /><br />Hope this helpsWesay GAINZhttps://www.blogger.com/profile/05676332880263742262noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-27757168850486696812017-07-25T10:27:24.793+01:002017-07-25T10:27:24.793+01:00This comment has been removed by a blog administrator.Minkyu Kimhttps://www.blogger.com/profile/12721284558759239431noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-3245778459413082002017-07-25T10:26:25.388+01:002017-07-25T10:26:25.388+01:00Dear Mr. Carver,
Most of all I always appreciate ...Dear Mr. Carver,<br /><br />Most of all I always appreciate you for sharing detailed & practical knowledge of quantitative trading.<br /><br />I have a few questions while reading your book & blog posts.<br /><br />I am trying to develop a trend following trading system with ETFs using the framework in your book.<br /><br />The trading system is long-only and constrained by leverage limit(100%).<br /><br />Under the constraints, what is the best way to use your framework properly?<br /><br />Is there any changes in calculating forecast scalars, forecast weights, FDM, IDM, etc?<br /><br /><br />My thought is...<br /><br />Solution 1.<br />- Maintain all the procedures in your framework as if I could long/short and have no leverage limit. (Suppose that I have 15% target vol)<br /><br />- When I calculate positions for trading, I just assign zero position for negative forecasts.<br /><br />And if sum of long positions exceeds my capital I scale down the positions so that the portfolio could not be leveraged.<br /><br /><br />Solution 2.<br />- Forecast scalar:<br />No change. I calculate forecasts and scale them(-20 ~ + 20).<br /><br />- Forecast weights, Correlation: <br />For each trading rules,<br /> + Calculate portfolio returns of pooled instruments according to the forecasts.<br /> + Returns for negative forecasts replace to zeros. (Zero position instead of short)<br /> + And I scale down the returns for positive forecasts when sum of long positions exceeds my capital.<br /> + Returns of trading rules are used when bootstrapping or calculating correlations.<br /> + Forecast weights are optimized using these returns.<br />- FDM: <br />+ Calculate FDM based on forecast weights and correlations among the forecasts as your framework. <br />+ Calculate the historical participations(= sum(long position)/myCapital) using new rescaled forecasts and forecast weights.<br />+ Check the Median(participations) for back-tested period.<br />+ If it exceeds 100% I scaled down FDM in order to get my portfolio not take too much risk.<br /><br />Frankly speaking I don't know what the right ways are. Both ways does not seem proper. Maybe it is because of my lack of understading.<br /><br />Would you give any advice?<br /><br />I am really looking forward to your 2nd book. Thanks for reading.<br /><br />Best regards,<br /><br />Michael KimUnknownhttps://www.blogger.com/profile/12721284558759239431noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-3375776947819864092017-07-18T15:24:35.373+01:002017-07-18T15:24:35.373+01:00To be honest I haven't given this much thought...To be honest I haven't given this much thought. I can see why it will affect the calculation of volatility, but it's not obvious to me how it affects correlation.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-59266450265846578612017-07-18T13:41:44.563+01:002017-07-18T13:41:44.563+01:00Hi Rob, what is your view on using returns (weekly...Hi Rob, what is your view on using returns (weekly in your case) to compute correlations vs log returns (as per this AQR paper). Why do you suppose some choose log returns, do you view any significant difference between the two? I found this http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1586656 on the topic, but still not seeing the fundamental reason to use one vs the other.<br />Appreciate your thoughts as always.JMW100https://www.blogger.com/profile/18171394406805453830noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-73731722140996345712017-07-18T09:23:49.998+01:002017-07-18T09:23:49.998+01:00The answer to your question(s) is that I have not ...The answer to your question(s) is that I have not tested the system with non daily data so there is no guarantee it will work. I am pretty sure there are several places where I assume the data is daily (and you have unearthed one of them); some data is always resampled daily whilst others are not, so there could be some issues with mismatching and what not.<br /><br />The legacy data with multiple snapshots per day is an oversight (my code should resample to daily before burning the legacy data - that might not have happened on the version of the data you are using) - and indeed it may be causing some slightly unpredictable results in the last few years.<br /><br />In summary I would need to do a lot of testing before I was confident the code would work with non daily data, so I really wouldn't trust it for those purposes yet.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-12479780945510610322017-07-18T05:09:43.228+01:002017-07-18T05:09:43.228+01:00P.S. for example, does the diversification multipl...P.S. for example, does the diversification multiplier need to be modified for interpreting 1-minute periods instead of sampling at end-of-day? What about volatility scaling floors currently set with daily period?Chad Bhttps://www.blogger.com/profile/13026562498196984544noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-78803010302485970382017-07-18T05:07:24.820+01:002017-07-18T05:07:24.820+01:00Also, since many of the instruments in the legacy ...Also, since many of the instruments in the legacy data have a lot of days near the final years of the records with more than one recorded value per day, it seems that using new CSVs with intraday data would be feasible in short order, but making sure to change the period of several calculations in other stages to recognize the periodicity on a minute scale, instead of days, no?<br /> Sorry in advance for my hasty monologue...Chad Bhttps://www.blogger.com/profile/13026562498196984544noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-91209452641985264062017-07-18T05:04:20.191+01:002017-07-18T05:04:20.191+01:00Good morning, Rob.
When I run your ch_15 system a...Good morning, Rob.<br /> When I run your ch_15 system along with default configs, trading rules, etc. unmodified, the stages run fine. If I substitute the legacyCSV files of several instruments with *intraday* 1-minute bars in the same 2-column format, both for the '_price' file and '_carrydata' (expiration months spaced from current date like you showed in legacy versions), spanning 5 days each, and re-running the system but changing nothing except reduction of the instrument_list entries, I get the error from line 530 (get_notional_position) of /systems/portfolio.py "No rules are cheap enough for CRUDE_W with threshold of 1.300 SR units! Raise threshold (...), add rules, or drop instrument."<br /> I raised it from the original 0.13 to 1.3, and in other tests as high as 100 (ridiculous value of course, just testing...), same result. Seems I'm overlooking a simple principle of the system, but I can't figure why, given the trading rules were left same. Can you offer a pointer?Chad Bhttps://www.blogger.com/profile/13026562498196984544noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-53775798141350342972017-05-03T16:21:17.640+01:002017-05-03T16:21:17.640+01:00Yes, I agree. That was also my initial intuition a...Yes, I agree. That was also my initial intuition as well. I compared the weekly non-overlapping approach with the overlapping 3-day approach over the same time-frame and the markets that are synchronous the correlation estimates are very similar. More importantly, for example when I ran it on the Hang Seng the rolling 3-day approach was quite close to the weekly approach. So obviously some slight differences but, as you say, the approach doesn't seem like something crazy. <br /><br />Your response is much appreciated.OTNY33https://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-59024878569757776382017-05-03T16:02:21.190+01:002017-05-03T16:02:21.190+01:00The guys at AQR know what they are doing and almos...The guys at AQR know what they are doing and almost without exception are smarter than me. So it would be surprising if they'd done something crazy.<br /><br />Using overlapping returns is generally frowned upon (eg in OLS regression, essentially this is a bias versus variance problem). Using overlapping returns artifically increases the data you have (you only really have completely new observations every 3 days) and that benefit must come at a cost.<br /><br />For correlations it *might* be okay; I rarely work overlapping returns so I don't know their properties well enough to say whether they are fine or not. My intuition and a few minutes of playing with excel suggests they will be fine with zero autocorrelation, but maybe not if autocorrelation comes in.<br /><br />But I don't see the point in using two types of returns - one to calculate vol, one to calculate correlation (in most software you work out a single covariance matrix which embodies both things).Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-69907677826716621052017-05-03T15:44:42.085+01:002017-05-03T15:44:42.085+01:00Thank you! The idea actually came from the Betting...Thank you! The idea actually came from the Betting Against Beta paper by the guys at AQR. They say they use overlapping(or rolling) 3-day log returns to calculate correlations to control for non-synchronous trading over 120 trading days. <br /><br />I think it is safe to say your disagreeing with their approach? <br />OTNY33https://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-70673978724367075392017-05-02T09:37:47.017+01:002017-05-02T09:37:47.017+01:00If you're using daily closing prices to calcul...If you're using daily closing prices to calculate returns across different markets then nonsynchrous is clearly an issue. <br /><br />BUT NEVER, EVER, EVERY USE ROLLING RETURNS!!!!! They will result in your volatility estimate being understated.<br /><br />Instead use non overlapping 3 day returns eg P_3 - P_0, P_6 - P_3, P_9 - P_6 where P_t is the price on day t.<br /><br />As for wether 3 days is enough, well even 2 days would help a bit with nonsynchrous data, although 3 days is better, and 5 days (which is what I use, eg weekly returns if you're in business day space) is better still.<br /><br />On a system trading at the kind of speed I trade at using 60 days worth of correlations probably is too short a period, since it isn't long enough to give you a stable estimate. It's also only 20 observations if you're using 3 day non overlapping returns (it's even worse for weekly returns of course, only 12 observations). Your estimate will be very noisy.<br /><br />Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-16848503328442688822017-04-29T22:43:35.474+01:002017-04-29T22:43:35.474+01:00Rob,
Is one way to estimate correlations with non...Rob,<br /><br />Is one way to estimate correlations with nonsynchronous trading to run correlations on rolling 3-day returns over a lookback of 60-days?(which I know is much shorter than yours)OTNY33https://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-45737020909482062362017-04-21T20:19:26.286+01:002017-04-21T20:19:26.286+01:00Any resources you may have or know of on the subje...Any resources you may have or know of on the subject would be greatly appreciated though!!ESHKDhttps://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-60012069341468567992017-04-21T20:16:51.539+01:002017-04-21T20:16:51.539+01:00Gotcha. It still appears more right to me to set t...Gotcha. It still appears more right to me to set the buffer around % allocations(as opposed to contracts). So obviously I'm missing something. I don't want to exhaust you so thank you for your response!ESHKDhttps://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-24908403691190798482017-04-21T06:34:44.724+01:002017-04-21T06:34:44.724+01:00ESHKD
"Did you estimate the buffer purely on ...ESHKD<br />"Did you estimate the buffer purely on a backtest? eg tried a bunch of different buffers for a give set of costs and settled on 10%?"<br /><br />In fact there are well known analytical methods for deriving the optimal buffer size (we should always avoid fitting when we can), and 10% is a pretty conservative value which I chose to avoid the complexity of including yet another table in my book (it's what I use in my own system, regardless of cost, so this is a simplification I'm personally fine with).<br /><br />"Using this logic then, if I have 1 contract, the upper buffer is 1.1 contracts. So if my optimal number of contracts goes above 1.1(or 1.5 pretty much) then I make a trade. So if my optimal number is 1.6 I just trade one contract?" <br /><br />Correct.<br /><br /><br />"Also, in the example I just gave, isnt the impact different for contracts of varying values. So 1 more contract in JGB is $1m vs 1 more contract in TYA being $125k?"<br /><br />Yes, but (a) it's contract risk that is important not notional size, and (b) there clearly isn't anything we can do about this!Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-3136379151374312122017-04-21T00:50:14.869+01:002017-04-21T00:50:14.869+01:00Also I apologize I should have referenced your boo...Also I apologize I should have referenced your book first. I wasn't fully aware that you calculate the buffer using actual contracts, not percents as I thought. <br /><br />The example in your book is: the upper buffer is 133.52 * 1.1 = 147 contracts. <br /><br />Using this logic then, if I have 1 contract, the upper buffer is 1.1 contracts. So if my optimal number of contracts goes above 1.1(or 1.5 pretty much) then I make a trade. So if my optimal number is 1.6 I just trade one contract? <br /><br />Also, in the example I just gave, isnt the impact different for contracts of varying values. So 1 more contract in JGB is $1m vs 1 more contract in TYA being $125k?<br /><br />I find this aspect interesting, greatly appreciate your thoughts.<br />ESHKDhttps://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-70275555668054683122017-04-20T23:55:01.588+01:002017-04-20T23:55:01.588+01:00Did you estimate the buffer purely on a backtest? ...Did you estimate the buffer purely on a backtest? eg tried a bunch of different buffers for a give set of costs and settled on 10%?ESHKDhttps://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-57023123267640944902017-04-18T06:55:17.289+01:002017-04-18T06:55:17.289+01:00Theoretically the optimal buffer size depends only...Theoretically the optimal buffer size depends only on costs. Higher costs means a larger buffer. Lower costs means a smaller buffer. I use a constant buffer size purely to make life simpler.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-23551738590596262062017-04-17T18:43:55.343+01:002017-04-17T18:43:55.343+01:00This is an interesting question. What is the reaso...This is an interesting question. What is the reasoning for a 10% buffer(like why not 5% or 15%)?. Two scenarios pop into my head.<br /><br />a.) Lets say you have 10 commodities and each are 9% above their optimal weights. That would leave you 90% more exposed to commodities than your model would call for. Obviously your willing to accept this risk(unless you have some other limits in place). Or say all the 10 commodities have optimal weights of -5% and your current position in all commodities is 4%. You should be -50% short commodities but instead your 40% long commodities. <br /><br />b.) With the 10% buffer there is room for path dependency. Taking the example above, if you establish positions in 10 commodities in January and the signals give them 4% weights each and say commodities dont move around much for, say 3-months, you end up being long 40% commodities for those three months. On the other hand, say you establish positions in February and the signals for all commodities is -5% and don't move around a lot for 3 months. You are now -50% short for a few months(2 overlapping with the January scenario). Certainly you can say well they didn't move much so overall the difference might not be that important. But in real time, say in March, we obviously don't know the next few months will be less volatile we just know we're either 40% long commodities or -50% short commodities.<br /><br />These are just a few thoughts. Obviously you can mitigate some of the problem by setting exposure limits and such. But the path dependency scenario would still be there(especially with larger buffers).<br /><br />Obviously I'm biased towards a smaller buffer. By how much I'm not sure that's why I'd love to get your thoughts on the matter! <br /><br />Or say you have 11 strategies equally weighted trading one market. Presumably each strategy is profitable after some conservative measure of costs without any buffering. If 10 of the strategies show now changes in weights and 1 strategy is now 100% long(9% increase in position) you'd be ignoring that valuable signal.<br /><br />Would love your thoughts!ESHKDhttps://www.blogger.com/profile/09457839737383784545noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-33241997813596701432017-03-21T06:24:19.942+00:002017-03-21T06:24:19.942+00:00To get the performance of a trading rule you run t...To get the performance of a trading rule you run through the position sizing method in the book allocating 100% to a given trading rule.<br /><br />1) No, that isn't how the system works at all. Read the rest of the book before asking any more questions.<br />2) yes - again this in discussed later in the book<br />3) No, I use continous positions. You need to read chapter 7 again as you don't seem to have quite got the gist.<br /><br /> f*w - lemada*w*sigma*w<br /><br />I don't think I've ever used this formula in my book, or on my blog, so I can't really explain it.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.com