tag:blogger.com,1999:blog-261139923818144971.post4666484474990195341..comments2024-03-29T07:20:07.753+00:00Comments on This Blog is Systematic: Historic data from native IB python APIRob Carverhttp://www.blogger.com/profile/10175885372013572770noreply@blogger.comBlogger70125tag:blogger.com,1999:blog-261139923818144971.post-57955557852708042202023-04-19T00:27:07.012+01:002023-04-19T00:27:07.012+01:00Right, 2 years of expired contracts then. :(
Anyo...Right, 2 years of expired contracts then. :(<br /><br />Anyone got ideas of the best place for price data for backtesting for a non-coder? I've been using Barchart for Excel but the data is riddled with errors and gaps.<br />Richard Johnstonhttps://www.blogger.com/profile/18150375457520792501noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-13074731791231047342023-04-18T18:47:20.901+01:002023-04-18T18:47:20.901+01:00Whatever IB has available.Whatever IB has available.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-20213133612199124232023-04-18T06:27:24.531+01:002023-04-18T06:27:24.531+01:00Does this method allow you to collect data for lon...Does this method allow you to collect data for long ago expired contracts, or is it limited to contracts expiring less than 2 years ago?Richard Johnstonhttps://www.blogger.com/profile/18150375457520792501noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-53138646322253759122021-05-07T10:22:26.877+01:002021-05-07T10:22:26.877+01:00Rob, thanks for writing this and the other related...Rob, thanks for writing this and the other related posts.<br /><br />I've been accumulating data using ib_insync for a while now, but this and the previous post have made OOP coding just that little bit clearer, from the consistency of thick mud, to that of a good pea and ham soup.<br /><br />Thanks again, and back to the reading :)Lake Newshttps://www.blogger.com/profile/03503911102858912173noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-17862732097442605042021-03-24T06:59:07.215+00:002021-03-24T06:59:07.215+00:00Yes, market data subscription is required.Yes, market data subscription is required.Jeroenhttps://www.blogger.com/profile/02819426964787793954noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-49053897444576991392021-03-24T00:21:26.715+00:002021-03-24T00:21:26.715+00:00Hello.
Will someone please tell me if a market da...Hello.<br /><br />Will someone please tell me if a market data subscription is required to download historical end-of-day data via the ibapi?<br /><br />I suspect so, but just wanted to make sure before subscribing.Lake Newshttps://www.blogger.com/profile/03503911102858912173noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-20860745321422543852019-05-22T20:55:49.710+01:002019-05-22T20:55:49.710+01:00Alright, I would like to thank you for your help R...Alright, I would like to thank you for your help Rob. <br /><br />I have been able to fix my connection issues and now my program does not seem to have any issues anymore.<br /><br />Again, thank you a lot for all your help and your code.<br /><br />RegardsJack Sandershttps://www.blogger.com/profile/07165471874056205569noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-74926348879233909302019-05-20T20:32:24.472+01:002019-05-20T20:32:24.472+01:00Alright So, I have added:
*if self.isConnected():...Alright So, I have added:<br /><br />*if self.isConnected():<br />**working = True<br />**print("Connection Established")<br />*if not self.isConnected():<br />**working = False <br />**print("Connection not Established")<br /><br />eventhough the ib terminal does not return:<br />"IB error id -1 errorcode 2104 string Market data farm connection is OK:usfarm"<br /><br />It does say:<br />"Connection Established"<br />but this is mentioned before the terminal returns:<br />"Getting full contract details from server"<br /><br />So after digging through the manual pages, I found that I should be able to call:<br />"EWrapper.error(self, reqId, errorCode, errorString)"<br />in order to verify that my connection is good and that I can now receive data.<br /><br />When I replace:<br />self.isConnected():<br /><br />with that code, it just returns me the code as a string, so I am not able to use that for some reason.<br /><br />But since "isConnected" does not seem to work, I think we should find a way to make python do something like this:<br /><br />If we do not get error-1 within 5 seconds, working = False & retry code.<br /><br />What I really still think is strange, is that it does not attempt to receive the data after the first connection is failed.<br /><br />Could you help me with that? :)<br /><br />Jack Sandershttps://www.blogger.com/profile/07165471874056205569noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-34117298253517632172019-05-20T18:59:19.997+01:002019-05-20T18:59:19.997+01:00UPDATE:
another weird thing, when I say:
self.is...UPDATE:<br /><br />another weird thing, when I say: <br />self.isConnected():<br />print("Connection established " + symbol)<br /><br />it says that a connection is established, eventhough it's not retreiving the data and gives me a timeout after 10 sec.<br />Jack Sandershttps://www.blogger.com/profile/07165471874056205569noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-5615119976921019512019-05-20T18:54:38.744+01:002019-05-20T18:54:38.744+01:00Unfortunately this does not seem to work.
first, ...Unfortunately this does not seem to work.<br /><br />first, apparently it is<br />self.isConnected(): (manual page says "IsConnected", but client.py says "isConnected"<br /><br />Then, trying the following:<br /><br />...<br />*if self.isConnected():<br />**working = True<br />*if not self.isConnected():<br />**working = False <br /><br />Does not work aswell, for some reason.<br /><br />Changing "if not..."by "except" gives me an error and says "statement expected"<br /><br />This is verry strange to me, as now we are calling for isConnected, which is False because it's not connected, but still it skips the loop and goes on with the next symbol<br /><br />Jack Sandershttps://www.blogger.com/profile/07165471874056205569noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-33678567829175618812019-05-20T18:25:34.770+01:002019-05-20T18:25:34.770+01:00Its not possible to modify self.connect as this is...Its not possible to modify self.connect as this is a method provided by the IB API objects. However we can check for a connection:<br /><br />working=False<br />while not working:<br />*self.connect (etc)<br />*if self.IsConnected():<br />**worked=True<br /><br />Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-59692078006637353552019-05-20T16:32:33.938+01:002019-05-20T16:32:33.938+01:00UPDATE:
Sorry, i forgot the
**worked = True
So, ...UPDATE:<br /><br />Sorry, i forgot the<br />**worked = True<br /><br />So, after adding that line AND deleting self.cancelHistoricalData, the program does not lose it's connection, so that is good. but when it fails to retreive data, it now skips whatever it failed to retreive and continues on with the next one (progress! :P)<br /><br />EDIT: (deleted comment and copied that to this reply so clean your blogpost a bit, sorry for the missclicks).<br /><br />Alright, so I have tried a couple new things so far. Basically I don't think that this works because self.connect(etc) does not return anything itself, it just tries to connect and because it tries it will automatically turn to true (I think). <br /><br />changing "pass" with the following things will not work aswell, for the same reason.<br /><br />continue, TestApp, working=False, finishableque, EClient, EWrapper.<br /><br />I think that it would work if we could add something like:<br /><br />if we get an output, working = true<br />else<br />working = false<br />retry<br /><br />the problem is that I do not know how to define that output, could you help me with that? (again, sorry for asking so many questions)<br /><br />Thanks!Jack Sandershttps://www.blogger.com/profile/07165471874056205569noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-87901847997167257362019-05-20T13:51:52.534+01:002019-05-20T13:51:52.534+01:00This comment has been removed by the author.Jack Sandershttps://www.blogger.com/profile/07165471874056205569noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-61016205620084033682019-05-20T13:44:36.734+01:002019-05-20T13:44:36.734+01:00Thank you again for your answer.
Unfortunately If...Thank you again for your answer.<br /><br />Unfortunately If i try to use that line of code, the program does nothing. I think that it should be specified somewhere in the TestClient class, but im not sure.Jack Sandershttps://www.blogger.com/profile/07165471874056205569noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-70639905389921262252019-05-20T13:39:20.518+01:002019-05-20T13:39:20.518+01:00Okay - replace * with tab:
working=False
while no...Okay - replace * with tab:<br /><br />working=False<br />while not working:<br />*try:<br />**self.connect (etc)<br />**worked=True<br />*except:<br />**passRob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-80720434771811289762019-05-20T13:38:32.253+01:002019-05-20T13:38:32.253+01:00
working=False
while not working:
try:
self.conne...<br /><br />working=False<br />while not working:<br />try:<br />self.connect (etc)<br />worked=True<br />except:<br />passRob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-54100162166847412272019-05-20T13:38:01.166+01:002019-05-20T13:38:01.166+01:00The tabs have vanished from that post so let's...The tabs have vanished from that post so let's try again with spaces:<br /><br />working=False<br />while not working:<br /> try:<br /> self.connect (etc)<br /> worked=True<br /> except:<br /> passRob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-52572803014350305222019-05-20T13:37:03.208+01:002019-05-20T13:37:03.208+01:00This is what I mean:
working=False
while not wor...This is what I mean: <br /><br />working=False<br />while not working:<br /> try:<br /> self.connect (etc)<br /> worked=True<br /> except:<br /> pass<br /><br />However I haven't tested it and I don't know if the self.connect() will throw an exception or like it if you try to repeat that command multiple times.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-31712928117427829952019-05-20T13:30:20.945+01:002019-05-20T13:30:20.945+01:00Thank you for you answer Rob!
Luckily I am not tr...Thank you for you answer Rob!<br /><br />Luckily I am not trying to make an automated trading system, I am just trying to download historical data. But thank you for your concerns.<br /><br />As for question 1, and I am sorry to ask another question, I am not quite sure how to achieve that, since I can't find the value that is supposed to be true (which should be connection). So I have tried a couple things:<br />while False<br /> try:<br /> self.connect(etc.)<br /> except:<br /> True<br /><br />This does not loop forever, and also gives me the ib 504 error: not connected.<br /><br />If I use while True the program does nothing<br /><br />And I have tried calling all the functions (while function: False/True try self.connect(etc.)<br /><br />So basically, since I do not know what's supposed to be True I can't specify what to do if it's false (If that makes sence)<br /><br />Would you be so kind to tell me what's supposed to be true or false and what to call when except happens? (or write that piece of code for me (: )<br /><br />As for question 2, I have been able to fix that by deleting everything that you wrote in the "def historicalData" function and replacing it with the following line:<br /><br />print(symbol, ";", bar.date.replace(' ', '; '), ";", bar.open, ";", bar.high, ";", bar.low, ";", bar.close, ";", bar.volume)<br /><br />First I would like to thank you again for trying to help me with my issue and for posting this code!<br /><br />Second I would like to ask you if you would be so kind to help me out with question 1, since I am unsure what the conditions are for the while loop that should be true or false.<br /><br />Really the only thing that is holding me back now for downloading bulk data is that connection issue, so it would be amazing if you could help me out with that!<br /><br />Have a nice day and thank you again!<br />Jack Sandershttps://www.blogger.com/profile/07165471874056205569noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-69603292013305281162019-05-20T10:12:05.693+01:002019-05-20T10:12:05.693+01:00Jack,
As a general point I think you need to learn...Jack,<br />As a general point I think you need to learn a *lot* more python before it would be safe to try writing a trading system using the language, because these are quite basic questions. I'm worried that you will be a danger to your own bank balance if you trade with real money at this stage. <br /><br />As a kindness I will answer your questions, but this blog is not meant to be a 'help you learn python' blog. There are better and more well qualified experts out there for that, than me:<br /><br />Q1. This is handled in the API not my code; to deal with this you could wrap the line 266 in the gist with a try: except: inside an infinite while loop<br /><br /><br />Q2: That isn't a string it's returning, but the string representation of historic_data which is produced by the print() command. So historic_data is a list of tuples. You wouldn't normally want to turn those into strings for a real trading system, but if you wanted to print them out as individual rows for diagnostic purposes then you'd do something like this:<br /><br />for one_tuple in list_of_tuples:<br /> print(",".join(one_tuple))<br /><br />Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-11521289489832709542019-05-18T21:53:19.580+01:002019-05-18T21:53:19.580+01:00Hi Rob,
Thanks for this post, really awesome that...Hi Rob,<br /><br />Thanks for this post, really awesome that you shared this with the community.<br /><br />I have 2 questions about your code, I have re-read your post a million times and tried some stuff, but i can't seem to figure it out. (I am also quite new to Python, did a 6 hour course and made some easy programs with it).<br /><br />Question 1:<br />When the program is not able to make a connection with the api, your code seems to re-try it 1 more time, and than it quits trying. (i think this related to "get bored waiting"). For some reason, I deal with this a lot (also when I am using a different code, seems to be either API related or related to my own internet connection, not sure)<br /><br />Basically, I would like this program to keep trying untill it retreives the data (basically, "I never get bored waiting") Could you tell me how to achieve this?<br /><br />Question 2:<br />If the program than returns the data, it returns a string that goes sideways for eternity (or untill we have retreived all the data for the contract) for example: <br /><br />'AAPL', ';', '20190517; ; 15:30:00', ';', 187.06, ';', 187.59, ';', 186.76, ';', 187.14, ';', 11626, '\n'), ('AAPL', ';', '20190517; ; 15:31:00', ';', 187.13, ';', 187.68, ';', 187.09, ';', 187.52, ';', 1794, '\n')<br /><br />As you can see with my '\n' command, i would like every output to be on a new row, but (I think) because your code writes a str, it writes the whole thing and does not uses the '\n' command.<br /><br />Could you tell me what I should change in order for your code to return the data every single row? I can't seem to find the related str command that does this.<br /><br />Again thanks a million times for posting this and it would be great if you could help me out with my issues.<br /><br />RegardsJack Sandershttps://www.blogger.com/profile/07165471874056205569noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-50575516736770275592019-02-16T20:39:38.167+00:002019-02-16T20:39:38.167+00:00As far as I can make out, when there is an error, ...As far as I can make out, when there is an error, the code hangs until timeout, and then continues processing. E.g. when calling contract_details_queue.get(timeout = MAX_WAIT_SECONDS) in your gist, if there is no match found, nothing happens for seconds, and then the error is thrown. Which would make sense since nothing is being placed into the queue by IB (Does it indeed behave this way or am I doing something wrong?) If so, is there a way to handle the error as soon as it occurs. Thanks as always...JMW100https://www.blogger.com/profile/18171394406805453830noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-2081432017401739432018-08-31T14:51:17.936+01:002018-08-31T14:51:17.936+01:00That is weird, but then anything to do with asynch...That is weird, but then anything to do with asynchrous operations can always cause weird stuff to happen (sorry that isn't helpful...)Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-55248380055646725512018-08-31T14:46:22.406+01:002018-08-31T14:46:22.406+01:00It's just to avoid exposing the innards of the...It's just to avoid exposing the innards of the class. What it's telling you is hopefully self documenting.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.comtag:blogger.com,1999:blog-261139923818144971.post-22586032086778224252018-08-31T14:45:14.219+01:002018-08-31T14:45:14.219+01:00API change, gist now updated.API change, gist now updated.Rob Carverhttps://www.blogger.com/profile/10175885372013572770noreply@blogger.com