Dukascopy - WARNING or timeout causes missing data

[halterc]

Hi. Thank you for this great package. However, I experienced a problem: When downloading Dukascopy forex tick data, if there is a:
WARNING - Didn't download on 1 attempt
or
WARNING - Problem downloading.. https://www.dukascopy.com/datafeed/GBPJPY/2012/10/03/14h_ticks.bi5 HTTPConnectionPool(host='datafeed.dukascopy.com', port=80): Max retries exceeded with url: /datafeed/GBPJPY/2012/10/03/14h_ticks.bi5 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x000001F49D7B09A0>, 'Connection to datafeed.dukascopy.com timed out. (connect timeout=10)')).. will try again 0 occasion

it says "retry". But I noticed that it will cause missing data (in blocks of 1 hour, probably missing 1 h chunk file). It also causes to retry downloading seemingly in a random order also chunk files that were (I assume, as there was no warning) correctly downloaded. For an example:

2023-06-16 12:14:01,292 - findatapy.market.datavendorweb - INFO - Downloading... 2022-08-21 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/07/21/00h_ticks.bi5
2023-06-16 12:14:04,706 - findatapy.market.datavendorweb - INFO - Downloading... 2022-08-22 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/07/22/00h_ticks.bi5
2023-06-16 12:14:08,215 - findatapy.market.datavendorweb - INFO - Downloading... 2022-08-23 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/07/23/00h_ticks.bi5
2023-06-16 12:14:11,705 - findatapy.market.datavendorweb - INFO - Downloading... 2022-08-24 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/07/24/00h_ticks.bi5
2023-06-16 12:14:15,614 - findatapy.market.datavendorweb - INFO - Downloading... 2022-08-25 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/07/25/00h_ticks.bi5
2023-06-16 12:14:18,989 - findatapy.market.datavendorweb - INFO - Downloading... 2022-08-26 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/07/26/00h_ticks.bi5
2023-06-16 12:14:22,176 - findatapy.market.datavendorweb - INFO - Downloading... 2022-08-27 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/07/27/00h_ticks.bi5
2023-06-16 12:14:25,274 - findatapy.market.datavendorweb - INFO - Downloading... 2022-08-28 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/07/28/00h_ticks.bi5
2023-06-16 12:14:28,754 - findatapy.market.datavendorweb - INFO - Downloading... 2022-08-29 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/07/29/00h_ticks.bi5
2023-06-16 12:14:32,250 - findatapy.market.datavendorweb - WARNING - Didn't download on 2 attempt... 
2023-06-16 12:14:33,773 - findatapy.market.datavendorweb - INFO - Downloading... 2022-08-30 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/07/30/00h_ticks.bi5
2023-06-16 12:14:36,856 - findatapy.market.datavendorweb - WARNING - Problem downloading.. https://www.dukascopy.com/datafeed/USDCHF/2022/07/28/13h_ticks.bi5 HTTPConnectionPool(host='datafeed.dukascopy.com', port=80): Max retries exceeded with url: /datafeed/USDCHF/2022/07/28/13h_ticks.bi5 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x0000021DAA1151B0>, 'Connection to datafeed.dukascopy.com timed out. (connect timeout=10)')).. will try again 0 occasion
2023-06-16 12:14:38,859 - findatapy.market.datavendorweb - INFO - Downloading... 2022-08-31 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/07/31/00h_ticks.bi5
2023-06-16 12:14:42,527 - findatapy.market.datavendorweb - INFO - Downloading... 2022-01-01 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/00/01/00h_ticks.bi5
2023-06-16 12:14:46,194 - findatapy.market.datavendorweb - INFO - Downloading... 2022-01-02 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/00/02/00h_ticks.bi5
2023-06-16 12:14:46,251 - findatapy.market.datavendorweb - INFO - Downloading... 2022-09-01 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/08/01/00h_ticks.bi5
2023-06-16 12:14:49,868 - findatapy.market.datavendorweb - INFO - Downloading... 2022-01-03 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/00/03/00h_ticks.bi5
2023-06-16 12:14:53,487 - findatapy.market.datavendorweb - INFO - Downloading... 2022-01-04 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/00/04/00h_ticks.bi5
2023-06-16 12:14:56,346 - findatapy.market.datavendorweb - INFO - Downloading... 2022-01-05 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/00/05/00h_ticks.bi5
2023-06-16 12:14:58,170 - findatapy.market.datavendorweb - INFO - Downloading... 2022-09-02 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/08/02/00h_ticks.bi5
2023-06-16 12:15:00,116 - findatapy.market.datavendorweb - INFO - Downloading... 2022-01-06 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/00/06/00h_ticks.bi5
2023-06-16 12:15:03,554 - findatapy.market.datavendorweb - INFO - Downloading... 2022-01-07 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/00/07/00h_ticks.bi5
2023-06-16 12:15:07,646 - findatapy.market.datavendorweb - INFO - Downloading... 2022-01-08 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/00/08/00h_ticks.bi5
2023-06-16 12:15:10,204 - findatapy.market.datavendorweb - INFO - Downloading... 2022-09-03 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/08/03/00h_ticks.bi5
2023-06-16 12:15:11,014 - findatapy.market.datavendorweb - INFO - Downloading... 2022-01-09 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/00/09/00h_ticks.bi5
2023-06-16 12:15:14,305 - findatapy.market.datavendorweb - INFO - Downloading... 2022-01-10 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/00/10/00h_ticks.bi5
2023-06-16 12:15:18,668 - findatapy.market.datavendorweb - INFO - Downloading... 2022-01-11 00:00:00 https://www.dukascopy.com/datafeed/USDCHF/2022/00/11/00h_ticks.bi5

Do you know what could be the culprit? Could it be the multiprocessing in combination with Windows? (edit: I will try it out with "thread" instead of "multiprocessing" and post back after the test)
Data connection should be fine, I'm in a large institution. Firewall should not be the problem.

For your information, I'm on Windows 11 and using the following code:

SwimPool()
market = Market(market_data_generator=MarketDataGenerator())
DataConstants.market_thread_technique = "multiprocessing"
md_request = MarketDataRequest(start_date='01 Jan ' + str(year),
                                finish_date='31 Dec ' + str(year),
                                category='fx', fields=['bid', 'ask', 'bidv', 'askv'],
                                freq='tick', data_source='dukascopy', tickers=ticker)
df = market.fetch_market(md_request)

Thank you very much for your help.

[saeedamen]

Yes, you should probably remove the line changing the market thread technique. Downloading from DukasCopy seems to work ok from my machine (the main problem is if you increase the threads in the configuration, which can cause server side rejection - in the newest version of the code, I've reduced the number of threads for this reason). You can also try download a day at a time and sleep between calls.

[halterc]

Thank you for your answer. I tried without using the "thread" or "multiprocess" option. Unfortunately, the problem persisted (at least on my computers) that there will be missing data when the download is retried (I don't know why).
I think that downloading data one day at a time would probably not solve my problem. Also, it would make things more complicated.
For downloading the data, I gave up and I sticked with Tickstory for downloading.

If I have already downloaded the data from Dukascopy, is there a possibility in findatapy to just unpack the .bi5 files for a ticker to a csv file, lets say as 1 min candles? That would be a really great functionality! :)
Thanks again for taking your time to create and maintain such a very useful library! 👍