Cloud Data Subscriber & Failed Downloads

Cloud Data Subscriber & Failed Downloads

Postby alaroy » Mon Aug 16, 2021 5:50 pm

Good evening!

We download a lot of data from PODAAC and occasionally something goes wrong partway through the download (real-world stuff like the a bad network connection the system going down at the wrong moment).

Does the cloud data subscriber script have mechanisms to deal with these cases? Ideally it should try to get the failed download again but quickly "give up" if the problem persists.

Thank you,
-Andrew L.
alaroy
 
Posts: 3
Joined: Thu Jun 17, 2021 8:30 am

Re: Cloud Data Subscriber & Failed Downloads

Postby mgangl » Tue Oct 05, 2021 9:25 am

Hi Andrew,

we've created a ticket for your suggestion here: https://github.com/podaac/data-subscriber/issues/17

The subscriber is pretty simple in what it tries to do. If it fails at any point during the "subscription" run, it will exit. This means the next time it is run, it will attempt to download the files that it saw during the failed run. This is _usually_ fine for downloads of several files- the cost of redownloading isn't too great. however, fi you're attempting to download many, many files, a single error would cause the need to re-download all of the files again.

So to answer your question explicitly: The subscriber will "exit" when a transient error is encountered, such as a network glitch.

The subscriber will not "remember" what it downloaded from a previous run, it only looks at data since a "last run" time, which is stored in the downloaded data directory.
mgangl
 
Posts: 22
Joined: Wed Apr 27, 2016 1:31 pm

Re: Cloud Data Subscriber & Failed Downloads

Postby alaroy » Tue Oct 05, 2021 12:54 pm

Thank you for your response & clarification!
alaroy
 
Posts: 3
Joined: Thu Jun 17, 2021 8:30 am


Return to Data Access