Cloud Data Subscriber & Failed Downloads

[alaroy]

Good evening!

We download a lot of data from PODAAC and occasionally something goes wrong partway through the download (real-world stuff like the a bad network connection the system going down at the wrong moment).

Does the cloud data subscriber script have mechanisms to deal with these cases? Ideally it should try to get the failed download again but quickly "give up" if the problem persists.

Thank you,
-Andrew L.

[mgangl]

Hi Andrew,

we've created a ticket for your suggestion here: https://github.com/podaac/data-subscriber/issues/17

The subscriber is pretty simple in what it tries to do. If it fails at any point during the "subscription" run, it will exit. This means the next time it is run, it will attempt to download the files that it saw during the failed run. This is _usually_ fine for downloads of several files- the cost of redownloading isn't too great. however, fi you're attempting to download many, many files, a single error would cause the need to re-download all of the files again.

So to answer your question explicitly: The subscriber will "exit" when a transient error is encountered, such as a network glitch.

The subscriber will not "remember" what it downloaded from a previous run, it only looks at data since a "last run" time, which is stored in the downloaded data directory.

[alaroy]

Thank you for your response & clarification!