Page 1 of 2

PO.DAAC Cloud Data Subscriber

PostPosted: Wed Jun 23, 2021 12:09 pm
by mgangl
Hello all!

We'd like to announce the availability of a new tool for subscribing to PO.DAAC datasets in the cloud- the PO.DAAC Data Subscriber.

The data subscriber allows you to run a few commands to pull the most recent data or data since a specified date to your local directories for use. This is very much like using wget to continuously pull data from PO.DAAC Drive.

The subscriber takes this one step further by allowing users to specify a bounding box to limit their subscriptions to a region of interest. Note this will not subset the data, but only download granules that intersect the data region. This is best for L2 datasets that don't have global, gridded coverages.

An example of this is to download some VIIRS_N20-OSPO-L2P-v2.61 data for a small region (Hawaii):

Code: Select all
podaac-data-subscriber -c VIIRS_N20-OSPO-L2P-v2.61 -d ./data -b="-178.334698,18.910361,-154.806773,28.402123"

If there are additional features or functions you'd like in the tool, please let us know!

Re: PO.DAAC Cloud Data Subscriber

PostPosted: Wed Aug 11, 2021 2:37 am
by pacomet

I've just tried to run podaac-data-subscriber but I'm getting an error with the username and password. I think the .netrc file in my home dir is ok but I get an error message when running:

meteo@PINAR:~/PROJECTES/SST/OISST$ podaac-data-subscriber -c OISST_UHR_NRT-GOS-L4-MED-v 2.0 -d ./DATA/
Error getting the token - check user name and password
WARN: No .update in the data directory. (Is this the first run?)
Downloaded: 0 files

Files Failed to download:0

CMR token deleting failed.

Any idea about the problem?
Thanks in advance and best regards

Re: PO.DAAC Cloud Data Subscriber

PostPosted: Wed Aug 11, 2021 7:15 am
by mgangl
Before we dig into the issue, i wanted to let you know that the dataset you are subscribing to is not in the cloud _yet_, but should be availbe for use in the cloud by November of 2021.

On to the potential issues...

To determine if your .netrc file is set up correctly, try and use 'wget' to grab a file from the PO.DAAC Cloud archive:

Code: Select all

if that works (after a lot of messages), and you see the file downloaded then you should be good. if not, let's look at the .netrc file.

It should look like the following:

Code: Select all
cat ~/.netrc

     login <EDL Username>
     password <EDL PASSWORD>

Let me know if that fixes or identifies the issue, or if those both look good and we need to troubleshoot a bit more.

Re: PO.DAAC Cloud Data Subscriber

PostPosted: Thu Aug 12, 2021 1:20 am
by pacomet
Hi and thanks

The wget command has run fine and .netrc in my home directory looks ok

-rw-rw-r-- 1 meteo meteo 3444058 ago 11 17:04

What next?

Re: PO.DAAC Cloud Data Subscriber

PostPosted: Thu Aug 12, 2021 1:41 am
by pacomet
If I remove .netrc I am not asked for the username or password

meteo@PINAR:~/PROJECTES/SST/MUR/TREBALL$ podaac-data-subscriber -c VIIRS_N20-OSPO-L2P-v2.61 -d ../DATA -b="-178.334698,18.910361,-154.806773,28.402123"
There's no .netrc file or the The endpoint isn't in the netrc file
Traceback (most recent call last):
File "/home/meteo/.local/bin/podaac-data-subscriber", line 8, in <module>
File "/home/meteo/.local/lib/python3.8/site-packages/subscriber/", line 208, in run
File "/home/meteo/.local/lib/python3.8/site-packages/subscriber/", line 97, in setup_earthdata_login_auth
manager.add_password(None, endpoint, username, password)
UnboundLocalError: local variable 'username' referenced before assignment

Re: PO.DAAC Cloud Data Subscriber

PostPosted: Thu Aug 12, 2021 9:35 am
by mgangl
Here are the things i'd look for:

* Make sure the .netrc file has the right permissions (chmod 600 ~/.netrc)
* Make sure there isn't a "CMR" entry in the .netrc endpoint
* My last guess would be if there is an issue with the password itself (some sort of special cahracter?). I'm not sure how to _fix_ that per se, but you might be able to try and see if your password is causing issues by retrieving a token directly from CMR:

from a terminal
Code: Select all
curl -X POST --header "Content-Type: application/xml" -d "<token><username>EDL username</username><password>EDL password</password><client_id>tempClient</client_id><user_ip_address></user_ip_address></token>"

Re: PO.DAAC Cloud Data Subscriber

PostPosted: Thu Aug 12, 2021 9:58 pm
by pacomet

I think .netrc meets all requirements.

curl -X POST --header "Content-Type: application/xml" ... est/tokens -d "<token><username>XXXXXXXX</username><password>XXXXXXXXX</password><client_id>tempClient</client_id><user_ip_address></user_ip_address></token>"

with this result

<errors><error>Failed to contact URS. Please wait for 1 minute and retry.</error></errors>

Re: PO.DAAC Cloud Data Subscriber

PostPosted: Tue Aug 17, 2021 8:51 am
by mgangl
Just an FYI, we are working with the CMR team to determine what might be happening with your token request. Is this error constant for you? Or do you get other errors during attempted use of the scripts/commands?

Re: PO.DAAC Cloud Data Subscriber

PostPosted: Wed Aug 18, 2021 3:37 am
by pacomet
Thank you very much for working on this.

This error appears every time I try podaac-data-subscriber with different options and it's always the same. No special characters in password. But I'm still using PODAAC Drive without any problem.

Re: PO.DAAC Cloud Data Subscriber

PostPosted: Wed Aug 18, 2021 10:49 am
by mgangl
So we know there were some issues with the service that authenticates the credentials, but they should have been fixed yesterday morning.

In the mean time, we've added a new feature to the subscriber to add some more logging. Can you please upgrade your subscriber install:

Code: Select all
python -m pip install git+
Successfully installed podaac-data-subscriber-1.1.2

And then set the log level so we get some more output:

Code: Select all

When you run the command now, you should see more ouput. Can you please post that here so we can get a better idea of what error codes you're getting? you should see entries like:

Code: Select all
DEBUG:root:Log level set to DEBUG
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1):
DEBUG:urllib3.connectionpool: "POST /legacy-services/rest/tokens HTTP/1.1" 201 None

a 201 is ideal, that means it's working. Anything else we may need to troubleshoot some more.

If you can paste that into a reply that will give us some more information.

Thank you for your patience