PO.DAAC Cloud Data Subscriber

[mgangl]

Hello all!

We'd like to announce the availability of a new tool for subscribing to PO.DAAC datasets in the cloud- the PO.DAAC Data Subscriber.

https://github.com/podaac/data-subscriber

The data subscriber allows you to run a few commands to pull the most recent data or data since a specified date to your local directories for use. This is very much like using wget to continuously pull data from PO.DAAC Drive.

The subscriber takes this one step further by allowing users to specify a bounding box to limit their subscriptions to a region of interest. Note this will not subset the data, but only download granules that intersect the data region. This is best for L2 datasets that don't have global, gridded coverages.

An example of this is to download some VIIRS_N20-OSPO-L2P-v2.61 data for a small region (Hawaii):

Code: Select all

podaac-data-subscriber -c VIIRS_N20-OSPO-L2P-v2.61 -d ./data -b="-178.334698,18.910361,-154.806773,28.402123"


If there are additional features or functions you'd like in the tool, please let us know!

[pacomet]

Hi

I've just tried to run podaac-data-subscriber but I'm getting an error with the username and password. I think the .netrc file in my home dir is ok but I get an error message when running:

meteo@PINAR:~/PROJECTES/SST/OISST$ podaac-data-subscriber -c OISST_UHR_NRT-GOS-L4-MED-v 2.0 -d ./DATA/
Error getting the token - check user name and password
WARN: No .update in the data directory. (Is this the first run?)
Downloaded: 0 files

Files Failed to download:0

CMR token deleting failed.

Any idea about the problem?
Thanks in advance and best regards

[mgangl]

Before we dig into the issue, i wanted to let you know that the dataset you are subscribing to is not in the cloud _yet_, but should be availbe for use in the cloud by November of 2021.

On to the potential issues...

To determine if your .netrc file is set up correctly, try and use 'wget' to grab a file from the PO.DAAC Cloud archive:

Code: Select all

wget https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/ASCATA-L2-Coastal/ascat_20210811_120300_metopa_76857_eps_o_coa_3202_ovw.l2.nc


if that works (after a lot of messages), and you see the file downloaded then you should be good. if not, let's look at the .netrc file.

It should look like the following:

Code: Select all

cat ~/.netrc

machine urs.earthdata.nasa.gov
     login <EDL Username>
     password <EDL PASSWORD>



Let me know if that fixes or identifies the issue, or if those both look good and we need to troubleshoot a bit more.

[pacomet]

Hi and thanks

The wget command has run fine and .netrc in my home directory looks ok

-rw-rw-r-- 1 meteo meteo 3444058 ago 11 17:04 ascat_20210811_120300_metopa_76857_eps_o_coa_3202_ovw.l2.nc

What next?

[pacomet]

If I remove .netrc I am not asked for the username or password

meteo@PINAR:~/PROJECTES/SST/MUR/TREBALL$ podaac-data-subscriber -c VIIRS_N20-OSPO-L2P-v2.61 -d ../DATA -b="-178.334698,18.910361,-154.806773,28.402123"
There's no .netrc file or the The endpoint isn't in the netrc file
Traceback (most recent call last):
File "/home/meteo/.local/bin/podaac-data-subscriber", line 8, in <module>
sys.exit(run())
File "/home/meteo/.local/lib/python3.8/site-packages/subscriber/podaac_data_subscriber.py", line 208, in run
setup_earthdata_login_auth(edl)
File "/home/meteo/.local/lib/python3.8/site-packages/subscriber/podaac_data_subscriber.py", line 97, in setup_earthdata_login_auth
manager.add_password(None, endpoint, username, password)
UnboundLocalError: local variable 'username' referenced before assignment

[mgangl]

Here are the things i'd look for:

* Make sure the .netrc file has the right permissions (chmod 600 ~/.netrc)
* Make sure there isn't a "CMR" entry in the .netrc endpoint
* My last guess would be if there is an issue with the password itself (some sort of special cahracter?). I'm not sure how to _fix_ that per se, but you might be able to try and see if your password is causing issues by retrieving a token directly from CMR:

from a terminal

Code: Select all

curl -X POST --header "Content-Type: application/xml" https://cmr.earthdata.nasa.gov/legacy-services/rest/tokens -d "<token><username>EDL username</username><password>EDL password</password><client_id>tempClient</client_id><user_ip_address>127.0.0.1</user_ip_address></token>"


[pacomet]

Hi

I think .netrc meets all requirements.

Tried

curl -X POST --header "Content-Type: application/xml" https://cmr.earthdata.nasa.gov/legacy-s ... est/tokens -d "<token><username>XXXXXXXX</username><password>XXXXXXXXX</password><client_id>tempClient</client_id><user_ip_address>127.0.0.1</user_ip_address></token>"

with this result

<errors><error>Failed to contact URS. Please wait for 1 minute and retry.</error></errors>

[mgangl]

Just an FYI, we are working with the CMR team to determine what might be happening with your token request. Is this error constant for you? Or do you get other errors during attempted use of the scripts/commands?

[pacomet]

Thank you very much for working on this.

This error appears every time I try podaac-data-subscriber with different options and it's always the same. No special characters in password. But I'm still using PODAAC Drive without any problem.

[mgangl]

So we know there were some issues with the service that authenticates the credentials, but they should have been fixed yesterday morning.

In the mean time, we've added a new feature to the subscriber to add some more logging. Can you please upgrade your subscriber install:

Code: Select all

python -m pip install git+https://github.com/podaac/data-subscriber.git
...
Successfully installed podaac-data-subscriber-1.1.2


And then set the log level so we get some more output:

Code: Select all

export SUBSCRIBER_LOGLEVEL=WARNING


When you run the command now, you should see more ouput. Can you please post that here so we can get a better idea of what error codes you're getting? you should see entries like:

Code: Select all

DEBUG:root:Log level set to DEBUG
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): cmr.earthdata.nasa.gov:443
DEBUG:urllib3.connectionpool:https://cmr.earthdata.nasa.gov:443 "POST /legacy-services/rest/tokens HTTP/1.1" 201 None


a 201 is ideal, that means it's working. Anything else we may need to troubleshoot some more.

If you can paste that into a reply that will give us some more information.

Thank you for your patience