Data subscriber error

[pacomet]

Hi, I'm successfully running podaac-data-subscriber in an Ubuntu 20.04 but had no success when moving to a somewhat older production machine running Ubuntu 18.04. This is the error message I'm getting when running

Code: Select all

podaac-data-subscriber -c AVHRR_OI-NCEI-L4-GLOB-v2.1 -d ./

Code: Select all

NOTE: .update found in the data directory. (The last run was at 2021-10-29T00:05:03Z
.)
Traceback (most recent call last):
  File "/home/meteo/.local/bin/podaac-data-subscriber", line 11, in <module>
    load_entry_point('podaac-data-subscriber==1.6.0', 'console_scripts', 'podaac-data-subscriber')()
  File "/home/meteo/.local/lib/python3.6/site-packages/subscriber/podaac_data_subscriber.py", line 339, in run
    with urlopen(url) as f:
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request


python version is 3.6.9, requests module is also installed, podaac-data-subscriber is installed and version is 1.6.0.

Thanks in advance for your help

[mgangl]

I was just able to successfuly use the subscriber and python 3.6.9.

Can you try running the subscriber with the --verbose option to see what URL it is trying to access?

Code: Select all

 podaac-data-subscriber -c AVHRR_OI-NCEI-L4-GLOB-v2.1 -d ./ --verbose
WARN: No .update in the data directory. (Is this the first run?)
Provider: POCLOUD
Updated Since: 2021-11-08T22:47:20Z
https://cmr.earthdata.nasa.gov/search/granules.umm_json?scroll=true&page_size=2000&sort_key=-start_date&provider=POCLOUD&ShortName=AVHRR_OI-NCEI-L4-GLOB-v2.1&updated_since=2021-11-08T22%3A47%3A20Z&token=*****&bounding_box=-180%2C-90%2C180%2C90
0 new granules ingested for AVHRR_OI-NCEI-L4-GLOB-v2.1 since 2021-11-08T22:47:20Z
Found 0 files to download
Downloading files with extensions: ['.nc', '.h5', '.zip']
Downloaded: 0 files

Files Failed to download:0

CMR token successfully deleted



[pacomet]

Please find the error message here

Code: Select all

meteo@BOIRA:~/PROJECTES/SST/NCEI/DATA/SST/NC2$ podaac-data-subscriber -c AVHRR_OI-NCEI-L4-GLOB-v2.1 -d ./ --verbose
NOTE: .update found in the data directory. (The last run was at 2021-10-29T00:05:03Z
.)
Provider: POCLOUD
Updated Since: 2021-10-29T00:05:03Z

https://cmr.earthdata.nasa.gov/search/granules.umm_json?scroll=true&page_size=2000&sort_key=-start_date&provider=POCLOUD&ShortName=AVHRR_OI-NCEI-L4-GLOB-v2.1&updated_since=2021-10-29T00%3A05%3A03Z%0A&token=99E378CA-AA90-A101-1DDE-84D9A597DE24&bounding_box=-180%2C-90%2C180%2C90
Traceback (most recent call last):
  File "/home/meteo/.local/bin/podaac-data-subscriber", line 11, in <module>
    load_entry_point('podaac-data-subscriber==1.6.0', 'console_scripts', 'podaac-data-subscriber')()
  File "/home/meteo/.local/lib/python3.6/site-packages/subscriber/podaac_data_subscriber.py", line 339, in run
    with urlopen(url) as f:
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request

I have tried removing the .update and using -sd option and it worked.

Code: Select all

meteo@BOIRA:~/PROJECTES/SST/NCEI/DATA/SST/NC2$ podaac-data-subscriber -c AVHRR_OI-NCEI-L4-GLOB-v2.1 -d ./ -sd 2021-11-01T00:10:00Z --verbose
WARN: No .update in the data directory. (Is this the first run?)
Temporal Range: 2021-11-01T00:10:00Z,2021-11-09T07:34:52Z
Provider: POCLOUD
Updated Since: 2021-11-01T00:10:00Z
https://cmr.earthdata.nasa.gov/search/granules.umm_json?scroll=true&page_size=2000&sort_key=-start_date&provider=POCLOUD&updated_since=2021-11-01T00%3A10%3A00Z&ShortName=AVHRR_OI-NCEI-L4-GLOB-v2.1&temporal=2021-11-01T00%3A10%3A00Z%2C2021-11-09T07%3A34%3A52Z&token=D9CD4F30-AAD6-E2E6-6F6E-664A0C10170A&bounding_box=-180%2C-90%2C180%2C90
3 new granules ingested for AVHRR_OI-NCEI-L4-GLOB-v2.1 since 2021-11-01T00:10:00Z
Found 9 files to download
Downloading files with extensions: ['.nc', '.h5', '.zip']
2021-11-09 08:35:03.463381 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/AVHRR_OI-NCEI-L4-GLOB-v2.1/20211103120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.1.nc
2021-11-09 08:35:07.179736 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/AVHRR_OI-NCEI-L4-GLOB-v2.1/20211102120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.1.nc
2021-11-09 08:35:11.024598 SUCCESS: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/AVHRR_OI-NCEI-L4-GLOB-v2.1/20211101120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.1.nc
Downloaded: 3 files

Files Failed to download:0

CMR token successfully deleted


Maybe the problem came from the .update file I manually created. Now it is fine with the new .update created.

[mgangl]

Ah I see. In the url of the error command, the “data-since” option ends with an extra “%0a” which is a line feed character- essentially the .update file you created, had the date, and then a new line (you hit enter/return) at the end, and this was causing problems.

We will update the script in future versions to be more permissive of such issues and remove them for the user. Glad it’s working now, but you should be able to create your update file as long as it has no newline at the end.