Download Multiple Data Files from PODAAC Drive Using wget

Re: Download Multiple Data Files from PODAAC Drive Using wge

Postby cmamos » Thu Apr 30, 2020 1:51 pm

yiboj wrote:Hi,
Thanks for your inquiry. Please try the following command and let us know if this works for you.

Code: Select all
$ wget --user=USER --password=PASS -r -nc -np -nH "https://podaac-tools.jpl.nasa.gov/drive/files/allData/ghrsst/data/L4/GLOB/NCDC/AVHRR_AMSR_OI/2002/"


Regards,

PODAAC DE


Thank you. Your suggestion is downloading 'Unix executable' files but I need the '...-fv02_0-AVHRR_AMSR_OI.nc.bz2' file that is in each day's directory. Is there a way to only download these files or is there a way to work with the 'unix executable files'?
cmamos
 
Posts: 4
Joined: Thu Apr 30, 2020 10:32 am

Re: Download Multiple Data Files from PODAAC Drive Using wge

Postby yiboj » Thu Apr 30, 2020 2:44 pm

Hi,
Thanks for your inquiry. Please try the following command and let us know if this works for you.
Code: Select all
$ wget --user=USER --password=PASS -r -nc -np -nH -R xml "https://podaac-tools.jpl.nasa.gov/drive/files/allData/ghrsst/data/L4/GLOB/NCDC/AVHRR_AMSR_OI/2002/"


Regards,

PODAAC DE
yiboj
 
Posts: 130
Joined: Mon Mar 30, 2015 11:22 am

Re: Download Multiple Data Files from PODAAC Drive Using wge

Postby cmamos » Thu Apr 30, 2020 3:00 pm

yiboj wrote:Hi,
Thanks for your inquiry. Please try the following command and let us know if this works for you.
Code: Select all
$ wget --user=USER --password=PASS -r -nc -np -nH -R xml "https://podaac-tools.jpl.nasa.gov/drive/files/allData/ghrsst/data/L4/GLOB/NCDC/AVHRR_AMSR_OI/2002/"


Regards,

PODAAC DE


Thanks for the new suggestion. However this still results in 'unix executable' files and no '.nc' files.
cmamos
 
Posts: 4
Joined: Thu Apr 30, 2020 10:32 am

Re: Download Multiple Data Files from PODAAC Drive Using wge

Postby yiboj » Thu Apr 30, 2020 7:50 pm

Hi,
The data files are compressed by gzip, the wget command will not unzip for you and you may have to do it by yourself.
Please let me know if you need further info.
Regards,
PODAAC DE
yiboj
 
Posts: 130
Joined: Mon Mar 30, 2015 11:22 am

Re: Download Multiple Data Files from PODAAC Drive Using wge

Postby cmamos » Fri May 01, 2020 4:26 am

yiboj wrote:Hi,
The data files are compressed by gzip, the wget command will not unzip for you and you may have to do it by yourself.
Please let me know if you need further info.
Regards,
PODAAC DE


The issue I have though is that the files that are downloaded using your previous code suggestions are not the zip files, they appear as unix executable files.

Using this code:
Code: Select all
wget --user=USER --password=PASSWORD -r -nc -np -nH -nd "https://podaac-tools.jpl.nasa.gov/drive/files/allData/ghrsst/data/L4/GLOB/NCDC/AVHRR_AMSR_OI/2002/"

The files that are downloaded appear as "152", "153", etc. (the day numbers from the 2002 directory) and the format is 'unix executable'. I tried uncompressing these files but that did not work.

The code works if I'm in the day directory, e.g.
Code: Select all
wget --user=USER --password=PASSWORD -r -nc -np -nH -nd "https://podaac-tools.jpl.nasa.gov/drive/files/allData/ghrsst/data/L4/GLOB/NCDC/AVHRR_AMSR_OI/2002/152/"
but not when I try to download all day files from the year directory.
cmamos
 
Posts: 4
Joined: Thu Apr 30, 2020 10:32 am

Re: Download Multiple Data Files from PODAAC Drive Using wge

Postby yiboj » Fri May 01, 2020 9:12 am

Hi,
You may need to update wget and let me know if that solves the problem.
Regards,
PODAAC DE
yiboj
 
Posts: 130
Joined: Mon Mar 30, 2015 11:22 am

Re: Download Multiple Data Files from PODAAC Drive Using wge

Postby mar77d » Sun Oct 18, 2020 7:41 pm

Hi,

I am trying to download multiple netcdf files from multiple years:

Code: Select all
wget --user=**** --password=**** -r -np -nc -nH "https://https://podaac-tools.jpl.nasa.gov/drive/files/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2.1/"


This downloads the the directory and subdirectories and readme.txt files but not datafiles. Same for this:

Code: Select all
wget --user=**** --password=**** -r -np -nc -nH -A "*.nc" "https://https://podaac-tools.jpl.nasa.gov/drive/files/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2.1/"


Note that downloading one single file works, so credentials are ok. Also I am using the latest wget version.

Thanks for your help.

Max
mar77d
 
Posts: 2
Joined: Sun Oct 18, 2020 7:04 pm

Re: Download Multiple Data Files from PODAAC Drive Using wge

Postby yiboj » Mon Oct 19, 2020 1:50 pm

Hi,
Thanks for you inquiry.
Please check your link and try again.
Regards,
PODAAC DE
yiboj
 
Posts: 130
Joined: Mon Mar 30, 2015 11:22 am

Re: Download Multiple Data Files from PODAAC Drive Using wge

Postby mar77d » Mon Oct 19, 2020 5:21 pm

Hi,

Obviously the repetition of https:// was a typo, the link was fine.

Like I said, I was able to download a single file, but cannot download them recursively. The code I've tried is:

Code: Select all
wget --user=**** --password=**** -r -np -nc -nH -A "*.nc" "https://podaac-tools.jpl.nasa.gov/drive/files/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2.1/"


And for a single file:

Code: Select all
wget --user=**** --password=**** "https://podaac-tools.jpl.nasa.gov/drive/files/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2.1/2016/001/20160101120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.1.nc"
mar77d
 
Posts: 2
Joined: Sun Oct 18, 2020 7:04 pm

Re: Download Multiple Data Files from PODAAC Drive Using wge

Postby yiboj » Tue Oct 20, 2020 12:56 pm

Hi,
Please try the following link and let me know if it works for you.

wget --user=**** --password=**** -r -np -nc -nH -nd "https://podaac-tools.jpl.nasa.gov/drive/files/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2.1/"

Regards,

PODAAC DE
yiboj
 
Posts: 130
Joined: Mon Mar 30, 2015 11:22 am

PreviousNext

Return to Data Access and Services

cron