Data Discovery & Download by PODAAC Web Service, wget/curl

Data Discovery & Download by PODAAC Web Service, wget/curl

Postby yiboj » Thu Dec 01, 2016 9:04 pm

PO.DAAC Web Services are application programming interfaces (APIs) that can be accessed through standard web protocols. User can check the APIs in the following link:

http://podaac.jpl.nasa.gov/ws/

The Granule Search web service is widely used and it finds granules meeting the specified search criteria in PO.DAAC level 2 swath datasets (individual orbits of a satellite), and level 3 & 4 gridded datasets (time averaged to span the globe). Coverage footprint polygons are used to enable spatial search on level 2 swath datasets. These coverage footprints do not reflect data gaps and will occasionally contain no data at all. Therefore, spatial search on level 2 data can return granules where there is no data in the selected bounding box even though the footprint indicates that there is.

We just released the new L2B QuikSCAT dataset (datasetId=PODAAC-QSX12-L2B31) a few weeks ago, and now we can search for the available data granules from 2005-03-01 to 2005-03-31 in a web browser using the following link:

http://podaac.jpl.nasa.gov/ws/search/gr ... rPage=1000

and then the UNIX utility command wget can be used to retrieve the returned granules from the web service.

Code: Select all
% wget -q -nH -nd "http://podaac.jpl.nasa.gov/ws/search/granule/?datasetId=PODAAC-QSX12-L2B31&startTime=2005-03-01&endTime=2005-03-31&itemsPerPage=1000" -O - | grep "FTP URL" | cut -f2 -d\" > granule_list.out
% wget -i granule_list.out


or by using curl utility command if it is installed in your system. Please note that curl command can not read the input file line by line.

Code: Select all
% curl -s "http://podaac.jpl.nasa.gov/ws/search/granule/?datasetId=PODAAC-QSX12-L2B31&startTime=2005-03-01&endTime=2005-03-31&itemsPerPage=1000" | grep "FTP URL" | cut -f2 -d\" > granule_list.out
% xargs -n 1 curl -O < granule_list.out


In three steps we have: 1) executed the web service call to retrieve the list of data granules which satisfy the input conditions; 2) extracted just the data granule FTP URLs and cleaned them up to fit the requirement for downloading; 3) downloaded the cleaned list of FTP URLs

The following code uses the same procedure except it uses OPeNDAP URL to download the data granules.

Code: Select all
% wget -q -nH -nd "http://podaac.jpl.nasa.gov/ws/search/granule/?datasetId=PODAAC-QSX12-L2B31&startTime=2005-03-01&endTime=2005-03-31&itemsPerPage=1000" -O - | grep "OPeNDAP URL" | cut -f2 -d\" | sed 's/.html/.nc/g' > granule_list.out
% wget -i granule_list.out


or by using curl utility command.

Code: Select all
% curl -s "http://podaac.jpl.nasa.gov/ws/search/granule/?datasetId=PODAAC-QSX12-L2B31&startTime=2005-03-01&endTime=2005-03-31&itemsPerPage=1000" | grep "OPeNDAP URL" | cut -f2 -d\" | sed 's/.html/.nc/g' > granule_list.out
% xargs -n 1 curl -O < granule_list.out
yiboj
 
Posts: 80
Joined: Mon Mar 30, 2015 11:22 am

Re: Data Discovery & Download by PODAAC Web Service, wget/c

Postby essay » Sat Dec 03, 2016 2:00 am

thank you for the code! 8-)
essay
 
Posts: 1
Joined: Sat Dec 03, 2016 1:51 am


Return to Data Access and Services