This data recipe shows how to download multiple data files from PO.DAAC using GNU wget utility command. GNU Wget is a free utility for non-interactive download of files from the Web. It supports http, https, and ftp protocols, as well as retrieval through http proxies. It is a Unix-based command-line tool, but is also available for other operating system, such as Windows, Mac OS X, etc.
1. wget Command Options
Here is the list of a few key options frequently used:
Do not create a hierarchy of directories when retrieving recursively. With this option turned on, all files will get saved to the current directory, without clobbering (if a name shows up more than once, the filenames will get extensions '.n').
The opposite of '-nd' —create a hierarchy of directories, even if one would not have been created otherwise. E.g. 'wget -x https://podaac.jpl.nasa.gov/robots.txt ' will save the downloaded file to podaac.jpl.nasa.gov/robots.txt.
Disable generation of host-prefixed directories. By default, invoking Wget with '-r https://podaac.jpl.nasa.gov/ ' will create a structure of directories beginning with podaac.jpl.nasa.gov/. This option disables such behavior.
Turn on recursive retrieving. The default maximum depth is 5.
Specify recursion maximum depth level depth.
Try to specify the criteria that match the kind of download you are trying to achieve. If you want to download only one page, use '--page-requisites' without any additional recursion. If you want to download things under one directory, use '-np' to avoid downloading things from other directories. If you want to download all the files from one directory, use '-l 1' to make sure the recursion depth never exceeds one.
2. Download multiple files from PO.DAAC Drive
In order to access PO.DAAC Drive, all users are required to be registered with NASA Earthdata Login. User can login to the PO.DAAC Drive using the following link https://podaac-tools.jpl.nasa.gov/drive/. Figure 1 shows the WebDAV/Programmatic API credentials which will be used later to access the files through wget command. Please note that the password is encrypted, it is different from the Earthdata Login password.
We take the GHRSST SST Level 2 AMSRE datset from REMSS as an example (Figure 2).
* To download one day data files
- Code: Select all
% wget --user=LOGIN --password=PASSWORD -r -nc -np -nH -nd -A "*.nc" "https://podaac-tools.jpl.nasa.gov/drive/files/allData/ghrsst/data/GDS2/L2P/AMSRE/REMSS/v7/2011/001/”
* To download one year data files and create sub-directory
- Code: Select all
% wget --user=LOGIN --password=PASSWORD -r -nc -np -nH -d -A "*.nc" "https://podaac-tools.jpl.nasa.gov/drive/files/allData/ghrsst/data/GDS2/L2P/AMSRE/REMSS/v7/2011/”
Please refer to the following link for more detail information:
Download and Install wget
wget Manual in PDF Format