This data recipe shows how to download multiple data files from PODAAC using GNU wget utility command. GNU Wget is a free utility for non-interactive download of files from the Web. It supports http, https, and ftp protocols, as well as retrieval through http proxies. It is a Unix-based command-line tool, but is also available for other operating system, such as Windows, Mac OS X, etc.
[b][color=#FF0000]1. wget Command Options[/color][/b]
Here is the list of a few key options frequently used:
[b]-nd[/b]
--no-directories
Do not create a hierarchy of directories when retrieving recursively. With this option turned on, all files will get saved to the current directory, without clobbering (if a name shows up more than once, the filenames will get extensions '.n').
[b]-x[/b]
--force-directories
The opposite of '-nd' —create a hierarchy of directories, even if one would not have been created otherwise. E.g. "wget -x http://podaac.jpl.nasa.gov/robots.txt" will save the downloaded file to podaac.jpl.nasa.gov/robots.txt.
[b]
-nH[/b]
--no-host-directories
Disable generation of host-prefixed directories. By default, invoking Wget with "-r http://podaac.jpl.nasa.gov/" will create a structure of directories beginning with podaac.jpl.nasa.gov/. This option disables such behavior.
[b]
-r[/b]
--recursive
Turn on recursive retrieving. The default maximum depth is 5.
[b]
-l depth[/b]
--level=depth
Specify recursion maximum depth level depth.
[i]Try to specify the criteria that match the kind of download you are trying to achieve. If you want to download only one page, use '--page-requisites' without any additional recursion. If you want to download things under one directory, use '-np' to avoid downloading things from other directories. If you want to download all the files from one directory, use '-l 1' to make
sure the recursion depth never exceeds one.[/i]
[b][color=#FF0000]2. Download multiple files from PODAAC FTP site[/color][/b]
Let's take GHRSST SST Level 2 datset from REMSS as an example, the dataset landing page is [url=https://podaac.jpl.nasa.gov/dataset/AMSRE-REMSS-L2P-v7a]https://podaac.jpl.nasa.gov/dataset/AMSRE-REMSS-L2P-v7a[/url]. The FTP link for this dataset is indicated by the red circle in Figure 1.
[attachment=1]amsr-e_ftp.png[/attachment]
* To download one day data files
[code]
% wget -r -nc -np -nH -nd -A "*.nc" "ftp://podaac-ftp.jpl.nasa.gov/allData/ghrsst/data/GDS2/L2P/AMSRE/REMSS/v7/2011/001"
[/code]
* To download one year data files and create sub-directory
[code]
% wget -r -nc -np -nH -d -A "*.nc" "ftp://podaac-ftp.jpl.nasa.gov/allData/ghrsst/data/GDS2/L2P/AMSRE/REMSS/v7/2011/"
[/code]
[b][color=#FF0000]3. Download multiple files from PODAAC Drive[/color][/b]
In order to access PODAAC Drive, all users are required to be registered with NASA Earthdata system. User can login to the PODAAC Drive using the following link [url=https://podaac-tools.jpl.nasa.gov/drive/]https://podaac-tools.jpl.nasa.gov/drive/[/url]. Figure 2 shows the WebDAV/Programmatic API credentials which will be used later to access the files through wget command. Please note that the password is encrypted, it is different from the Earthdata Login password.
[attachment=0]podaac_drive.png[/attachment]
Again we take the GHRSST SST Level 2 datset from REMSS as an example.
* To download one day data files
[code]
% wget --user=LOGIN --password=PASSWORD -r -nc -np -nH -nd -A "*.nc" "https://podaac-tools.jpl.nasa.gov/drive/files/OceanTemperature/ghrsst/data/GDS2/L2P/AMSRE/REMSS/v7/2011/001/"
[/code]
* To download one year data files and create sub-directory
[code]
% wget --user=LOGIN --password=PASSWORD -r -nc -np -nH -d -A "*.nc" "https://podaac-tools.jpl.nasa.gov/drive/files/OceanTemperature/ghrsst/data/GDS2/L2P/AMSRE/REMSS/v7/2011/"
[/code]
Please refer to the following link for more detail information:
[url=https://www.gnu.org/software/wget/]Download and Install wget[/url]
[url=https://www.gnu.org/software/wget/manual/wget.pdf]wget Manual in PDF Format[/url]
