Download Multiple Data Files from PODAAC Drive Using curl
Posted: Thu Sep 20, 2018 8:54 amPODAAC Drive can be accessed both by wget and curl command on linux system. The recipe "Download Multiple Data Files from PODAAC Drive Using wget" shows how to use wget command, and this recipe will focus on the curl command.
The major difference between wget and curl is that wget can download files recursively and curl can upload file to the server.
The curl command allows you to download as well as upload data through the command line in Linux. Following is its syntax:
1. curl Command Options
Here is the list of a few key options frequently used:
2. Download multiple files from PODAAC Drive
In order to access PODAAC Drive, all users are required to be registered with NASA Earthdata system. User can login to the PODAAC Drive using the following link https://podaac-tools.jpl.nasa.gov/drive/. Figure 1 shows the WebDAV/Programmatic API credentials which will be used later to access the files through wget command. Please note that the password is encrypted, it is different from the Earthdata Login password.
Again we take the GHRSST SST Level 2 datset from REMSS as an example (see Figure 2).
* To download one data file
* To download one day data files
The shell script can also be accessed here.
curl is a tool to transfer data from or to a server, using one of the
supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP,
IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS,
TELNET and TFTP). The command is designed to work without user interaction.
curl offers a busload of useful tricks like proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, Metalink, and more. As you will see below, the number of features will make your head spin!
The major difference between wget and curl is that wget can download files recursively and curl can upload file to the server.
The curl command allows you to download as well as upload data through the command line in Linux. Following is its syntax:
- Code: Select all
curl [options] [URL...]
1. curl Command Options
Here is the list of a few key options frequently used:
| url | One or multiple URLs that will be fetched in sequence.. |
| -u user:password | The username and password to use for server authentication. |
| -o filename | Write output to filename instead of stdout. |
| -O | Write output to a local file named like the remote file we get. (Only the file part of the remote file is used, the path is cut off.) |
| --compressed | Request a compressed response using one of the algorithms curl supports (gzip), and save the uncompressed document. |
2. Download multiple files from PODAAC Drive
In order to access PODAAC Drive, all users are required to be registered with NASA Earthdata system. User can login to the PODAAC Drive using the following link https://podaac-tools.jpl.nasa.gov/drive/. Figure 1 shows the WebDAV/Programmatic API credentials which will be used later to access the files through wget command. Please note that the password is encrypted, it is different from the Earthdata Login password.
- Figure 1
- podaac_drive.png (57.44 KiB) Viewed 13689 times
Again we take the GHRSST SST Level 2 datset from REMSS as an example (see Figure 2).
- Figure 2
- amsr-e_ftp.png (113.75 KiB) Viewed 13689 times
* To download one data file
- Code: Select all
% curl -O -u LOGIN:PASSWORD https://podaac-tools.jpl.nasa.gov/drive/files/OceanTemperature/ghrsst/data/GDS2/L2P/AMSRE/REMSS/v7/2011/001/20110101013456-REMSS-L2P_GHRSST-SSTsubskin-AMSRE-l2b_v07a_r46074.dat-v02.0-fv01.0.nc
* To download one day data files
- Code: Select all
driveroot='https://podaac-tools.jpl.nasa.gov/'
rootdir='https://podaac-tools.jpl.nasa.gov/drive/files/OceanTemperature/ghrsst/data/GDS2/L2P/AMSRE/REMSS/v7/2011/001/'
login_web='LOGIN:PASSWORD'
for file in $(curl -u $login_web -s $rootdir |
grep href |
sed 's/.*href="//' |
sed 's/".*//' |
grep '.nc$'); do
curl -u $login_web -O $driveroot$file
done
The shell script can also be accessed here.