Download Multiple Data Files from PODAAC Drive Using curl

Programming scripts on how to read, analyze and access PO.DAAC data

Download Multiple Data Files from PODAAC Drive Using curl

Postby yiboj » Thu Sep 20, 2018 8:54 am

PODAAC Drive can be accessed both by wget and curl command on linux system. The recipe "Download Multiple Data Files from PODAAC Drive Using wget" shows how to use wget command, and this recipe will focus on the curl command.

curl is a tool to transfer data from or to a server, using one of the
supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP,
IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS,
TELNET and TFTP). The command is designed to work without user interaction.

curl offers a busload of useful tricks like proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, Metalink, and more. As you will see below, the number of features will make your head spin!


The major difference between wget and curl is that wget can download files recursively and curl can upload file to the server.

The curl command allows you to download as well as upload data through the command line in Linux. Following is its syntax:
Code: Select all
curl [options] [URL...]


1. curl Command Options

Here is the list of a few key options frequently used:







urlOne or multiple URLs that will be fetched in sequence..
-u user:passwordThe username and password to use for server authentication.
-o filenameWrite output to filename instead of stdout.
-OWrite output to a local file named like the remote file we get. (Only the file part of the remote file is used, the path is cut off.)
--compressedRequest a compressed response using one of the algorithms curl supports (gzip), and save the uncompressed document.


2. Download multiple files from PODAAC Drive

In order to access PODAAC Drive, all users are required to be registered with NASA Earthdata system. User can login to the PODAAC Drive using the following link https://podaac-tools.jpl.nasa.gov/drive/. Figure 1 shows the WebDAV/Programmatic API credentials which will be used later to access the files through wget command. Please note that the password is encrypted, it is different from the Earthdata URS password.
podaac_drive.png
Figure 1
podaac_drive.png (57.44 KiB) Viewed 762 times


Again we take the GHRSST SST Level 2 datset from REMSS as an example (see Figure 2).
amsr-e_ftp.png
Figure 2
amsr-e_ftp.png (113.75 KiB) Viewed 762 times


* To download one data file
Code: Select all
% curl -O -u LOGIN:PASSWORD https://podaac-tools.jpl.nasa.gov/drive/files/OceanTemperature/ghrsst/data/GDS2/L2P/AMSRE/REMSS/v7/2011/001/20110101013456-REMSS-L2P_GHRSST-SSTsubskin-AMSRE-l2b_v07a_r46074.dat-v02.0-fv01.0.nc


* To download one day data files
Code: Select all
driveroot='https://podaac-tools.jpl.nasa.gov/'
rootdir='https://podaac-tools.jpl.nasa.gov/drive/files/OceanTemperature/ghrsst/data/GDS2/L2P/AMSRE/REMSS/v7/2011/001/'
login_web='LOGIN:PASSWORD'

for file in $(curl -u $login_web -s $rootdir |
                  grep href |
                  sed 's/.*href="//' |
                  sed 's/".*//' |
                  grep '.nc$'); do
    curl -u $login_web -O $driveroot$file
done


The shell script can also be accessed here.
yiboj
 
Posts: 79
Joined: Mon Mar 30, 2015 11:22 am

Return to Data Recipes

cron