LFTP and podaac drive

LFTP and podaac drive

Postby alexfore » Tue Jun 04, 2019 1:58 pm

I've been using LFTP to mirror things for many protocols. It doesn't work right with PODAAC drive:
Code: Select all
lftp -u USER,PASSWORD https://podaac-tools.jpl.nasa.gov/drive/files
cd /drive/files/OceanWinds/ascat/preview/L2/metop_a/25km/2019/
mirror -c

You'll see it downloads every directory as a file. The HTTPS podaac drive interface doesn't seem to be fully standards compliant is what I think is going on.
alexfore
 
Posts: 2
Joined: Tue Jun 04, 2019 1:48 pm

Re: LFTP and podaac drive

Postby alexfore » Wed Jun 05, 2019 7:09 am

Anyways the only acceptable option I have found is to use rclone:

https://rclone.org/

wget makes too much junk, curl is flakey, lftp doesn't work right, and I need to do things in user space (no webdav mounts)
alexfore
 
Posts: 2
Joined: Tue Jun 04, 2019 1:48 pm

Re: LFTP and podaac drive

Postby podaac » Thu Jun 06, 2019 1:37 pm

Thank you for your feedback.

We are currently looking into this issue and will get back to you as soon as possible.

Thank you for your continued patience.
podaac
Site Admin
 
Posts: 337
Joined: Mon Oct 22, 2012 4:00 pm

Re: LFTP and podaac drive

Postby jroberts » Wed Jun 12, 2019 6:08 am

Are there any updates to accessing the drive via LFTP? I've also used LFTP for many years to handle downloads across numerous protocols (including HTTPS).

I am using the latest LFTP (4.8.4, CentOS x86_64)). Any connection attempt results in:
cd: Access failed: 401 Unauthorized (/drive/files)

I know that LFTP allows you to set a number of options ... perhaps something needs to be set in the ~/.lftp/rc file to allow the connection to be made??

However, I can use wget (with user/pass) to download directly from the drive so Im fairly sure Im using the correct information. But, Im having separate issues with wget and recursively downloading.

Any advice is welcome!
jroberts
 
Posts: 3
Joined: Tue Jun 11, 2019 1:24 pm

Re: LFTP and podaac drive

Postby jamesbro » Tue Jun 18, 2019 11:30 am

Hello jroberts,

I've connected just fine using lftp (version 4.4.8) on both a CentOS and RedHat machine without having to configure the /lftp/.rc file.

I would say to first, please re-verify the credentials you are using. Once you log back into Earthdata, use this link below to see your username and password.

https://podaac-tools.jpl.nasa.gov/drive/

After you retrieve the credentials from the link, you should be able to connect just fine.

Command I used to connect
lftp -u USERNAME,PASSWORD ACCESS_URL

If you still are not able to connect:
One, please reply back with the results.
Two, try pinging podaac-tools.jpl.nasa.gov
ping podaac-tools.jpl.nasa.gov
jamesbro
 
Posts: 3
Joined: Fri Jun 07, 2019 7:48 am

Re: LFTP and podaac drive

Postby jroberts » Tue Jun 18, 2019 11:58 am

Hello jamesbro,

I re-verified my credentials but still can't get it to work. I've tried lftp 4.8.4 from both a Mac (High Sierra) and a Linux box (CentOS).

The command was (password is commented out):
lftp -u jroberts,##### https://podaac-tools.jpl.nasa.gov/drive/files
cd: Access failed: 401 Unauthorized (/drive/files)

... and with debug turned on:
---- Resolving host address...
---- 1 address found: 137.78.248.120
---- Connecting to podaac-tools.jpl.nasa.gov (137.78.248.120) port 443
---- Sending request...
---> HEAD /drive/files HTTP/1.1
---> Host: podaac-tools.jpl.nasa.gov
---> User-Agent: lftp/4.8.4
---> Accept: */*
---> Connection: keep-alive
--->
Certificate depth: 2; subject: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA; issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
Certificate depth: 1; subject: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA; issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
Certificate depth: 0; subject: /C=US/ST=California/L=Pasadena/O=NASA Jet Propulsion Laboratory/OU=OCIO/CN=*.jpl.nasa.gov; issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
Certificate verification: subjectAltName: ‘podaac-tools.jpl.nasa.gov’ matched
<--- HTTP/1.1 401 Unauthorized
<--- Date: Tue, 18 Jun 2019 19:50:27 GMT
<--- Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5
<--- Strict-Transport-Security: max-age=31536000; includeSubDomains;
<--- Content-Type: text/html; charset=iso-8859-1
<--- Strict-Transport-Security: max-age=631138519; includeSubDomains
<--- Access-Control-Allow-Origin: *
<--- Keep-Alive: timeout=15, max=2048
<--- Connection: Keep-Alive
<---
cd: Access failed: 401 Unauthorized (/drive/files)
---- Closing HTTP connection

_______________

ping podaac-tools.jpl.nasa.gov
PING podaac-tools.jpl.nasa.gov (137.78.248.120): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
Request timeout for icmp_seq 5
Request timeout for icmp_seq 6
Request timeout for icmp_seq 7
Request timeout for icmp_seq 8

Pinging seems to result in a timeout. It almost seems to me like I need to somehow let the server *know* that I am a registered Earthdata user ... for example, when I use a browser to get to the drive, first I have to enter by Earthdata login ... and then it recognizes I have valid credentials and lets me see the https:// drive access url. Is there something I should be doing at the command line prior to issuing the lftp command? My only counterpoint to this potential is that I can access a specific file URL when using wget with the same user/pass credentials (though I do have a .netrc file with earthdata login info that is needed when downloading, for example, MERRA-2 data files). See https://wiki.earthdata.nasa.gov/display ... L+And+Wget

I was wondering if there is an analogue needed to make lftp work?
jroberts
 
Posts: 3
Joined: Tue Jun 11, 2019 1:24 pm

Re: LFTP and podaac drive

Postby jamesbro » Tue Jun 18, 2019 12:49 pm

Thank you for your prompt reply! We will update you with troubleshooting steps once we dig further into the information you provided us. Until then, please keep using the "wget" command to retrieve the files needed.

Thank you again for the information.
jamesbro
 
Posts: 3
Joined: Fri Jun 07, 2019 7:48 am

Re: LFTP and podaac drive

Postby jamesbro » Thu Jun 20, 2019 5:43 am

Good morning jroberts. Just want you to know that we are still working on this issue. I do have another question though.

When your running this LFTP command, is your CentOS machine: Completely off the network (as in working from home); VPN'ed into the network; or is this machine on the actual NASA/JPL network?
jamesbro
 
Posts: 3
Joined: Fri Jun 07, 2019 7:48 am

Re: LFTP and podaac drive

Postby jroberts » Thu Jun 20, 2019 7:01 am

Thanks for continuing to work on this.

This occurs attempting to connect from two machines (MacOS and CentOS) both on the NASA MSFC network. It also occurred attempting to connect from a remote server on another network (that requires a VPN to use to login (though not on the NASA network) ... not sure what their restrictions are reaching outward, however).

However, it did work successfully (sort of) from another server I have access to which is located on the west coast (and which required a VPN to login to, but also is not the NASA network). By "sort of," I mean that it could connect and `ls` directories and subdirectories that I directly connect to (i.e. specify the full path when connecting), but once connected I could not seem to navigate around using the `cd` commands.

I attempted to use lftp from the NCCS Discover system as well (version 4.8.3) but received a TLS handshake error ... this is something I've seen with older versions of lftp.

I was able to do what I needed by using wget() and obtaining a list of all the individual URLs through the Earthdata search portal. Globbing with wget also seemed to be problematic for me at times ... sometimes it worked and sometimes it didnt ... but as long as I had the complete URL wget worked fine.
jroberts
 
Posts: 3
Joined: Tue Jun 11, 2019 1:24 pm


Return to PO.DAAC Drive