Using Python to subset large gridded datasets (v2)

Re: Using Python to subset large gridded datasets (v2)

Postby dgroenen » Mon Aug 01, 2016 10:34 am

Thank you PODACC. It is working now!
dgroenen
 
Posts: 2
Joined: Fri Jul 29, 2016 8:53 am

Re: Using Python to subset large gridded datasets (v2)

Postby robert.williamson » Wed Aug 10, 2016 2:58 am

Hello. I have attempted to run the subset_dataset.py script on the MODIS L3 Aqua/Terra products as follows:

python subset_dataset.py -s 20020704 -f 20151231 -b 10 40 -40 -20 -x MODIS_TERRA_L3_SST_THERMAL_DAILY_4KM_DAYTIME_V2014.0

and receive the following result:

'Granule file format may not be in netcdf or no latitude or longitude info for dataset: MODIS_TERRA_L3_SST_THERMAL_DAILY_4KM_DAYTIME_V2014.0'

The python scripts has worked fine on other datasets so I assume there may be a problem with the MODIS L3. Do you have any suggestions/comments?

I have also now noticed this with the OSTIA L4 product, which downloaded without a problem a few months ago. ODYSSEA, K10, MW_IR all download ok still.
robert.williamson
 
Posts: 1
Joined: Wed Aug 10, 2016 2:52 am

Re: Using Python to subset large gridded datasets (v2)

Postby yiboj » Mon Aug 15, 2016 1:08 pm

Hi,
Thanks for the inquiry. The python script is using OPENDAP to subset the datasets. But unfortunately the v2014,0 MODIS datasets are not in OPENDAP catalog yet, so the script does not work for these datasets. I will check with the system engineer team and let you know once the OPENDAP works with these datastes.
yiboj
 
Posts: 80
Joined: Mon Mar 30, 2015 11:22 am

Re: Using Python to subset large gridded datasets (v2)

Postby mariakatosvic » Fri Sep 16, 2016 10:46 pm

i think Shortname is the required field for the script to run and the script will print out the help menu without the shortname. can it be found from PODAAC web portal.where
Last edited by mariakatosvic on Tue Sep 20, 2016 3:41 am, edited 1 time in total.
mariakatosvic
 
Posts: 2
Joined: Fri Sep 16, 2016 4:23 am

Re: Using Python to subset large gridded datasets (v2)

Postby yiboj » Sat Sep 17, 2016 8:11 pm

Hi,
Thanks for the inquiry. The following post in the General Information FAQ section shows the dataset shortname and where you can find it.

viewtopic.php?f=12&t=418
yiboj
 
Posts: 80
Joined: Mon Mar 30, 2015 11:22 am

Re: Using Python to subset large gridded datasets (v2)

Postby rabernat » Mon Dec 05, 2016 5:23 pm

This is a perfect case for xarray.
http://xarray.pydata.org/en/stable/index.html

See the OPeNDAP example in the docs
http://xarray.pydata.org/en/stable/io.html#opendap
rabernat
 
Posts: 4
Joined: Mon Dec 05, 2016 5:08 pm

Re: Using Python to subset large gridded datasets (v2)

Postby yiboj » Tue Dec 06, 2016 9:39 am

thanks for sharing the info, xarray is definitely a great tool to manipulate netcdf file
yiboj
 
Posts: 80
Joined: Mon Mar 30, 2015 11:22 am

Re: Using Python to subset large gridded datasets (v2)

Postby rsdpc » Wed Oct 24, 2018 3:31 pm

Hi, I using this script to subset data from AVHRR_OI-NCEI-L4-GLOB-v2.0. I enter this:

Code: Select all
subset_dataset.py -s 19810901 -f 20171014 -b -6 -5.5 36 36.2 -x


and receive the following result:

Code: Select all
Please wait while program searching for the granules ...
Longitude range: -6.000000 to -5.500000
Latitude range: 36.000000 to 36.200000
  every 1 pixel(s) is obtained

grid dimensions will be ( 4 x 2 )
OK to download?  [yes or no]: yes

The script will need curl or wget on the system, please install them first before running the script !
Program will exit now !


I have installed both wget and curl on the system (including them in PATH folder and as environment variables), and both of them work well. However the script doesn't seem to recognize they are installed somehow. Is there any solution for this? :? :?

Thanks in advance
rsdpc
 
Posts: 2
Joined: Wed Oct 24, 2018 3:12 pm

Re: Using Python to subset large gridded datasets (v2)

Postby rsdpc » Thu Oct 25, 2018 2:03 am

Hi again, I tried to set up everything in another computer (with Windows 7 instead of Windows 10) but now I get a different problem:

Code: Select all
C:\Python27\Scripts>subset_dataset.py -s 19810901 -f 20171014 -b -6 -5.5 36 36.2
 -x AVHRR_OI-NCEI-L4-GLOB-v2.0

Please wait while program searching for the granules ...

Traceback (most recent call last):
  File "C:\Python27\Scripts\subset_dataset.py", line 542, in <module>
    standalone_main()
  File "C:\Python27\Scripts\subset_dataset.py", line 237, in standalone_main
    response = urllib.urlopen(wsurl)
  File "C:\Python27\lib\urllib.py", line 87, in urlopen
    return opener.open(url)
  File "C:\Python27\lib\urllib.py", line 213, in open
    return getattr(self, name)(url)
  File "C:\Python27\lib\urllib.py", line 443, in open_https
    h.endheaders(data)
  File "C:\Python27\lib\httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "C:\Python27\lib\httplib.py", line 882, in _send_output
    self.send(msg)
  File "C:\Python27\lib\httplib.py", line 844, in send
    self.connect()
  File "C:\Python27\lib\httplib.py", line 1263, in connect
    server_hostname=server_hostname)
  File "C:\Python27\lib\ssl.py", line 369, in wrap_socket
    _context=self)
  File "C:\Python27\lib\ssl.py", line 617, in init
    self.do_handshake()
  File "C:\Python27\lib\ssl.py", line 846, in do_handshake
    self._sslobj.do_handshake()
IOError: [Errno socket error] [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:7
26)


Can anyone help me? Thanks
rsdpc
 
Posts: 2
Joined: Wed Oct 24, 2018 3:12 pm

Re: Using Python to subset large gridded datasets (v2)

Postby podaac » Fri Oct 26, 2018 12:27 pm

Thank you for your inquiry.

We are currently looking into this matter and will get back to you as soon as possible.

Thank you for your continued patience.
podaac
Site Admin
 
Posts: 316
Joined: Mon Oct 22, 2012 4:00 pm

PreviousNext

Return to Data Access and Services

cron