Using Python to subset large gridded datasets (v2)

Using Python to subset large gridded datasets (v2)

Postby yiboj » Mon Mar 30, 2015 12:52 pm

This python script is the upgraded version of the previous post by Mike and Ed on Using Python and OPeNDAP to subset large gridded datasets. The script combines the PODAAC web granule spatial and temporal search service, and the OPeNDAP to subset and download most of the Level 3 and Level 4 datasets from PODAAC server.

The current version only works with Level 3 and Level 4 gridded datasets in netcdf format. Subsetting for level 2 datasets and hdf format is currently in progress and will be posted here once it is available.

Please note that you have to have the python software, wget or curl installed on the computer.

You can download the script subset_dataset.py here.

For example, to extract MUR data from 2010-01-01 to 2010-02-05 for a region bounded by -140 to -110 degrees longitude and 20 to 30 degrees latitude, this request can be made on the UNIX command line:

% ./subset_dataset.py -s 20100101 -f 20100205 -b -140 -110 20 30 -x MUR-JPL-L4-GLOB-v4.1

where JPL-L4UHfnd-GLOB-MUR is the short name for MUR dataset. Shortname is the required field for the script to run and the script will print out the help menu without the shortname. Shortname can be found from PODAAC web portal.

User could also download the Docker image for this recipe from dockerhub, and run the recipe using jupyter nootbook:
https://hub.docker.com/r/podaacdatarecipes/dataset_subsetting/
yiboj
 
Posts: 79
Joined: Mon Mar 30, 2015 11:22 am

Re: Using Python to subset large gridded datasets (2)

Postby hliuhliu » Mon Aug 10, 2015 2:31 pm

Hi, I tested out other level 3 and 4 datasets but it shows No granules found for dataset.
I've tried the following datasets commands:
python subset_dataset.py -s 20120101 -f 20120103 -b -140 -110 20 30 -x AQUARIUS_L3_SSS_SMI_DAILY_V4
python subset_dataset.py -s 20060101 -f 20060103 -b -140 -110 20 30 -x QSCAT_LEVEL_2B_OWV_COMP_12
python subset_dataset.py -s 20060101 -f 20060103 -b -140 -110 20 30 -x QSCAT_LEVEL_3_V2

Non of the above works, it shows:
Please wait while program searching for the granules ...

No granules found for dataset: QSCAT_LEVEL_2B_OWV_COMP_12
or No latitude or longitude info for dataset: QSCAT_LEVEL_2B_OWV_COMP_12
Program will exit now !


Please guide my on how to subset opendap data other than MURS data.
hliuhliu
 
Posts: 1
Joined: Mon Aug 10, 2015 2:17 pm

Re: Using Python to subset large gridded datasets (2)

Postby yiboj » Mon Aug 24, 2015 8:35 am

Hi, Thanks for the inquiry, I will update the scripts and please check back later. Thanks.
yiboj
 
Posts: 79
Joined: Mon Mar 30, 2015 11:22 am

Re: Using Python to subset large gridded datasets (v2)

Postby omkarr » Mon Jun 27, 2016 10:58 am

Hi,

I think there is an error in the line 217 and 218 where the 'wsurl' assigned in line 217 is being over written in line 218 and hence missing so many user input parameters while sending a get request.

Thanks,
Omkar
omkarr
 
Posts: 25
Joined: Wed Jun 15, 2016 12:34 am

Re: Using Python to subset large gridded datasets (v2)

Postby yiboj » Mon Jun 27, 2016 9:02 pm

Hi Omkar,

Thanks for pointing out the error in the code, we will make the corresponding update.
Thanks,
yiboj
 
Posts: 79
Joined: Mon Mar 30, 2015 11:22 am

Re: Using Python to subset large gridded datasets (v2)

Postby omkarr » Fri Jul 01, 2016 12:34 pm

Hi,

Is this the part of the code [0] where the lats and lons are being subsetted, I am a little bit confused over here.

One more thing, does the extract granule webservice return "GRIDDED" l3 and l4 datasets??

Thanks,
Omkar

[0]
Code: Select all
  nlon = nj
  nlat = ni

  dint_lon = (lon1-lon0)/float(nlon)
  dint_lat = (lat1-lat0)/float(nlat)

  [i0,i1]=boundingindex(lon0,dint_lon,nlon,box[0],box[1])
  [j0,j1]=boundingindex(lat0,dint_lat,nlat,box[2],box[3])
  if i0>i1 or j0>j1:
    sys.exit('No grid point in your domain box.')

  # modify the max grid indices, as necessary:
  if ig>1:
    i1=max(span(i0,i1,ig))
    j1=max(span(j0,j1,ig))
omkarr
 
Posts: 25
Joined: Wed Jun 15, 2016 12:34 am

Re: Using Python to subset large gridded datasets (v2)

Postby yiboj » Thu Jul 07, 2016 1:13 pm

The codes in this part is to find the index of latitude and longitude of the bounding box which is the region where data is extracted.

The Grided level 3 and level 4 granules are returned from the script.
yiboj
 
Posts: 79
Joined: Mon Mar 30, 2015 11:22 am

Re: Using Python to subset large gridded datasets (v2)

Postby omkarr » Thu Jul 07, 2016 1:21 pm

Thank you very much.
omkarr
 
Posts: 25
Joined: Wed Jun 15, 2016 12:34 am

Re: Using Python to subset large gridded datasets (v2)

Postby dgroenen » Fri Jul 29, 2016 8:57 am

Hi,

I'm having problems accessing and using subset_dataset.py to get data from JPL PODACC.

I've typed in:

python subset_dataset.py -s 19820101 -f 20050101 -x AVHRR_OI-NCEI-L4-GLOB-v2.0 -n -b -110.0 -60.0 5.0 40.0

It worked yesterday just fine but is not loading today. I think it's a problem with the server. It says "Please wait while program searching for granules..." and then just freezes up.

Thank you,
Danielle
dgroenen
 
Posts: 2
Joined: Fri Jul 29, 2016 8:53 am

Re: Using Python to subset large gridded datasets (v2)

Postby podaac » Mon Aug 01, 2016 10:16 am

Dear Danielle,

Thank you for bringing this issue to our attention.

Unfortunately, we encountered a server glitch and as a result our server ran out of resources.

Please try your data request again, as the issue has been resolved.

We greatly appreciate your patience over the last couple days and apologize for any inconvenience this may have caused you.

Respectfully,
PO.DAAC User Services and Interfaces Team
podaac
Site Admin
 
Posts: 316
Joined: Mon Oct 22, 2012 4:00 pm

Next

Return to Data Access and Services

cron