Page 1 of 3

Using Python to subset large gridded datasets (v2)

PostPosted: Mon Mar 30, 2015 12:52 pm
by yiboj
This python script is the upgraded version of the previous post by Mike and Ed on Using Python and OPeNDAP to subset large gridded datasets. The script combines the PODAAC web granule spatial and temporal search service, and the OPeNDAP to subset and download most of the Level 3 and Level 4 datasets from PODAAC server.

The current version only works with Level 3 and Level 4 gridded datasets in netcdf format. Subsetting for level 2 datasets and hdf format is currently in progress and will be posted here once it is available.

Please note that you have to have the python software, wget or curl installed on the computer.

You can download the script subset_dataset.py here.

For example, to extract MUR data from 2010-01-01 to 2010-02-05 for a region bounded by -140 to -110 degrees longitude and 20 to 30 degrees latitude, this request can be made on the UNIX command line:

% ./subset_dataset.py -s 20100101 -f 20100205 -b -140 -110 20 30 -x MUR-JPL-L4-GLOB-v4.1

where JPL-L4UHfnd-GLOB-MUR is the short name for MUR dataset. Shortname is the required field for the script to run and the script will print out the help menu without the shortname. Shortname can be found from PODAAC web portal.

User could also download the Docker image for this recipe from dockerhub, and run the recipe using jupyter nootbook:
https://hub.docker.com/r/podaacdatarecipes/dataset_subsetting/

Re: Using Python to subset large gridded datasets (2)

PostPosted: Mon Aug 10, 2015 2:31 pm
by hliuhliu
Hi, I tested out other level 3 and 4 datasets but it shows No granules found for dataset.
I've tried the following datasets commands:
python subset_dataset.py -s 20120101 -f 20120103 -b -140 -110 20 30 -x AQUARIUS_L3_SSS_SMI_DAILY_V4
python subset_dataset.py -s 20060101 -f 20060103 -b -140 -110 20 30 -x QSCAT_LEVEL_2B_OWV_COMP_12
python subset_dataset.py -s 20060101 -f 20060103 -b -140 -110 20 30 -x QSCAT_LEVEL_3_V2

Non of the above works, it shows:
Please wait while program searching for the granules ...

No granules found for dataset: QSCAT_LEVEL_2B_OWV_COMP_12
or No latitude or longitude info for dataset: QSCAT_LEVEL_2B_OWV_COMP_12
Program will exit now !


Please guide my on how to subset opendap data other than MURS data.

Re: Using Python to subset large gridded datasets (2)

PostPosted: Mon Aug 24, 2015 8:35 am
by yiboj
Hi, Thanks for the inquiry, I will update the scripts and please check back later. Thanks.

Re: Using Python to subset large gridded datasets (v2)

PostPosted: Mon Jun 27, 2016 10:58 am
by omkarr
Hi,

I think there is an error in the line 217 and 218 where the 'wsurl' assigned in line 217 is being over written in line 218 and hence missing so many user input parameters while sending a get request.

Thanks,
Omkar

Re: Using Python to subset large gridded datasets (v2)

PostPosted: Mon Jun 27, 2016 9:02 pm
by yiboj
Hi Omkar,

Thanks for pointing out the error in the code, we will make the corresponding update.
Thanks,

Re: Using Python to subset large gridded datasets (v2)

PostPosted: Fri Jul 01, 2016 12:34 pm
by omkarr
Hi,

Is this the part of the code [0] where the lats and lons are being subsetted, I am a little bit confused over here.

One more thing, does the extract granule webservice return "GRIDDED" l3 and l4 datasets??

Thanks,
Omkar

[0]
Code: Select all
  nlon = nj
  nlat = ni

  dint_lon = (lon1-lon0)/float(nlon)
  dint_lat = (lat1-lat0)/float(nlat)

  [i0,i1]=boundingindex(lon0,dint_lon,nlon,box[0],box[1])
  [j0,j1]=boundingindex(lat0,dint_lat,nlat,box[2],box[3])
  if i0>i1 or j0>j1:
    sys.exit('No grid point in your domain box.')

  # modify the max grid indices, as necessary:
  if ig>1:
    i1=max(span(i0,i1,ig))
    j1=max(span(j0,j1,ig))

Re: Using Python to subset large gridded datasets (v2)

PostPosted: Thu Jul 07, 2016 1:13 pm
by yiboj
The codes in this part is to find the index of latitude and longitude of the bounding box which is the region where data is extracted.

The Grided level 3 and level 4 granules are returned from the script.

Re: Using Python to subset large gridded datasets (v2)

PostPosted: Thu Jul 07, 2016 1:21 pm
by omkarr
Thank you very much.

Re: Using Python to subset large gridded datasets (v2)

PostPosted: Fri Jul 29, 2016 8:57 am
by dgroenen
Hi,

I'm having problems accessing and using subset_dataset.py to get data from JPL PODACC.

I've typed in:

python subset_dataset.py -s 19820101 -f 20050101 -x AVHRR_OI-NCEI-L4-GLOB-v2.0 -n -b -110.0 -60.0 5.0 40.0

It worked yesterday just fine but is not loading today. I think it's a problem with the server. It says "Please wait while program searching for granules..." and then just freezes up.

Thank you,
Danielle

Re: Using Python to subset large gridded datasets (v2)

PostPosted: Mon Aug 01, 2016 10:16 am
by podaac
Dear Danielle,

Thank you for bringing this issue to our attention.

Unfortunately, we encountered a server glitch and as a result our server ran out of resources.

Please try your data request again, as the issue has been resolved.

We greatly appreciate your patience over the last couple days and apologize for any inconvenience this may have caused you.

Respectfully,
PO.DAAC User Services and Interfaces Team