Using Python to subset large gridded datasets (v2)

[yiboj]

This python script is the upgraded version of the previous post by Mike and Ed on Using Python and OPeNDAP to subset large gridded datasets. The script combines the PODAAC web granule spatial and temporal search service, and the OPeNDAP to subset and download most of the Level 3 and Level 4 datasets from PODAAC server.

The current version only works with Level 3 and Level 4 gridded datasets in netcdf format. Subsetting for level 2 datasets and hdf format is currently in progress and will be posted here once it is available.

Please note that you have to have the python software, wget or curl installed on the computer.

You can download the script subset_dataset.py here.

For example, to extract MUR data from 2010-01-01 to 2010-02-05 for a region bounded by -140 to -110 degrees longitude and 20 to 30 degrees latitude, this request can be made on the UNIX command line:

% ./subset_dataset.py -s 20100101 -f 20100205 -b -140 -110 20 30 -x MUR-JPL-L4-GLOB-v4.1

where JPL-L4UHfnd-GLOB-MUR is the short name for MUR dataset. Shortname is the required field for the script to run and the script will print out the help menu without the shortname. Shortname can be found from PODAAC web portal.

User could also download the Docker image for this recipe from dockerhub, and run the recipe using jupyter nootbook:
https://hub.docker.com/r/podaacdatarecipes/dataset_subsetting/

[hliuhliu]

Hi, I tested out other level 3 and 4 datasets but it shows No granules found for dataset.
I've tried the following datasets commands:
python subset_dataset.py -s 20120101 -f 20120103 -b -140 -110 20 30 -x AQUARIUS_L3_SSS_SMI_DAILY_V4
python subset_dataset.py -s 20060101 -f 20060103 -b -140 -110 20 30 -x QSCAT_LEVEL_2B_OWV_COMP_12
python subset_dataset.py -s 20060101 -f 20060103 -b -140 -110 20 30 -x QSCAT_LEVEL_3_V2

Non of the above works, it shows:
Please wait while program searching for the granules ...

No granules found for dataset: QSCAT_LEVEL_2B_OWV_COMP_12
or No latitude or longitude info for dataset: QSCAT_LEVEL_2B_OWV_COMP_12
Program will exit now !


Please guide my on how to subset opendap data other than MURS data.

[yiboj]

Hi, Thanks for the inquiry, I will update the scripts and please check back later. Thanks.

[omkarr]

Hi,

I think there is an error in the line 217 and 218 where the 'wsurl' assigned in line 217 is being over written in line 218 and hence missing so many user input parameters while sending a get request.

Thanks,
Omkar

[yiboj]

Hi Omkar,

Thanks for pointing out the error in the code, we will make the corresponding update.
Thanks,

[omkarr]

Hi,

Is this the part of the code [0] where the lats and lons are being subsetted, I am a little bit confused over here.

One more thing, does the extract granule webservice return "GRIDDED" l3 and l4 datasets??

Thanks,
Omkar

[0]

Code: Select all

  nlon = nj
  nlat = ni

  dint_lon = (lon1-lon0)/float(nlon)
  dint_lat = (lat1-lat0)/float(nlat)

  [i0,i1]=boundingindex(lon0,dint_lon,nlon,box[0],box[1])
  [j0,j1]=boundingindex(lat0,dint_lat,nlat,box[2],box[3])
  if i0>i1 or j0>j1:
    sys.exit('No grid point in your domain box.')

  # modify the max grid indices, as necessary:
  if ig>1:
    i1=max(span(i0,i1,ig))
    j1=max(span(j0,j1,ig))

[yiboj]

The codes in this part is to find the index of latitude and longitude of the bounding box which is the region where data is extracted.

The Grided level 3 and level 4 granules are returned from the script.

[omkarr]

Thank you very much.

[dgroenen]

Hi,

I'm having problems accessing and using subset_dataset.py to get data from JPL PODACC.

I've typed in:

python subset_dataset.py -s 19820101 -f 20050101 -x AVHRR_OI-NCEI-L4-GLOB-v2.0 -n -b -110.0 -60.0 5.0 40.0

It worked yesterday just fine but is not loading today. I think it's a problem with the server. It says "Please wait while program searching for granules..." and then just freezes up.

Thank you,
Danielle

[podaac]

Dear Danielle,

Thank you for bringing this issue to our attention.

Unfortunately, we encountered a server glitch and as a result our server ran out of resources.

Please try your data request again, as the issue has been resolved.

We greatly appreciate your patience over the last couple days and apologize for any inconvenience this may have caused you.

Respectfully,
PO.DAAC User Services and Interfaces Team