CLOSED - PO.DAAC OPENDAP UAT Testing Feedback (03/2015)

Open-source Project for a Network Data Access Protocol

CLOSED - PO.DAAC OPENDAP UAT Testing Feedback (03/2015)

Postby podaac » Tue Mar 17, 2015 12:52 pm

This Topic is no longer active. Test Period ended in April 2015

PO.DAAC has recently deployed the latest version of HYRAX 1.10.1 (OPeNDAP) in the new User Acceptance Test (UAT) Server and would like to invite the users, to test their scripts for access performance and provide feedback:

http://opendap-uat.jpl.nasa.gov


The Backend Server (BES) was also updated to utilize an optimized uncompression software for datasets that have been stored with compression and have been identified as a bottleneck in the current operational OPENDAP.


The PO.DAAC THREDDS and LAS servers were also upgraded to the latest versions and selected datasets/files were made available for testing.

http://opendap-uat.jpl.nasa.gov/thredds (v4.3.23)

http://opendap-uat.jpl.nasa.gov/las (v8.2)


(The work on categorizing the list is still work in progress, but users can check out the new layout of LAS.)


Note:
There may be some files from selected datasets that are already in the HYRAX local disk CACHE directory in opendap-uat, which it will use as immediate access to the data. There will be some files that it will be accessed for the first time that will be uncompressed to the CACHE directory first before it performs any reading/subsetting of the data (expect this time to be slower than to the files that already exist in the CACHE directory.)


Please provide the following information from your testing (*)


1. List of Datasets (URL Locations) you are accessing *

2. Run your script via http://opendap-uat.jpl.nasa.gov/ capture performance (this initiates uncompression, if not already in CACHE)
Benchmark time for (2) *

3. Run the same script call again and capture performance (this will access data from CACHE)
Benchmark time for (3) *

4. Run your script with http://opendap.jpl.nasa.gov (currently operational server) and capture performance.
Benchmark time for (4) *

5. Additional feedback.


Your support in testing this new server is greatly appreciated.

Please provide your testing results/feedback by Friday 03/20/2015

Thank you.
PO.DAAC Team
podaac
Site Admin
 
Posts: 322
Joined: Mon Oct 22, 2012 4:00 pm

Re: PO.DAAC OPENDAP UAT Testing Feedback (03/2015)

Postby bobsimons » Wed Mar 25, 2015 10:05 am

Regarding the new PO.DAAC OPENDAP UAT THREDDS:

This catalog.xml
http://opendap-uat.jpl.nasa.gov/thredds ... atalog.xml
listed in the new OpenDAP UAT THREDDS catalog at
http://opendap-uat.jpl.nasa.gov/thredds ... g_old.html
failed.

Also, many of the datasets which are referenced by the new OpenDAP UAT THREDDS catalog do not work.
(Or at least didn't when I initially tested them. The ones I rechecked still have the problem.)
Here are the URLs for the .das requests (so you can easily test them and see the error message) which fail:
http://opendap-uat.jpl.nasa.gov/thredds ... 0.ncml.das
http://opendap-uat.jpl.nasa.gov/thredds ... E.ncml.das
http://opendap-uat.jpl.nasa.gov/thredds ... E.ncml.das
http://opendap-uat.jpl.nasa.gov/thredds ... E.ncml.das
http://opendap-uat.jpl.nasa.gov/thredds ... E.ncml.das
http://opendap-uat.jpl.nasa.gov/thredds ... E.ncml.das
http://opendap-uat.jpl.nasa.gov/thredds ... E.ncml.das
http://opendap-uat.jpl.nasa.gov/thredds ... E.ncml.das
http://opendap-uat.jpl.nasa.gov/thredds ... E.ncml.das
http://opendap-uat.jpl.nasa.gov/thredds ... E.ncml.das
http://opendap-uat.jpl.nasa.gov/thredds ... E.ncml.das
http://opendap-uat.jpl.nasa.gov/thredds ... E.ncml.das
http://opendap-uat.jpl.nasa.gov/thredds ... E.ncml.das
http://opendap-uat.jpl.nasa.gov/thredds ... et.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... gy.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... gy.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... nc.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... ml.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... ml.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... ml.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... ml.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... ml.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... ml.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... ml.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... ml.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... ml.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... ml.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... ml.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... ml.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... ml.jnl.das
http://opendap-uat.jpl.nasa.gov/thredds ... ml.jnl.das
bobsimons
 
Posts: 5
Joined: Wed Mar 25, 2015 9:26 am

Re: PO.DAAC OPENDAP UAT Testing Feedback (03/2015)

Postby mgierach09 » Wed Mar 25, 2015 11:37 am

The new version of LAS is a significant improvement over the last with additional capabilities, such as difference mode (attached below) and correlation viewer! For the latter, if it could be enhanced to permit correlation of variables in different datasets versus just a comparison with variables within an individual dataset it would be extremely useful.

Some other comments:

1. There is saturation in several dataset colorbars. This can be remedied by setting the _FillValue or setting the color range itself.
2. MODIS Aqua 4km daily ocean color is not displaying, providing the error message “The map scale file was not found”.
3. SeaWiFS 9km daily ocean color is not displaying, providing the error message “The map scale file was not found”.
4. The “animate” capability is not working for hdf files (e.g., Aquarius, MODIS ocean color, SeaWiFS ocean color).
5. The “difference mode” capability when comparing plots did not work for any GHRSST dataset or ocean color dataset (e.g., K10, DMI, GAMSSA, ODYSSEA, OSTIA, MODIS chl-a and SeaWiFS chl-a).
Attachments
Screen Shot 2015-03-25 at 11.14.41 AM.png
Screen Shot 2015-03-25 at 11.14.41 AM.png (211.43 KiB) Viewed 4448 times
mgierach09
 
Posts: 3
Joined: Mon Sep 29, 2014 5:27 pm

Re: PO.DAAC OPENDAP UAT Testing Feedback (03/2015)

Postby bobsimons » Thu Mar 26, 2015 7:55 am

There are three datasets with unsorted time axes.
Here is a URL (so you can see the problem) and related information for each dataset.

http://opendap-uat.jpl.nasa.gov/thredds ... l.asc?time[1311:1315]
time[1312]=824430960 > [1313]=824428800
824430960 seconds since 1981-01-01 = 2007-02-16T00:36:00Z
824428800 seconds since 1981-01-01 = 2007-02-16T00:00:00Z

http://opendap-uat.jpl.nasa.gov/thredds ... l.asc?time[2895:2898]
time[2896]=976410720 > [2897]=976406400
976410720 seconds since 1981-01-01 = 2011-12-11T01:12:00Z
976406400 seconds since 1981-01-01 = 2011-12-11T00:00:00Z

http://opendap-uat.jpl.nasa.gov/thredds ... c.asc?time[61:64]
[62]=689040000 > [63]=-347166000
689040000 seconds since 1981-01-01 = 2002-11-02T00:00:00Z

The CF standard
(http://cf-pcmdi.llnl.gov/documents/cf-c ... erminology ,
see the definition for "coordinate variable") and some client
software require that coordinate variables have values in
sorted (monotonic) order.

Sometimes, the non-sortedness reflects a problem (e.g., that
a given file is misnamed, so THREDDS orders it incorrectly
when THREDDS aggregates the dataset). But in the third case above, it
appears to be an incorrect value within one file.

Could you please look into fixing these datasets?

Thank you.
bobsimons
 
Posts: 5
Joined: Wed Mar 25, 2015 9:26 am

Re: PO.DAAC OPENDAP UAT Testing Feedback (03/2015)

Postby bobsimons » Thu Mar 26, 2015 8:07 am

For these datasets
http://opendap-uat.jpl.nasa.gov/thredds ... c.asc?time[0:4]
http://opendap-uat.jpl.nasa.gov/thredds ... c.asc?time[0:4]
the time axis datatype is String.

I know that THREDDS/ncml has an option to grab time values from file names
and make them into String time axis values, but the CF metadata standard defines
"coordinate variables" as
"as a *numeric* data type with values that are ordered monotonically"
http://cfconventions.org/Data/cf-conven ... erminology

I know that for some other datasets, you use other techniques so that
the time values are numeric. Can you please do that for these datasets?

Thank you.
bobsimons
 
Posts: 5
Joined: Wed Mar 25, 2015 9:26 am

Re: PO.DAAC OPENDAP UAT Testing Feedback (03/2015)

Postby bobsimons » Fri Mar 27, 2015 4:25 pm

Some of the datasets have CF metadata, but many just have minimal COARDS metadata. I know it is a huge, long-term effort, but it would be great if you could move towards fuller support for CF metadata. Thanks.
bobsimons
 
Posts: 5
Joined: Wed Mar 25, 2015 9:26 am

Re: PO.DAAC OPENDAP UAT Testing Feedback (03/2015)

Postby rsumagay » Tue Apr 07, 2015 3:49 pm

Hello Bob,

Thank you for all the feedbacks you've provided. Here's just some quick response. I apologize for this delayed response.. I was thoroughly investigating through the issues you pointed out. My comments below.

The list of bad links you provided are indeed links that needed to be removed from the server. The old catalog included some older catalogs that have been moved to a newer catalog.
And for the 'jnl' files they are expected to return the message you are seeing. There were some missing MODIS Annual datasets that were missing in the catalog that are currently being created.

As for the unsorted time axes via. This is quite interesting and this appears to be in the catalog links to the L2P datasets. The pre-generated NCML files pulled time values for each granules and listed them accordingly. I've also tested using the THREDDS configured dataset to that creates the aggregation dynamically as virtual netCDF file. I looked at this file and it shows similar time information as the pre-generated NCML file. What kind of error did you get when you tried to import this in your tool?

(Pre-generated NCML file)
<netcdf id="891" location = "http://opendap-uat.jpl.nasa.gov/opendap/allData/ghrsst/data/L2P_GRIDDED/WSAT/REMSS/2006/008/20060108-WSAT-REMSS-L2P_GRIDDED_25-wsat_20060108v7-v01.nc.gz" coordValue="789523200,789608880"/>
<netcdf id="892" location = "http://opendap-uat.jpl.nasa.gov/opendap/allData/ghrsst/data/L2P_GRIDDED/WSAT/REMSS/2006/009/20060109-WSAT-REMSS-L2P_GRIDDED_25-wsat_20060109v7-v01.nc.gz" coordValue="789609960,789696024"/>
<netcdf id="893" location = "http://opendap-uat.jpl.nasa.gov/opendap/allData/ghrsst/data/L2P_GRIDDED/WSAT/REMSS/2006/010/20060110-WSAT-REMSS-L2P_GRIDDED_25-wsat_20060110v7-v01.nc.gz" coordValue="789696000,789782400"/>
<netcdf id="894" location = "http://opendap-uat.jpl.nasa.gov/opendap/allData/ghrsst/data/L2P_GRIDDED/WSAT/REMSS/2006/011/20060111-WSAT-REMSS-L2P_GRIDDED_25-wsat_20060111v7-v01.nc.gz" coordValue="789782400,789868800"/>
<netcdf id="895" location = "http://opendap-uat.jpl.nasa.gov/opendap/allData/ghrsst/data/L2P_GRIDDED/WSAT/REMSS/2006/012/20060112-WSAT-REMSS-L2P_GRIDDED_25-wsat_20060112v7-v01.nc.gz" coordValue="789868800,789955200"/>
<netcdf id="896" location = "http://opendap-uat.jpl.nasa.gov/opendap/allData/ghrsst/data/L2P_GRIDDED/WSAT/REMSS/2006/013/20060113-WSAT-REMSS-L2P_GRIDDED_25-wsat_20060113v7-v01.nc.gz" coordValue="789955200,790041240"/>
<netcdf id="897" location = "http://opendap-uat.jpl.nasa.gov/opendap/allData/ghrsst/data/L2P_GRIDDED/WSAT/REMSS/2006/014/20060114-WSAT-REMSS-L2P_GRIDDED_25-wsat_20060114v7-v01.nc.gz" coordValue="790041600,790126920"/>
<netcdf id="898" location = "http://opendap-uat.jpl.nasa.gov/opendap/allData/ghrsst/data/L2P_GRIDDED/WSAT/REMSS/2006/015/20060115-WSAT-REMSS-L2P_GRIDDED_25-wsat_20060115v7-v01.nc.gz" coordValue="790128360,790214424"/>
<netcdf id="899" location = "http://opendap-uat.jpl.nasa.gov/opendap/allData/ghrsst/data/L2P_GRIDDED/WSAT/REMSS/2006/016/20060116-WSAT-REMSS-L2P_GRIDDED_25-wsat_20060116v7-v01.nc.gz" coordValue="790214400,790300440"/>


(Dynamically created aggregation)
<netcdf id='/usr/ftp/allData/ghrsst/data/L2P_GRIDDED/WSAT/REMSS/2006/008/20060108-WSAT-REMSS-L2P_GRIDDED_25-wsat_20060108v7-v01.nc.gz' ncoords='2' >
<cache varName='time' >789523200 789608880 </cache>
</netcdf>
<netcdf id='/usr/ftp/allData/ghrsst/data/L2P_GRIDDED/WSAT/REMSS/2006/009/20060109-WSAT-REMSS-L2P_GRIDDED_25-wsat_20060109v7-v01.nc.gz' ncoords='2' >
<cache varName='time' >789609960 789696024 </cache>
</netcdf>
<netcdf id='/usr/ftp/allData/ghrsst/data/L2P_GRIDDED/WSAT/REMSS/2006/010/20060110-WSAT-REMSS-L2P_GRIDDED_25-wsat_20060110v7-v01.nc.gz' ncoords='2' >
<cache varName='time' >789696000 789782400 </cache>
</netcdf>
<netcdf id='/usr/ftp/allData/ghrsst/data/L2P_GRIDDED/WSAT/REMSS/2006/011/20060111-WSAT-REMSS-L2P_GRIDDED_25-wsat_20060111v7-v01.nc.gz' ncoords='2' >
<cache varName='time' >789782400 789868800 </cache>
</netcdf>
<netcdf id='/usr/ftp/allData/ghrsst/data/L2P_GRIDDED/WSAT/REMSS/2006/012/20060112-WSAT-REMSS-L2P_GRIDDED_25-wsat_20060112v7-v01.nc.gz' ncoords='2' >
<cache varName='time' >789868800 789955200 </cache>
</netcdf>


As for the Negative Value in the Dataset (REMO_OI_SST)... Thank you for catching this. It is in the Granule and the Data Provider has been informed and will provide fixed granule file.

As we negotiate to bring in new datasets, we are requiring the data providers to confirm with the CF compliancy. Can you point to me an example of some datasets that have minimal COARDS metadata. I know that our HDF formatted files (ie MODIS L3 SST has minimum metadata).

I also looked into the 2 links that shows time axis datatype as String.
I updated this UAT server to grab time as numeric. The link is updated to point to the following. This link is accessible from the catalog_old.xml and will be relocated/configured in the normal catalog.
http://opendap-uat.jpl.nasa.gov/thredds ... B-AVHRR_OI

Thank you again for your excellent feedback.
By the way since this UAT server is a new server did you find and improved performance to the access?

Thanks,
Rosanna
rsumagay
 
Posts: 3
Joined: Tue Jul 22, 2014 2:26 pm

Re: PO.DAAC OPENDAP UAT Testing Feedback (03/2015)

Postby rsumagay » Tue Apr 07, 2015 4:01 pm

mgierach09 wrote:.....
Some other comments:

1. There is saturation in several dataset colorbars. This can be remedied by setting the _FillValue or setting the color range itself.

The MODIS/SEAWIFS chlorophyll datasets have been fixed to define a _FillValue to the data so that it can be equated in the Scaling Equations.
As for the MODIS L3 SST, I've attempted to force some relevant attributes that I expected for THREDDS/LAS to recognize and use in the Scale_equation. It looks like the SST variable is getting recognized as Unsigned Integer as it should.. but the Attributes already defined in the variable are not getting applied correctly even with the forced changes. I'm still looing in to this issue.

mgierach09 wrote:.....
2. MODIS Aqua 4km daily ocean color is not displaying, providing the error message “The map scale file was not found”.

Please try the link again.

mgierach09 wrote:.....
3. SeaWiFS 9km daily ocean color is not displaying, providing the error message “The map scale file was not found”.

Please try the link again.

mgierach09 wrote:.....
4. The “animate” capability is not working for hdf files (e.g., Aquarius, MODIS ocean color, SeaWiFS ocean color).

Will require the LAS developers to look into this issue.

mgierach09 wrote:.....
5. The “difference mode” capability when comparing plots did not work for any GHRSST dataset or ocean color dataset (e.g., K10, DMI, GAMSSA, ODYSSEA, OSTIA, MODIS chl-a and SeaWiFS chl-a).

I've sent an email to the LAS developers about this issue.. and they are looking into it. They did noticed that these are high resolution datasets and if you are to choose a smaller region then the comparison works.

Once I get further response from the LAS developers then I'll add to this FORUM post.
rsumagay
 
Posts: 3
Joined: Tue Jul 22, 2014 2:26 pm

Re: PO.DAAC OPENDAP UAT Testing Feedback (03/2015)

Postby rsumagay » Thu Apr 16, 2015 3:22 pm

mgierach09 wrote:4. The “animate” capability is not working for hdf files (e.g., Aquarius, MODIS ocean color, SeaWiFS ocean color).

Status on this:
I sent an email to the LAS support email list. I investigated further and it looks like "animate" does not work when the dataset time selection has more than one pull down button (ie. [YYYY][month][day]) example:
[2015] [Apr] [01] (see: AQUARIUS L3 SSS SMI DAILY V3

It does work on dataset with only one time selection button. For example:
[2015-Aug-2011 00:00:00] (see: AQUARIUS L3 SSS SMI 7DAY V3)

Currently waiting for LAS folks to reply.
rsumagay
 
Posts: 3
Joined: Tue Jul 22, 2014 2:26 pm


Return to PO.DAAC OPeNDAP

cron