Generating time Variable to Aggregate MODIS L3 Data Files

[yiboj]

The MODIS L3 dataset, such as the MODIS Terra Level 3 SST Mid-IR Daily dataset, has no time variable. Therefore it is impossible to aggregate the data files with time. In this post, we will show you how to create time variable using NCO utility, and how to aggregate the files. In the end, we will put the NCO command into python script.

Let's take MODIS Terra Level 3 SST Mid-IR Daily data file T2017001.L3m_DAY_SST4_sst4_4km.nc on January 1, 2017 as example, and we will extract the global attribute time_coverage_end as the time value and convert it to second since since 1981-01-01 00:00:00.

First we retrieve the time_coverage_end value:

Code: Select all

$> ncks -M -m T2017001.L3m_DAY_SST4_sst4_4km.nc | grep -E 'time_coverage_end' | cut -f 11 -d ' '
2017-01-01T11:50:09.000Z


The time value is 1136116209 after converted to second since 1981-01-01 00:00:00. We will show how to calculate the seconds in the python script later.

Next, we create the time vaiable and assign the necessary attributes to it:

Code: Select all

$> ncap2 -O -s 'time=1136116209' T2017001.L3m_DAY_SST4_sst4_4km.nc T2017001.L3m_DAY_SST4_sst4_4km.nc
$> ncatted -O -a long_name,time,c,c,'reference time of sst field' T2017001.L3m_DAY_SST4_sst4_4km.nc
$> ncatted -O -a axis,time,c,c,'T' T2017001.L3m_DAY_SST4_sst4_4km.nc
$> ncatted -O -a units,time,c,c,'seconds since 1981-01-01 00:00:00 UTC' T2017001.L3m_DAY_SST4_sst4_4km.nc
$> ncatted -O -a comment,time,c,c,'Nominal time of analyzed fields' T2017001.L3m_DAY_SST4_sst4_4km.nc


Now we can put all the above command into python script as follows:

Code: Select all

#! /usr/bin/env python
#
import sys,os
import time
from datetime import datetime, date, timedelta
from math import floor,ceil
from optparse import OptionParser
import subprocess

import numpy as np

###############
# subroutines #
###############
def diffdates(d1, d2):
    #2007-12-31T08:50:07.000Z
    #Date format: %Y-%m-%dT%H:%M:%S:%Z
    #return (time.mktime(time.strptime(d2,"%Y-%m-%dT%H:%M:%S.%fZ")) -
    #           time.mktime(time.strptime(d1, "%Y-%m-%dT%H:%M:%S.%fZ")))
    td=datetime.strptime(d2,"%Y-%m-%dT%H:%M:%S.%fZ")-datetime.strptime(d1,"%Y-%m-%dT%H:%M:%S.%fZ")
    return int(round(td.total_seconds()))

def standalone_main():

    filename = "T2017001.L3m_DAY_SST4_sst4_4km.nc"
    fcmd = "ncks -M -m "+filename+" | grep -E 'time_coverage_end' | cut -f 11 -d ' '"
    date_str = os.popen(fcmd).readlines()
    p = subprocess.Popen(fcmd, stdout=subprocess.PIPE, shell=True)
    (date_str, err) = p.communicate()
    adate_str = date_str.rstrip()
    print adate_str
    bdate_str = "1981-01-01T00:00:00.000Z"
    adiff = diffdates(bdate_str, adate_str)
    acmd = "ncap2 -O -s 'time="+str(int(adiff)) + "' "+filename+" "+filename
    os.system(acmd)
    os.system("ncatted -O -a long_name,time,c,c,'reference time of sst field' "+filename)
    os.system("ncatted -O -a axis,time,c,c,'T' "+filename)
    os.system("ncatted -O -a units,time,c,c,'seconds since 1981-01-01 00:00:00 UTC' "+filename)
    os.system("ncatted -O -a comment,time,c,c,'Nominal time of analyzed fields' "+filename)

if __name__ == "__main__":
        standalone_main()


Finally, after we update some data files with new time variable, we can use NCO utility to aggregate all the data files as show below:

Code: Select all

$> ncecat `ls *.nc` temp.nc