[MITgcm-devel] mnc and "global" files

[Ed Hill]

Hi folks,

Jean-Michel, Baylor, Daniel, and I recently discussed the lack of
"global" files for mnc/netCDF and have come up with the following scheme
which is designed to be very general/flexible:

  1) For mnc, we won't create a single "global" file like mdsio 
     does.  It just doesn't work well for non-cube domains.  Instead, 
     we will have a "global" format that is PER FACE since each 
     face is logically rectangular, readily maps to netCDF format, 
     and can be easily cut up into one or more tiles.

  2) Given the non-MPI- and non-multithread-writing-safety of 
     netCDF v3 we will (at least initially) only support the 
     READING of "global" (again, think "per-face") files.  The 
     creation of "global" per-face files from collections of 
     per-tile files can be done by a post-processing script.  We 
     have a MatLAB script that does it.

  3) The naming scheme that J-M and I propose is:

        PER FACE:  BASENAME.MYITER.f_FACENUM.nc
          eg:  state.0000000000.f_000001.nc
               phiHydLow.0017280000.f_000003.nc
               dynDiag.0000864000.f_000006.nc

        PER TILE:  BASENAME.MYITER.t_TILENUM.nc
          eg:  state.0000000000.t_000001.nc
               phiHydLow.0017280000.t_000003.nc
               dynDiag.0000864000.t_000201.nc

     where:
       BASENAME gives some indication of the type or source 
         of the variables within the file
       MYITER is a 10-digit number (much like mdsio) containing 
         the model iteration count (myIter) at which the file 
         is created.  Typically, files will start at nIter0 
         and more files will be created as the netCDF files 
         either reach capacity (remember, there is a 2GB file 
         size limit on many filesystems so we can only fit a 
         finite number of time steps in each file) or reach 
         a specified time period (so its easy to create a new 
         set of files every month or year or ...). 
       FACENUM or TILENUM is, respectively, either a global 
         face index (prefaced by "f_") or a global tile index 
         (prefaced by "t_").  A MatLAB script will be written 
         to spatially assemble tile files into "global" per-
         face files.

  4) A new flag or flags (and probably some logic) will be added 
     to allow the specification of how much time or how many 
     model iterations should pass before a new set of netCDF 
     files are created.  New files will use the then-current 
     myIter value for their names so that the correct file 
     sequence is easily recognized.

So, does anyone have any vetoes or suggestions for improvement?

Ed


-- 
Edward H. Hill III, PhD
office:  MIT Dept. of EAPS;  Rm 54-1424;  77 Massachusetts Ave.
             Cambridge, MA 02139-4307
emails:  eh3 at mit.edu                ed at eh3.com
URLs:    http://web.mit.edu/eh3/    http://eh3.com/
phone:   617-253-0098
fax:     617-253-4464