[MITgcm-support] mnc questions

[Michael Schaferkotter (Contractor)]

martin:
indeed i 'was' looking to store each variable in a separate netcdf file.
thanx for appreciating the intent.

i also appreciate ed/s remarks and strategy of using nco. i believe that works in theory, however
i haven/t been able to demonstrate that in practice with 'large' domains.

for the time being, i went along with ed (to a point).

using

FC='pgf77'
CC='gcc'

LIBS='-L/common/netcdf/x86/3.6.0-p1/pgf-6.02/lib -lnetcdf'
INCLUDES='-I/common/netcdf/x86/3.6.0-p1/pgf-6.02/include'

i was able to create a single state.*.nc file but of size

[me at mach mnc_test_20051217_0001]$ ls -l state.*.nc
-rw-r--r--  1 me gruppe 4444119560 Dec 18 11:27 state.0000.000001.nc

even though mnc_max_fsize default is 'supposed' to cap at < 2GB. (not sure why such a big file then)

ed:
using your suggestion via nco and ncks particularly (which we/ve used here for a couple of years now)

extract the 2nd and 7th time step of all variables from the state file.

  ncks -d T,1,1  state.0000000000.t001.nc s2.nc      
  ncks -d T,6,6  state.0000000000.t001.nc s7.nc

let/s grab all variables at the initial step.
[me at mach mnc_test_20051217_0001]$ ncks -d T,0,0 state.0000.000001.nc t0.nc
ncks: ERROR unable to determine method for remote retrieval of state.0000.000001.nc

let/s grab all temperature data
[me at mach mnc_test_20051217_0001]$ ncks -v Temp state.0000.000001.nc t.nc
ncks: ERROR unable to determine method for remote retrieval of state.0000.000001.nc

i suspect the file is too big for temperorary files or swap or something.

note. the state .nc file has 169 time steps and the domain is not large (this is my 'little' test problem)

the original file looks like:
[me at mach mnc_test_20051217_0001]$ ncdump state.0000.000001.nc | more
netcdf state.0000.000001 {
dimensions:
        T = UNLIMITED ; // (169 currently)
        Xp1 = 86 ;
        Y = 61 ;
        Z = 105 ;
        X = 85 ;
        Yp1 = 62 ;
variables:
        int T(T) ;
                T:long_name = "iteration_count" ;
        double model_time(T) ;
                model_time:long_name = "Model Time" ;
                model_time:units = "s" ;
        double Xp1(Xp1) ;
        double Y(Y) ;
        double Z(Z) ;
        double U(T, Z, Y, Xp1) ;
                U:units = "m/s" ;
                U:coordinates = "XU YU RC iter" ;
        double X(X) ;
        double Yp1(Yp1) ;
        double V(T, Z, Yp1, X) ;
                V:units = "m/s" ;
                V:coordinates = "XV YV RC iter" ;
        double Temp(T, Z, Y, X) ;
                Temp:units = "degC" ;
                Temp:long_name = "potential_temperature" ;
                Temp:coordinates = "XC YC RC iter" ;
        double S(T, Z, Y, X) ;
                S:long_name = "salinity" ;
                S:coordinates = "XC YC RC iter" ;
        double Eta(T, Y, X) ;
                Eta:long_name = "free-surface_r-anomaly" ;
                Eta:units = "m" ;
                Eta:coordinates = "XC YC iter" ;
        double W(T, Z, Y, X) ;
                W:units = "m/s" ;
                W:coordinates = "XC YC RC iter" ;
        double phi_nh(T, Z, Y, X) ;

so the file looks 'OK'.

for a smaller file, things work:

[me at mach mnc_test_20051219_0001]$ ls -l state.0000.000001.nc 
-rw-r--r--  1 me gruppe 52598716 Dec 19 13:17 state.0000.000001.nc

[me at mach mnc_test_20051219_0001]$ ncks -d T,0,0 state.0000.000001.nc t0.nc

now here/s another experiment.

a 10GB file is written.

[me at mach run]$ ls -l mnc_test_20051219_0001/
total 11470788
-rw-r--r--  1 me gruppe    14085928 Dec 19 13:35 grid.0000.000001.nc
-rw-r--r--  1 me gruppe  1659412424 Dec 19 14:37 phiHyd.0000.000001.nc
-rw-r--r--  1 me gruppe    15808016 Dec 19 14:37 phiHydLow.0000.000001.nc
-rw-r--r--  1 me gruppe 10045280876 Dec 19 14:37 state.0000.000001.nc

so something is really amiss.

try to build another way.

i tried to build with g77 and gcc, but genmake2 disabled mnc because of some failure of the netcdf test programs,
using g77 and gcc with and without name-mangling yields 

*********************************************************************
WARNING: the "mnc" package was enabled but tests failed to compile
  NetCDF applications.  Please check that:

  1) NetCDF is correctly installed for this compiler and
  2) the LIBS variable (within the "optfile") specifies the correct 
       NetCDF library to link against.

  Due to this failure, the "mnc" package is now DISABLED.

the bright side is that with 
FC_NAMEMANGLE="#define FC_NAMEMANGLE(X) X ## _" we do get to use the 
stat function

===  Checking system libraries  ===
  Do we have the system() command using g77...  yes
  Do we have the fdate() command using g77...  yes
  Do we have the etime() command using g77...  yes
  Can we call simple C routines (here, "cloc()") using g77...  yes
  Can we use stat() through C calls...  yes
  Can we create NetCDF-enabled binaries...  no

things are certainly not as expected.

here is the data.mnc file:

[me at mach run]$ cat data.mnc
# Example "data.mnc" file
 &MNC_01
 mnc_use_outdir=.TRUE.,
 mnc_outdir_str='mnc_test_',
 mnc_outdir_date=.TRUE.,
 snapshot_mnc=.TRUE.,
 monitor_mnc=.FALSE.,
 mnc_max_fsize=3.0e+06,
 &

i understand why mnc_max_fsize has no effect, but i don/t understand why the files that are being written
are larger than 2GB. (snapshot_mnc has been toggled and the file size is still > 2GB.)

michael