[MITgcm-support] gluemnc

[Oliver Jahn]

Hi,

I have a version of the python script gluemncbig that can write netcdf3
files with 64-bit offsets (the “64-bit Offset Format”).  Every
variable/time slice still has to be below 2GB but the file can be
larger.  I'll dig it up and check the changes into cvs.

Oliver


On 2014-06-20 04:00, Martin Losch wrote:
> Hi John,
> 
> as far as I remember, netcdf3 only allows file sizes of up to 2GB, and you error messages basically tell the same story.
> 
> If your nco can already do netcdf4, then you’ll probably have to go through gluemnc and add the appropriate options (—64 or —4) when files are combined, as suggested by your error messages. It’s probably enough to do it for occurences of “ncrcat” and “ncks”
> 
> Alternatively you could try to use the python script utils/python/MITgcmutils/scripts/gluemncbig
> that is usually much faster, but to be honest, I don’t know about file sizes (ie. netcdf4 support). Maybe Oliver can say something about that?
> 
> Martin
> 
> On Jun 19, 2014, at 11:53 PM, John Pender <jgpender at alaska.edu> wrote:
> 
>> Hi All-
>> I have a question about gluemnc.  I have been getting netcdf output from MITgcm for a long time, which has been working fine with output data and with the grid files.  I run in multiprocessor mode so there is a netcdf output file for each tile and I glob them together using gluemnc.  This works great on smaller grids (240x200 by 100 layers) but I've started with a much denser grid (1200x1000 by 100 layers) and suddenly - for the grid file - every single 2D field (drC, Depth, etc) is missing in the resulting grid.glob.nc file though the 1D fields ( RC, drF, etc) show up fine.  The computer here (pacman at ARSC) has a couple options for the nco module
>>
>> nco/4.3.0.gnu-4.7.3(default)
>> nco/4.3.1.gnu-4.7.3 
>>
>> and I get the same problem either way.  All the way at the end of the log file for the gluemnc run for the grid I see some new error messages alluding to large file size, which is exactly my problem.  There is also a mention of the distinction between netcdf3 and netcdf4 but I've been assuming MITgcm used netcdf4 by default.
>>
>> …
>> ERROR NC_EVARSIZE One or more variable sizes violate format constraints
>> HINT: NC_EVARSIZE errors can occur when attempting to aggregate netCDF3
>> classic files together into outputs that exceed the capacity of the netCDF3
>> classic file format, e.g., a variable with size in excess of 2^31 bytes.
>> In this case, try altering the output file type to netCDF3 classic with 64-bit offsets (with --64)
>> or to netCDF4 (with -4). For more details, see http://nco.sf.net/nco.html#fl_fmt
>> ...
>>
>> The total size of the grid material is about 3GB.
>>
>> I'm hoping others have run into this and there's a simple fix, but I'm psychologically prepared for the worst.
>>
>> Thanks very much,
>>
>> John
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
> 
> 
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support
>