[MITgcm-support] Data formats and archiving hints

[Christopher L. Wolfe]

Jody,

Just a few comments on MDS:

  - You don't need to write per-tile MDS files. If you set  
globalFiles=.TRUE. or useSingleCpuIO=.TRUE. in data&PARM01, you'll  
just get one file per field.

  - MDS meta files written by the diagnostics package *do* contain a  
"modeltime" field (at least in recent model releases), but the "dump"  
files do not. The "modeltime" variable field is however commented out  
in the meta file and is not read by rdmds. You could probably write a  
simple Matlab script to read this field, though.

I'll have to let someone else comment on netCDF. It's something I've  
always meant to start using, but have never gotten around to it ...

Christopher

On Aug 2, 2009, at 11:48 AM, Klymak Jody wrote:

>
> Hi all,
>
> As an amateur numerical modeller using the MITgcm I thought I'd ask
> for folks' data format and archiving ideas/advice.
>
> I do my analysis in Matlab, and am unlikely to change that.  I've been
> writing the bare binary files (mds?) and reading those in fine with
> the matlab rdmds.m function.  It works very well, and I appreciate the
> effort that went into it.
>
> However, as I get to larger simulations (ahem, larger for me means 16
> or 32 tiles instead of 4 or 8), I start to wonder about the thousands
> of tile files on my machine, and if that is really the most efficient
> way for me to be storing my data.  So:
>
> Is there an inherent advantage to switching to netcdf?
>
> To be honest I'm not sure what files are produced from the netcdf
> output - it looks like they are per-tile, and monolithic in that one
> file contains the whole run for that tile?  If correct,  how fast are
> they to read in matlab?  I'm running a simulation that will reach 3Gb/
> tile.
>
> Is there more meta information?  I am always flumoxed that there is no
> "time" in the MDS meta files, so I have to figure out what dt was for
> my run and multiply by iteration number.
>
> Parallel discussion:  How do folks organize and keep track of their
> model runs?  I have a large number now, and quite frankly I forget
> which ones are trash, and which ones I am using for my latest paper.
> Sure, I have to be more organized, but rather than invent the wheel,
> I'd love to hear how folks who have been doing this for a while keep
> track.  Being lazy, automagic methods are always appreciated...
>
> Thanks for any thoughts folks feel like sharing...
>
> Cheers,  Jody
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support