[MITgcm-support] Data formats and archiving hints
Klymak Jody
jklymak at uvic.ca
Sun Aug 2 14:48:20 EDT 2009
Hi all,
As an amateur numerical modeller using the MITgcm I thought I'd ask
for folks' data format and archiving ideas/advice.
I do my analysis in Matlab, and am unlikely to change that. I've been
writing the bare binary files (mds?) and reading those in fine with
the matlab rdmds.m function. It works very well, and I appreciate the
effort that went into it.
However, as I get to larger simulations (ahem, larger for me means 16
or 32 tiles instead of 4 or 8), I start to wonder about the thousands
of tile files on my machine, and if that is really the most efficient
way for me to be storing my data. So:
Is there an inherent advantage to switching to netcdf?
To be honest I'm not sure what files are produced from the netcdf
output - it looks like they are per-tile, and monolithic in that one
file contains the whole run for that tile? If correct, how fast are
they to read in matlab? I'm running a simulation that will reach 3Gb/
tile.
Is there more meta information? I am always flumoxed that there is no
"time" in the MDS meta files, so I have to figure out what dt was for
my run and multiply by iteration number.
Parallel discussion: How do folks organize and keep track of their
model runs? I have a large number now, and quite frankly I forget
which ones are trash, and which ones I am using for my latest paper.
Sure, I have to be more organized, but rather than invent the wheel,
I'd love to hear how folks who have been doing this for a while keep
track. Being lazy, automagic methods are always appreciated...
Thanks for any thoughts folks feel like sharing...
Cheers, Jody
More information about the MITgcm-support
mailing list