[MITgcm-support] netcdf4 and useSingleCpuIO = .TRUE.,

[Ryan Abernathey]

My impression is that this is a feature that many people would like but no
one wants to spend the time implementing. Parallel netCDF would solve many
simultaneous problems related to I/O and postprocessing which are growing
more severe as model runs get bigger and bigger.

Because MITgcm is not supported by a modeling center (like GFDL), it is
hard to prioritize these sort of "infrastructure" enhancements over
science-driven enhancements (new parameterizations, etc.)...even though the
infrastructure would accelerate the science in the long run. I have floated
the idea of applying for an NSF SSE grant to support such I/O enhancements
in MITgcm
https://www.nsf.gov/pubs/2016/nsf16532/nsf16532.htm

What is needed is some guidance from the MITgcm "leadership" about the long
term roadmap for the model.

-Ryan


On Fri, Feb 3, 2017 at 12:20 PM, Jody Klymak <jklymak at uvic.ca> wrote:

>
> Hi All,
>
> I was wondering if anyone had investigated implementing useSingleCpuIO (or
> its equivalent) for mnc via netcdf4?  It seems vaguely possible that it is
> relatively simple as changing `NF_CREATE` to `NF90_CREATE` (
> https://unidata.ucar.edu/software/netcdf/workshops/
> 2010/pnetcdf/UsingF90Parallel.html) and changing the tile-write structure
> to all write to the same file.
>
> However, I’m sure its not *that* easy, and I didn’t want to look into it
> further if its a) impractical, or b) someone has already started.
>
> This comes from machine-memory issues I have working on my super computer
> making mds files impractical (even using Ryan’s excellent `xmitgcm`), and a
> dislike of having to glue 1024 nc files together into a single file.  I
> guess I’m willing to be told that I’m being dumb and the gluemnc step is
> not a big deal.
>
> I would also like, if possible, for the *.nc files to be written one per
> time dump, simply because I often only deal with the last few time steps,
> and having Tb-sized files seems clunky.  But maybe that is just silly.
>
> Thanks a lot,   Jody
>
>
>
> --
> Jody Klymak
> http://web.uvic.ca/~jklymak/
>
>
>
>
>
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20170203/c8dd1bc8/attachment.htm>