[MITgcm-devel] Re: mnc pickups

[Ed Hill]

On Tue, 2005-05-24 at 09:33 -0400, Baylor Fox-Kemper wrote:
> Hi Ed,
>    I just wanted to let you know what's going on with the netcdf pickup 
> files.  There is a bit of an issue/frustration from the community which 
> I think is preventing folks from using netcdf.

Hi Baylor,

I hope you don't mind me moving this discussion over to the devel list.
The points you make are very good and I think others ought to see them.


> 1) Lack of global writes.  This is a slight problem but not so bad with 
> 2)

This one is tough to fix.  Unfortunately, the netCDF libraries are
neither multi-thread nor multi-processor safe.  So only one thread or
MPI process can write to one file at any given time.  There are ways
that we can enforce this with mutexes and MPI barriers but its going to
be slow and complicated.

Looking ahead, the next generation of netCDF (netCDF v4 is based on
HDF5) is supposed to solve a lot of these parallel read-write problems
in an elegant fashion:

  http://hdf.ncsa.uiuc.edu/whatishdf5.html

So probably the best short-to-mid-term solution is to handle this with
post-processing scripts/programs.


> 2) utils/matlab/interpickups.m   I wrote this little script to take any 
> set of pickups on any number of processors and tiles and convert it to 
> another set of pickups on any number of processors and tiles.  It 
> doesn't use exchanges at all (currently it assumes periodic in x and 
> bounded in y, but that's easy to change), so it won't work on cube 
> sphere.

I haven't had time to look at this but its a really good idea.  Both
Chris and Jean-Michel have asked for a similar tool.  Hopefully, we can
take your version and extend it to work for more general topologies like
the cube.


> 3) Lack of global reads.  This is a bigger problem, as it requires one 
> to use 2) every time the number of processors is changed.  This is a 
> real pain in the behind, since, for example, my pickup files are 120 
> 2.4 Meg files, which amounts to about 30 minutes of interpolating time.

Yes!  Global reads is something that I should have done from the very
beginning.  It'll take only a week or two to code and debug it so
hopefully it can be done this summer.


> 4) Naming inconsistency:  The other problem with mnc pickups is that 
> the write name is different from the read name.  The write name is 
> pickup.iter.#.proc.nc where # is usually 0000, but I suppose could be 
> 0001 if (Lord help you) your pickup files were greater than 2 gb.  This 
> feature should probably be disabled, since the model looks for pickup 
> files named pickup.iter.proc.nc, so one has to run a script to remove 
> the 0000.

The naming issue is very easy to handle.  We can either:

  (a) leave out the sequence number when its zero, or
  (b) have the model, by default, assume a sequence number of "0000"
      for the pickups.

Please let me know which you prefer.  I like (b).


> 5) Minor point:  pickups don't contain model_time, which is a real pain.

This is an easy fix.  I'll talk to Andrea and Jean-Michel and make sure
that the times work correctly with the recent changes.



> A snippet from an email I just got from a collaborator:
> 
> "If you make global pickup files (hopefully NOT netcdf!), I should be 
> able to
> use any number of processors."

You can always use a mix of netCDF and MDSIO.  Its not (and never has
been!) an all-or-nothing choice.  Folks certainly can use MDSIO for the
pickups and netCDF for the other forms of output.


> So, in summary, I think that 2) alleviates much of the problem with 1). 
>   That is, with an example of a global pickup file (perhaps the model 
> could even write this from node #1 at the zeroth time-step?) then it is 
> easy to create a global pickup from any collection of pickups.  Someone 
> should probably write a version of 2) that works on the cube sphere, 
> too.  4) and 5) can be corrected in a matter of minutes I am sure.  3) 
> is the big sticking point and it should be fixed asap, as right now it 
> really cripples the use of mnc pickups.

Yes, I can check in changes for #4 & #5 later this week.

And in my copious spare time [;-)] I'll try to make some progress on #3
and #2 (for more general topologies).

Ed

-- 
Edward H. Hill III, PhD
office:  MIT Dept. of EAPS;  Rm 54-1424;  77 Massachusetts Ave.
             Cambridge, MA 02139-4307
emails:  eh3 at mit.edu                ed at eh3.com
URLs:    http://web.mit.edu/eh3/    http://eh3.com/
phone:   617-253-0098
fax:     617-253-4464