[MITgcm-devel] [MITgcm-support] Single reduction CG solver

[Jean-Michel Campin]

Martin,

forgot something:
> 3. select one or a few verification experiments to use this solver.
I would propose (but I don't have strong preference) one of the 2 
following set-up: cfc_example or global_with_exf
(both 3-D, hydrostatic, with bathy, and not with already too many 
"specific" things tested)
And if we pick global_with_exf, we can leave one of the 2 test
(std or yearly) with current solver, and use cg2d_sr for the other.
Do you have an other one in mind ?
Cheers,
Jean-Michel

On Fri, Nov 20, 2009 at 02:16:06PM -0500, Jean-Michel Campin wrote:
> Hi Martin,
> 
> It would be good to have this checked in as you propose.
> Just few remarks:
> > 2. add a runtime flag (to PARM03)
>   I would prefer PARAM02 (eliptic solver parameter)
> 
> o No need to worry about JAM, I already started to remove
>   those bits.
> o Can see later on if we want to do something with dynamics
>   allocation in GLOBAL_SUM_VEC but for now, just use it as this.
> 
> Cheers,
> Jean-Michel
> 
> On Fri, Nov 20, 2009 at 09:32:28AM +0100, Martin Losch wrote:
> > Hi Jean-Michel,
> >
> > anything with cg3d will not happen soon (at my end). So go ahead and do 
> > your stuff.
> > However, we should consider putting Christopher's contribution into the 
> > respository, maybe even with a runtime flag. Unfortunately, I do not 
> > currently use any large scale parallel computers, so I won't be able to 
> > test anything, but I can volunteer to add this stuff to the repository. I 
> > would
> > 1. use CPP flags
> > 2. add a runtime flag (to PARM03)
> > 3. select one or a few verification experiments to use this solver.
> >
> > Christopher's code does not include everything, eg. no "JAM" (whatever) 
> > that is, and no vectorization stuff. I can deal with the latter, but not 
> > the former.
> >
> > Martin
> >
> > On Nov 19, 2009, at 7:03 PM, Jean-Michel Campin wrote:
> >
> >> Hi Martin,
> >>
> >> I am right now working on some modif of the 3-D solver
> >> to start to have it with z* (for now, just linear-free-surf),
> >> (but nothing to do with optimisation). I might check-in this
> >> when it passes a coupled of tests. I though it would be good
> >> to sync with you if you planned to implement Christopher's
> >> method.
> >>
> >> Cheers,
> >> Jean-Michel
> >>
> >> On Thu, Nov 19, 2009 at 04:07:39PM +0100, Martin Losch wrote:
> >>> Hi Christopher,
> >>>
> >>> that sounds very good. Do you think that the effect will be similar  
> >>> for
> >>> cg3d?
> >>>
> >>> Are the results between the two cg2d's different? If so, by how much?
> >>>
> >>> Martin
> >>> On Nov 19, 2009, at 1:45 AM, Christopher L. Wolfe wrote:
> >>>
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I've written an implementation of d'Azevedo, Eijkhout, and Romine's
> >>>> (1999) single reduction CG solver for the MITgcm. This method uses a
> >>>> rearrangement of the standard conjugate gradient method so that the
> >>>> required scalars can be determined by a single call to  
> >>>> MPI_Allreduce.
> >>>> For problems running on a large number of processors, the  
> >>>> decreased MPI
> >>>> overhead can significantly increase the efficiency of the conjugate
> >>>> gradient solver.
> >>>>
> >>>> The attached figure shows the scaling of the 2D CG solver for a  
> >>>> fixed
> >>>> problem size of 1024x1024 as the number of processors increase on  
> >>>> the
> >>>> Cray XT4 "Franklin." For less than 2^6 processors, the original cg2d
> >>>> solver is slightly more efficient than the single reduction solver
> >>>> since the latter requires slightly more matrix-vector  
> >>>> multiplications.
> >>>> However, for processor counts over 2^7, the single reduction  
> >>>> solver's
> >>>> performance is significantly better than the original cg2d.
> >>>>
> >>>> I've attached a tarball of the new CG routine cg2d_sr plus the
> >>>> modified files CG2D.h, ini_cg2d.F, and solve_for_pressure.F. To use
> >>>> the single reduce solver, simply drop these files into an experiment
> >>>> directory and compile with "ALLOW_CG2D_SR" defined. I've also  
> >>>> included
> >>>> at benchmarking suite, which can also be compiled like a standard
> >>>> MITgcm experiment.
> >>>>
> >>>> I've only implemented the single reduce solver for the 2D case  
> >>>> since I
> >>>> just use the hydrostatic model, but the 3D implementation should be
> >>>> straightforward. A paper deriving the single reduction method can be
> >>>> found at http://www.netlib.org/lapack/lawnspdf/lawn56.pdf
> >>>>
> >>>> Feel free to contact me with any questions or comments.
> >>>>
> >>>> Cheers,
> >>>> Christopher
> >>>>
> >>>> -----------------------------------------------------------
> >>>> Dr. Christopher L. Wolfe              	   858-534-4560
> >>>> Climate, Atmospheric Science, and Physical Oceanography
> >>>> Scripps Institution of Oceanography, UCSD  clwolfe at ucsd.edu
> >>>> -----------------------------------------------------------
> >>>>
> >>>> <strong.eps>
> >>>>
> >>>>
> >>>> <cg2d_sr.tar.gz>
> >>>> <cg2d_sr_bench.tar.gz>
> >>>>
> >>>> _______________________________________________
> >>>> MITgcm-support mailing list
> >>>> MITgcm-support at mitgcm.org
> >>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
> >>>
> >>>
> >>> _______________________________________________
> >>> MITgcm-support mailing list
> >>> MITgcm-support at mitgcm.org
> >>> http://mitgcm.org/mailman/listinfo/mitgcm-support
> >
> >
> > _______________________________________________
> > MITgcm-devel mailing list
> > MITgcm-devel at mitgcm.org
> > http://mitgcm.org/mailman/listinfo/mitgcm-devel
> 
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel