[MITgcm-devel] [MITgcm-support] Single reduction CG solver

[Martin Losch]

Hi Jean-Michel,

anything with cg3d will not happen soon (at my end). So go ahead and  
do your stuff.
However, we should consider putting Christopher's contribution into  
the respository, maybe even with a runtime flag. Unfortunately, I do  
not currently use any large scale parallel computers, so I won't be  
able to test anything, but I can volunteer to add this stuff to the  
repository. I would
1. use CPP flags
2. add a runtime flag (to PARM03)
3. select one or a few verification experiments to use this solver.

Christopher's code does not include everything, eg. no  
"JAM" (whatever) that is, and no vectorization stuff. I can deal with  
the latter, but not the former.

Martin

On Nov 19, 2009, at 7:03 PM, Jean-Michel Campin wrote:

> Hi Martin,
>
> I am right now working on some modif of the 3-D solver
> to start to have it with z* (for now, just linear-free-surf),
> (but nothing to do with optimisation). I might check-in this
> when it passes a coupled of tests. I though it would be good
> to sync with you if you planned to implement Christopher's
> method.
>
> Cheers,
> Jean-Michel
>
> On Thu, Nov 19, 2009 at 04:07:39PM +0100, Martin Losch wrote:
>> Hi Christopher,
>>
>> that sounds very good. Do you think that the effect will be similar  
>> for
>> cg3d?
>>
>> Are the results between the two cg2d's different? If so, by how much?
>>
>> Martin
>> On Nov 19, 2009, at 1:45 AM, Christopher L. Wolfe wrote:
>>
>>>
>>> Hi all,
>>>
>>> I've written an implementation of d'Azevedo, Eijkhout, and Romine's
>>> (1999) single reduction CG solver for the MITgcm. This method uses a
>>> rearrangement of the standard conjugate gradient method so that the
>>> required scalars can be determined by a single call to  
>>> MPI_Allreduce.
>>> For problems running on a large number of processors, the  
>>> decreased MPI
>>> overhead can significantly increase the efficiency of the conjugate
>>> gradient solver.
>>>
>>> The attached figure shows the scaling of the 2D CG solver for a  
>>> fixed
>>> problem size of 1024x1024 as the number of processors increase on  
>>> the
>>> Cray XT4 "Franklin." For less than 2^6 processors, the original cg2d
>>> solver is slightly more efficient than the single reduction solver
>>> since the latter requires slightly more matrix-vector  
>>> multiplications.
>>> However, for processor counts over 2^7, the single reduction  
>>> solver's
>>> performance is significantly better than the original cg2d.
>>>
>>> I've attached a tarball of the new CG routine cg2d_sr plus the
>>> modified files CG2D.h, ini_cg2d.F, and solve_for_pressure.F. To use
>>> the single reduce solver, simply drop these files into an experiment
>>> directory and compile with "ALLOW_CG2D_SR" defined. I've also  
>>> included
>>> at benchmarking suite, which can also be compiled like a standard
>>> MITgcm experiment.
>>>
>>> I've only implemented the single reduce solver for the 2D case  
>>> since I
>>> just use the hydrostatic model, but the 3D implementation should be
>>> straightforward. A paper deriving the single reduction method can be
>>> found at http://www.netlib.org/lapack/lawnspdf/lawn56.pdf
>>>
>>> Feel free to contact me with any questions or comments.
>>>
>>> Cheers,
>>> Christopher
>>>
>>> -----------------------------------------------------------
>>> Dr. Christopher L. Wolfe              	   858-534-4560
>>> Climate, Atmospheric Science, and Physical Oceanography
>>> Scripps Institution of Oceanography, UCSD  clwolfe at ucsd.edu
>>> -----------------------------------------------------------
>>>
>>> <strong.eps>
>>>
>>>
>>> <cg2d_sr.tar.gz>
>>> <cg2d_sr_bench.tar.gz>
>>>
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support