[MITgcm-devel] [MITgcm-support] Single reduction CG solver

[Jean-Michel Campin]

Hi Martin,

It would be good to have this checked in as you propose.
Just few remarks:
> 2. add a runtime flag (to PARM03)
  I would prefer PARAM02 (eliptic solver parameter)

o No need to worry about JAM, I already started to remove
  those bits.
o Can see later on if we want to do something with dynamics
  allocation in GLOBAL_SUM_VEC but for now, just use it as this.

Cheers,
Jean-Michel

On Fri, Nov 20, 2009 at 09:32:28AM +0100, Martin Losch wrote:
> Hi Jean-Michel,
>
> anything with cg3d will not happen soon (at my end). So go ahead and do 
> your stuff.
> However, we should consider putting Christopher's contribution into the 
> respository, maybe even with a runtime flag. Unfortunately, I do not 
> currently use any large scale parallel computers, so I won't be able to 
> test anything, but I can volunteer to add this stuff to the repository. I 
> would
> 1. use CPP flags
> 2. add a runtime flag (to PARM03)
> 3. select one or a few verification experiments to use this solver.
>
> Christopher's code does not include everything, eg. no "JAM" (whatever) 
> that is, and no vectorization stuff. I can deal with the latter, but not 
> the former.
>
> Martin
>
> On Nov 19, 2009, at 7:03 PM, Jean-Michel Campin wrote:
>
>> Hi Martin,
>>
>> I am right now working on some modif of the 3-D solver
>> to start to have it with z* (for now, just linear-free-surf),
>> (but nothing to do with optimisation). I might check-in this
>> when it passes a coupled of tests. I though it would be good
>> to sync with you if you planned to implement Christopher's
>> method.
>>
>> Cheers,
>> Jean-Michel
>>
>> On Thu, Nov 19, 2009 at 04:07:39PM +0100, Martin Losch wrote:
>>> Hi Christopher,
>>>
>>> that sounds very good. Do you think that the effect will be similar  
>>> for
>>> cg3d?
>>>
>>> Are the results between the two cg2d's different? If so, by how much?
>>>
>>> Martin
>>> On Nov 19, 2009, at 1:45 AM, Christopher L. Wolfe wrote:
>>>
>>>>
>>>> Hi all,
>>>>
>>>> I've written an implementation of d'Azevedo, Eijkhout, and Romine's
>>>> (1999) single reduction CG solver for the MITgcm. This method uses a
>>>> rearrangement of the standard conjugate gradient method so that the
>>>> required scalars can be determined by a single call to  
>>>> MPI_Allreduce.
>>>> For problems running on a large number of processors, the  
>>>> decreased MPI
>>>> overhead can significantly increase the efficiency of the conjugate
>>>> gradient solver.
>>>>
>>>> The attached figure shows the scaling of the 2D CG solver for a  
>>>> fixed
>>>> problem size of 1024x1024 as the number of processors increase on  
>>>> the
>>>> Cray XT4 "Franklin." For less than 2^6 processors, the original cg2d
>>>> solver is slightly more efficient than the single reduction solver
>>>> since the latter requires slightly more matrix-vector  
>>>> multiplications.
>>>> However, for processor counts over 2^7, the single reduction  
>>>> solver's
>>>> performance is significantly better than the original cg2d.
>>>>
>>>> I've attached a tarball of the new CG routine cg2d_sr plus the
>>>> modified files CG2D.h, ini_cg2d.F, and solve_for_pressure.F. To use
>>>> the single reduce solver, simply drop these files into an experiment
>>>> directory and compile with "ALLOW_CG2D_SR" defined. I've also  
>>>> included
>>>> at benchmarking suite, which can also be compiled like a standard
>>>> MITgcm experiment.
>>>>
>>>> I've only implemented the single reduce solver for the 2D case  
>>>> since I
>>>> just use the hydrostatic model, but the 3D implementation should be
>>>> straightforward. A paper deriving the single reduction method can be
>>>> found at http://www.netlib.org/lapack/lawnspdf/lawn56.pdf
>>>>
>>>> Feel free to contact me with any questions or comments.
>>>>
>>>> Cheers,
>>>> Christopher
>>>>
>>>> -----------------------------------------------------------
>>>> Dr. Christopher L. Wolfe              	   858-534-4560
>>>> Climate, Atmospheric Science, and Physical Oceanography
>>>> Scripps Institution of Oceanography, UCSD  clwolfe at ucsd.edu
>>>> -----------------------------------------------------------
>>>>
>>>> <strong.eps>
>>>>
>>>>
>>>> <cg2d_sr.tar.gz>
>>>> <cg2d_sr_bench.tar.gz>
>>>>
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>
>>>
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
>
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel