[MITgcm-devel] [MITgcm-support] Single reduction CG solver

[Martin Losch]

I forgot to say, that S/R GLOBAL_VEC_SUM_R8 (Ed's code) actually uses  
dynamic array allocation:
>       INTEGER ndim, nval, myThid
>       Real*4 sumPhi(ndim,nSx,nSy)
> CEOP
>
> C     !LOCAL VARIABLES:
> C     mpiRC    :: MPI return code
>       INTEGER i, bi,bj
>       Real*8  tmp1(nval), tmp2(nval)
Is that a problem in general? (using this routine instead of  
global_sum_tile, etc.  is the part of the cg2d_sr, that makes it  
"single reduction")

Martin

On Nov 20, 2009, at 9:32 AM, Martin Losch wrote:

> Hi Jean-Michel,
>
> anything with cg3d will not happen soon (at my end). So go ahead and  
> do your stuff.
> However, we should consider putting Christopher's contribution into  
> the respository, maybe even with a runtime flag. Unfortunately, I do  
> not currently use any large scale parallel computers, so I won't be  
> able to test anything, but I can volunteer to add this stuff to the  
> repository. I would
> 1. use CPP flags
> 2. add a runtime flag (to PARM03)
> 3. select one or a few verification experiments to use this solver.
>
> Christopher's code does not include everything, eg. no  
> "JAM" (whatever) that is, and no vectorization stuff. I can deal  
> with the latter, but not the former.
>
> Martin
>
> On Nov 19, 2009, at 7:03 PM, Jean-Michel Campin wrote:
>
>> Hi Martin,
>>
>> I am right now working on some modif of the 3-D solver
>> to start to have it with z* (for now, just linear-free-surf),
>> (but nothing to do with optimisation). I might check-in this
>> when it passes a coupled of tests. I though it would be good
>> to sync with you if you planned to implement Christopher's
>> method.
>>
>> Cheers,
>> Jean-Michel
>>
>> On Thu, Nov 19, 2009 at 04:07:39PM +0100, Martin Losch wrote:
>>> Hi Christopher,
>>>
>>> that sounds very good. Do you think that the effect will be  
>>> similar for
>>> cg3d?
>>>
>>> Are the results between the two cg2d's different? If so, by how  
>>> much?
>>>
>>> Martin
>>> On Nov 19, 2009, at 1:45 AM, Christopher L. Wolfe wrote:
>>>
>>>>
>>>> Hi all,
>>>>
>>>> I've written an implementation of d'Azevedo, Eijkhout, and Romine's
>>>> (1999) single reduction CG solver for the MITgcm. This method  
>>>> uses a
>>>> rearrangement of the standard conjugate gradient method so that the
>>>> required scalars can be determined by a single call to  
>>>> MPI_Allreduce.
>>>> For problems running on a large number of processors, the  
>>>> decreased MPI
>>>> overhead can significantly increase the efficiency of the conjugate
>>>> gradient solver.
>>>>
>>>> The attached figure shows the scaling of the 2D CG solver for a  
>>>> fixed
>>>> problem size of 1024x1024 as the number of processors increase on  
>>>> the
>>>> Cray XT4 "Franklin." For less than 2^6 processors, the original  
>>>> cg2d
>>>> solver is slightly more efficient than the single reduction solver
>>>> since the latter requires slightly more matrix-vector  
>>>> multiplications.
>>>> However, for processor counts over 2^7, the single reduction  
>>>> solver's
>>>> performance is significantly better than the original cg2d.
>>>>
>>>> I've attached a tarball of the new CG routine cg2d_sr plus the
>>>> modified files CG2D.h, ini_cg2d.F, and solve_for_pressure.F. To use
>>>> the single reduce solver, simply drop these files into an  
>>>> experiment
>>>> directory and compile with "ALLOW_CG2D_SR" defined. I've also  
>>>> included
>>>> at benchmarking suite, which can also be compiled like a standard
>>>> MITgcm experiment.
>>>>
>>>> I've only implemented the single reduce solver for the 2D case  
>>>> since I
>>>> just use the hydrostatic model, but the 3D implementation should be
>>>> straightforward. A paper deriving the single reduction method can  
>>>> be
>>>> found at http://www.netlib.org/lapack/lawnspdf/lawn56.pdf
>>>>
>>>> Feel free to contact me with any questions or comments.
>>>>
>>>> Cheers,
>>>> Christopher
>>>>
>>>> -----------------------------------------------------------
>>>> Dr. Christopher L. Wolfe              	   858-534-4560
>>>> Climate, Atmospheric Science, and Physical Oceanography
>>>> Scripps Institution of Oceanography, UCSD  clwolfe at ucsd.edu
>>>> -----------------------------------------------------------
>>>>
>>>> <strong.eps>
>>>>
>>>>
>>>> <cg2d_sr.tar.gz>
>>>> <cg2d_sr_bench.tar.gz>
>>>>
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>
>>>
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
>
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel