[MITgcm-support] Single reduction CG solver

[Martin Losch]

Hi Christopher,

again, thanks for your code.

> (Oddly, several of the verification tests fail even if I try to run  
> them with a fresh out-of-the-box MITgcm. Perhaps I don't have my  
> compiler configured correctly.)

Please have a look at
<http://mitgcm.org/public/testing.html>
You'll see that the tests only pass perfectly on "faulks" (the  
reference) and machines with similar architecture and compilers. Some  
verification experiments (e.g. lab_sea) are (for some unknown reasons)  
very sensitive to details of the numerical and some compilers do  
things a little different from the reference (g77). It's important  
that the difference to the reference does not become large (although  
for the lab_sea case, they are (e.g. <http://mitgcm.org/testing/results/2009_11/tr_sx8_20091115_0/summary.txt 
 >)
So probably nothing wrong with your compiler configuration.

Martin

On Nov 19, 2009, at 6:49 PM, Christopher L. Wolfe wrote:

>
> Hi Martin,
>
> I imagine the algorithm would work just as well for the 3D problem,  
> though the point where the single-reduction method becomes more  
> efficient than the standard method might be different. My impression  
> is that the 3D solver does a lot more work per reduction, so it time  
> to solution might not be as dominated by reduction overhead.
>
> The two cg2d's give the same results per iteration to within round- 
> off error. My benchmarking code writes in single precision and the  
> difference between the two methods is always single-precision zero,  
> even after thousands if iterations. I ran it through some of the  
> verification tests using testreport and all the ones that passed  
> with the "vanilla" CG routine also passed with the single-reduce  
> routine. (Oddly, several of the verification tests fail even if I  
> try to run them with a fresh out-of-the-box MITgcm. Perhaps I don't  
> have my compiler configured correctly.)
>
> Cheers,
> Christopher
>
> On Nov 19, 2009, at 7:07 AM, Martin Losch wrote:
>
>> Hi Christopher,
>>
>> that sounds very good. Do you think that the effect will be similar
>> for cg3d?
>>
>> Are the results between the two cg2d's different? If so, by how much?
>>
>> Martin
>> On Nov 19, 2009, at 1:45 AM, Christopher L. Wolfe wrote:
>>
>>>
>>> Hi all,
>>>
>>> I've written an implementation of d'Azevedo, Eijkhout, and Romine's
>>> (1999) single reduction CG solver for the MITgcm. This method uses a
>>> rearrangement of the standard conjugate gradient method so that the
>>> required scalars can be determined by a single call to
>>> MPI_Allreduce. For problems running on a large number of processors,
>>> the decreased MPI overhead can significantly increase the efficiency
>>> of the conjugate gradient solver.
>>>
>>> The attached figure shows the scaling of the 2D CG solver for a
>>> fixed problem size of 1024x1024 as the number of processors increase
>>> on the Cray XT4 "Franklin." For less than 2^6 processors, the
>>> original cg2d solver is slightly more efficient than the single
>>> reduction solver since the latter requires slightly more matrix-
>>> vector multiplications. However, for processor counts over 2^7, the
>>> single reduction solver's performance is significantly better than
>>> the original cg2d.
>>>
>>> I've attached a tarball of the new CG routine cg2d_sr plus the
>>> modified files CG2D.h, ini_cg2d.F, and solve_for_pressure.F. To use
>>> the single reduce solver, simply drop these files into an experiment
>>> directory and compile with "ALLOW_CG2D_SR" defined. I've also
>>> included at benchmarking suite, which can also be compiled like a
>>> standard MITgcm experiment.
>>>
>>> I've only implemented the single reduce solver for the 2D case since
>>> I just use the hydrostatic model, but the 3D implementation should
>>> be straightforward. A paper deriving the single reduction method can
>>> be found at http://www.netlib.org/lapack/lawnspdf/lawn56.pdf
>>>
>>> Feel free to contact me with any questions or comments.
>>>
>>> Cheers,
>>> Christopher
>>>
>>> -----------------------------------------------------------
>>> Dr. Christopher L. Wolfe              	   858-534-4560
>>> Climate, Atmospheric Science, and Physical Oceanography
>>> Scripps Institution of Oceanography, UCSD  clwolfe at ucsd.edu
>>> -----------------------------------------------------------
>>>
>>> <strong.eps>
>>>
>>>
>>> <cg2d_sr.tar.gz>
>>> <cg2d_sr_bench.tar.gz>
>>>
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support