[MITgcm-support] Single reduction CG solver

[Christopher L. Wolfe]

Hi all,

I've written an implementation of d'Azevedo, Eijkhout, and Romine's  
(1999) single reduction CG solver for the MITgcm. This method uses a  
rearrangement of the standard conjugate gradient method so that the  
required scalars can be determined by a single call to MPI_Allreduce.  
For problems running on a large number of processors, the decreased  
MPI overhead can significantly increase the efficiency of the  
conjugate gradient solver.

The attached figure shows the scaling of the 2D CG solver for a fixed  
problem size of 1024x1024 as the number of processors increase on the  
Cray XT4 "Franklin." For less than 2^6 processors, the original cg2d  
solver is slightly more efficient than the single reduction solver  
since the latter requires slightly more matrix-vector multiplications.  
However, for processor counts over 2^7, the single reduction solver's  
performance is significantly better than the original cg2d.

I've attached a tarball of the new CG routine cg2d_sr plus the  
modified files CG2D.h, ini_cg2d.F, and solve_for_pressure.F. To use  
the single reduce solver, simply drop these files into an experiment  
directory and compile with "ALLOW_CG2D_SR" defined. I've also included  
at benchmarking suite, which can also be compiled like a standard  
MITgcm experiment.

I've only implemented the single reduce solver for the 2D case since I  
just use the hydrostatic model, but the 3D implementation should be  
straightforward. A paper deriving the single reduction method can be  
found at http://www.netlib.org/lapack/lawnspdf/lawn56.pdf

Feel free to contact me with any questions or comments.

Cheers,
Christopher

-----------------------------------------------------------
Dr. Christopher L. Wolfe              	   858-534-4560
Climate, Atmospheric Science, and Physical Oceanography
Scripps Institution of Oceanography, UCSD  clwolfe at ucsd.edu
-----------------------------------------------------------




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20091118/cc6855e8/attachment-0004.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: strong.eps
Type: application/postscript
Size: 10178 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20091118/cc6855e8/attachment-0001.eps>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20091118/cc6855e8/attachment-0005.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cg2d_sr.tar.gz
Type: application/x-gzip
Size: 9499 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20091118/cc6855e8/attachment-0002.gz>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20091118/cc6855e8/attachment-0006.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cg2d_sr_bench.tar.gz
Type: application/x-gzip
Size: 15608 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20091118/cc6855e8/attachment-0003.gz>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20091118/cc6855e8/attachment-0007.htm>