[MITgcm-support] inefficient pressure solver

[David Wang]

Hi MITgcmers,

We have experienced problems with MITgcm on a small local cluster (24-node
dual AMD Opteron quad-core processors "shanghai" with Infiniband using
OpenMPI 1.3.2). The symptom is that when we increase the number of
processors (nProcs), the pressure solver cg2d takes a progressively larger
share (SOLVE_FOR_PRESSURE in STDOUT.0000) of the total walltime (ALL in
STDOUT.0000), and this percentage is much larger than on other clusters
(specifically TACC's Ranger and Lonestar).

Some 1-year hydrostatic, implicit free-surface test runs with the grid
points of 360x224x46, asynchronous timestepping (1200.s/43200.s) result in
the following statistics:

nodes    cores    ALL (sec)    SOLVE_FOR_PRESSURE (sec)
SOLVE_FOR_PRESSURE/ALL (%)
1    8    1873    93    4.97%
2    16    922    129    13.99%
4    32    682    310    45.45%

And with 96 cores, this percentage soars to about 80%!

However, our experience with TACC's Ranger and Lonestar shows that this
percentage does increase with the number of processors, but never above 40%.
TACC's machines use mvapich. So we also tested mvapich on our local cluster
but found no better luck.

We have no idea why the cg2d pressure solver runs so inefficiently on our
cluster. If anyone can kindly provide a few clues, we will very much
appreciate them.

Thanks,
David

-- 
turn and live.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20090713/898b917e/attachment.htm>