[MITgcm-support] inefficient pressure solver

[David Hebert]

David,

I recall discussion earlier in the year about difficulties with quad 
core processors and memory bandwidth. Could this be what you are seeing 
as you increase cores?

David

David Wang wrote:
> Hi MITgcmers,
>
> We have experienced problems with MITgcm on a small local cluster 
> (24-node dual AMD Opteron quad-core processors "shanghai" with 
> Infiniband using OpenMPI 1.3.2). The symptom is that when we increase 
> the number of processors (nProcs), the pressure solver cg2d takes a 
> progressively larger share (SOLVE_FOR_PRESSURE in STDOUT.0000) of the 
> total walltime (ALL in STDOUT.0000), and this percentage is much 
> larger than on other clusters (specifically TACC's Ranger and Lonestar).
>
> Some 1-year hydrostatic, implicit free-surface test runs with the grid 
> points of 360x224x46, asynchronous timestepping (1200.s/43200.s) 
> result in the following statistics:
>
> nodes    cores    ALL (sec)    SOLVE_FOR_PRESSURE (sec)    
> SOLVE_FOR_PRESSURE/ALL (%)
> 1    8    1873    93    4.97%
> 2    16    922    129    13.99%
> 4    32    682    310    45.45%
>
> And with 96 cores, this percentage soars to about 80%!
>
> However, our experience with TACC's Ranger and Lonestar shows that 
> this percentage does increase with the number of processors, but never 
> above 40%. TACC's machines use mvapich. So we also tested mvapich on 
> our local cluster but found no better luck.
>
> We have no idea why the cg2d pressure solver runs so inefficiently on 
> our cluster. If anyone can kindly provide a few clues, we will very 
> much appreciate them.
>
> Thanks,
> David
>
> -- 
> turn and live.
> ------------------------------------------------------------------------
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support
>