[MITgcm-support] MITGCM on dual-dual-core Opteron

[Constantinos Evangelinos]

On Thu 15 Jun 2006 17:00, Jason Goodman wrote:

> I'd like to be able to use the quad-core nodes as dual-core nodes
> sometimes, but MITGCM's performance using 2 cores per node is
> terrible.  The 275 chip has the same clock speed as the 248, so you
> would think that each MITGCM process would run on a separate chip,
> each with its separate memory, and performance would be equal to the
> 248.  That's exactly what happens with CAM.  But with MITGCM on my
> system, the processes hop around from CPU to CPU.  Sometimes both
> processes will be on the same chip, sometimes they'll be on different
> chips, the scheduler moves them around like crazy.  And of course,
> when they're on the same chip, you get a memory bottleneck.
>
> Why does this happen?  Why MITGCM and not CAM?  Is there anything I
> can change in my OS configuration or MITGCM configuration to stop it?

For MPI codes you need a NUMA-aware MPI to do this using the numactl utility 
or numa library calls. There are patched versions of MPICH and MPICH2 around 
(there's a report somewhere from Phil Mucci) and OpenMPI has a numa module.
That should work.

Constantinos
-- 
Dr. Constantinos Evangelinos
Department of Earth, Atmospheric and Planetary Sciences
Massachusetts Institute of Technology