[MITgcm-support] timing on ibm p690 with sp4+mpi_jump

[Constantinos Evangelinos]

On Mon 19 Mar 2007 06:42, Martin Losch wrote:

> we are encountering poor (and unexpected) scaling behavior on a IBM
> p690 (I think P5 processores) and xlf95 10.??, see attached plot and

Your system in Juelich has Power4+ @1.7GHz.

> comparison of 3 runs with 54/54 (tiles/processors), 216/54, 216/216
> (from left to right). The model configuration is the high resolution
> cubed sphere (with exch2), with 16 extra passive tracers and we used
> tools/build_options/sp4+mpi_jump
>
> What I find interesting is that the 54/54 is slower than the 216/54.
> I would have expected a larger overhead because of more exchanges in
> the second case, but apparently that's not the case (blocking
> exchanges even takes less time in the 216/54 case).
>
> I do not send timings from a 150 and a 360 cpu run, but the 360 cpu
> run is actually SLOWER than the 216 cpu run.

All this is intriguing...

> "Blocking exchanges" takes much more time in 216/216 compared to
> 216/54. Other parts of the code, eg. exf_getforcing is very slow,
> too, in the 216/216 case (I use singleCPUio, maybe not a good idea?).

You can try I/O per tile but keep in mind that GPFS may not like it. Matt has 
had very bad performance problems on the SDSC Power4 system.

> I don't know exactly how this is handled but in the code I could not
> find that exchanging between tiles on one cpu is any different from
> exchanging between different cpus (the MPI_Isend command appears to
> be the same, eg. in exch2_send_rl1.F). 

It should actually be done as memory copies instead of MPI calls to oneself 
which end up doing the same thing with more overhead.

> Can I interpret this 
> circumstance (slow blocking exchanges) as the exchange between
> processors being slow in principle? Is there a way to maybe overcome
> this problem, that is, are there parameters/CPP-flags that we can
> tune/try?

Try playing with increasing MP_EAGER_LIMIT.

Constantinos 
-- 
Dr. Constantinos Evangelinos
Department of Earth, Atmospheric and Planetary Sciences
Massachusetts Institute of Technology