[MITgcm-support] timing on ibm p690 with sp4+mpi_jump

[Martin Losch]

Hi,

we are encountering poor (and unexpected) scaling behavior on a IBM  
p690 (I think P5 processores) and xlf95 10.??, see attached plot and  
comparison of 3 runs with 54/54 (tiles/processors), 216/54, 216/216  
(from left to right). The model configuration is the high resolution  
cubed sphere (with exch2), with 16 extra passive tracers and we used  
tools/build_options/sp4+mpi_jump

What I find interesting is that the 54/54 is slower than the 216/54.  
I would have expected a larger overhead because of more exchanges in  
the second case, but apparently that's not the case (blocking  
exchanges even takes less time in the 216/54 case).

I do not send timings from a 150 and a 360 cpu run, but the 360 cpu  
run is actually SLOWER than the 216 cpu run.

"Blocking exchanges" takes much more time in 216/216 compared to  
216/54. Other parts of the code, eg. exf_getforcing is very slow,  
too, in the 216/216 case (I use singleCPUio, maybe not a good idea?).
I don't know exactly how this is handled but in the code I could not  
find that exchanging between tiles on one cpu is any different from  
exchanging between different cpus (the MPI_Isend command appears to  
be the same, eg. in exch2_send_rl1.F). Can I interpret this  
circumstance (slow blocking exchanges) as the exchange between  
processors being slow in principle? Is there a way to maybe overcome  
this problem, that is, are there parameters/CPP-flags that we can  
tune/try?

Martin

-------------- next part --------------
A non-text attachment was scrubbed...
Name: code.tgz
Type: application/octet-stream
Size: 18177 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20070319/25484968/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Skalierungstests.eps
Type: application/postscript
Size: 21189 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20070319/25484968/attachment.eps>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: data.gz
Type: application/x-gzip
Size: 1457 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20070319/25484968/attachment.gz>
-------------- next part --------------

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Zeitvergleich.txt
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20070319/25484968/attachment.txt>