[MITgcm-support] building with MPI on a dual-core mac

[Constantinos Evangelinos]

On Wednesday 22 July 2009 10:52:14 pm Klymak Jody wrote:
> Thanks a lot Constantinos,
>
> that is very clear!
>
> (PID.TID 0000.0001)   Seconds in section "ALL
> [THE_MODEL_MAIN]":
> (PID.TID 0000.0001)           User time:   1318.5500146746635
> (PID.TID 0000.0001)         System time:   522.38001954555511
> (PID.TID 0000.0001)     Wall clock time:   1977.2751381397247
>
> For 2 machines over gigabit enet.  For me, the 10% imbalance is
> probably acceptable given the cost of infiniband cards....
>
> Cheers,  Jody

Well actually all this says is that the model doesn't idle too much using 
GigEth - which is good - but it does not mean that that 522 sec of system 
time is (a) due to communications mainly or to I/O (the rest of the timing 
information can shed some light on this) or that the communication part could 
not be lessened with IB or another faster interconnect. 

By the way, I do not mean this as a commercial plug (I have not bought 
anything from them and have no stake in their company - they are simply an 
example of a cheap source) but these days one can come by cheap low end IB 
equipment (SDR):
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idcategory=6&idproduct=12
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idcategory=2&idproduct=2
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idcategory=7&idproduct=10
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idcategory=7&idproduct=13

The issue is the latency of IB (~3-4 us for a zero-byte message with the right 
MPI stack for this low end IB card) vs. the latency of GigEth (25-50 us for a 
zero-byte message with the right MPI stack and a lot of kernel TCP tuning). 
The bandwidth is also much higher of course (~900MB/s vs 125) but MITgcm does 
not send messages that are large enough to see that asymptotic bandwidth 
regime.

Constantinos
-- 
Dr. Constantinos Evangelinos
Department of Earth, Atmospheric and Planetary Sciences
Massachusetts Institute of Technology