[MITgcm-support] mpi runs, throughput, latency, InfiniBand, Myrinet

[Nicolas Wienders]

Dear MITgcmers,

We are running a 12 processor job using 3 four AMD processor boxes.

First, we have noticed that the network communication between nodes
is about 1 to 2 MB bi-directional (total), between nodes (using the  
utility 'bwm').
This is not much data. This makes us wonder if the slowdown is not
throughput but latency. 1Gb Ethernet is a high latency networking  
technology
in contrast to InfiniBand and Myrinet.


1. What is the general nonde to node I/O for MITgcm processes? Is 1  
to 2 MB
    bidirectional normal, high or low?

2. Bandwidth vs' Latency. If 1 to 2 MB is normal, would the jobs run  
faster
    using Myrinet or Infiniband?

3. Does MITgcm support Myrinet or Infinband and which is recommended?

4. Is 10Gb ethernet an option to Myrinet or Infinband since 10Gb  
Ethernet
    uses a TCP offload engine on the network adapters? I mention this  
because
    10Gb Ethernet is an emerging technology. To overcome the latency and
    contention issues found in 10/100/1000 Mb Ethernet, 10Gb Ethernet
    introduces some new, latency improving, features. It seems that 10Gb
    Ethernet will be a great technology, but will it be great for  
cluster
    computing is the next question.


Note: It seems that Infiniband is a better choice than Myrinet due to  
the
       adoption of open standards.

Note: It appears that Infiniband technology is slightly cheaper than  
10GbE
       at this time. Low density switches are about 8K to 10K and  
network
       cards for the nodes are about 1K each.


I would appreciate any comment/experience you would have on this matter.

Thanks a lot. Happy afternoon (29ºC here today...)





--

Nicolas Wienders                   (850) 644-1987 (office)
Oceanography - OSB415                294-3542 (cell)
Florida State University                    644-2581 (fax)
Tallahassee, FL - 32306    wienders at ocean.fsu.edu


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20070228/02c41053/attachment.htm>