[MITgcm-support] Optimal System/CPUtype for MITgcm?

[Gøran Brostrøm]

Ok, another quick reply.
MITgcm is very craving on processor-memory bandwidth, and 
multiprocessors have bad performance for MITgcm. In the old Spec cpu 
2000 (http://www.spec.org/cpu2000/), which is now retired, it was a good 
idea to look at the shallow water code 171.swim. This code is somewhat 
extreme and MITgcm is not as "bad" as this code on memory bandwidth (but 
almost). For the newer Spec cpu 2006 I would look at some of the fluid 
dynamic codes but I am not sure which one (do not use the atmospheric 
case as atmospheric models are much less craving on memory bandwidth 
than ocean models). Look into these test and you will learn a lot about 
computer performance.

Well, hope this helps.
Gøran

> Hi there,
>
> I have been meaning to reply to this email for a long time, but never 
> got around to it. As far as I can see there has been no feedback 
> whatsoever, so I'll be the first, although I am by no means a specialist.
>
> This is my experience:
>
> The MITgcm code has (as I recently learned) a relatively low 
> "computational intensity" (FLOPS/memory access); incidentally, that 
> seems to be true for any fluid dynamics code, and is to some extend 
> related to the type of equations that we are solving. It is intented 
> for both parallel and vector architecture machines.
> For a good performance you obviously need fast cpus (with fast memory 
> access) and reasonable bandwidth between cpus.
>
> - A machine where this was realized to a high degree is our Cray XD1 
> with amd (single core) opteron cpus. There I could not notice any 
> overhead for applications ranging from 1 to 64 cpus (or machine does 
> not have more).
> - IBM p690, P4 work very well, too, as long as the individual nodes 
> have a fast connection. We have small IBM where jobs scale linear up 
> to 8CPU on one node, but do not scale at all across nodes. For very 
> MPI-intensive jobs (may passive tracers) I had to modify the way the 
> exchanges are handled for ptracers even on the best configuration.
> - I get by far the best throughput on our NEC-SX8-R (vector) computer. 
> My best results are on the order of 7.5GFLOPS on 1CPU and 5GFLOPS/CPU 
> on 24CPU (3nodes), so 120GFLOPS (I am sure that Jens-Olaf will correct 
> me on these numbers)
> - recently, we've had a bad experience on an Altix with Intel quadcore 
> cpus (see this thread: 
> <http://forge.csail.mit.edu/pipermail/mitgcm-support/2008-October/005731.html> 
> and the following postings). In the end I never closed the thread, but 
> these are conclusions: On that particular machine there were two 
> problems: 1. the connection of nodes was slow (not sure whether this 
> is related to hardward or MPI implementation). 2. the memory bandwidth 
> of the quadcore chips is not sufficient to hand 4cores, so that the 
> memory bandwidth intensive MITgcm (see "computational intensity" 
> above), does not scale when you use more than 2cores per chip. This 
> observation is very similar to what Jeff Blundell found: 
> <http://forge.csail.mit.edu/pipermail/mitgcm-support/2008-October/005749.html> 
>
>
> My conclusions are: If I were ever responsible for buying a large 
> computer to run something like the MITgcm on (not very likely that 
> someone will trust me with that), I would try to avoid 
> Mulitcore-chips, and take care that the connection between cpus is fast.
>
> Martin
>
>
>
> On 5 Nov 2008, at 19:59, m. r. schaferkotter wrote:
>
>>
>> greetings;
>>
>> of late we/ve been less than enthused about the MITgcm performance on 
>> Cray XT4 Opteron quadcore system.
>>
>> given the opportunity of choice of system/cputype pair, i would be 
>> most interested in community views regarding
>> the model performance on the system/cputype pairs below:
>>
>> which system/cputype would give best performance (speed)?
>> which system/cputype should be avoided?
>>
>>
>> SYSTEM                          CPUtype
>>
>> HP XC                                       Opteron
>> SGI Altix 3700                          Itanium 2
>> SGI Altix 4700                          Dual-Core Intel 2
>>
>> IBM Cluster 1350                    Opteron
>> Linux Networx ATC                 Intel Dempsey
>> Linux Networx ATC                 Intel Woodcrest
>> Linux Networx Evolocity II      Intel Xeon EMT64T
>> Linux Networx LS-V                Opteron
>>
>> Sun Fire X2200                        Opteron
>> Sun Fire X4600                        Opteron
>>
>> Cray XT3                                   Opteron
>> Crat XT4                                    Opteron
>>
>> Dell PowerEdge 1955             Intel Woodcrest
>> Cray XD1                                   Opteron
>>
>> IBM P5+ Power5+                   Opteron
>> Cray XT5                                  Opteron
>> IBM Power6                             Power6
>> IBM P575+                               Power5+
>>
>>
>> m. r. schaferkotter
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support