[MITgcm-support] (no subject)

[Jody Klymak]

Hi Steve, 

What is your value of nonHydrostatic in the `data` file, and what are your cg3d convergence criteria?  nonhydrostatic can make the code run 4-10xs slower, and I think has a bunch more inter-tile communications.  

How much memory does each node have?  

How often are you writing out data? 

How often do you have to read in data?  

Recently, I was running 1024 x 256 x 400 in *hydrostatic* mode on 128 processors (haise.navo.hpc.mil), and I would get 60 time steps a minute.  So, for your total domain being 16 times smaller, it seems you are maybe running 10-20 times slower.  So I bet you are running in non-hydrostatic mode, or you are inducing a lot of i/o for some reason. 

Good luck - hopefully the actual gurus have other suggestions.

Cheers,   Jody

> On 14 Jul 2016, at  8:45 AM, Stephen Cousins <steve.cousins at maine.edu> wrote:
> 
> Hi,
> 
> I'm trying to help researchers from the University of Maine to run MITgcm. The model runs they think it should run much faster. 
> 
> I have run or helped run many models while working for the Ocean Modeling Group however this is the first time I have encountered MITgcm. 
>  
> With Rutgers ROMS there is a method of running a number of tiles per sub-domain and it seems that MITgcm can do that too. The reason for doing so with ROMS was (I believe) to try to get the tiles to fit in cache to increase performance. Is that the reason for doing so with MITgcm? We have tried a number of combinations with not much luck. 
> 
> For testing, the full domain we have is 600 x 520 x 21 using 64 processes and getting only 30 time steps per minute. I wondered if the domain was too small for that many processes so I reduced the number of processes but that didn't help. The plan is to triple the resolution in each horizontal direction and double in the vertical. 
> 
> Our cluster has nodes with Intel E5-2600v3 processors totaling 24 cores per node with FDR-10 Infiniband. The way the jobs were specified, some compute nodes had many processes (like 20) on them and some had only 1 or 2. I experimented and found that by using only 4 cores per node and only 48 cores, it ran close to twice as fast as with 64 cores and a mix of the numbers of cores per node. To me this indicates that the inter-process-communication is high and it is saturating the memory bandwidth of the nodes with large process counts. That might point to the subdomains being too small halo region being a significant proportion of the subdomain) but in that case when I decreased the run to 16 cores I would have thought that it would have improved things quite a bit. I haven't profiled the code yet. I thought it might be quicker to write to you to get some information first. 
> 
> Can you please explain what the optimal layout is for performance? Is there an optimal size subdomain that you know of for these processors? Optimal number of tiles per subdomain? Also can you explain at a somewhat high level any other factors to consider when running the model to get better performance? Also, are there Intel Haswell CPU-specific compiler flags (we're using the Intel compilers with MVAPICH2) that you can recommend to us? Finally, is there a benchmark case where we can verify that we are getting the expected performance?
> 
> Thanks very much,
> 
> Steve
> -- 
> ________________________________________________________________
>  Steve Cousins             Supercomputer Engineer/Administrator
>  Advanced Computing Group            University of Maine System
>  244 Neville Hall (UMS Data Center)              (207) 561-3574
>  Orono ME 04469                      steve.cousins at maine.edu <http://maine.edu/>
> 
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support

--
Jody Klymak    
http://web.uvic.ca/~jklymak/





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20160714/5f0fb157/attachment.htm>