[MITgcm-support] MPI scaling of coupled aim-ocn runs

[David Ferreira]

Andrew,
It is a hardware problem, the best combination of processor depends 
strongly on your machine.
For example, on one of the machines I used for the C32 coupled model, 
there was no improvement
beyond 1cou/3ocn/6atm. When I moved to the Columbia machine, then 
1cou/12ocn/12atm
made things faster.
If you can, test another machine. Otherwise, I don't know what you could 
try. Maybe multithreading
could help, but I've never used it.
david


Andrew Keats wrote:
> Hi David,
>
> Thanks for your help.  Are you using a 32x32 cube sphere grid, or a 
> higher resolution?  I am only seeing decent scaling up to 8 processors 
> (6 atm, 1 ocn, 1 coupler) for the cs32 grid (the verification 
> experiment).  I checked the output lines as you suggested and with a 
> 6:1 atm:ocn division, the ocean component is still spending more time 
> waiting than the atmosphere.  Would it be possible for me to take a 
> look at your "SIZE.h" and "data" files to compare our setups?
>



> Andrew
>
> On 7-Dec-09, at 5:47 PM, David Ferreira wrote:
>
>> Hi Andrew,
>>> I have just begun setting up a coupled run based on the code in the 
>>> verification/cpl_aim+ocn directory.  By editing SIZE.h's I have set 
>>> up runs with different combinations of processes (1 ocn / 1 atm ; 2 
>>> ocn / 1 atm ; 1 ocn / 2 atm ; 2 ocn / 2 atm ; 1 ocn / 3 atm) and the 
>>> fastest run seems to be the one with 1 ocn / 2 atm.
>> When running the coupled model, I usually uses 18 or 24 processors: 1 
>> coupler/6 ocn/12 atm or 1 coupler/12ocn/12atm. There is little or no 
>> benefits using more cpus.
>>
>>> We are eventually hoping to use this code to do multiple 
>>> thousand-year runs for the last glaciation (at some point will 
>>> attempt coupling with Tarasov's glacial system model).  Does anyone 
>>> know what the best configuration of processes/threads is for running 
>>> the coupled model?  I haven't tried multithreading yet, only MPI.  
>>> Also, is there a way to tell how well synchronized the atmosphere 
>>> and ocean processes are?  Does it happen that one ends up waiting 
>>> for the other if the ratio of the time steps is off?
>> Yes it happens, usually the ocean waits for the atmosphere to finish 
>> but this can be the reverse if
>> for example the ocean carries lots of tracers.
>> You can find out which component waits for the other by looking at 
>> the time each spends in the coupler
>> (given by "CPL_EXPORT-IMPORT  [FORWARD_STEP]").
>> At this point, there is no other possibility than the ocean and 
>> atmosphere exchanging information
>> every ocean time-step.
>>
>>
>>
>>> Many thanks,
>>>
>>> Andrew
>>>
>>>
>>>
>>> -- 
>>> Andrew Keats
>>> NSERC Postdoctoral Fellow
>>> Department of Physics and Physical Oceanography
>>> Memorial University of Newfoundland
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>
>>
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support