[MITgcm-support] MPI scaling of coupled aim-ocn runs

[Andrew Keats]

I see.  What kind of hardware is inside Columbia?  I currently only  
have access to Xeon and Opteron boxes...

Cheers

Andrew

On 10-Dec-09, at 11:36 AM, David Ferreira wrote:

> Andrew,
> It is a hardware problem, the best combination of processor depends  
> strongly on your machine.
> For example, on one of the machines I used for the C32 coupled  
> model, there was no improvement
> beyond 1cou/3ocn/6atm. When I moved to the Columbia machine, then  
> 1cou/12ocn/12atm
> made things faster.
> If you can, test another machine. Otherwise, I don't know what you  
> could try. Maybe multithreading
> could help, but I've never used it.
> david
>
>
> Andrew Keats wrote:
>> Hi David,
>>
>> Thanks for your help.  Are you using a 32x32 cube sphere grid, or  
>> a higher resolution?  I am only seeing decent scaling up to 8  
>> processors (6 atm, 1 ocn, 1 coupler) for the cs32 grid (the  
>> verification experiment).  I checked the output lines as you  
>> suggested and with a 6:1 atm:ocn division, the ocean component is  
>> still spending more time waiting than the atmosphere.  Would it be  
>> possible for me to take a look at your "SIZE.h" and "data" files  
>> to compare our setups?
>>
>
>
>
>> Andrew
>>
>> On 7-Dec-09, at 5:47 PM, David Ferreira wrote:
>>
>>> Hi Andrew,
>>>> I have just begun setting up a coupled run based on the code in  
>>>> the verification/cpl_aim+ocn directory.  By editing SIZE.h's I  
>>>> have set up runs with different combinations of processes (1  
>>>> ocn / 1 atm ; 2 ocn / 1 atm ; 1 ocn / 2 atm ; 2 ocn / 2 atm ; 1  
>>>> ocn / 3 atm) and the fastest run seems to be the one with 1  
>>>> ocn / 2 atm.
>>> When running the coupled model, I usually uses 18 or 24  
>>> processors: 1 coupler/6 ocn/12 atm or 1 coupler/12ocn/12atm.  
>>> There is little or no benefits using more cpus.
>>>
>>>> We are eventually hoping to use this code to do multiple  
>>>> thousand-year runs for the last glaciation (at some point will  
>>>> attempt coupling with Tarasov's glacial system model).  Does  
>>>> anyone know what the best configuration of processes/threads is  
>>>> for running the coupled model?  I haven't tried multithreading  
>>>> yet, only MPI.  Also, is there a way to tell how well  
>>>> synchronized the atmosphere and ocean processes are?  Does it  
>>>> happen that one ends up waiting for the other if the ratio of  
>>>> the time steps is off?
>>> Yes it happens, usually the ocean waits for the atmosphere to  
>>> finish but this can be the reverse if
>>> for example the ocean carries lots of tracers.
>>> You can find out which component waits for the other by looking  
>>> at the time each spends in the coupler
>>> (given by "CPL_EXPORT-IMPORT  [FORWARD_STEP]").
>>> At this point, there is no other possibility than the ocean and  
>>> atmosphere exchanging information
>>> every ocean time-step.
>>>
>>>
>>>
>>>> Many thanks,
>>>>
>>>> Andrew
>>>>
>>>>
>>>>
>>>> -- 
>>>> Andrew Keats
>>>> NSERC Postdoctoral Fellow
>>>> Department of Physics and Physical Oceanography
>>>> Memorial University of Newfoundland
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>
>>>
>>>
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
>
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support