[MITgcm-devel] [MITgcm-support] BLOCKING_EXCHANGES slowdown when using pkg/ptracers on Columbia

[Martin Losch]

I agree with Jean-Michel, it is puzzling that even 1 ptracer is enough to cause this problem (and in this case my hack won't do anything).

As I said, my solution is a hack, probably specific to the platform I was using, that made my simulations (with 16 tracers) fast enough so that I could continue to with the integration, I never bothered to go into detail with this.

Just as additional information. In my/our case we were runing on JUMP, a IBM690/P4 cluster in Jülich (now offline) and observed bad scaling with # of cpu, with the hack we were happy with the scaling and did not investigate things any further (see here for a terrible gray paper: <http://epic.awi.de/Publications/Los2008a.pdf> that describes the problem).

Martin

On Mar 11, 2010, at 10:48 PM, Jean-Michel Campin wrote:

> Hi,
> 
> I think it would still be usefull to understand why Dimitris
> see such a slow down, specially with only 1 tracer (and in this case,
> Martin's solution, to copy back and forth to an array which has
> exactly the same shape, is unlikely to speed up the run).
> After, the 8 tracers case scale like the 1 tracer, and is easy to 
> interpret if we understand the 1 tracer case.
> 
> Jean-Michel
> 
> On Thu, Mar 11, 2010 at 05:30:43PM +0100, Martin Losch wrote:
>> Hi Constantinos,
>> 
>> I am moving this to the devel-list:
>> 
>> I would have added the code long ago, if it had been clean, but it isn't. Jean-Michel might even remove my cvs privileges if he only _sees_ the code changes (o:
>> 
>> I am happy to provide my changes (they are with very much outdated code), and Dimitris can see if it works for him. Then we should think about how to include this in a more general context, but that's way beyond my technical (and currently even time) capabilities.
>> 
>> Martin
>> PS. Here comes the code (between checkpoint59n and p).
>> 
> 
> 
>> 
>> 
>> On Mar 11, 2010, at 5:02 PM, Constantinos Evangelinos wrote:
>> 
>>> On Thursday 11 March 2010 05:52:23 am Martin Losch wrote:
>>> 
>>>> I observed a huge slowdown with ptracers as well, the solution was to copy
>>>> ptracers into a new 5D-array where the 3rd index (k) is nPtracer*Nr long,
>>> 
>>> You mean 3D I take it.
>>> 
>>>> and then use one exchange for that (and I think I had to hack the
>>>> corresponding exchange routine to have it allow such large arrays). Can
>>>> provide code if required.
>>> 
>>> Since more than one person has this problem it might be a good time to add the 
>>> code (as an IFDEF) in the main code.
>>> 
>>> Constantinos
>>> -- 
>>> Dr. Constantinos Evangelinos
>>> Department of Earth, Atmospheric and Planetary Sciences
>>> Massachusetts Institute of Technology
>>> 
>>> 
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>> 
> 
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> 
> 
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel