[MITgcm-support] BLOCKING_EXCHANGES slowdown when using pkg/ptracers on Columbia

[Martin Losch]

Hi Dimitris, 

I observed a huge slowdown with ptracers as well, the solution was to copy ptracers into a new 5D-array where the 3rd index (k) is nPtracer*Nr long, and then use one exchange for that (and I think I had to hack the corresponding exchange routine to have it allow such large arrays). Can provide code if required.

Martin
On Mar 10, 2010, at 11:48 PM, Dimitris Menemenlis wrote:

> Has anyone experienced huge slow-downs in blocking_exchanges when using pkg/ptracers?
> 
> An is running 8 ptracers for a 9-km Arctic configuration on Columbia.
> For a 5-day integration without ptracers, BLOCKING_EXCHANGES requires 55 s.
> With 1 ptracers, BLOCKING_EXCHANGES requires 1774 s.
> With 8 ptracers, BLOCKING_EXCHANGES requires 7000 s
> or approximately 80% of integration time.
> 
> If we comment out the call to
> PTRACERS_FIELDS_BLOCKING_EXCH
> in do_fields_blocking_exchanges.F
> the code speeds up once again.
> 
> This is puzzling since do_fields_blocking_exchanges.F already includes
> 3-D blocking exchanges, for for theta and salt.   For some reason
> the theta and salt blocking exchanges occur some 100 times
> faster than a ptracers blocking exchange even though
> the same underlying routine is used.
> 
> Any suggestions on what to try next?
> 
> Dimitris
> 
> Dimitris Menemenlis <menemenlis at jpl.nasa.gov>
> Jet Propulsion Lab, California Institute of Technology
> MS 300-323, 4800 Oak Grove Dr, Pasadena CA 91109-8099, USA
> tel: 818-354-1656;  cell: 818-625-6498;  fax: 818-393-6720
> 
> 
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support