[MITgcm-support] BLOCKING_EXCHANGES slowdown when using pkg/ptracers on Columbia

[Dimitris Menemenlis]

Has anyone experienced huge slow-downs in blocking_exchanges when using pkg/ptracers?

An is running 8 ptracers for a 9-km Arctic configuration on Columbia.
For a 5-day integration without ptracers, BLOCKING_EXCHANGES requires 55 s.
With 1 ptracers, BLOCKING_EXCHANGES requires 1774 s.
With 8 ptracers, BLOCKING_EXCHANGES requires 7000 s
or approximately 80% of integration time.

If we comment out the call to
PTRACERS_FIELDS_BLOCKING_EXCH
in do_fields_blocking_exchanges.F
the code speeds up once again.

This is puzzling since do_fields_blocking_exchanges.F already includes
3-D blocking exchanges, for for theta and salt.   For some reason
the theta and salt blocking exchanges occur some 100 times
faster than a ptracers blocking exchange even though
the same underlying routine is used.

Any suggestions on what to try next?

Dimitris

Dimitris Menemenlis <menemenlis at jpl.nasa.gov>
Jet Propulsion Lab, California Institute of Technology
MS 300-323, 4800 Oak Grove Dr, Pasadena CA 91109-8099, USA
tel: 818-354-1656;  cell: 818-625-6498;  fax: 818-393-6720