[MITgcm-devel] changing default (MPI) Global-Sum

[Jean-Michel Campin]

Hi,

I am thinking of changing the MPI-global-sum default
(in eesuapp/inc/CPP_EEOPTIONS.h) to:
#define GLOBAL_SUM_ORDER_TILES
(which only affects MPI built of global_sum_tile.F)
to make the global-sum independent of tile to proc mapping.

For a given tile-size, we found that, with present default global-sum 
and with some version of MPI, the results not only depends on the number 
of MPI-procs (nPx,nPy,nSx,nSy) but also on proc to node repartition
(which is less straightforward to control on some platform).

This new option (GLOBAL_SUM_ORDER_TILES) fixes this issue and 
should not slow down MITgcm runs (except may be with very large # of procs)
since it relies on the same MPI call (MPI_Allreduce) than the present default
(see comments in CPP_EEOPTIONS.h).
It produces identical results as the slower alternative (#define GLOBAL_SUM_SEND_RECV).

Right now, most of the tested experiments have a specific CPP_EEOPTIONS.h
(or CPP_EEOPTIONS.h_mpi) with either GLOBAL_SUM_SEND_RECV or GLOBAL_SUM_ORDER_TILES
defined. Did not notice any problem in these verification exp. since
GLOBAL_SUM_ORDER_TILES was added (on Aug 25, 2015).
If you find problems when trying "#define GLOBAL_SUM_ORDER_TILES", please
report to the list.

Cheers,
Jean-Michel