[MITgcm-support] Zonal filter with nPx>1

[Jean-Michel Campin]

Hi Roland,

Regarding solution differences due to changes the tile-domain decomposition:
In most cases, it comes from the global-sum used in CG solver(s).

There are 2 things that address this issue (but they slow down the model):
1) with the same tile decomposition (sNx,sNy) but different processor
   mapping (difference in any nSx,nSy,nPx,nPy) you can compile with:
   > #define GLOBAL_SUM_SEND_RECV
  in CPP_EEOPTIONS.h
  (stantard version: eesupp/inc/CPP_EEOPTIONS.h)
  but I realised that this is not going to help you since the current
  implemetation of the zonal filter does not even allow to have nSx different
  from 1 !
2) you can try to compile with:
  > #define CG2D_SINGLECPU_SUM
  also in eesupp/inc/CPP_EEOPTIONS.h
  This is even slower than (1) but should make the results independent
  of the tile decomposition if the only global-sum used is in CG2D 
  (hydrostatic simulation, without forcing global balance of surface
   forcing, without OBCs balance, no seaice dynamics).

Cheers,
Jean-Michel

On Thu, Nov 13, 2014 at 01:55:10PM +0000, Roland Young wrote:
> Hi Jean,
> 
> OK thanks.
> 
> I have coded a version that works for nPx>1 (at least for nSx=nSy=1). It gathers the field from all the processors along a longitude row and applies the filter on the west-most processor before scattering the filtered field back to the other processors.
> 
> I was wondering whether, in general, the MITgcm is bit-reproducible with respect to domain decomposition? I’d like to know whether my edited zonal filter does the same thing as the standard version but I can’t be sure as the output is not quite bit-reproducible. 
> 
> For example, I ran the following configurations:
> 
> (a) 16x1 standard zonal filter
> (b) 16x1 edited zonal filter
> 
> Output from (a) and (b) is bit reproducible.
> 
> (c) 4x4 edited zonal filter
> (d) 8x2 edited zonal filter
> (e) 8x1 standard zonal filter
> (f) 32x1 standard zonal filter
> 
> Output from (a) is not bit-reproducible wrt (c), (d), (e), or (f) (although identical to 1 part in 10^8 in U and V, and 1 part in 10^16 in theta, after 10 time steps)
> 
> (g) 16x1 no zonal filter
> (h) 4x4 no zonal filter
> 
> Output from (g) and (h) is not bit-reproducible (but are identical to 1 part in 10^8 in U and V, and 1 part in 10^16 in theta after 2 time steps - any more it crashes as the zonal filter isn’t on)
> 
> Bit-reproducibility is not critical for my work so I can probably proceed as-is, but are there any parts of the code that are known to be not bit-reproducible across domain decompositions? I’m using the global atmosphere with mom_vecinv.
> 
> Thanks,
> 
> Roland
> 
>  
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support