[MITgcm-support] results differences with number of processors

[Matthew Mazloff]

Hello

I believe the differences are round-off discrepancies amplified by chaos. 

You will have reproducibility for two runs with the same tiling by setting 
#define GLOBAL_SUM_ORDER_TILES
in CPP_EEOPTIONS
if on the same machine. And this likely is not even needed if you are on one node. 

With different tiling, you may be able to get reproducibility with 
#define CG2D_SINGLECPU_SUM
in CPP_EEOPTIONS, but I’m not sure. 

-Matt







> On Nov 22, 2024, at 7:51 AM, Helber, Robert W CIV USN NRL DET SSC MS (USA) <robert.w.helber.civ at us.navy.mil> wrote:
> 
> Dear MITgcm Support,
> 
> We are seeing some differences in the resulting temperature fields from two Deep Convection example experiments.  Using the Deep Convection example, we ran one simulation using 4 processes:
> 
>      PARAMETER (
>     &           sNx =  50,
>     &           sNy =  50,
>     &           OLx =   2,
>     &           OLy =   2,
>     &           nSx =   1,
>     &           nSy =   1,
>     &           nPx =   2,
>     &           nPy =   2,
>     &           Nx  = sNx*nSx*nPx,
>     &           Ny  = sNy*nSy*nPy,
>     &           Nr  =  50)
> 
> Then another simulation using 16 processes:
> 
>      PARAMETER (
>     &           sNx =  25,
>     &           sNy =  25,
>     &           OLx =   2,
>     &           OLy =   2,
>     &           nSx =   1,
>     &           nSy =   1,
>     &           nPx =   4,
>     &           nPy =   4,
>     &           Nx  = sNx*nSx*nPx,
>     &           Ny  = sNy*nSy*nPy,
>     &           Nr  =  50)
> 
> After 335 hours, we are seeing differences, in the convection region down to 300 m, that are about 0.04 C.
> 
> The only difference between the two simulations are the two PARAMETER specifications in the SIZE.h files.
> 
> Is this expected behavior?  If so, why does the number of processes change the results?
> 
> Thanks for any advice.
> 
> Bob Helber
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> https://urldefense.com/v3/__http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support__;!!Mih3wA!FDSBbsdHYxfdVI98gmw4ttGiB-mHlj-kdcchBdTwKea_zQYcWTN5rxI8sA63rBsbwLvf8XiqJU5aFX6QcHIB6L5QSXJD3U4b0Q$