[MITgcm-support] Different output shortly after restart

[Jean-Michel Campin]

Hi Roland,

In general, we do care about getting an exact restart, and 
this is tested regularly (for forward run) on different platforms & compilers:
and reported on this page: 
 http://mitgcm.org/public/testing.html
lines with "restart" in column "Type".

The reasons for not getting an exact restart could be:
1) using a combination of MITgcm options that are not currently
  tested in any of the verification experiment, and there is a problem
  with the code in this particular case.
2) the compiler optimisation option "changes the code" and the resulting
  executable does not always guarantee an exact restart;
  we have this problem with few experiments/compilers , for example here:
  http://mitgcm.org/testing/results/2014_12/rs_acesgrid-pgi_20141210_0/summary.txt
  where isomip does not restart perfectly with the compiler optimisation
  on, but does well with genmake2 -ieee option (no optimisation).
3) in your case, I would suspect first the netcdf pickup:
  for various reasons (code pourly maintained, slower, not tested, with few
  missing capability ... ), it is recommanded not to use NetCDF pickup files
  but instead plain binary pickup files.
 Note that in all the verification experiments that use pkg/mnc I/O,
 the netcdf pickup is always disabled:
> pickup_write_mnc=.FALSE.,
> pickup_read_mnc=.FALSE.,

Could you try again a short test without netcdf pickup option and
check the restart ?

Cheers,
Jean-Michel

On Fri, Dec 12, 2014 at 03:37:21PM +0000, Roland Young wrote:
> Hi,
> 
> I have been running a simulation that I had to restart after 1000
> days. To test whether the restart began OK I ran it for 1 day (from
> 900d to 901d, so I could compare the log output too), but found some
> of the fields had changed very much during that short time, in
> particular the meridional velocity field. The model is of a global
> atmosphere, with base at 18 bar and top at vacuum.
> 
> You can see some plots of the model state after 900d in the initial
> run here:
> 
> http://dl.dropbox.com/u/70735993/U_V_900d_10mb.png
> 
> and after 1000d here:
> 
> http://dl.dropbox.com/u/70735993/U_V_1000d_10mb.png
> 
> As you can see there isn't much change over that time. Here are the
> same fields at 901d after the restart:
> 
> http://dl.dropbox.com/u/70735993/U_V_901d_10mb.png
> 
> The features in the v field do decay away quite quickly - they are
> gone after around 50d. However, I understood that the model should
> restart without any perturbation to the flow, but this clearly isn't
> the case, so I'm a little worried about whether this does anything
> to the flow on a longer timescale.
> 
> What might be going on here? I had a look at the STDOUT and monitor
> output at 900d in the initial run and just before the restarted
> model began its first timestep at 900d. The headers in the STDOUT
> file are identical (except where I expect them to differ e.g. start
> time), and the %MON output at 900d is identical except for the
> following three lines. In the initial run at 900d:
> 
> %MON am_eta_mean                  =  -3.0794405228060E+11
> %MON am_uZo_mean                  =   3.3860138312055E+11
> %MON am_tot_mean                  =   3.0657330839944E+10
> 
> At the start of the short run at 900d:
> 
> %MON am_eta_mean                  =   0.0000000000000E+00
> %MON am_uZo_mean                  =   3.3860198695900E+11
> %MON am_tot_mean                  =   3.3860198695900E+11
> 
> (At 901d am_eta_mean is nonzero in the short run.) In the subsequent
> timestep, the various diagnostics I have added for my own code all
> produce identical output in the two runs, but the following
> diagnostic is different:
> 
> Initial run at 900d+1ts:
> 
> cg2d: Sum(rhs),rhsMax =   9.09494701772928E-12  2.47635718311010E-01
> 
> Short run at 900d+1ts:
> 
> cg2d: Sum(rhs),rhsMax =   1.36424205265939E-11  2.88289188872306E-01
> 
> Are the pickup.####.t###.nc files the only files required for the
> restart, or am I missing something? I can post my input/data if that
> would be useful.
> 
> Thanks,
> 
> Roland
> 
> 
> 
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support