[MITgcm-support] Different output shortly after restart

[Roland Young]

Hi,

I have been running a simulation that I had to restart after 1000 days. 
To test whether the restart began OK I ran it for 1 day (from 900d to 
901d, so I could compare the log output too), but found some of the 
fields had changed very much during that short time, in particular the 
meridional velocity field. The model is of a global atmosphere, with 
base at 18 bar and top at vacuum.

You can see some plots of the model state after 900d in the initial run 
here:

http://dl.dropbox.com/u/70735993/U_V_900d_10mb.png

and after 1000d here:

http://dl.dropbox.com/u/70735993/U_V_1000d_10mb.png

As you can see there isn't much change over that time. Here are the same 
fields at 901d after the restart:

http://dl.dropbox.com/u/70735993/U_V_901d_10mb.png

The features in the v field do decay away quite quickly - they are gone 
after around 50d. However, I understood that the model should restart 
without any perturbation to the flow, but this clearly isn't the case, 
so I'm a little worried about whether this does anything to the flow on 
a longer timescale.

What might be going on here? I had a look at the STDOUT and monitor 
output at 900d in the initial run and just before the restarted model 
began its first timestep at 900d. The headers in the STDOUT file are 
identical (except where I expect them to differ e.g. start time), and 
the %MON output at 900d is identical except for the following three 
lines. In the initial run at 900d:

%MON am_eta_mean                  =  -3.0794405228060E+11
%MON am_uZo_mean                  =   3.3860138312055E+11
%MON am_tot_mean                  =   3.0657330839944E+10

At the start of the short run at 900d:

%MON am_eta_mean                  =   0.0000000000000E+00
%MON am_uZo_mean                  =   3.3860198695900E+11
%MON am_tot_mean                  =   3.3860198695900E+11

(At 901d am_eta_mean is nonzero in the short run.) In the subsequent 
timestep, the various diagnostics I have added for my own code all 
produce identical output in the two runs, but the following diagnostic 
is different:

Initial run at 900d+1ts:

cg2d: Sum(rhs),rhsMax =   9.09494701772928E-12  2.47635718311010E-01

Short run at 900d+1ts:

cg2d: Sum(rhs),rhsMax =   1.36424205265939E-11  2.88289188872306E-01

Are the pickup.####.t###.nc files the only files required for the 
restart, or am I missing something? I can post my input/data if that 
would be useful.

Thanks,

Roland