[Mitgcm-support] problems upgrading to r1_p8

[mitgcm-support at dev.mitgcm.org]

Hi, 

I've been running a 1deg N. Atl. model happily with c40_pre9. I thought I
should upgrade my code so I built a r1_p8 version. Picking up from my
c40_pre9 dump I can only run for a few 100 steps before the solver
diverges.

Side by side comparison of the codes shows: identical input/data.* files
and identical output to stdout at startup (excluding a few trivial
differences). My code mods include: rotated spherical grid, RBCs, and
movie. I'm using KPP.

Any suggestions?  Detailed diagnostics below.

Thanks, Tom.



The solver outputs are different from the first step:

r1_p8    (MITgcm_release1_p8/twnh/run/mitgcmuv.out):

 cg2d: Sum(rhs),rhsMax =  -6.14057184440686E-09  9.40447027078642E-01
(PID.TID 0000.0001)                    cg2d_init_res =
1.21108678713545E-01
(PID.TID 0000.0001)                       cg2d_iters =    52
(PID.TID 0000.0001)                         cg2d_res =
6.31509548748506E-14

c40_pre9 (MITgcm_c40pre9/twnh2/run/mitgcmuv.out):

 cg2d: Sum(rhs),rhsMax =  -6.17300681318017E-09  9.35722761934761E-01
(PID.TID 0000.0001)                    cg2d_init_res =
8.29898708457235E-02
(PID.TID 0000.0001)                       cg2d_iters =    50
(PID.TID 0000.0001)                         cg2d_res =
9.22957373770799E-14


MONITOR output shows identical max and mins at the first step, but
slightly different means and sds. After a few 10s of steps the max, min
speeds are different. 


Finally, r1_p8 explodes:

(PID.TID 0000.0001)                    cg2d_init_res =
9.66464443750035E-01
(PID.TID 0000.0001)                       cg2d_iters =    49
(PID.TID 0000.0001)                         cg2d_res =
6.09015534503417E-14
 cg2d: Sum(rhs),rhsMax =  -5.45781517202748E-09  1.10643568700360E+00
(PID.TID 0000.0001)                    cg2d_init_res =
1.72170539524943E+00
(PID.TID 0000.0001)                       cg2d_iters =    49
(PID.TID 0000.0001)                         cg2d_res =
8.16034307437118E-14
 cg2d: Sum(rhs),rhsMax =  -1.82372121571017E-09  3.31101489078477E+00
(PID.TID 0000.0001)                    cg2d_init_res =
1.64995278126615E+00
(PID.TID 0000.0001)                       cg2d_iters =    49
(PID.TID 0000.0001)                         cg2d_res =
7.57149619083152E-14
 cg2d: Sum(rhs),rhsMax =  -9.35809749136318E-10  6.45432834545128E+00
(PID.TID 0000.0001)                    cg2d_init_res =
2.01011457828059E+00
(PID.TID 0000.0001)                       cg2d_iters =    49
(PID.TID 0000.0001)                         cg2d_res =
8.30761672148342E-14
 cg2d: Sum(rhs),rhsMax =  -2.25735723516363E-10  2.67677846388646E+01
(PID.TID 0000.0001)                    cg2d_init_res =
1.63509084933505E+00
(PID.TID 0000.0001)                       cg2d_iters =    49
(PID.TID 0000.0001)                         cg2d_res =
7.44134148753651E-14
 cg2d: Sum(rhs),rhsMax =  -2.11718107238565E-11  2.85458845271579E+02
(PID.TID 0000.0001)                    cg2d_init_res =
1.32604314372823E+00
(PID.TID 0000.0001)                       cg2d_iters =    48
(PID.TID 0000.0001)                         cg2d_res =
8.60880146965341E-14
 cg2d: Sum(rhs),rhsMax =  -1.23077366847397E-13  4.91080350803837E+04
(PID.TID 0000.0001)                    cg2d_init_res =
1.10846133766567E+00
(PID.TID 0000.0001)                       cg2d_iters =    50
(PID.TID 0000.0001)                         cg2d_res =
6.50546394178715E-14
 cg2d: Sum(rhs),rhsMax =   5.52812183728078E-16  1.00849245459929E+14
(PID.TID 0000.0001)                    cg2d_init_res =
1.43461843658355E+00
(PID.TID 0000.0001)                       cg2d_iters =    52
(PID.TID 0000.0001)                         cg2d_res =
7.57551170873034E-14
 cg2d: Sum(rhs),rhsMax =   7.02077927881976E-17  7.48278838805958E+50
(PID.TID 0000.0001)                    cg2d_init_res =
1.24921675417006E+00
(PID.TID 0000.0001)                       cg2d_iters =    51
(PID.TID 0000.0001)                         cg2d_res =
8.84530102944944E-14
 cg2d: Sum(rhs),rhsMax =  -1.08880737624419E-16  9.91672579756393+197

The MONITOR values a few 10s of steps earlier look OK:

(PID.TID 0000.0001) %MON advcfl_uvel_max              =
1.0766191062590E-02
(PID.TID 0000.0001) %MON advcfl_vvel_max              =
1.0979402537837E-02
(PID.TID 0000.0001) %MON advcfl_wvel_max              =
6.1385124825574E-03
(PID.TID 0000.0001) %MON advcfl_W_hf_max              =
1.1204467069580E-02