[Mitgcm-support] Re: Boyin and Linux

[mitgcm-support at dev.mitgcm.org]

Well it seems weird but I did two runs of the exact same executable
with the exact same input ( c32/verification/exp2 but longer)  on 
synoptic2.mit.edu.

First run gives nan after 1267 steps, second run has exact same Sum(rhs)
for 1267 steps but continues through to 1268 fine.
The second run does also die with nan but much later ( 2000+ steps ).

My very first test on synoptic2 ran for 26000 steps and then
gave nan. 

For the 1267 and 2000+ runs I turned off dumpfreq, chkptFreq,
pchkptFreq in case it was something connected to I/O.

For the 26000+ run I/O was dumpfreq on every 20 timesteps.

Currently running same executable and input as the 
1267,2000+ setup on a separate test machine. On that machine it is still 
going OK at 17000 steps.

1267 and 2000+ run trace of Sum(rhs) below
 

[cnh at synoptic2 exe]$ sum mitgcmuv data
32488   653 mitgcmuv
36779     2 data
[cnh at synoptic2 exe]$ grep Sum output.o | head -1269 | tail
 cg2d: Sum(rhs) =          -2.90089050003495E-10
 cg2d: Sum(rhs) =          -2.90791973833748E-10
 cg2d: Sum(rhs) =          -2.90593216156765E-10
 cg2d: Sum(rhs) =          -2.91278681729956E-10
 cg2d: Sum(rhs) =          -2.91099117033511E-10
 cg2d: Sum(rhs) =          -2.91509663630229E-10
 cg2d: Sum(rhs) =          -2.92033605631126E-10
 cg2d: Sum(rhs) =          -2.92030150061962E-10
 cg2d: Sum(rhs) =                            nan
 cg2d: Sum(rhs) =                            nan
[cnh at synoptic2 exe]$ cd ../exe2
[cnh at synoptic2 exe2]$ sum mitgcmuv data
32488   653 mitgcmuv
36779     2 data
[cnh at synoptic2 exe2]$ grep Sum output.o | head -1269 | tail
 cg2d: Sum(rhs) =          -2.90089050003495E-10
 cg2d: Sum(rhs) =          -2.90791973833748E-10
 cg2d: Sum(rhs) =          -2.90593216156765E-10
 cg2d: Sum(rhs) =          -2.91278681729956E-10
 cg2d: Sum(rhs) =          -2.91099117033511E-10
 cg2d: Sum(rhs) =          -2.91509663630229E-10
 cg2d: Sum(rhs) =          -2.92033605631126E-10
 cg2d: Sum(rhs) =          -2.92030150061962E-10
 cg2d: Sum(rhs) =          -2.91861243506553E-10
 cg2d: Sum(rhs) =          -2.92456281414388E-10