[MITgcm-devel] (not so) funny things happen in seaice_lsr and pickups

[Martin Losch]

Hi all,

just to let you know that we are experiencing problems with the LSR  
sea ice solver on the C-grid: At unpredictable points of the  
integration, it appears to become instable and blows up. I have not  
been able to isolate this in all cases, because a small issue with  
pickups hampers this:

Apparently, starting from pickup is NOT exact. We have tried the  
famous 2+2=4 test with our 8CPU job on our SX8 (cc to Olaf, who's been  
mostly involved in this) and found no difference between the cg2d  
output (and other output). However, when we run an experiment for a  
longer time, the same test fails, e.g., 2160+2160 != 4320 (we can  
provide plots if required). I assume that this is expected, because  
double precision is not more than double precisioin and in the cg2d  
output (and other monitor output) there are always only 15 digits, and  
we don't know about the 16th one, correct? Anyway, this tiny pickup  
issue hinders me from approaching the point of model crash with  
pickups, because after starting from a pickup, the model integrate  
beyond the problem and crashes (sometimes) at a much later time. This  
is to say, that the problem in seaice_lsr (the problem only appears  
when the C-LSR solver is used) very sensitive; the code crashes  
without any warning from one time step to the other. A while ago, in a  
different case I was able to get close enough the point of crashing to  
do some diagnostics, but its almost impossible to identify, why the  
model explodes. I am assuming that for random pathological cases one  
or more matrix entries are nearly zero, which then prevents the solver  
from converging.

Any comments? Any similar experience?

I run this code in so many different configurations, and I have these  
problems only very seldom/randomly, so I am a little at a loss where I  
should continue looking, so any hint is appreciated.

Martin