[MITgcm-devel] (not so) funny things happen in seaice_lsr and pickups

[Martin Losch]

Hi Jean-Michel,

just to make it clear: the 2+2=4 tests were OK to the very last digit,  
also with aggressive optimization; only after longer intergrations,  
the (very small) differences did occur.
I agree, it could be very well something that is hidden in some crappy  
seaice code, or the optimization on the SX8. However, the reason why I  
was mentioning this is not really the pickups themselves, but that  
these tiny differences, however they come about, cause the seaice  
model to crash or not to crash, and it's very difficult to debug (but  
that's because of the pickups). I was hoping that other people may  
have similar experience and I could collect situations in which this  
happens and try to track down the problem this way.

M.
On Feb 18, 2009, at 4:58 PM, Jean-Michel Campin wrote:

> Hi Martin,
>
> A short comment about restart:
> 1) it's tested (and I mean, not only the cg2d output, but we
> do a diff of the real*8 binary pickup files, including
> pickup_seaice, and we get zero diff) every day, with pgi compiler
> and gfortran, but always with zero optimisation.
> 2) it does not guarantee that the seaice restart is always OK:
> what I found with thsice (and it could be the same with
> seaice) is that there are so many different cases that
> we would need to run for ~1 year to be sure that all
> the pieces of code have really been tested.
> 3) there is always a possibility that the optimisation level
> you are using on SX8 is not safe for some part of the seaice
> code.
>
> Jean-Michel
>
> On Wed, Feb 18, 2009 at 12:21:41PM +0100, Martin Losch wrote:
>> Hi all,
>>
>> just to let you know that we are experiencing problems with the LSR  
>> sea
>> ice solver on the C-grid: At unpredictable points of the  
>> integration, it
>> appears to become instable and blows up. I have not been able to  
>> isolate
>> this in all cases, because a small issue with pickups hampers this:
>>
>> Apparently, starting from pickup is NOT exact. We have tried the  
>> famous
>> 2+2=4 test with our 8CPU job on our SX8 (cc to Olaf, who's been  
>> mostly
>> involved in this) and found no difference between the cg2d output  
>> (and
>> other output). However, when we run an experiment for a longer  
>> time, the
>> same test fails, e.g., 2160+2160 != 4320 (we can provide plots if
>> required). I assume that this is expected, because double precision  
>> is
>> not more than double precisioin and in the cg2d output (and other  
>> monitor
>> output) there are always only 15 digits, and we don't know about  
>> the 16th
>> one, correct? Anyway, this tiny pickup issue hinders me from  
>> approaching
>> the point of model crash with pickups, because after starting from a
>> pickup, the model integrate beyond the problem and crashes  
>> (sometimes) at
>> a much later time. This is to say, that the problem in seaice_lsr  
>> (the
>> problem only appears when the C-LSR solver is used) very sensitive;  
>> the
>> code crashes without any warning from one time step to the other. A  
>> while
>> ago, in a different case I was able to get close enough the point of
>> crashing to do some diagnostics, but its almost impossible to  
>> identify,
>> why the model explodes. I am assuming that for random pathological  
>> cases
>> one or more matrix entries are nearly zero, which then prevents the
>> solver from converging.
>>
>> Any comments? Any similar experience?
>>
>> I run this code in so many different configurations, and I have these
>> problems only very seldom/randomly, so I am a little at a loss  
>> where I
>> should continue looking, so any hint is appreciated.
>>
>> Martin
>>
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel