[MITgcm-support] NaNQ

[Göran Broström]

Hi all 
 
I have used this solution for quite some time and it works well. It is probably 
a good idea to include anything like this in the code. My experience is that 
when rhsMax becomes greater then 1e3 it always blows up. However, it is not 
completely failsafe. On one machine I work on it works very well :) On another 
machine the model the model seems to work fine until it hangs at writing 
output. Sometimes it it writes a pickup file, sometimes it hangs when writing 
the pickupfile. However, on restart it chrashes after the first time step (and 
the quickfix using rhsmax stops the run). It appears that this problem is 
specific the the computer that I use. 
 
Well, anyway, using rhsmax to stop the run is a quick fix that takes care of 
most of the cases when the model is heading for a crash. 
 
Sincerely 
Göran 
 
 
 
 
> Hi Sergey, 
> 
> I can't take credit for this solution...it was a mitgcm thread from   
> awhile back...it works very well.... 
> 
>          IF(rhsMax.gt.1e10) stop 'overflow!!!!' 
>          _END_MASTER( myThid ) 
>         ENDIF 
> 
> 
> after 
> 
>       IF ( debugLevel .GE. debLevZero ) THEN 
>          _BEGIN_MASTER( myThid ) 
>          write(standardmessageunit,'(A,1P2E22.14)') 
>       &  ' cg2d: Sum(rhs),rhsMax = ', 
>       &                                  sumRHS,rhsMax 
> 
> in 
> 
> cg2d.F 
> 
> this will kill the model before it reaches nans.  (you might want to   
> change the tolerance of rhsMax.)  (not sure what to do if using non-  
> hydro...maybe put it in cg3d too) 
> 
> JMC, can we check something like this into the model? 
> 
> -Matt 
> 
> 
> 
> 
> 
> On Mar 20, 2007, at 11:27 AM, Sergey Vinogradov wrote: 
> 
>> Hi 
>> Does anyone know how to make MITgcm break its execution once it   
>> starts to produce NaNQs? I'm not sure if it is platform/compiler-   
>> dependent; NaNQs are being dumped if a numerical instability is   
>> encountered (or by some other reasons??) but the model continues to   
>> run further and burn precious CPU time. 
>> Thanks, 
>> 
>> --  
>> Sergey Vinogradov, Ph.D.,    :: Oceanography 
>> Atmospheric and Environmental Research, Inc. 
>> 131 Hartwell Ave., Lexington, MA 02421,  USA 
>> Phone: 1-781-761-2256         sergey at aer.com 
>> Fax:   1-781-761-2299     http://www.aer.com 
>> 
>> _______________________________________________ 
>> MITgcm-support mailing list 
>> MITgcm-support at mitgcm.org 
>> http://mitgcm.org/mailman/listinfo/mitgcm-support 
> 
> _______________________________________________ 
> MITgcm-support mailing list 
> MITgcm-support at mitgcm.org 
> http://mitgcm.org/mailman/listinfo/mitgcm-support