[MITgcm-support] Bug in seaice code since checkpoint67s?

[Martin Losch]

Hi again,

it turns out that PR#348 <https://github.com/MITgcm/MITgcm/pull/348> is the cuprit (i.e. the change in the seaice energy diagnostics). I have no idea why the clearly unrelated code change affects this experiment. The run suddenly crashes after 50 timestep when trying to add a variable tmpscal0 (which is a global sum/mean) to a 2D field in the balancing code of seaice_growth.F. The code runs without problems when I write the value of tmpscal0 to STDOUT.

The problem appears on a vector computer SX-ACE (“stan”, which we test every night) for a specific configuration and appears to be a problem with aggressive optimization. Adding seaice_growth.F to the list of NOOPTFILES (which are compiled with slightly less aggressive optimization) in tools/build_options/SUPER-UX_SX-ACE_sxf90_awi fixes the problem. 
(Not compiling the diagnostics package also fixes the problem, but clearly that’s not desirable).

Given that this computer (SX-ACE stan) is on its way out (will retire in spring next year), I don’t want to pursue this issue too much and leave it at that.

Martin

> On 19. Oct 2020, at 07:46, Martin Losch <Martin.Losch at awi.de> wrote:
> 
> Hi Christoph,
> did you get anywhere with this? Between checkpoints 67r and s, this happened to pkg/seaice (according doc/tag-index):
> 
> o pkg/seaice:
>  - Replace hard-coded minimum for DWATN by runtime parameter "SEAICEdWatMin":
>    * allows to have zero drag by setting both SEAICEdWatMin and
>      SEAICE_waterDrag to zero. Non-zero drag had been implemented because
>      previous LSR solver could not handle zero entries on the main diagonal.
>    * this fixes a bug (unexpected behavior) reported at MITgcm-support by
>      Kaley Shrestha, but also allows to modify the somewhat arbitrary
>      default minimum value of SEAICEdWatMin = 0.25 ;
>    * freedrift code still requires non-zero drag coefficients,
>      S/R SEAICE_CHECK catches this case.
> o pkg/seaice:
>  - Add new mechanical energy diagnostics ;
>  - Move computation of stress diagnostics to seaice_dynsolver.F ; this change
>    affects diagnostics SIpress, SItensil, SIpress, SIzeta, SIdelta, SIshear,
>    because they are now filled after the current time step update, when they
>    are actually available. Note: previously, for the first time step, these
>    variables (and diagnostics) were not properly defined (or zero).
> 
> I guess the diagnostics part should not affect you, but changing DWATN may. Your crash in seaice_growth is probably only an indirect consequence. Can you try to revert this part of the code:
> <https://github.com/MITgcm/MITgcm/commit/5867b94c2f580c6e279f189e3f0b20250ebd8014#diff-8ab9e567a535b5bfbce9f547b746ace3a0a35465d9a0e812ee8add896f07d9a0>, in particular the part after line 102 in seaice_oceandrag_coeffs.F?
> 
> Martin
> 
>> On 13. Oct 2020, at 17:11, Christoph Voelker <christoph.voelker at awi.de> wrote:
>> 
>> Dear all,
>> 
>> I think I have discovered a problem with the current seaice code. I try to run a global MITgcm setup with a salinity tracer contained in seaice. I do so by setting
>> 
>> #define ALLOW_SITRACER
>> 
>> in SEAICE_OPTIONS.h. Otherwise I have no changed seaice code in my code directory. I was able to execute the model then with the additional tracer in previous code versions, but since I recently updated my code I get an error message
>> 
>>    * 253 Invalid operation PROG=seaice_growth ELN=6491(4003ee940)
>> 
>> and the model crashes. I have tried to reproduce the error again with older code versions (to make sure that it is not something that is within my code directory), by checking out older code, then going through the canonical three steps in compilation, and running. I did this for checkpoints 67m and 67r without any problem, but when I switched to checkpoint67s I got the error message and crash again.
>> 
>> So it seems something has happened between checkpoint67r and 67s that makes the model to crash when you switch on the seaice-tracer code. I will try to understand that myself, but maybe some of you who know the code better than me can directly say what the problem could be..
>> 
>> Cheers, Christoph
>> 
>> PS: I actually stumbled across this because I have a code version (started by Razib Vhuiyan) that adds iron in dust as another tracer to seaice; originally I encountered the error with this code after updating and doing some other changes. So I first thought the problem would be on my side, but I think I can now exclude that, because I now tested without any code additions from my side.
>> 
>> -- 
>> Christoph Voelker
>> Alfred Wegener Institute
>> Helmholtz Centre for Polar and Marine Research
>> Am Handelshafen 12
>> 27570 Bremerhaven, Germany
>> e: Christoph.Voelker at awi.de
>> t: +49 471 4831 1848
>> 
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
> 
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support