[MITgcm-support] cheapAML+global_ocean.90x40x15: it's all NaNs

Fri Nov 18 18:46:00 EST 2016

Hi MITgcm list and Prof. Deramble

I found a fix for the NaNs ...
... but unfortunately I hit another problem.

A) The fix for the NaNs

The year 2000 NCEP Reanalysis2 2m height
specific humidity data I used as input to the cheapAML
had a few isolated negative values.
They are very small negative values, probably due to
the NCEP model overshooting in the Reanalysis procedure.
They are isolated in time, and clustered around 1-8 grid cells
(in the Reanalysis grid).
So, even Reanalysis data needs to be double checked ...

The most conspicuous and spread out of these negative values
happened near Antarctica on day 257 (Sep/13/2000),
causing the cheapAML+global_ocean.90x40x15,
to always fail on the next day (258).

I suppose negative specific humidity may raise havoc in
the cheapAML calculations.

After I replaced those spurious negative values by zeros,
my model setup started running right.
No more NaNs!

So, I was happy ...

**

B) The new problem: a boiling ocean

Alas, my happiness was short lived.
After 162 years of integration,
the model bombed again,
now with extreme (ocean) potential temperatures.

The error message looks like this:

[1,1]<stderr>:: S/R MON_SOLUTION, stops due to EXTREME Pot.Temp
[1,10]<stderr>:STOP ABNORMAL END: S/R MON_SOLUTION, stops due to EXTREME 
Pot.Temp

The monitor was showing a steady maximum potential temperature,
fluctuating around 32-33 degrees C.
Then, all of a sudden, it goes crazy:
... 32, 32, 37, 78, 121, 93 ... and KABOOM!!!
The model bombed ...  :(

Here is what the monitor on STDOUT.0000 showed right before bombing:

(PID.TID 0000.0001) %MON dynstat_theta_max            =  3.2582251686287E+01
(PID.TID 0000.0001) %MON dynstat_theta_max            = 
3.2503291876356E+01
(PID.TID 0000.0001) %MON dynstat_theta_max            = 
3.2440986258708E+01
(PID.TID 0000.0001) %MON dynstat_theta_max            = 
3.2391082759296E+01
(PID.TID 0000.0001) %MON dynstat_theta_max            = 
3.2427201547426E+01
(PID.TID 0000.0001) %MON dynstat_theta_max            = 
3.7598466799827E+01
(PID.TID 0000.0001) %MON dynstat_theta_max            = 
7.8087402901902E+01
(PID.TID 0000.0001) %MON dynstat_theta_max            = 
1.2131618320702E+02
(PID.TID 0000.0001) %MON dynstat_theta_max            = 
9.3091090119057E+01

I checked the 2m air temperature NCEP Reanalysis2
data, which the cheapAML also uses, just in case.
However, the overall range is 197 to 318 K (-76 to 45 degrees C)
(It's a global grid, including polar regions, Sahara, etc,
and the data was converted to degrees C for the MITgcm input.)
In any case, I presume any extreme air temperature value would
have made the model bomb earlier, as the spurious negative
specific humidity values did, not wait to bomb out after 162 years.

**

C) QUESTIONS:

1) What could be causing the ocean to boil somewhere?
(Unfortunately the monitor doesn't tell where the problem happened.
I'd guess that is some narrow/shallow ocean area.)

2) Is there any way to keep the potential temperature under control,
and prevent the high temperature burst?

3) Should I turn off the SST & SSS relaxation, or perhaps
change their restoring time scales?
(I.e., guessing the boiling potential temperature happens
near the surface.)

FYI, I currently have in data PARM03:

# 2 months restoring timescale for temperature
  tauThetaClimRelax = 5270400.0,
# 6 months restoring timescale for salinity
  tauSaltClimRelax  = 15811200.0,

and in data PARM05

# sst and sss files for relaxation at the surface
  thetaClimFile=  'lev_sst.bin',
  saltClimFile=   'lev_sss.bin',

which are the boilerplate sst and sss files used in the
global_ocean.90x40x15 verification experiment.

****

I'd appreciate any insights or suggestions.

Thank you,
Gus Correa

On 11/15/2016 09:23 PM, Gus Correa wrote:
> Hi Prof. Deremble
>
> Thank you very much for your help,
> and for your kind offer to run the experiment yourself.
>
> FYI, I am using MITgcm_c65x.
>
> **
>
> 1) Files on the ftp site:
>
> I put the binary forcing files for the cheapAML+global_ocean.90x40x15
> experiment, along with the "data" (namelist) files that I used
> in our ftp server:
>
> ftp.ldeo.columbia.edu
>
> which you can access via anonymous sftp or
> through a browser, with the URL:
>
> ftp://ftp.ldeo.columbia.edu/
>
> Via sftp, the directory with the files is:
>
> /home/ftp-pub/gus/2deremble
>
> If you use the browser it spells this way:
>
> ftp://ftp.ldeo.columbia.edu/pub/gus/2deremble/
>
> If you have any problem to copy the files, please let me know.
>
> **
>
> 2) More attempts to run the experiment still produce NaNs
>
> Also, I ran the experiment with cheapamlYperiodic = .FALSE.
> With FluxFormula=COARE3 it gives me NaNs on step 2 if I use
> cheapaml_ntim = 10.
> If I increase cheapaml_ntim  to 100, 1000, or 10000,
> it starts the NaNs on step 258 (on STDERR.0000 at least).
> With FluxFormula=LANL it starts the NaNs on step 2,
> regardless of the value of cheapaml_ntim.
>
> **
>
> 3) Question
>
> This experiment is on a 4x4 degrees grid.
> Could it be that the cheapAML is inherently unstable
> at this coarse resolution? (I hope not!)
>
> **
>
> Many thanks,
> Gus Correa
>
>
> On 11/10/2016 03:41 PM, Bruno Deremble wrote:
>>
>> HI Gus,
>>
>> I mean, you need
>> cheapamlYperiodic = .FALSE.
>>
>> because your domain is not periodic in y.
>>
>> Can you put your configuration and init files somewhere? I can try to
>> look at it.
>>
>> bruno
>>
>
> On 11/09/2016 02:45 PM, Gus Correa wrote:
>> Hi Prof. Deramble
>>
>> Thank you very much for your help.
>> I made the changes you suggested, but I continue to have NaNs.
>> Please, see my answers and new questions inline.
>>
>> On 11/07/2016 06:16 PM, Bruno Deremble wrote:
>>>
>>> Hi Gus,
>>>
>>> your config looks good to me.
>>> a few minor points:
>>>
>>> I think you don't want
>>> cheapamlYperiodic = .TRUE.
>>> because you are on a lat/lon grid
>>
>> I was confused by the comments on the code,
>> that are probably applicable only to the cheapAML
>> box model in the MITgcm verification experiment.
>> Anyway, I switched to cheapamlYperiodic = .TRUE.
>>
>>>
>>>
>>> cheapaml_ntim = 10000
>>> is probably too much. I would set
>>> cheapaml_ntim ~ Umax(atm)/Umax(ocean) so O(10)
>>>
>>
>> Yes, I started with 5, then 10, based on
>> the same scaling argument you wrote above.
>> However, because I only got NaNs, I thought I might
>> have a CFL problem in the cheapAML, so I increased it
>> progressively to 100, 1000, 10000, but that didn't help either.
>> I switched back to cheapaml_ntim = 10.
>>
>>>
>>> I would use COARE3 (and not LANL)
>>> FluxFormula='COARE3'
>>>
>>
>> I applied this change also.
>> However, I wonder if, in order to use COARE3,
>> I need additional input files to the cheapAML (i.e.,
>> not just Usurf,Vsurf,Qsurf,Tsurf, ShortWavesurf).
>> Do I? Maybe I need also a wave model file?
>> Not sure.
>>
>>> If none of these work, can you give me a link to your binary files? I
>>> can try to run your config later this week.
>>>
>>> bruno
>>>
>>>
>>
>> Thank you,
>> Gus Correa
>>
>>
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>