[MITgcm-devel] pickups

[Martin Losch]

Hi Jean-Michel,

after understanding that there are issues with optimization and restarts 
on ollie, I still do not understand, what is happening to our 4km arctic 
configuration. Turning off the optimization (genmake2 -devel) does not 
help at all.

After further tests, the problem is clearly related to obcs: when I turn 
off obcs, the restarts (2+2=4) are perfect. With obcs turned on, this is 
my data.obcs:

  &OBCS_PARM01
  OB_Jnorth =   1680*1536,
  OB_Ieast  =   1536*1680,
  OB_Iwest  =   1536*1,
#
  useOBCSprescribe=.TRUE.,
#
  OBNsFile='../obcs_2012/OBNs_arctic_1680x1536.stable',
  OBNtFile='../obcs_2012/OBNt_arctic_1680x1536.stable',
  OBNuFile='../obcs_2012/OBNu_arctic_1680x1536.bin',
  OBNvFile='../obcs_2012/OBNv_arctic_1680x1536.balance',
  OBWsFile='../obcs_2012/OBWs_arctic_1680x1536.stable',
  OBWtFile='../obcs_2012/OBWt_arctic_1680x1536.stable',
  OBWuFile='../obcs_2012/OBWu_arctic_1680x1536.bin',
  OBWvFile='../obcs_2012/OBWv_arctic_1680x1536.bin',
  OBEsFile='../obcs_2012/OBEs_arctic_1680x1536.stable',
  OBEtFile='../obcs_2012/OBEt_arctic_1680x1536.stable',
  OBEuFile='../obcs_2012/OBEu_arctic_1680x1536.bin',
  OBEvFile='../obcs_2012/OBEv_arctic_1680x1536.bin',
#
  &

When I set useOBCSprescribe = .False., then restarts are perfect 
(2+2=4). I used exf_iprec_obcs = 32 and 64 and it makes no difference. 
The the few differences are along the "Northern" boundary (and along the 
"Western" boundary, but even fewer points), and confined to the row Ny-1 
(and column Nx-1), so not on the open boundary itself, but one grid 
point inward, i.e. the values on the open boundaries itself are OK. The 
max/min of the differences are symmetric, e.g. for temperature they are: 
9.53674e-07/-9.53674e-07

In OBCS_OPTIONS.h, only these flags are defined:

#define ALLOW_OBCS_NORTH
#define ALLOW_OBCS_EAST
#define ALLOW_OBCS_WEST
#define ALLOW_OBCS_PRESCRIBE

The way I understand exf/obcs, is that at the startup it reads 2 slices 
of the (in my case) monthly data and then it interpolates until it 
reaches the time when a new obcs fields needs to be loaded. With a 
timestep of 240 and a start date of Dec28, 2000, the two initial fields 
should always be the same for the 2+2=4 test. I just don't get it.

Do you have any idea, what may be going on here? What else can I test?

Martin

On 6/9/20 3:38 PM, Jean-Michel Campin wrote:
> Hi Martin,
>
> I would start by running "tools/tst_2+2" with this set-up and the same executable:
>   it just run compare 2-time-steps + 2 time-steps run with a 4 time-step run
> And this script is (relatively) easy to run from a run-dir that already comtains
> output (e.g., after running testreport).
> The advantage of checking with very short segment is that it's easier to assess if
> it's machine precision problem or a more serious bug. With longer segment and depending
> on the set-up, machine truncation errors can make the solution diverge and error to
> grow, so it's less easy to check where the problem is.
>
> In the past, we had issues (for some set-up) with some intel compiler optimisation level,
> for example on NAS machine (in optfile: linux_amd64_ifort+mpi_ice_nas, I thing the option
> "-fp-model precise" was added for this purpose). And since you run daily testreport on
> ollie, I think it would be useful to also run "tools/do_tst_2+2" after testreport
> (it's not difficult to set-up, and there are many examples in MITgcm_contrib/test_scripts
> and it does not take so long to run this 2+2=4 test for all expriments).
> This will allow to check how set-up dependent this is.
>
> Cheers,
> Jean-Michel
>
> On Tue, Jun 09, 2020 at 01:56:55PM +0200, Martin Losch wrote:
>> Hi Jean-Michel and others,
>>
>> on our Cray CS400 ollie, but also elsewhere without the throrough experimenation, we experience the following issue with pickup files. Yuqing has carried out the following experiment with the 4km Arctic configuration (e.g. Gunnar Spreen et al 2017): She ran two model simulations where one uses a pickup frequency of 3 days, i.e. it stops every 3 days and restarts, and the other one uses one of 10 days. If the pickups were perfect these two runs should be the same. Bottom line, they are not, please find attached some time series plots based on daily averages illustrating this:
>>
>> fig 1: RMS difference between 3 day and 10 day pchk of sea ice/ surface ocean diagnostics in Jan 2001.
>>
>> fig 2: RMS difference between 3 day and 10 day pchk of sea ice/ surface ocean diagnostics for the full year of 2001
>>
>> fig 3: Mean difference between 3 day and 10 day pchk of sea ice and ocean diagnostics in Jan 2001
>>
>> So apparantly the two runs diverge until they seem to reach some kind of ???steady state??? of RMS differences. An RMS difference of 2cm for EtaN is not small, I believe. In this configuration, (which uses seaice, kpp, cal/exf, obcs, salt_plume and exch2), we don???t do anything fancy or experimental.
>>
>> Have you seen something like this before?
>>
>> Martin
>>
>>
>>
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mailman.mitgcm.org/mailman/listinfo/mitgcm-devel
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mailman.mitgcm.org/mailman/listinfo/mitgcm-devel