[MITgcm-devel] pickups

[Jean-Michel Campin]

Hi Martin,

I would start by running "tools/tst_2+2" with this set-up and the same executable:
 it just run compare 2-time-steps + 2 time-steps run with a 4 time-step run
And this script is (relatively) easy to run from a run-dir that already comtains
output (e.g., after running testreport).
The advantage of checking with very short segment is that it's easier to assess if
it's machine precision problem or a more serious bug. With longer segment and depending
on the set-up, machine truncation errors can make the solution diverge and error to 
grow, so it's less easy to check where the problem is.

In the past, we had issues (for some set-up) with some intel compiler optimisation level,
for example on NAS machine (in optfile: linux_amd64_ifort+mpi_ice_nas, I thing the option
"-fp-model precise" was added for this purpose). And since you run daily testreport on 
ollie, I think it would be useful to also run "tools/do_tst_2+2" after testreport
(it's not difficult to set-up, and there are many examples in MITgcm_contrib/test_scripts
and it does not take so long to run this 2+2=4 test for all expriments).
This will allow to check how set-up dependent this is.

Cheers,
Jean-Michel

On Tue, Jun 09, 2020 at 01:56:55PM +0200, Martin Losch wrote:
> Hi Jean-Michel and others,
> 
> on our Cray CS400 ollie, but also elsewhere without the throrough experimenation, we experience the following issue with pickup files. Yuqing has carried out the following experiment with the 4km Arctic configuration (e.g. Gunnar Spreen et al 2017): She ran two model simulations where one uses a pickup frequency of 3 days, i.e. it stops every 3 days and restarts, and the other one uses one of 10 days. If the pickups were perfect these two runs should be the same. Bottom line, they are not, please find attached some time series plots based on daily averages illustrating this:
> 
> fig 1: RMS difference between 3 day and 10 day pchk of sea ice/ surface ocean diagnostics in Jan 2001.
> 
> fig 2: RMS difference between 3 day and 10 day pchk of sea ice/ surface ocean diagnostics for the full year of 2001
> 
> fig 3: Mean difference between 3 day and 10 day pchk of sea ice and ocean diagnostics in Jan 2001
> 
> So apparantly the two runs diverge until they seem to reach some kind of ???steady state??? of RMS differences. An RMS difference of 2cm for EtaN is not small, I believe. In this configuration, (which uses seaice, kpp, cal/exf, obcs, salt_plume and exch2), we don???t do anything fancy or experimental.
> 
> Have you seen something like this before?
> 
> Martin
> 
> 
> 

> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mailman.mitgcm.org/mailman/listinfo/mitgcm-devel