[MITgcm-devel] pickups

[Martin Losch]

Hi there,

thanks for the feedback, everyone.

I am pretty sure that the 2+2=4 tests will pass if done on the same compute node. Yuqing is now trying to do that “by hand” for her setup. I will try to set up the automatic tests  for ollie.

I was suprised that the runs diverge so quickly and by so much. But if that’s expected, then it is what it is.

Martin


> On 9. Jun 2020, at 15:38, Jean-Michel Campin <jmc at mit.edu> wrote:
> 
> Hi Martin,
> 
> I would start by running "tools/tst_2+2" with this set-up and the same executable:
> it just run compare 2-time-steps + 2 time-steps run with a 4 time-step run
> And this script is (relatively) easy to run from a run-dir that already comtains
> output (e.g., after running testreport).
> The advantage of checking with very short segment is that it's easier to assess if
> it's machine precision problem or a more serious bug. With longer segment and depending
> on the set-up, machine truncation errors can make the solution diverge and error to 
> grow, so it's less easy to check where the problem is.
> 
> In the past, we had issues (for some set-up) with some intel compiler optimisation level,
> for example on NAS machine (in optfile: linux_amd64_ifort+mpi_ice_nas, I thing the option
> "-fp-model precise" was added for this purpose). And since you run daily testreport on 
> ollie, I think it would be useful to also run "tools/do_tst_2+2" after testreport
> (it's not difficult to set-up, and there are many examples in MITgcm_contrib/test_scripts
> and it does not take so long to run this 2+2=4 test for all expriments).
> This will allow to check how set-up dependent this is.
> 
> Cheers,
> Jean-Michel
> 
> On Tue, Jun 09, 2020 at 01:56:55PM +0200, Martin Losch wrote:
>> Hi Jean-Michel and others,
>> 
>> on our Cray CS400 ollie, but also elsewhere without the throrough experimenation, we experience the following issue with pickup files. Yuqing has carried out the following experiment with the 4km Arctic configuration (e.g. Gunnar Spreen et al 2017): She ran two model simulations where one uses a pickup frequency of 3 days, i.e. it stops every 3 days and restarts, and the other one uses one of 10 days. If the pickups were perfect these two runs should be the same. Bottom line, they are not, please find attached some time series plots based on daily averages illustrating this:
>> 
>> fig 1: RMS difference between 3 day and 10 day pchk of sea ice/ surface ocean diagnostics in Jan 2001.
>> 
>> fig 2: RMS difference between 3 day and 10 day pchk of sea ice/ surface ocean diagnostics for the full year of 2001
>> 
>> fig 3: Mean difference between 3 day and 10 day pchk of sea ice and ocean diagnostics in Jan 2001
>> 
>> So apparantly the two runs diverge until they seem to reach some kind of ???steady state??? of RMS differences. An RMS difference of 2cm for EtaN is not small, I believe. In this configuration, (which uses seaice, kpp, cal/exf, obcs, salt_plume and exch2), we don???t do anything fancy or experimental.
>> 
>> Have you seen something like this before?
>> 
>> Martin
>> 
>> 
>> 
> 
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mailman.mitgcm.org/mailman/listinfo/mitgcm-devel
> 
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mailman.mitgcm.org/mailman/listinfo/mitgcm-devel