<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Martin, I don’t have experience with Cray but on pleiades supercomputer (<a href="https://www.nas.nasa.gov/hecc/resources/pleiades.html" class="">https://www.nas.nasa.gov/hecc/resources/pleiades.html</a>) this non-repeatabilty for eddying MITgcm configurations is pretty common. I am pretty sure that I have seen it occur when:<div class="">- we use maximum optimization, which does not guarantee that precise order of operations will always be the same,</div><div class="">- we use recompiled code with different compilers,</div><div class="">- we use the same executable but on different nodes of the machine, e.g., Broadwell instead of Haswell, or</div><div class="">- we use the same executable and nodes after operating system upgrades.</div><div class=""><div class=""><div><br class=""></div><div>Dimitris</div><div><br class=""><blockquote type="cite" class=""><div class="">On Jun 9, 2020, at 4:56 AM, Martin Losch <<a href="mailto:Martin.Losch@awi.de" class="">Martin.Losch@awi.de</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Jean-Michel and others,<div class=""><br class=""></div><div class="">on our Cray CS400 ollie, but also elsewhere without the throrough experimenation, we experience the following issue with pickup files. Yuqing has carried out the following experiment with the 4km Arctic configuration (e.g. Gunnar Spreen et al 2017): She ran two model simulations where one uses a pickup frequency of 3 days, i.e. it stops every 3 days and restarts, and the other one uses one of 10 days. If the pickups were perfect these two runs should be the same. Bottom line, they are not, please find attached some time series plots based on daily averages illustrating this:</div><div class=""><div class=""><br class=""></div><div class=""><div class=""><div class=""><div class="">fig 1: RMS difference between 3 day and 10 day pchk of sea ice/ surface ocean diagnostics in Jan 2001.<br class=""><br class="">fig 2: RMS difference between 3 day and 10 day pchk of sea ice/ surface ocean diagnostics for the full year of 2001<br class=""><br class="">fig 3: Mean difference between 3 day and 10 day pchk of sea ice and ocean diagnostics in Jan 2001</div><div class=""><br class=""></div><div class="">So apparantly the two runs diverge until they seem to reach some kind of “steady state” of RMS differences. An RMS difference of 2cm for EtaN is not small, I believe. In this configuration, (which uses seaice, kpp, cal/exf, obcs, salt_plume and exch2), we don’t do anything fancy or experimental.</div><div class=""><br class=""></div><div class="">Have you seen something like this before?</div><div class=""><br class=""></div><div class="">Martin<br class=""><br class=""><br class=""><span id="cid:06914F59-FC05-42B4-AE3E-7970BAD04950@dmawi.de"><10-3day-pick-up-diff_RMS.png></span><span id="cid:A7D71949-75C3-467D-B511-1266265AD564@dmawi.de"><10-3day-pick-up-diff_RMS_annual.png></span><span id="cid:A3502353-2024-454B-A2AF-C0D89D900F14@dmawi.de"><10-3day-pick-up-diff.png></span></div></div></div><br class=""></div></div></div>_______________________________________________<br class="">MITgcm-devel mailing list<br class=""><a href="mailto:MITgcm-devel@mitgcm.org" class="">MITgcm-devel@mitgcm.org</a><br class="">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-devel<br class=""></div></blockquote></div><br class=""></div></div></body></html>