<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hello,</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">I am trying to diagnose a blowup in my simulation by first reducing the checkpoint frequency (so I can get as close as possible to the blowup on successive restarts) and then setting the monitor frequency to monitor every
timestep, so I can catch the exact timestep when the model blows up and output some fields to figure out what's going on.</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Reducing the checkpoint frequency from monthly to daily worked just fine, giving me<span style="font-size: 12pt;"> the pickup file within 1 day of the blowup. Now I've reduced the checkpoint frequency to hourly, meaning
I would expect the model to blow up at some point before 24 checkpoints are written. </span><span style="font-size: 12pt;">However, the model has now </span><span style="font-size: 12pt;">written 335 hourly checkpoints and counting (almost 14 days), meaning
it's soared past the point where it originally blew up in the daily-checkpoint simulation!</span></p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">I've already checked that the model is bit-reproducible at least for the first 10 minutes of a simulation, and that it remains bit-reproducible when I change the checkpoint frequency. <span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">So
there's nothing wrong with my compilers in that respect.</span> I'm guessing the key difference here is that I'm restarting from a checkpoint. Should I expect that process to be bit-reproducible? In other words, is it suspicious that my blowup disappears when
I stop and restart the model?</p>
<p style="margin-top:0;margin-bottom:0"><span style="font-size: 12pt;"></span></p>
<p style="margin-top:0;margin-bottom:0"><span style="font-size: 12pt;"></span></p>
<p style="margin-top:0;margin-bottom:0"><span style="font-size: 12pt;"><br>
</span></p>
<p style="margin-top:0;margin-bottom:0">Many thanks,</p>
<p style="margin-top:0;margin-bottom:0">Kaitlin</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"></p>
<p style="margin-top:0;margin-bottom:0"></p>
</div>
<hr>
<small>This message (and any attachments) is for the recipient only. NERC is subject to the Freedom of Information Act 2000 and the contents of this email and any reply you make may be disclosed by NERC unless it is exempt from release under the Act. Any material
supplied to NERC may be stored in an electronic records management system</small>.
<hr>
</body>
</html>