<div dir="ltr"><div>Hi,</div><div><br></div><div>Replying to this:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>The code simply stopped producing any new results, however, it was still running</div></blockquote><div><br></div><div>I have encountered this issue as well, when running on a cluster with SLURM queue management (batch script to sbmit job). The error is called and written in output files, but execution is not stopped (only when time limit is reached).</div><div>The model continue running and burning core hours for nothing.<br></div><div><br></div><div>Enrico<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mer 25 mar 2020 alle ore 21:06 Matthew Mazloff <<a href="mailto:mmazloff@ucsd.edu">mmazloff@ucsd.edu</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">Hello<div><br></div><div>The code crashed trying to read a file. The file is size NY*NZ*NT so I suspect it is an eastern or western boundary condition file. Make sure your files are long enough.</div><div><br></div><div>-Matt</div><div><br><div><br><blockquote type="cite"><div>On Mar 25, 2020, at 12:24 PM, Yangxin He <<a href="mailto:y67he@uwaterloo.ca" target="_blank">y67he@uwaterloo.ca</a>> wrote:</div><br><div><div id="gmail-m_6573036946965186473divtagdefaultwrapper" dir="ltr" style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-size:12pt;font-family:Calibri,Helvetica,sans-serif"><div style="margin-top:0px;margin-bottom:0px">Hello there,</div><div style="margin-top:0px;margin-bottom:0px"><br></div><div style="margin-top:0px;margin-bottom:0px">Recently several jobs of mine died of no reason. The error message is</div><div style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Courier;color:rgb(255,238,149);background-color:rgb(21,102,47)"><span style="font-variant-ligatures:no-common-ligatures">[y67he@gra-login1 b6]$ more sim-29315632.err </span></div><div style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Courier;color:rgb(255,238,149);background-color:rgb(21,102,47)"><span style="font-variant-ligatures:no-common-ligatures">ABNORMAL END: S/R MDS_READ_SEC_YZ</span></div><div style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Courier;color:rgb(255,238,149);background-color:rgb(21,102,47)"><span style="font-variant-ligatures:no-common-ligatures">srun: Job step aborted: Waiting up to 62 seconds for job step to finish.</span></div><div style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Courier;color:rgb(255,238,149);background-color:rgb(21,102,47)"><span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error: *** JOB 29315632 ON gra228 CANCELLED AT 2020-03-25T08:30:08 DUE TO TIME LIMIT ***</span></div><p style="margin-top:0px;margin-bottom:0px"></p><div style="margin:0px;font-stretch:normal;font-size:12px;line-height:normal;font-family:Courier;color:rgb(255,238,149);background-color:rgb(21,102,47)"><span style="font-variant-ligatures:no-common-ligatures">slurmstepd: error: *** STEP 29315632.0 ON gra228 CANCELLED AT 2020-03-25T08:30:08 DUE TO TIME LIMIT ***</span></div><div style="margin-top:0px;margin-bottom:0px">The time limit was not the problem. The code simply stopped producing any new results, however, it was still running.</div><div style="margin-top:0px;margin-bottom:0px">This is confusing, because I have been using the same set up for a while and this only started to happen in the past few weeks.</div><div style="margin-top:0px;margin-bottom:0px"><br></div><div style="margin-top:0px;margin-bottom:0px">I ran my code on graham in compute Canada, and the people there suggested it may be the problem in the code.</div><div style="margin-top:0px;margin-bottom:0px">Can anyone shed any lights on this?</div><div style="margin-top:0px;margin-bottom:0px"><br></div><div style="margin-top:0px;margin-bottom:0px">Thanks</div><div style="margin-top:0px;margin-bottom:0px"><br></div><div style="margin-top:0px;margin-bottom:0px">Yangxin</div></div><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">_______________________________________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">MITgcm-support mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><a href="mailto:MITgcm-support@mitgcm.org" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">MITgcm-support@mitgcm.org</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a></div></blockquote></div><br></div></div>_______________________________________________<br>
MITgcm-support mailing list<br>
<a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
<a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
</blockquote></div>