<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Well it definitely died while trying to read something for the obcs:<div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: Courier; color: rgb(255, 238, 149); background-color: rgb(21, 102, 47);" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">ABNORMAL END: S/R MDS_READ_SEC_YZ</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">Do you also give boundary files that have a start time and period given in data.exf?</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">E.g.:</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""> obcsWstartdate1 = 20081216,<br class=""> obcsWstartdate2 = 00000,<br class=""> obcsWperiod = 2629800,<br class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">Matt</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class=""></span></div><div><br class=""><blockquote type="cite" class=""><div class="">On Mar 25, 2020, at 1:11 PM, Yangxin He <<a href="mailto:y67he@uwaterloo.ca" class="">y67he@uwaterloo.ca</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div id="divtagdefaultwrapper" dir="ltr" style="caret-color: rgb(0, 0, 0); font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;" class=""><div style="margin-top: 0px; margin-bottom: 0px;" class="">Hi Matt,</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">This would be really confusing.</div><div style="margin-top: 0px; margin-bottom: 0px;" class="">My file seems to be the right size, and the run died after running fine for 34 tidal periods. If the size of boundary files is the problem, then the run would have died in the beginning?</div><div style="margin-top: 0px; margin-bottom: 0px;" class="">Another thing is, I have been using this set up for about a year now, and it was running fine only until recently.</div><div style="margin-top: 0px; margin-bottom: 0px;" class=""><br class=""></div><div style="margin-top: 0px; margin-bottom: 0px;" class="">Yangxin</div></div><hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 857.5px;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><font face="Calibri, sans-serif" style="font-size: 11pt;" class=""><b class="">From:</b><span class="Apple-converted-space"> </span>MITgcm-support <<a href="mailto:mitgcm-support-bounces@mitgcm.org" class="">mitgcm-support-bounces@mitgcm.org</a>> on behalf of Matthew Mazloff <<a href="mailto:mmazloff@ucsd.edu" class="">mmazloff@ucsd.edu</a>><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Wednesday, March 25, 2020 4:06:38 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span><a href="mailto:mitgcm-support@mitgcm.org" class="">mitgcm-support@mitgcm.org</a><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [MITgcm-support] jobs died suddenly</font><div class=""> </div></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">Hello<div class=""><br class=""></div><div class="">The code crashed trying to read a file. The file is size NY*NZ*NT so I suspect it is an eastern or western boundary condition file. Make sure your files are long enough.</div><div class=""><br class=""></div><div class="">-Matt</div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Mar 25, 2020, at 12:24 PM, Yangxin He <<a href="mailto:y67he@uwaterloo.ca" class="">y67he@uwaterloo.ca</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div id="divtagdefaultwrapper" dir="ltr" class="" style="caret-color: rgb(0, 0, 0); font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"><div class="" style="margin-top: 0px; margin-bottom: 0px;">Hello there,</div><div class="" style="margin-top: 0px; margin-bottom: 0px;"><br class=""></div><div class="" style="margin-top: 0px; margin-bottom: 0px;">Recently several jobs of mine died of no reason. The error message is</div><div class="" style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Courier; color: rgb(255, 238, 149); background-color: rgb(21, 102, 47);"><span class="" style="font-variant-ligatures: no-common-ligatures;">[y67he@gra-login1 b6]$ more sim-29315632.err </span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Courier; color: rgb(255, 238, 149); background-color: rgb(21, 102, 47);"><span class="" style="font-variant-ligatures: no-common-ligatures;">ABNORMAL END: S/R MDS_READ_SEC_YZ</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Courier; color: rgb(255, 238, 149); background-color: rgb(21, 102, 47);"><span class="" style="font-variant-ligatures: no-common-ligatures;">srun: Job step aborted: Waiting up to 62 seconds for job step to finish.</span></div><div class="" style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Courier; color: rgb(255, 238, 149); background-color: rgb(21, 102, 47);"><span class="" style="font-variant-ligatures: no-common-ligatures;">slurmstepd: error: *** JOB 29315632 ON gra228 CANCELLED AT 2020-03-25T08:30:08 DUE TO TIME LIMIT ***</span></div><p class="" style="margin-top: 0px; margin-bottom: 0px;"></p><div class="" style="margin: 0px; font-stretch: normal; font-size: 12px; line-height: normal; font-family: Courier; color: rgb(255, 238, 149); background-color: rgb(21, 102, 47);"><span class="" style="font-variant-ligatures: no-common-ligatures;">slurmstepd: error: *** STEP 29315632.0 ON gra228 CANCELLED AT 2020-03-25T08:30:08 DUE TO TIME LIMIT ***</span></div><div class="" style="margin-top: 0px; margin-bottom: 0px;">The time limit was not the problem. The code simply stopped producing any new results, however, it was still running.</div><div class="" style="margin-top: 0px; margin-bottom: 0px;">This is confusing, because I have been using the same set up for a while and this only started to happen in the past few weeks.</div><div class="" style="margin-top: 0px; margin-bottom: 0px;"><br class=""></div><div class="" style="margin-top: 0px; margin-bottom: 0px;">I ran my code on graham in compute Canada, and the people there suggested it may be the problem in the code.</div><div class="" style="margin-top: 0px; margin-bottom: 0px;">Can anyone shed any lights on this?</div><div class="" style="margin-top: 0px; margin-bottom: 0px;"><br class=""></div><div class="" style="margin-top: 0px; margin-bottom: 0px;">Thanks</div><div class="" style="margin-top: 0px; margin-bottom: 0px;"><br class=""></div><div class="" style="margin-top: 0px; margin-bottom: 0px;">Yangxin</div></div><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">_______________________________________________</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">MITgcm-support mailing list</span><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><a href="mailto:MITgcm-support@mitgcm.org" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">MITgcm-support@mitgcm.org</a><br class="" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a></div></blockquote></div><br class=""></div></div><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">MITgcm-support mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="mailto:MITgcm-support@mitgcm.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">MITgcm-support@mitgcm.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a></div></blockquote></div><br class=""></div></body></html>