<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">It may be something with reading the data* files. I have had issues with this in the past when using many cores, though it has been a long time since I have seen this issue arise. <div> To try something other than the default you can try</div><div>USE_FORTRAN_SCRATCH_FILES </div><div>I’m not sure how it works but it only impacts this part of the code so definitely safe to try</div><div><br></div><div>or you can try </div><div>SINGLE_DISK_IO </div><div>but this eliminates IO from all other cores, and will thus suppress error messages. That said, if multiple cores trying to process data* at once is your issue this will resolve it.</div><div><br></div><div>Matt</div><div><br></div><div><br></div><div><div><br></div><div><br><blockquote type="cite"><div>On Mar 3, 2023, at 2:14 PM, mario wrk <wrkmario@gmail.com> wrote:</div><br class="Apple-interchange-newline"><div><div dir="ltr">Thanks for pointing that out! the scratch files point to data.exch2 and data.ctrl I excluded some pkg one by one, but I still have some similar issues, <div>in the end, there is a segmentation fault, srun: error: l10551: task 256: Segmentation fault, now I highly suspect it might be some compiler issues, cuz it was good with some verification examples, but my own configuration substantially increased resolution and I wanted to compile/run in multi-processors</div><div>I also noticed there were similar discussions before: <a href="https://urldefense.com/v3/__http://mailman.mitgcm.org/pipermail/mitgcm-support/2018-June/011593.html__;!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8RLRznInA$">http://mailman.mitgcm.org/pipermail/mitgcm-support/2018-June/011593.html</a> </div><div><br></div><div>Best, </div><div>Mario</div><div><div><br></div><div> 255: forrtl: severe (28): CLOSE error, unit 11, file "Unknown"<br> 255: Image PC Routine Line Source <br> 255: libifcoremt.so.5 00001555553EFBDE for__exit_handler Unknown Unknown<br> 255: libifcoremt.so.5 00001555553FC78E for__signal_handl Unknown Unknown<br> 255: libpthread-2.28.s 0000155550C3BC20 Unknown Unknown Unknown<br> 255: <a href="https://urldefense.com/v3/__http://libc-2.28.so__;!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8RezGD5pg$">libc-2.28.so</a> 000015555095316B unlink Unknown Unknown<br> 255: libifcoremt.so.5 00001555553E18B1 for__close_proc Unknown Unknown<br> 255: libifcoremt.so.5 00001555553E0EB0 for_close Unknown Unknown<br> 255: mitgcmuv_ad 0000000000902A3F Unknown Unknown Unknown<br> 255: mitgcmuv_ad 00000000009E2F4A Unknown Unknown Unknown<br> 255: mitgcmuv_ad 00000000009DB0E5 Unknown Unknown Unknown<br> 255: mitgcmuv_ad 00000000009F7D38 Unknown Unknown Unknown<br> 255: mitgcmuv_ad 000000000097DDEA Unknown Unknown Unknown<br> 714: mitgcmuv_ad 0000000000403852 Unknown Unknown Unknown<br> 714: <a href="https://urldefense.com/v3/__http://libc-2.28.so__;!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8RezGD5pg$">libc-2.28.so</a> 0000155550887493 __libc_start_main Unknown Unknown<br> 714: mitgcmuv_ad 000000000040375E Unknown Unknown Unknown<br> 255: mitgcmuv_ad 0000000000403852 Unknown Unknown Unknown<br> 255: <a href="https://urldefense.com/v3/__http://libc-2.28.so__;!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8RezGD5pg$">libc-2.28.so</a> 0000155550887493 __libc_start_main Unknown Unknown<br> 255: mitgcmuv_ad 000000000040375E Unknown Unknown Unknown<br>1481: forrtl: error (78): process killed (SIGTERM)<br>1481: Image PC Routine Line Source <br>1481: libifcoremt.so.5 00001555553FC76C for__signal_handl Unknown Unknown<br>1481: libpthread-2.28.s 0000155550C3BC20 Unknown Unknown Unknown<br>1481: libpthread-2.28.s 0000155550C3B1D6 __open64 Unknown Unknown<br>1481: libifcoremt.so.5 00001555554911B1 for__open_proc Unknown Unknown<br>1481: libifcoremt.so.5 000015555540CCBE for_open Unknown Unknown<br>1481: mitgcmuv_ad 000000000097F1F5 Unknown Unknown Unknown<br>1481: mitgcmuv_ad 000000000090122F Unknown Unknown Unknown<br>1481: mitgcmuv_ad 00000000009E2F4A Unknown Unknown Unknown<br>1481: mitgcmuv_ad 00000000009DB0E5 Unknown Unknown Unknown<br>1481: mitgcmuv_ad 00000000009F7D38 Unknown Unknown Unknown<br>1481: mitgcmuv_ad 000000000097DDEA Unknown Unknown Unknown<br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 3, 2023 at 6:35 PM <<a href="mailto:mitgcm-support-request@mitgcm.org">mitgcm-support-request@mitgcm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send MITgcm-support mailing list submissions to<br>
<a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://urldefense.com/v3/__http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support__;!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8QuLWdRUA$" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:mitgcm-support-request@mitgcm.org" target="_blank">mitgcm-support-request@mitgcm.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:mitgcm-support-owner@mitgcm.org" target="_blank">mitgcm-support-owner@mitgcm.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of MITgcm-support digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. too many values for NAMELIST variable (mario wrk)<br>
2. Re: too many values for NAMELIST variable<br>
(Menemenlis, Dimitris (US 329B))<br>
3. Re: too many values for NAMELIST variable<br>
(Carroll, Dustin (US 329C-Affiliate))<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Fri, 3 Mar 2023 17:48:03 +0300<br>
From: mario wrk <<a href="mailto:wrkmario@gmail.com" target="_blank">wrkmario@gmail.com</a>><br>
To: <a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a><br>
Subject: [MITgcm-support] too many values for NAMELIST variable<br>
Message-ID:<br>
<<a href="mailto:CAAfDP0dnjOKcAO682mC1%2Bz1BDGmAYXdOaC3-iLQfJa0E8kinjA@mail.gmail.com" target="_blank">CAAfDP0dnjOKcAO682mC1+z1BDGmAYXdOaC3-iLQfJa0E8kinjA@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Dear MITgcm community,<br>
I was running a high resolution model in parallel with OpenMPI and got the<br>
error below.<br>
Does anyone have a clue?<br>
Best,<br>
Mario<br>
<br>
<br>
799: forrtl: severe (18): too many values for NAMELIST variable, unit 11,<br>
file .........run_ad/scratch1.000000799, line 3315, position 7<br>
799: Image PC Routine Line<br>
Source<br>
799: libifcoremt.so.5 00001555553E6E79 for__io_return Unknown<br>
Unknown<br>
799: libifcoremt.so.5 000015555542F3F5 for_read_seq_nml Unknown<br>
Unknown<br>
799: mitgcmuv_ad 0000000000786AC8 Unknown Unknown<br>
Unknown<br>
799: mitgcmuv_ad 000000000077F9A4 Unknown Unknown<br>
Unknown<br>
799: mitgcmuv_ad 0000000000862732 Unknown Unknown<br>
Unknown<br>
799: mitgcmuv_ad 00000000008BD780 Unknown Unknown<br>
Unknown<br>
799: mitgcmuv_ad 00000000004037C2 Unknown Unknown<br>
Unknown<br>
799: <a href="https://urldefense.com/v3/__http://libc-2.28.so__;!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8RezGD5pg$" rel="noreferrer" target="_blank">libc-2.28.so</a> 0000155550887493 __libc_start_main Unknown<br>
Unknown<br>
799: mitgcmuv_ad 00000000004036CE Unknown Unknown<br>
Unknown<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="https://urldefense.com/v3/__http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20230303/14bbf386/attachment-0001.html__;!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8R3N7lOzA$" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20230303/14bbf386/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Fri, 3 Mar 2023 15:11:44 +0000<br>
From: "Menemenlis, Dimitris (US 329B)"<br>
<<a href="mailto:dimitris.menemenlis@jpl.nasa.gov" target="_blank">dimitris.menemenlis@jpl.nasa.gov</a>><br>
To: MITgcm Support <<a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a>><br>
Subject: Re: [MITgcm-support] too many values for NAMELIST variable<br>
Message-ID: <<a href="mailto:541236B8-2F43-46B3-A2BE-67CC43AD6D94@jpl.nasa.gov" target="_blank">541236B8-2F43-46B3-A2BE-67CC43AD6D94@jpl.nasa.gov</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
There is a problem with one of your runtime parameter files (data or data.*) in your runtime directory run_ad.<br>
<br>
On Mar 3, 2023, at 6:48 AM, mario wrk <<a href="mailto:wrkmario@gmail.com" target="_blank">wrkmario@gmail.com</a>> wrote:<br>
<br>
Dear MITgcm community,<br>
I was running a high resolution model in parallel with OpenMPI and got the error below.<br>
Does anyone have a clue?<br>
Best,<br>
Mario<br>
<br>
<br>
799: forrtl: severe (18): too many values for NAMELIST variable, unit 11, file .........run_ad/scratch1.000000799, line 3315, position 7<br>
799: Image PC Routine Line Source<br>
799: libifcoremt.so.5 00001555553E6E79 for__io_return Unknown Unknown<br>
799: libifcoremt.so.5 000015555542F3F5 for_read_seq_nml Unknown Unknown<br>
799: mitgcmuv_ad 0000000000786AC8 Unknown Unknown Unknown<br>
799: mitgcmuv_ad 000000000077F9A4 Unknown Unknown Unknown<br>
799: mitgcmuv_ad 0000000000862732 Unknown Unknown Unknown<br>
799: mitgcmuv_ad 00000000008BD780 Unknown Unknown Unknown<br>
799: mitgcmuv_ad 00000000004037C2 Unknown Unknown Unknown<br>
799: <a href="https://urldefense.com/v3/__http://libc-2.28.so__;!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8RezGD5pg$" rel="noreferrer" target="_blank">libc-2.28.so</a><<a href="https://urldefense.com/v3/__https://urldefense.us/v3/__http:/*libc-2.28.so__;!!PvBDto6Hs4WbVuu7!M2W-DPfuY1q_PomytTcSW2ZHGL19dSVErDKaeBk6O3OIi_YZzV1x60iLZjKuaNrKDqjCzquEc3Kh4E8hzb2sHWWSPPDd$__;Lw!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8SGL_NBUw$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__http://libc-2.28.so__;!!PvBDto6Hs4WbVuu7!M2W-DPfuY1q_PomytTcSW2ZHGL19dSVErDKaeBk6O3OIi_YZzV1x60iLZjKuaNrKDqjCzquEc3Kh4E8hzb2sHWWSPPDd$</a>> 0000155550887493 __libc_start_main Unknown Unknown<br>
799: mitgcmuv_ad 00000000004036CE Unknown Unknown Unknown<br>
_______________________________________________<br>
MITgcm-support mailing list<br>
<a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
<a href="https://urldefense.com/v3/__https://urldefense.us/v3/__http:/*mailman.mitgcm.org/mailman/listinfo/mitgcm-support__;!!PvBDto6Hs4WbVuu7!M2W-DPfuY1q_PomytTcSW2ZHGL19dSVErDKaeBk6O3OIi_YZzV1x60iLZjKuaNrKDqjCzquEc3Kh4E8hzb2sHXBtL2F1$__;Lw!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8SaL3uv1Q$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support__;!!PvBDto6Hs4WbVuu7!M2W-DPfuY1q_PomytTcSW2ZHGL19dSVErDKaeBk6O3OIi_YZzV1x60iLZjKuaNrKDqjCzquEc3Kh4E8hzb2sHXBtL2F1$</a><br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="https://urldefense.com/v3/__http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20230303/3f126b2a/attachment-0001.html__;!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8RTkw7Kfw$" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20230303/3f126b2a/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Fri, 3 Mar 2023 15:35:12 +0000<br>
From: "Carroll, Dustin (US 329C-Affiliate)"<br>
<<a href="mailto:dustin.carroll@jpl.nasa.gov" target="_blank">dustin.carroll@jpl.nasa.gov</a>><br>
To: "<a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a>" <<a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a>><br>
Subject: Re: [MITgcm-support] too many values for NAMELIST variable<br>
Message-ID:<br>
<<a href="mailto:SJ0PR09MB90307EB60388A8CD177F8130BDB39@SJ0PR09MB9030.namprd09.prod.outlook.com" target="_blank">SJ0PR09MB90307EB60388A8CD177F8130BDB39@SJ0PR09MB9030.namprd09.prod.outlook.com</a>><br>
<br>
Content-Type: text/plain; charset="windows-1252"<br>
<br>
To follow up on Dimitris? comment, if you open the file ?scratch1.000000799? in your run_ad directory<br>
and look at line 3315, position 7 this will tell you where the syntax error / incorrect parameter value<br>
occurred in your data.* file.<br>
<br>
From: MITgcm-support <<a href="mailto:mitgcm-support-bounces@mitgcm.org" target="_blank">mitgcm-support-bounces@mitgcm.org</a>> on behalf of Menemenlis, Dimitris (US 329B) <<a href="mailto:dimitris.menemenlis@jpl.nasa.gov" target="_blank">dimitris.menemenlis@jpl.nasa.gov</a>><br>
Date: Friday, March 3, 2023 at 7:12 AM<br>
To: MITgcm Support <<a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a>><br>
Subject: [EXTERNAL] Re: [MITgcm-support] too many values for NAMELIST variable<br>
There is a problem with one of your runtime parameter files (data or data.*) in your runtime directory run_ad.<br>
<br>
<br>
On Mar 3, 2023, at 6:48 AM, mario wrk <<a href="mailto:wrkmario@gmail.com" target="_blank">wrkmario@gmail.com</a>> wrote:<br>
<br>
Dear MITgcm community,<br>
I was running a high resolution model in parallel with OpenMPI and got the error below.<br>
Does anyone have a clue?<br>
Best,<br>
Mario<br>
<br>
<br>
799: forrtl: severe (18): too many values for NAMELIST variable, unit 11, file .........run_ad/scratch1.000000799, line 3315, position 7<br>
799: Image PC Routine Line Source<br>
799: libifcoremt.so.5 00001555553E6E79 for__io_return Unknown Unknown<br>
799: libifcoremt.so.5 000015555542F3F5 for_read_seq_nml Unknown Unknown<br>
799: mitgcmuv_ad 0000000000786AC8 Unknown Unknown Unknown<br>
799: mitgcmuv_ad 000000000077F9A4 Unknown Unknown Unknown<br>
799: mitgcmuv_ad 0000000000862732 Unknown Unknown Unknown<br>
799: mitgcmuv_ad 00000000008BD780 Unknown Unknown Unknown<br>
799: mitgcmuv_ad 00000000004037C2 Unknown Unknown Unknown<br>
799: <a href="https://urldefense.com/v3/__http://libc-2.28.so__;!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8RezGD5pg$" rel="noreferrer" target="_blank">libc-2.28.so</a><<a href="https://urldefense.com/v3/__https://urldefense.us/v3/__http:/libc-2.28.so__;!!PvBDto6Hs4WbVuu7!M2W-DPfuY1q_PomytTcSW2ZHGL19dSVErDKaeBk6O3OIi_YZzV1x60iLZjKuaNrKDqjCzquEc3Kh4E8hzb2sHWWSPPDd$__;!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8QNGFHtAQ$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__http:/libc-2.28.so__;!!PvBDto6Hs4WbVuu7!M2W-DPfuY1q_PomytTcSW2ZHGL19dSVErDKaeBk6O3OIi_YZzV1x60iLZjKuaNrKDqjCzquEc3Kh4E8hzb2sHWWSPPDd$</a>> 0000155550887493 __libc_start_main Unknown Unknown<br>
799: mitgcmuv_ad 00000000004036CE Unknown Unknown Unknown<br>
_______________________________________________<br>
MITgcm-support mailing list<br>
<a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
<a href="https://urldefense.com/v3/__https://urldefense.us/v3/__http:/*mailman.mitgcm.org/mailman/listinfo/mitgcm-support__;!!PvBDto6Hs4WbVuu7!M2W-DPfuY1q_PomytTcSW2ZHGL19dSVErDKaeBk6O3OIi_YZzV1x60iLZjKuaNrKDqjCzquEc3Kh4E8hzb2sHXBtL2F1$__;Lw!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8SaL3uv1Q$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support__;!!PvBDto6Hs4WbVuu7!M2W-DPfuY1q_PomytTcSW2ZHGL19dSVErDKaeBk6O3OIi_YZzV1x60iLZjKuaNrKDqjCzquEc3Kh4E8hzb2sHXBtL2F1$</a><br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="https://urldefense.com/v3/__http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20230303/c74ea89b/attachment.html__;!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8RsC1HGuw$" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20230303/c74ea89b/attachment.html</a>><br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
MITgcm-support mailing list<br>
<a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
<a href="https://urldefense.com/v3/__http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support__;!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8QuLWdRUA$" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
<br>
<br>
------------------------------<br>
<br>
End of MITgcm-support Digest, Vol 237, Issue 1<br>
**********************************************<br>
</blockquote></div>
_______________________________________________<br>MITgcm-support mailing list<br>MITgcm-support@mitgcm.org<br>https://urldefense.com/v3/__http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support__;!!Mih3wA!Gw4tc7h4d5xgbTQUaiW0X933fMbQFlPW_FA6f78Qjf2PAq-yIVJFR10JAMMJ4xttCi3NmbCTS8QuLWdRUA$ <br></div></blockquote></div><br></div></body></html>