<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Martin,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you for following up on this, so it is not the xx_* files prep issue but something different.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I will try obtaining more information and see if I can trace the lines, and I believe the character length is good...</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Regards,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Yohei</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> MITgcm-support <mitgcm-support-bounces@mitgcm.org> on behalf of Martin Losch <Martin.Losch@awi.de><br>
<b>Sent:</b> 13 May 2024 13:45<br>
<b>To:</b> MITgcm Support <mitgcm-support@mitgcm.org><br>
<b>Subject:</b> Re: [MITgcm-support] Segmentation Fault in MITgcm Adjoint Simulations</font>
<div> </div>
</div>
<div style="line-break:after-white-space">the xx_* files in the zeroth iteration are generated and initialised as zeros by the model, no need to specify them.
<div><br>
</div>
<div>Martin<br id="x_lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On 13. May 2024, at 14:40, Yohei Takano - BAS <yokano@bas.ac.uk> wrote:</div>
<br class="x_Apple-interchange-newline">
<div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Hi Martin,</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Thank you for clarifying and I agree, need to trave the line for #2 0x0000000000519133 in ctrl_map_genarr2d_ad_ ()</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
(which requires debug mode (devel) from my understanding, figuring out the other issue on this now).</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
I was tracing the forward/adjoint code found the fnamegenIn (e.g "fnamegenIn is xx_*.$iter.data, so the required access records starts at")</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
and for example for tauu, I have xx_ files, like "xx_tauu.effective.0000000000.data". The character length is not 80 but it is updated ones</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
with MAX_LEN_FNAME and it's 512 as in current code.</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
I might ask before but do we need to prepare xx_* files before running the adjoint sensitivity simulations? I think the codes generates it when</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
not provided so we thought it is okay...</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Regards,</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Yohei</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div id="x_appendonsend" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
</div>
<hr tabindex="-1" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none; display:inline-block; width:1025.078125px">
<span style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important"></span>
<div id="x_divRplyFwdMsg" dir="ltr" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b><span class="x_Apple-converted-space"> </span>MITgcm-support <mitgcm-support-bounces@mitgcm.org> on behalf of Martin Losch <Martin.Losch@awi.de><br>
<b>Sent:</b><span class="x_Apple-converted-space"> </span>13 May 2024 13:08<br>
<b>To:</b><span class="x_Apple-converted-space"> </span>MITgcm Support <mitgcm-support@mitgcm.org><br>
<b>Subject:</b><span class="x_Apple-converted-space"> </span>Re: [MITgcm-support] Segmentation Fault in MITgcm Adjoint Simulations</font>
<div> </div>
</div>
<div class="x_BodyFragment" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font size="2"><span style="font-size:11pt">
<div class="x_PlainText">Hi Yohei,<br>
<br>
I think you need to find out what the hex-code in this line of your traceback mean (i.e. which line):<br>
<br>
#2 0x0000000000519133 in ctrl_map_genarr2d_ad_ ()<br>
<br>
The error message is pretty clear, and since the last MITgcm routine is the taf-generated ctrl_map_genarr2d_ad, it should happen somewhere in that routine. But since this routine depends on your setup (CPP-flags, etc), it’s hard to see from here, where this
should happen. In the forward code there are write statements like:<br>
<br>
> WRITE(fnamegenIn,'(2A,I10.10)')<br>
> & ctrlDir(1:ilDir)//fnamebase(1:ilgen),'.',optimcycle<br>
> WRITE(fnamegenOut,'(2A,I10.10)')<br>
> & ctrlDir(1:ilDir)//fnamebase(1:ilgen),'.effective.',optimcycle<br>
<br>
which also appear in the ad-code, so I guess this is where it happens. What are the lengths of your “fnamegenin/out”? In current code they are “max_len_fnam” = 512, but I remember that we just changed that recently from 80.<br>
<br>
Martin<br>
<br>
> On 11. May 2024, at 16:55, Yohei Takano - BAS <yokano@bas.ac.uk> wrote:<br>
><span class="x_Apple-converted-space"> </span><br>
> Hello everyone,<br>
><span class="x_Apple-converted-space"> </span><br>
> I followed Martin's suggestion commenting out the STRING part, but the error (core dump) still exists<br>
> although the error slightly changed in core backtrace.<br>
><span class="x_Apple-converted-space"> </span><br>
> -----<br>
> #0 0x0000153e0f8793be in __cray_memcpy_ROME () from /opt/cray/pe/cce/15.0.0/cce/x86_64/lib/libu.so.1<br>
> #1 0x0000153e0f88700e in __cray_memmove_ROME () from /opt/cray/pe/cce/15.0.0/cce/x86_64/lib/libu.so.1<br>
> #2 0x0000000000519133 in ctrl_map_genarr2d_ad_ ()<br>
> #3 0x000000000050a0c0 in ctrl_map_ini_genarr_ad_ ()<br>
> #4 0x0000000000505f4e in ctrl_init_variables_ad_ ()<br>
> #5 0x000000000068c07f in packages_init_variables_ad_ ()<br>
> #6 0x000000000042ca7f in initialise_varia_ad_ ()<br>
> #7 0x000000000041e819 in adthe_main_loop_ ()<br>
> #8 0x0000000000a2c303 in the_model_main_ ()<br>
> #9 0x0000000000978dc0 in main_ ()<br>
> -----<br>
><span class="x_Apple-converted-space"> </span><br>
> I tried with debug mode (-devel) but I am having other issue with this on the HPC I am using (Archer2),<br>
> turns out MITgcm is trying to write into a string and is writing too many characters, given the declared length of the string.<br>
> I couldn't trace where this comes from yet (talking to my colleague, he said it could be anywhere...) but now I am a bit stuck.<br>
><span class="x_Apple-converted-space"> </span><br>
> -----<br>
> lib-4211 : UNRECOVERABLE library error<br>
> A WRITE operation tried to write a record that was too long.<br>
><span class="x_Apple-converted-space"> </span><br>
> Encountered during a sequential formatted WRITE to an internal file (character variable)<br>
> -----<br>
><span class="x_Apple-converted-space"> </span><br>
> For now, I will try with more stable (solid) adjoint setup and see if the adjoint simulations stably run on the HPC.<br>
> My colleague suggested to try simpler adjoint setup in verification such as below so I will start and try from this setup again.<br>
><span class="x_Apple-converted-space"> </span><a href="https://github.com/MITgcm/MITgcm/tree/master/verification/tutorial_tracer_adjsens">https://github.com/MITgcm/MITgcm/tree/master/verification/tutorial_tracer_adjsens</a><br>
> (I see this has been updated until recent but perhaps Martin can follow up on this).<br>
><span class="x_Apple-converted-space"> </span><br>
> Thank you for the suggestion and I will keep updated when something progresses.<br>
><span class="x_Apple-converted-space"> </span><br>
> Regards,<br>
><span class="x_Apple-converted-space"> </span><br>
> Yohei<span class="x_Apple-converted-space"> </span><br>
> <span class="x_Apple-converted-space"> </span><br>
><span class="x_Apple-converted-space"> </span><br>
> From: MITgcm-support <mitgcm-support-bounces@mitgcm.org> on behalf of Yohei Takano - BAS <yokano@bas.ac.uk><br>
> Sent: 10 May 2024 14:18<br>
> To: MITgcm Support <mitgcm-support@mitgcm.org><br>
> Subject: Re: [MITgcm-support] Segmentation Fault in MITgcm Adjoint Simulations<br>
> Thank you again for your continuous support Martin. I think I managed to finally trace the routine; I will check how things<br>
> are working now.<br>
><span class="x_Apple-converted-space"> </span><br>
> Regards,<br>
><span class="x_Apple-converted-space"> </span><br>
> YoheiFrom: MITgcm-support <mitgcm-support-bounces@mitgcm.org> on behalf of Martin Losch <Martin.Losch@awi.de><br>
> Sent: 10 May 2024 13:17<br>
> To: MITgcm Support <mitgcm-support@mitgcm.org><br>
> Subject: Re: [MITgcm-support] Segmentation Fault in MITgcm Adjoint Simulations<br>
> based on your backtrace message, your code will have a routine “ctrl_map_genarr2d_ad” somewhere (because it appears in the backtrace message). “grep” is your friend:<br>
><span class="x_Apple-converted-space"> </span><br>
> cd build<br>
> grep -i ctrl_map_genarr2d_ad *_ad.f<br>
><span class="x_Apple-converted-space"> </span><br>
> Martin<br>
><span class="x_Apple-converted-space"> </span><br>
> > On 10. May 2024, at 05:30, Yohei Takano - BAS <yokano@bas.ac.uk> wrote:<br>
> ><span class="x_Apple-converted-space"> </span><br>
> > Hi Matin, all,<br>
> ><span class="x_Apple-converted-space"> </span><br>
> > Sorry again, I followed up and checked the adjoint code but turns out "ctrl_map_genarr2d_ad.F" code does not exist<br>
> > (after make adall) and something strange might be happenning... (more specifically I don't see the forward ctrl_map_genarr2d.F<br>
> > code in checkpoint68r but instead I see "ctrl_map_genarr.F" but still the adjoint code (ctrl_map_genarr_ad.F) is not generated...<br>
> ><span class="x_Apple-converted-space"> </span><br>
> > I am not sure why this happened (I checked the code and ctrl package is included).<br>
> ><span class="x_Apple-converted-space"> </span><br>
> > Best regards,<br>
> ><span class="x_Apple-converted-space"> </span><br>
> > YoheiFrom: MITgcm-support <mitgcm-support-bounces@mitgcm.org> on behalf of Martin Losch <Martin.Losch@awi.de><br>
> > Sent: 09 May 2024 19:56<br>
> > To: MITgcm Support <mitgcm-support@mitgcm.org><br>
> > Subject: Re: [MITgcm-support] Segmentation Fault in MITgcm Adjoint Simulations<br>
> > Hi Yohei,<br>
> ><span class="x_Apple-converted-space"> </span><br>
> > Since this is in the AD-code produced by TAF (ctrl_map_genarr2d_ad.F), that’s where you need to put the print statements (in the corresponding subroutine, you’ll findi it in your build directory after run "make adall" (or "make adtaf”)<br>
> ><span class="x_Apple-converted-space"> </span><br>
> > Martin<br>
> ><span class="x_Apple-converted-space"> </span><br>
> >> On 9. May 2024, at 09:32, Yohei Takano - BAS <yokano@bas.ac.uk> wrote:<br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> Hi Martin,<br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> Thank you so much for the details, good to know where I can dig in related to the issue.<br>
> >> In pkg/ctrl, I think I found the relevant part of the code (although the code/file name is now different from what you mentioned,<br>
> >> because of update? ,... and I use checkpint68r). In ctrl_map_genarr.F (instead of ctrl_map_genarr2d_ad.F), I found the relevant<br>
> >> subroutine (CTRL_MAP_GENARR2D) and I think I can try adding some "print-hallo" statement to trace what is happening.<br>
> >> (other thing I spotted is that suroutine CTRL_MAP_GENARR2D is no calling tauu or tauv in the code, but I might have missed something,<br>
> >> still digging the codes).<br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> I am now running with the debugging options turned on, hope I can obtain line numbers.<br>
> >> I will also try commenting out the string comparison part (commenting this part will not impact the adjoint simulations... correct?).<br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> Best regards,<br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> Yohei<br>
> >><span class="x_Apple-converted-space"> </span><br>
> >><span class="x_Apple-converted-space"> </span><br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> From: MITgcm-support <mitgcm-support-bounces@mitgcm.org> on behalf of Martin Losch <Martin.Losch@awi.de><br>
> >> Sent: 09 May 2024 13:17<br>
> >> To: MITgcm Support <mitgcm-support@mitgcm.org><br>
> >> Subject: Re: [MITgcm-support] Segmentation Fault in MITgcm Adjoint Simulations<br>
> >> Hi Yohei,<br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> this is difficult to debug without access to your specific hpc computer. Segmentation fault usually means some memory violation (array boundaries, or in your case maybe the length of a string, because of the _f90_string_compare).<br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> It looks like it’s failing in the AD-part of initialising the ctrl variables, i.e. when the gradients are finally written to the adxx_tauu files (in this case).<br>
> >> My favorite debugger is the primitive “print-hallo” debugger: I would add print statements just before the call of ctrl_map_genarr2d_ad(tauu,tauu_ad, …) in ctrl_map_ini_genarr_ad.f or even within ctrl_map_genarr2d_ad (which will give you a lot of output)
to try to figure out what’s going on.<br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> You can also run it with debugging options turned on, then the backtrace should give you linenumbers instead of hex-code<br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> If it runs 1 out of 5 times, the problem is most likely related to your computer or the way your computer handles internal writes (again because of the _F90_STRING_COMPARE). There are few string comparisons in ctrl_map_genarr2d_ad like:<br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> > do k2 = 1, maxctrlproc<br>
> >> > if (xx_genarr2d_preproc(k2,iarr) == 'noscaling') then<br>
> >> > doscaling = .false.<br>
> >> > endif<br>
> >> > if (xx_genarr2d_preproc_c(k2,iarr) == 'log10ctrl') then<br>
> >> > dolog10ctrl = .true.<br>
> >> > log10initval = xx_genarr2d_preproc_r(k2,iarr)<br>
> >> > endif<br>
> >> > end do<br>
> >><span class="x_Apple-converted-space"> </span><br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> maybe you can try commenting them out for a test?<br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> Martin<br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> > On 9. May 2024, at 07:09, Yohei Takano - BAS <yokano@bas.ac.uk> wrote:<br>
> >> ><br>
> >> > Hello again,<br>
> >> ><br>
> >> > Following up on this topic, what I got from the core dump information is<br>
> >> ><br>
> >> > -----<br>
> >> > (gdb) backtrace<br>
> >> > #0 0x00001534138194ea in __memcmp_avx2_movbe () from /lib64/libc.so.6<br>
> >> > #1 0x00001534143fed66 in _F90_STRING_COMPARE () from /opt/cray/pe/cce/15.0.0/cce/x86_64/lib/libfi.so.1<br>
> >> > #2 0x0000000000519160 in ctrl_map_genarr2d_ad_ ()<br>
> >> > #3 0x000000000050a0c0 in ctrl_map_ini_genarr_ad_ ()<br>
> >> > #4 0x0000000000505f4e in ctrl_init_variables_ad_ ()<br>
> >> > #5 0x000000000068c28f in packages_init_variables_ad_ ()<br>
> >> > #6 0x000000000042ca7f in initialise_varia_ad_ ()<br>
> >> > #7 0x000000000041e819 in adthe_main_loop_ ()<br>
> >> > #8 0x0000000000a2c513 in the_model_main_ ()<br>
> >> > #9 0x0000000000978fd0 in main_ ()<br>
> >> > -----<br>
> >> ><br>
> >> > and STDOUT stops at this point (stops at the same point every time I run the model).<br>
> >> ><br>
> >> > ...<br>
> >> > (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: adxx_tauu.0000000000.data<br>
> >> > (PID.TID 0000.0001) MDS_WRITE_FIELD: it,rec,kS,kL,kH= 0 -42 15 1 1 file=adxx_tauu.0000000000<br>
> >> > (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: adxx_tauu.effective.00000000<br>
> >> > ...<br>
> >> ><br>
> >> > I think it is related to data access issues but still not sure why it runs successfully from time to time.<br>
> >> > I am not sure if this is related but, in this configuration, I don't provide 2d or 3d control files in advance (i.e. xx_tauu.*data, meta etc.)<br>
> >> > since I heard this will be generated during the run (and okay for adjoint sensitivity experiments, please correct me if I am wrong).<br>
> >> ><br>
> >> > Thank you again in advance, please let me know if you have thoughts (or encountered similar issues before).<br>
> >> ><br>
> >> > Best regards,<br>
> >> ><br>
> >> > Yohei<br>
> >> ><br>
> >> ><br>
> >> > From: MITgcm-support <mitgcm-support-bounces@mitgcm.org> on behalf of Yohei Takano - BAS <yokano@bas.ac.uk><br>
> >> > Sent: 07 May 2024 12:24<br>
> >> > To: mitgcm-support@mitgcm.org <mitgcm-support@mitgcm.org><br>
> >> > Subject: [MITgcm-support] Segmentation Fault in MITgcm Adjoint Simulations<br>
> >> > Hello all,<br>
> >> ><br>
> >> > My colleague and I have been working on adjoint sensitivity setup & simulations<br>
> >> > based on global coarse resolution biogeochemistry model (<a href="https://github.com/MITgcm/MITgcm/tree/master/verification/tutorial_global_oce_biogeo">https://github.com/MITgcm/MITgcm/tree/master/verification/tutorial_global_oce_biogeo</a>),<br>
> >> > similar configurations but we have our own customization.<br>
> >> ><br>
> >> > We manage to compile the model and run adjoint test simulations. However, the model crashes<br>
> >> > (with segmentation fault) towards the end of adjoint simulations and we would like your advice to<br>
> >> > figure out why this is happening. The strange part is that we manage to successfully run at one point<br>
> >> > but when try to reproduce with the exact same setting (on the same HPC) it starts to crash again...<br>
> >> > Roughly speaking 1 out of 5 times it runs successfully but most of the time crashes at the same point.<br>
> >> > We are puzzled because we are using the exact same settings/executable every time and wondering what causes<br>
> >> > this unstable situation. The environment should be the same everytime we run the model.<br>
> >> ><br>
> >> > Does anyone have similar experiences? Here is the code/configuration I have been working on with my colleague Dani Jones.<br>
> >> ><span class="x_Apple-converted-space"> </span><a href="https://github.com/ytakano3/MITgcm_BGC_Model_Config">https://github.com/ytakano3/MITgcm_BGC_Model_Config</a><br>
> >> ><br>
> >> > In "global_ocn3deg_bgcv0" you see "code_ad" and "input_ad/input_ad_kpp_atmco2pv0" and you can compile<br>
> >> > (with TAF) and run the model. It is 2 years test run. When I am trying this, it fails (i.e. segmentation fault) most of the time but<br>
> >> > again say 1 in 5 times the model runs successfully... We would like to figure out why this is happening, why it gets unstable so<br>
> >> > let me know if you have thoughts on this.<br>
> >> ><br>
> >> > Sorry about the long e-mail but please let me know if anything is unclear. Thank you in advance.<br>
> >> ><br>
> >> > Regards,<br>
> >> ><br>
> >> > Yohei<br>
> >> > <span class="x_Apple-converted-space"> </span><br>
> >> ><br>
> >> > This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately
and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks
before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses.<br>
> >> ><br>
> >> > _______________________________________________<br>
> >> > MITgcm-support mailing list<br>
> >> > MITgcm-support@mitgcm.org<br>
> >> ><span class="x_Apple-converted-space"> </span><a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> >><span class="x_Apple-converted-space"> </span><br>
> >><span class="x_Apple-converted-space"> </span><br>
> >> _______________________________________________<br>
> >> MITgcm-support mailing list<br>
> >> MITgcm-support@mitgcm.org<br>
> >><span class="x_Apple-converted-space"> </span><a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> >> _______________________________________________<br>
> >> MITgcm-support mailing list<br>
> >> MITgcm-support@mitgcm.org<br>
> >><span class="x_Apple-converted-space"> </span><a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> ><span class="x_Apple-converted-space"> </span><br>
> ><span class="x_Apple-converted-space"> </span><br>
> > _______________________________________________<br>
> > MITgcm-support mailing list<br>
> > MITgcm-support@mitgcm.org<br>
> ><span class="x_Apple-converted-space"> </span><a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
><span class="x_Apple-converted-space"> </span><br>
><span class="x_Apple-converted-space"> </span><br>
> _______________________________________________<br>
> MITgcm-support mailing list<br>
> MITgcm-support@mitgcm.org<br>
><span class="x_Apple-converted-space"> </span><a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> _______________________________________________<br>
> MITgcm-support mailing list<br>
> MITgcm-support@mitgcm.org<br>
><span class="x_Apple-converted-space"> </span><a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
<br>
<br>
_______________________________________________<br>
MITgcm-support mailing list<br>
<a href="mailto:MITgcm-support@mitgcm.org">MITgcm-support@mitgcm.org</a><br>
<a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
</div>
</span></font></div>
<span style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important">_______________________________________________</span><br style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<span style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important">MITgcm-support
mailing list</span><br style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<a href="mailto:MITgcm-support@mitgcm.org" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px">MITgcm-support@mitgcm.org</a><br style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a></div>
</blockquote>
</div>
<br>
</div>
</div>
</body>
</html>