<div dir="ltr"><div>Hi Martin,</div><div><br></div><div>Oh, I think I found the issue. I wasn't using the right procedure to prepare the run. As a result, some of the packages needed for the adjoint run were turned off (e.g. autodiff). I've gotten past that error now - thanks for helping me talk it through. <br><br>I'll aim to make the documentation for global_oce_cs32 a little more detailed for that case.<br><br>Best wishes,<br>Dan<br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 24, 2020 at 11:56 AM Dan Jones <<a href="mailto:dcjones.work@gmail.com">dcjones.work@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Martin,</div><div><br></div><div>I've switched to BAS local HPC. I started from scratch and cloned MITgcm and verification_other. And I tried compiling and running the serial case. Unfortunately, I am still getting the same error message:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>(PID.TID 0000.0001) MDS_READ_FIELD: opening global file: adxx_diffkr.0000000000.data</div><div>(PID.TID 0000.0001) MDS_WRITE_FIELD: it,rec,kS,kL,kH= 0 1 50 1 50 file=adxx_diffkr.0000000000</div><div>(PID.TID 0000.0001) MDS_READ_FIELD: opening global file: xx_kapredi.0000000000.data</div><div>(PID.TID 0000.0001) MDS_READ_FIELD: opening global file: adxx_kapredi.0000000000.data</div><div>(PID.TID 0000.0001) MDS_WRITE_FIELD: it,rec,kS,kL,kH= 0 1 50 1 50 file=adxx_kapredi.0000000000</div><div>(PID.TID 0000.0001) MDS_READ_FIELD: opening global file: xx_kapgm.0000000000.data</div><div>(PID.TID 0000.0001) MDS_READ_FIELD: opening global file: adxx_kapgm.0000000000.data</div><div>(PID.TID 0000.0001) MDS_WRITE_FIELD: it,rec,kS,kL,kH= 0 1 50 1 50 file=adxx_kapgm.0000000000</div><div>(PID.TID 0000.0001) MDS_READ_FIELD: filename: adm_horflux_vol.0000000000.data</div><div>(PID.TID 0000.0001) MDS_READ_FIELD: File does not exist</div><div><br></div></blockquote>That's really strange. However, when I run "testreport" in serial mode, I don't get the above error! So I guess the "File does not exist" issue is specific to my parallel adjoint runs for some reason? Specifically:<div><br><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>run: ../verification/testreport -of ../tools/build_options/linux_amd64_scihub -command ./mitgcmuv_ad -adm -ncad -t global_oce_cs32</div><div>on : Linux bslws05 3.10.0-1127.10.1.el7.x86_64 #1 SMP Wed Jun 3 14:28:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux</div><div><br></div><div> OPTFILE=/data/expose/MITgcm_development/MITgcm/tools/build_options/linux_amd64_scihub</div><div><br></div><div>Adjoint generated by TAF</div><div><br></div><div>default 10</div><div>G D M C A F</div><div>e p a R o d D</div><div>n n k u s G G</div><div>2 d e n t r r</div><div><br></div><div>Y Y Y N .. .. .. N/O global_oce_cs32 (e=0, w=48)</div><div>Y Y Y N .. .. .. N/O global_oce_cs32.sens</div><div>Start time: Thu 24 Sep 10:47:11 BST 2020</div><div>End time: Thu 24 Sep 11:05:41 BST 2020</div><div><br></div></blockquote>The "global_oce_cs32" run fails with this error:<br><br><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div> fail at i,j= 16 17 ; rStarFacC,H,eta =********** 5.158000E+03 -2.329336E+07<br> fail at i,j= 16 17 ; rStarFacS,H,eta =********** 5.091000E+03 2.292655E+05 -2.329336E+07<br> fail at i,j= 17 17 ; rStarFacC,H,eta =********** 5.158000E+03 -2.329336E+07<br>WARNING: r*FacC < hFacInf at 2 pts : bi,bj,Thid,Iter= 12 1 1 7<br>WARNING: r*FacS < hFacInf at 1 pts : bi,bj,Thid,Iter= 12 1 1 7<br>STOP in CALC_R_STAR : too SMALL rStarFac[C,W,S] !<br></div><div><br></div></blockquote>Whereas the "global_oce_cs32.sens" run does finish, albeit with a note/warning:<br><br><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL<br>STOP NORMAL END<br>PROGRAM MAIN: Execution ended Normally<br></div><div><br></div></blockquote>In short: the "adm_horflux_vol" and "adm_boxmean_theta" files *do* exist in the testreport run. But they do not exist in my manual run. I don't yet have any idea what the difference might be. <br><br>I'll try to see if anyone else can replicate my error. Any other thoughts at the moment? </div><div><br></div><div>Thanks so much again for your help thus far. <br><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br></div></blockquote><div>Best wishes,<br>Dan<br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 23, 2020 at 5:07 PM <<a href="mailto:mitgcm-support-request@mitgcm.org" target="_blank">mitgcm-support-request@mitgcm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send MITgcm-support mailing list submissions to<br>
<a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:mitgcm-support-request@mitgcm.org" target="_blank">mitgcm-support-request@mitgcm.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:mitgcm-support-owner@mitgcm.org" target="_blank">mitgcm-support-owner@mitgcm.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of MITgcm-support digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: verification_other case global_oce_cs32 fails at runtime<br>
in adjoint mode (Martin Losch)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Wed, 23 Sep 2020 11:54:25 +0200<br>
From: Martin Losch <<a href="mailto:Martin.Losch@awi.de" target="_blank">Martin.Losch@awi.de</a>><br>
To: MITgcm Support <<a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a>><br>
Subject: Re: [MITgcm-support] verification_other case global_oce_cs32<br>
fails at runtime in adjoint mode<br>
Message-ID: <<a href="mailto:63C0735F-3275-4680-8322-3D96116B10A2@awi.de" target="_blank">63C0735F-3275-4680-8322-3D96116B10A2@awi.de</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi Dan,<br>
<br>
the ?N? means that the run did not complete sucessfully. genmake2, make depend, and make all have a Y, meaning sucessful completion.<br>
<br>
I reran the verification_other/global_oce_cs experiment on 24 CPUs on our Cray CS400 with ifort and TAF. and this is the result:<br>
<br>
> run: ../verification/testreport -MPI 24 -j 16 -of ../tools/build_options/linux_ia64_ifort_ollie -command 'srun ./mitgcmuv_ad' -adm -ncad -t global_oce_cs32<br>
> on : Linux prod-0115 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux<br>
> <br>
> OPTFILE=/work/ollie/mlosch/test_ollie/MITgcm_ifort/tools/build_options/linux_ia64_ifort_ollie<br>
> <br>
> Adjoint generated by TAF<br>
> <br>
> default 10<br>
> G D M C A F<br>
> e p a R o d D<br>
> n n k u s G G<br>
> 2 d e n t r r<br>
> <br>
> Y Y Y Y 16>16< 4 pass global_oce_cs32 (e=0, w=48)<br>
> Y Y Y Y 16>16<16 pass global_oce_cs32.sens<br>
> Start time: Wed Sep 23 11:46:44 CEST 2020<br>
> End time: Wed Sep 23 11:48:14 CEST 2020<br>
<br>
note that my ?mpirun? is actually ?srun?, because we use SLURM. So I have no issue here and I have no idea why this is different for you.<br>
Can I suggest that you start from scratch again: Get a new clone of MITgcm and verification_other and retry? <br>
<br>
Martin<br>
<br>
<br>
<br>
> On 23. Sep 2020, at 11:20, Dan Jones <<a href="mailto:dcjones.work@gmail.com" target="_blank">dcjones.work@gmail.com</a>> wrote:<br>
> <br>
> Hi Martin,<br>
> <br>
> Yes, apologies, I had commented out parts of the cost function for my testing purposes. If I just run the unmodified case, I get:<br>
> <br>
> global fc = 0.543536615356758E+08<br>
> <br>
> But this error still persists:<br>
> <br>
> MDS_READ_FIELD: filename: adm_boxmean_theta.0000000000.data<br>
> <br>
> If I create a blank adm file for it to read, then the computation continues but quickly runs into a different error. And in any case, adding a blank file before running shouldn't be necessary.<br>
> <br>
> I ran the "testreport" on a couple of adjoint cases in the "verification" directory. Both "tutorial_global_oce_optim" and "tutorial_dic_adjoffline" passed. But when I try to run this set of commands:<br>
> <br>
> cd verification_other<br>
> ../verification/testreport -t global_oce_cs32/ -adm -j 4 - optfile=../tools/build_options/linux_ia64_cray_archer<br>
> <br>
> Then I get the following result:<br>
> <br>
> Y Y Y N .. .. .. N/O global_oce_cs32/ (e=0, w=48)<br>
> Y Y Y N .. .. .. N/O global_oce_cs32/.sens<br>
> <br>
> Can you remind me what the last "N" stands for again? What has failed? This line of output seems suspicious:<br>
> <br>
> ../verification/testreport: line 845: 32072 Killed ./mitgcmuv_ad > output_adm.txt<br>
> <br>
> That line in "testreport" has to do with DIVA, according to the comments. Unfortunately, "output_adm.txt" is blank. <br>
> <br>
> Are you able to get the adjoint part of the "input_ad.sens" experiment to work, after the cost function has been calculated? <br>
> <br>
> Best wishes,<br>
> Dan<br>
> <br>
> On Tue, Sep 22, 2020 at 5:17 PM <<a href="mailto:mitgcm-support-request@mitgcm.org" target="_blank">mitgcm-support-request@mitgcm.org</a>> wrote:<br>
> Send MITgcm-support mailing list submissions to<br>
> <a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a><br>
> <br>
> To subscribe or unsubscribe via the World Wide Web, visit<br>
> <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> or, via email, send a message with subject or body 'help' to<br>
> <a href="mailto:mitgcm-support-request@mitgcm.org" target="_blank">mitgcm-support-request@mitgcm.org</a><br>
> <br>
> You can reach the person managing the list at<br>
> <a href="mailto:mitgcm-support-owner@mitgcm.org" target="_blank">mitgcm-support-owner@mitgcm.org</a><br>
> <br>
> When replying, please edit your Subject line so it is more specific<br>
> than "Re: Contents of MITgcm-support digest..."<br>
> <br>
> <br>
> Today's Topics:<br>
> <br>
> 1. Re: verification_other case global_oce_cs32 fails at runtime<br>
> in adjoint mode (Martin Losch)<br>
> <br>
> <br>
> ----------------------------------------------------------------------<br>
> <br>
> Message: 1<br>
> Date: Tue, 22 Sep 2020 17:40:44 +0200<br>
> From: Martin Losch <<a href="mailto:Martin.Losch@awi.de" target="_blank">Martin.Losch@awi.de</a>><br>
> To: MITgcm Support <<a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a>><br>
> Subject: Re: [MITgcm-support] verification_other case global_oce_cs32<br>
> fails at runtime in adjoint mode<br>
> Message-ID: <<a href="mailto:5B5A14D1-A1DF-435B-B468-E0807AC4C437@awi.de" target="_blank">5B5A14D1-A1DF-435B-B468-E0807AC4C437@awi.de</a>><br>
> Content-Type: text/plain; charset="utf-8"<br>
> <br>
> Hi Dan,<br>
> <br>
> I get very different output and results (e.g. global_fc= 0.543870?D+08).<br>
> <br>
> Can you try to run testreport on this?<br>
> <br>
> cd verification_other<br>
> ../verification/testreport -t global_oce_cs32 -adm $otheroptions<br>
> where $otheroptions, are, e.g. -j 4, -devel, etc.. I you want to use your MPI job you?ll need '-MPI 24' and maybe "-command ?mpirun -n 24 ./mitgcmuv?" to run on 24 cpus. check the help section of testreport for details.<br>
> <br>
> The output should be ?fairly? independent of the number of CPUs. I use the default 24 tiles (but run sequentially on one cpu)<br>
> <br>
> Martin<br>
> <br>
> <br>
> > On 22. Sep 2020, at 16:52, Dan Jones <<a href="mailto:dcjones.work@gmail.com" target="_blank">dcjones.work@gmail.com</a>> wrote:<br>
> > <br>
> > Hi Martin,<br>
> > <br>
> > Thanks for your quick reply! I can't get the serial case to run on ARCHER, unfortunately. I think for now I'm stuck testing in parallel. When I run "grep m_boxmean_theta *.f", I get exactly the same results as you. I'm also using MITgcm/verification_other from GitHub. <br>
> > <br>
> > Here is a bit more of the output which will hopefully help. This is from a case where I tried to use the "m_horflux_vol" case instead of the "m_boxmean_theta" case. I'm using data.ecco from the input_ad.sens directory. <br>
> > <br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: xx_kapgm.0000000000.data<br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: xx_kapredi.0000000000.data<br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: xx_diffkr.0000000000.data<br>
> > (PID.TID 0000.0001) --> f_gencost =-0.348173207824978E+08 2<br>
> > (PID.TID 0000.0001) --> f_genarr3d = 0.000000000000000E+00 1<br>
> > (PID.TID 0000.0001) --> f_genarr3d = 0.000000000000000E+00 2<br>
> > (PID.TID 0000.0001) --> f_genarr3d = 0.000000000000000E+00 3<br>
> > (PID.TID 0000.0001) --> fc =-0.348173207824978E+08<br>
> > (PID.TID 0000.0001) early fc = 0.000000000000000E+00<br>
> > (PID.TID 0000.0001) local fc = 0.000000000000000E+00<br>
> > (PID.TID 0000.0001) global fc = -0.348173207824978E+08<br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: xx_diffkr.0000000000.data<br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: adxx_diffkr.0000000000.data<br>
> > (PID.TID 0000.0001) MDS_WRITE_FIELD: it,rec,kS,kL,kH= 0 1 50 1 50 file=adxx_diffkr.0000000000<br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: xx_kapredi.0000000000.data<br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: adxx_kapredi.0000000000.data<br>
> > (PID.TID 0000.0001) MDS_WRITE_FIELD: it,rec,kS,kL,kH= 0 1 50 1 50 file=adxx_kapredi.0000000000<br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: xx_kapgm.0000000000.data<br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: adxx_kapgm.0000000000.data<br>
> > (PID.TID 0000.0001) MDS_WRITE_FIELD: it,rec,kS,kL,kH= 0 1 50 1 50 file=adxx_kapgm.0000000000<br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: filename: adm_horflux_vol.0000000000.data<br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: File does not exist<br>
> > <br>
> > So it's after the cost function has been calculated, as the model is getting ready to perform the adjoint steps. It's able to read/write for the existing controls (kapgm, kapredi, diffkr). But it's apparently not creating an "ad" file for the general objective function term "horflux". That's why I was wondering if I should manually create a blank file first, as an ad-hoc fix. Any thoughts? <br>
> > <br>
> > Best wishes,<br>
> > Dan<br>
> > <br>
> > On Mon, Sep 21, 2020 at 8:37 PM <<a href="mailto:mitgcm-support-request@mitgcm.org" target="_blank">mitgcm-support-request@mitgcm.org</a>> wrote:<br>
> > Send MITgcm-support mailing list submissions to<br>
> > <a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a><br>
> > <br>
> > To subscribe or unsubscribe via the World Wide Web, visit<br>
> > <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> > or, via email, send a message with subject or body 'help' to<br>
> > <a href="mailto:mitgcm-support-request@mitgcm.org" target="_blank">mitgcm-support-request@mitgcm.org</a><br>
> > <br>
> > You can reach the person managing the list at<br>
> > <a href="mailto:mitgcm-support-owner@mitgcm.org" target="_blank">mitgcm-support-owner@mitgcm.org</a><br>
> > <br>
> > When replying, please edit your Subject line so it is more specific<br>
> > than "Re: Contents of MITgcm-support digest..."<br>
> > <br>
> > <br>
> > Today's Topics:<br>
> > <br>
> > 1. verification_other case global_oce_cs32 fails at runtime in<br>
> > adjoint mode (Dan Jones)<br>
> > 2. Re: verification_other case global_oce_cs32 fails at runtime<br>
> > in adjoint mode (Martin Losch)<br>
> > <br>
> > <br>
> > ----------------------------------------------------------------------<br>
> > <br>
> > Message: 1<br>
> > Date: Mon, 21 Sep 2020 10:00:47 +0100<br>
> > From: Dan Jones <<a href="mailto:dcjones.work@gmail.com" target="_blank">dcjones.work@gmail.com</a>><br>
> > To: <a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a><br>
> > Subject: [MITgcm-support] verification_other case global_oce_cs32<br>
> > fails at runtime in adjoint mode<br>
> > Message-ID:<br>
> > <CAPj3iHRxhUOCDT5m7H8uj8cg9dc=_<a href="mailto:oYVssQnvYhEA%2B_ALjeR6w@mail.gmail.com" target="_blank">oYVssQnvYhEA+_ALjeR6w@mail.gmail.com</a>><br>
> > Content-Type: text/plain; charset="utf-8"<br>
> > <br>
> > Hello.<br>
> > <br>
> > Apologies for the cross-posting - I've posted this as a GitHub issue, but I<br>
> > thought I should put it here as well.<br>
> > <br>
> > I am trying to build and test the global_oce_cs32 verification_other<br>
> > exercise using the code in the input_ad.sens directory. The forward case<br>
> > compiles and runs without error. The adjoint case (built using TAF)<br>
> > compiles without error, but at runtime I receive the following error in<br>
> > STDOUT:<br>
> > <br>
> > (PID.TID 0000.0001) MDS_READVEC_LOC: open file: south30_maskT<br>
> > (PID.TID 0000.0001) MDS_RD_REC_RL: iRec,Dim = 9 1<br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: filename: adm_boxmean_theta.0000000000.data<br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: File does not exist<br>
> > <br>
> > and this error in STDERR:<br>
> > <br>
> > (PID.TID 0000.0001) *** ERROR *** MDS_READ_FIELD: filename:<br>
> > adm_boxmean_theta.0000000000.data<br>
> > (PID.TID 0000.0001) *** ERROR *** MDS_READ_FIELD: File does not exist<br>
> > <br>
> > My MITgcm source code is up-to-date with the master. I am running on<br>
> > <a href="http://archer.ac.uk" rel="noreferrer" target="_blank">archer.ac.uk</a> <<a href="https://www.archer.ac.uk/" rel="noreferrer" target="_blank">https://www.archer.ac.uk/</a>> in parallel mode using 24 cores.<br>
> > <br>
> > What should I try here? I haven't run into this error before using other<br>
> > adjoint setups, at least not that I can recall. Should I just create an<br>
> > empty "dummy" file to start with? Thanks in advance for any help/guidance.<br>
> > <br>
> > Best regards,<br>
> > Dan<br>
> > <br>
> > <br>
> > --------------------------------------------------------------<br>
> > Dr Dan Jones / British Antarctic Survey<br>
> > <a href="http://danjonesocean.com" rel="noreferrer" target="_blank">danjonesocean.com</a> <<a href="http://www.danjonesocean.com" rel="noreferrer" target="_blank">http://www.danjonesocean.com</a>> / @DanJonesOcean<br>
> > --------------------------------------------------------------<br>
> > -------------- next part --------------<br>
> > An HTML attachment was scrubbed...<br>
> > URL: <<a href="http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20200921/ddb38a00/attachment-0001.html" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20200921/ddb38a00/attachment-0001.html</a>><br>
> > <br>
> > ------------------------------<br>
> > <br>
> > Message: 2<br>
> > Date: Mon, 21 Sep 2020 14:56:23 +0200<br>
> > From: Martin Losch <<a href="mailto:Martin.Losch@awi.de" target="_blank">Martin.Losch@awi.de</a>><br>
> > To: MITgcm Support <<a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a>><br>
> > Subject: Re: [MITgcm-support] verification_other case global_oce_cs32<br>
> > fails at runtime in adjoint mode<br>
> > Message-ID: <<a href="mailto:FF2DD1AB-462E-4089-90CA-89B9552DF7D8@awi.de" target="_blank">FF2DD1AB-462E-4089-90CA-89B9552DF7D8@awi.de</a>><br>
> > Content-Type: text/plain; charset="utf-8"<br>
> > <br>
> > Hi Dan,<br>
> > <br>
> > I tried this on my linux box without MPI and I cannot reproduce your problem (I used MITgcm/verification_other.git and not the CVS MITgcm_contrib/verification_other, which appears to be out of date). I grepped the code for ?m_boxmean_theta? and only found this:<br>
> > <br>
> > (base) bkli04l006::build (master)> grep m_boxmean_theta *.f<br>
> > ad_input_code_ad.f: $'m_boxmean_theta') then<br>
> > ad_input_code_ad.f: $'m_boxmean_theta') then<br>
> > ad_input_code_ad.f: $'m_boxmean_theta') then<br>
> > ad_input_code_ad.f: $'m_boxmean_theta') then<br>
> > ad_input_code.f: if (gencost_barfile(kgen)(1:15).EQ.'m_boxmean_theta') then<br>
> > ad_taf_output.f: $'m_boxmean_theta') then<br>
> > ad_taf_output.f: $'m_boxmean_theta') then<br>
> > ad_taf_output.f: $'m_boxmean_theta') then<br>
> > ad_taf_output.f: $'m_boxmean_theta') then<br>
> > ecco_check.f: & (gencost_barfile(k)(1:15).EQ.'m_boxmean_theta').OR.<br>
> > ecco_phys.f: if (gencost_barfile(kgen)(1:15).EQ.'m_boxmean_theta') then<br>
> > <br>
> > (and I made sure that there?s this is really just m_boxmean_theta). Where in your code (which routine) does the model try to read adm_boxmean_theta?<br>
> > <br>
> > Martin<br>
> > > On 21. Sep 2020, at 11:00, Dan Jones <<a href="mailto:dcjones.work@gmail.com" target="_blank">dcjones.work@gmail.com</a>> wrote:<br>
> > > <br>
> > > Hello. <br>
> > > <br>
> > > Apologies for the cross-posting - I've posted this as a GitHub issue, but I thought I should put it here as well.<br>
> > > <br>
> > > I am trying to build and test the global_oce_cs32 verification_other exercise using the code in the input_ad.sens directory. The forward case compiles and runs without error. The adjoint case (built using TAF) compiles without error, but at runtime I receive the following error in STDOUT:<br>
> > > <br>
> > > (PID.TID 0000.0001) MDS_READVEC_LOC: open file: south30_maskT <br>
> > > (PID.TID 0000.0001) MDS_RD_REC_RL: iRec,Dim = 9 1<br>
> > > (PID.TID 0000.0001) MDS_READ_FIELD: filename: adm_boxmean_theta.0000000000.data<br>
> > > (PID.TID 0000.0001) MDS_READ_FIELD: File does not exist<br>
> > > <br>
> > > and this error in STDERR:<br>
> > > <br>
> > > (PID.TID 0000.0001) *** ERROR *** MDS_READ_FIELD: filename: adm_boxmean_theta.0000000000.data<br>
> > > (PID.TID 0000.0001) *** ERROR *** MDS_READ_FIELD: File does not exist<br>
> > > <br>
> > > My MITgcm source code is up-to-date with the master. I am running on <a href="http://archer.ac.uk" rel="noreferrer" target="_blank">archer.ac.uk</a> in parallel mode using 24 cores.<br>
> > > <br>
> > > What should I try here? I haven't run into this error before using other adjoint setups, at least not that I can recall. Should I just create an empty "dummy" file to start with? Thanks in advance for any help/guidance.<br>
> > > <br>
> > > Best regards,<br>
> > > Dan<br>
> > > <br>
> > > --------------------------------------------------------------<br>
> > > Dr Dan Jones / British Antarctic Survey<br>
> > > <a href="http://danjonesocean.com" rel="noreferrer" target="_blank">danjonesocean.com</a> / @DanJonesOcean<br>
> > > --------------------------------------------------------------<br>
> > > _______________________________________________<br>
> > > MITgcm-support mailing list<br>
> > > <a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
> > > <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> > <br>
> > <br>
> > <br>
> > ------------------------------<br>
> > <br>
> > Subject: Digest Footer<br>
> > <br>
> > _______________________________________________<br>
> > MITgcm-support mailing list<br>
> > <a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
> > <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> > <br>
> > <br>
> > ------------------------------<br>
> > <br>
> > End of MITgcm-support Digest, Vol 207, Issue 10<br>
> > ***********************************************<br>
> > _______________________________________________<br>
> > MITgcm-support mailing list<br>
> > <a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
> > <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> <br>
> <br>
> <br>
> ------------------------------<br>
> <br>
> Subject: Digest Footer<br>
> <br>
> _______________________________________________<br>
> MITgcm-support mailing list<br>
> <a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
> <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> <br>
> <br>
> ------------------------------<br>
> <br>
> End of MITgcm-support Digest, Vol 207, Issue 12<br>
> ***********************************************<br>
> _______________________________________________<br>
> MITgcm-support mailing list<br>
> <a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
> <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
<br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
MITgcm-support mailing list<br>
<a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
<a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
<br>
<br>
------------------------------<br>
<br>
End of MITgcm-support Digest, Vol 207, Issue 15<br>
***********************************************<br>
</blockquote></div></div></div></div>
</blockquote></div></div>