<div dir="ltr"><div>Hi Martin,<br><br>Yes, apologies, I had commented out parts of the cost function for my testing purposes. If I just run the unmodified case, I get:<br><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>global fc = 0.543536615356758E+08</div><div><br></div></blockquote>But this error still persists:<div><br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>MDS_READ_FIELD: filename: adm_boxmean_theta.0000000000.data</div></blockquote><div><br></div><div>If I create a blank adm file for it to read, then the computation continues but quickly runs into a different error. And in any case, adding a blank file before running shouldn't be necessary.</div><div><br></div><div>I ran the "testreport" on a couple of adjoint cases in the "verification" directory. Both "tutorial_global_oce_optim" and "tutorial_dic_adjoffline" passed. But when I try to run this set of commands:<br><br></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>cd verification_other</div></div><div><div> ../verification/testreport -t global_oce_cs32/ -adm -j 4 - optfile=../tools/build_options/linux_ia64_cray_archer</div></div></blockquote><div><div><div class="gmail_quote"><div dir="ltr" class="gmail_attr"><br></div><div class="gmail_attr">Then I get the following result:</div><div class="gmail_attr"><br></div></div></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><div class="gmail_quote"><div class="gmail_attr">Y Y Y N .. .. .. N/O global_oce_cs32/ (e=0, w=48)</div></div></div></div><div><div><div class="gmail_quote"><div class="gmail_attr">Y Y Y N .. .. .. N/O global_oce_cs32/.sens</div><div class="gmail_attr"><br></div></div></div></div></blockquote>Can you remind me what the last "N" stands for again? What has failed? This line of output seems suspicious:<div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div class="gmail_quote"><div class="gmail_attr"><br></div><div class="gmail_attr">../verification/testreport: line 845: 32072 Killed ./mitgcmuv_ad > output_adm.txt<br></div><div class="gmail_attr"><br></div></div></blockquote>That line in "testreport" has to do with DIVA, according to the comments. Unfortunately, "output_adm.txt" is blank. <br><br>Are you able to get the adjoint part of the "input_ad.sens" experiment to work, after the cost function has been calculated? <br><br>Best wishes,<br>Dan<br><div><div><div class="gmail_quote"><div class="gmail_attr"><br></div><div dir="ltr" class="gmail_attr">On Tue, Sep 22, 2020 at 5:17 PM <<a href="mailto:mitgcm-support-request@mitgcm.org">mitgcm-support-request@mitgcm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Send MITgcm-support mailing list submissions to<br>
<a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:mitgcm-support-request@mitgcm.org" target="_blank">mitgcm-support-request@mitgcm.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:mitgcm-support-owner@mitgcm.org" target="_blank">mitgcm-support-owner@mitgcm.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of MITgcm-support digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: verification_other case global_oce_cs32 fails at runtime<br>
in adjoint mode (Martin Losch)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 22 Sep 2020 17:40:44 +0200<br>
From: Martin Losch <<a href="mailto:Martin.Losch@awi.de" target="_blank">Martin.Losch@awi.de</a>><br>
To: MITgcm Support <<a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a>><br>
Subject: Re: [MITgcm-support] verification_other case global_oce_cs32<br>
fails at runtime in adjoint mode<br>
Message-ID: <<a href="mailto:5B5A14D1-A1DF-435B-B468-E0807AC4C437@awi.de" target="_blank">5B5A14D1-A1DF-435B-B468-E0807AC4C437@awi.de</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi Dan,<br>
<br>
I get very different output and results (e.g. global_fc= 0.543870?D+08).<br>
<br>
Can you try to run testreport on this?<br>
<br>
cd verification_other<br>
../verification/testreport -t global_oce_cs32 -adm $otheroptions<br>
where $otheroptions, are, e.g. -j 4, -devel, etc.. I you want to use your MPI job you?ll need '-MPI 24' and maybe "-command ?mpirun -n 24 ./mitgcmuv?" to run on 24 cpus. check the help section of testreport for details.<br>
<br>
The output should be ?fairly? independent of the number of CPUs. I use the default 24 tiles (but run sequentially on one cpu)<br>
<br>
Martin<br>
<br>
<br>
> On 22. Sep 2020, at 16:52, Dan Jones <<a href="mailto:dcjones.work@gmail.com" target="_blank">dcjones.work@gmail.com</a>> wrote:<br>
> <br>
> Hi Martin,<br>
> <br>
> Thanks for your quick reply! I can't get the serial case to run on ARCHER, unfortunately. I think for now I'm stuck testing in parallel. When I run "grep m_boxmean_theta *.f", I get exactly the same results as you. I'm also using MITgcm/verification_other from GitHub. <br>
> <br>
> Here is a bit more of the output which will hopefully help. This is from a case where I tried to use the "m_horflux_vol" case instead of the "m_boxmean_theta" case. I'm using data.ecco from the input_ad.sens directory. <br>
> <br>
> (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: xx_kapgm.0000000000.data<br>
> (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: xx_kapredi.0000000000.data<br>
> (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: xx_diffkr.0000000000.data<br>
> (PID.TID 0000.0001) --> f_gencost =-0.348173207824978E+08 2<br>
> (PID.TID 0000.0001) --> f_genarr3d = 0.000000000000000E+00 1<br>
> (PID.TID 0000.0001) --> f_genarr3d = 0.000000000000000E+00 2<br>
> (PID.TID 0000.0001) --> f_genarr3d = 0.000000000000000E+00 3<br>
> (PID.TID 0000.0001) --> fc =-0.348173207824978E+08<br>
> (PID.TID 0000.0001) early fc = 0.000000000000000E+00<br>
> (PID.TID 0000.0001) local fc = 0.000000000000000E+00<br>
> (PID.TID 0000.0001) global fc = -0.348173207824978E+08<br>
> (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: xx_diffkr.0000000000.data<br>
> (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: adxx_diffkr.0000000000.data<br>
> (PID.TID 0000.0001) MDS_WRITE_FIELD: it,rec,kS,kL,kH= 0 1 50 1 50 file=adxx_diffkr.0000000000<br>
> (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: xx_kapredi.0000000000.data<br>
> (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: adxx_kapredi.0000000000.data<br>
> (PID.TID 0000.0001) MDS_WRITE_FIELD: it,rec,kS,kL,kH= 0 1 50 1 50 file=adxx_kapredi.0000000000<br>
> (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: xx_kapgm.0000000000.data<br>
> (PID.TID 0000.0001) MDS_READ_FIELD: opening global file: adxx_kapgm.0000000000.data<br>
> (PID.TID 0000.0001) MDS_WRITE_FIELD: it,rec,kS,kL,kH= 0 1 50 1 50 file=adxx_kapgm.0000000000<br>
> (PID.TID 0000.0001) MDS_READ_FIELD: filename: adm_horflux_vol.0000000000.data<br>
> (PID.TID 0000.0001) MDS_READ_FIELD: File does not exist<br>
> <br>
> So it's after the cost function has been calculated, as the model is getting ready to perform the adjoint steps. It's able to read/write for the existing controls (kapgm, kapredi, diffkr). But it's apparently not creating an "ad" file for the general objective function term "horflux". That's why I was wondering if I should manually create a blank file first, as an ad-hoc fix. Any thoughts? <br>
> <br>
> Best wishes,<br>
> Dan<br>
> <br>
> On Mon, Sep 21, 2020 at 8:37 PM <<a href="mailto:mitgcm-support-request@mitgcm.org" target="_blank">mitgcm-support-request@mitgcm.org</a>> wrote:<br>
> Send MITgcm-support mailing list submissions to<br>
> <a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a><br>
> <br>
> To subscribe or unsubscribe via the World Wide Web, visit<br>
> <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> or, via email, send a message with subject or body 'help' to<br>
> <a href="mailto:mitgcm-support-request@mitgcm.org" target="_blank">mitgcm-support-request@mitgcm.org</a><br>
> <br>
> You can reach the person managing the list at<br>
> <a href="mailto:mitgcm-support-owner@mitgcm.org" target="_blank">mitgcm-support-owner@mitgcm.org</a><br>
> <br>
> When replying, please edit your Subject line so it is more specific<br>
> than "Re: Contents of MITgcm-support digest..."<br>
> <br>
> <br>
> Today's Topics:<br>
> <br>
> 1. verification_other case global_oce_cs32 fails at runtime in<br>
> adjoint mode (Dan Jones)<br>
> 2. Re: verification_other case global_oce_cs32 fails at runtime<br>
> in adjoint mode (Martin Losch)<br>
> <br>
> <br>
> ----------------------------------------------------------------------<br>
> <br>
> Message: 1<br>
> Date: Mon, 21 Sep 2020 10:00:47 +0100<br>
> From: Dan Jones <<a href="mailto:dcjones.work@gmail.com" target="_blank">dcjones.work@gmail.com</a>><br>
> To: <a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a><br>
> Subject: [MITgcm-support] verification_other case global_oce_cs32<br>
> fails at runtime in adjoint mode<br>
> Message-ID:<br>
> <CAPj3iHRxhUOCDT5m7H8uj8cg9dc=_<a href="mailto:oYVssQnvYhEA%2B_ALjeR6w@mail.gmail.com" target="_blank">oYVssQnvYhEA+_ALjeR6w@mail.gmail.com</a>><br>
> Content-Type: text/plain; charset="utf-8"<br>
> <br>
> Hello.<br>
> <br>
> Apologies for the cross-posting - I've posted this as a GitHub issue, but I<br>
> thought I should put it here as well.<br>
> <br>
> I am trying to build and test the global_oce_cs32 verification_other<br>
> exercise using the code in the input_ad.sens directory. The forward case<br>
> compiles and runs without error. The adjoint case (built using TAF)<br>
> compiles without error, but at runtime I receive the following error in<br>
> STDOUT:<br>
> <br>
> (PID.TID 0000.0001) MDS_READVEC_LOC: open file: south30_maskT<br>
> (PID.TID 0000.0001) MDS_RD_REC_RL: iRec,Dim = 9 1<br>
> (PID.TID 0000.0001) MDS_READ_FIELD: filename: adm_boxmean_theta.0000000000.data<br>
> (PID.TID 0000.0001) MDS_READ_FIELD: File does not exist<br>
> <br>
> and this error in STDERR:<br>
> <br>
> (PID.TID 0000.0001) *** ERROR *** MDS_READ_FIELD: filename:<br>
> adm_boxmean_theta.0000000000.data<br>
> (PID.TID 0000.0001) *** ERROR *** MDS_READ_FIELD: File does not exist<br>
> <br>
> My MITgcm source code is up-to-date with the master. I am running on<br>
> <a href="http://archer.ac.uk" rel="noreferrer" target="_blank">archer.ac.uk</a> <<a href="https://www.archer.ac.uk/" rel="noreferrer" target="_blank">https://www.archer.ac.uk/</a>> in parallel mode using 24 cores.<br>
> <br>
> What should I try here? I haven't run into this error before using other<br>
> adjoint setups, at least not that I can recall. Should I just create an<br>
> empty "dummy" file to start with? Thanks in advance for any help/guidance.<br>
> <br>
> Best regards,<br>
> Dan<br>
> <br>
> <br>
> --------------------------------------------------------------<br>
> Dr Dan Jones / British Antarctic Survey<br>
> <a href="http://danjonesocean.com" rel="noreferrer" target="_blank">danjonesocean.com</a> <<a href="http://www.danjonesocean.com" rel="noreferrer" target="_blank">http://www.danjonesocean.com</a>> / @DanJonesOcean<br>
> --------------------------------------------------------------<br>
> -------------- next part --------------<br>
> An HTML attachment was scrubbed...<br>
> URL: <<a href="http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20200921/ddb38a00/attachment-0001.html" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20200921/ddb38a00/attachment-0001.html</a>><br>
> <br>
> ------------------------------<br>
> <br>
> Message: 2<br>
> Date: Mon, 21 Sep 2020 14:56:23 +0200<br>
> From: Martin Losch <<a href="mailto:Martin.Losch@awi.de" target="_blank">Martin.Losch@awi.de</a>><br>
> To: MITgcm Support <<a href="mailto:mitgcm-support@mitgcm.org" target="_blank">mitgcm-support@mitgcm.org</a>><br>
> Subject: Re: [MITgcm-support] verification_other case global_oce_cs32<br>
> fails at runtime in adjoint mode<br>
> Message-ID: <<a href="mailto:FF2DD1AB-462E-4089-90CA-89B9552DF7D8@awi.de" target="_blank">FF2DD1AB-462E-4089-90CA-89B9552DF7D8@awi.de</a>><br>
> Content-Type: text/plain; charset="utf-8"<br>
> <br>
> Hi Dan,<br>
> <br>
> I tried this on my linux box without MPI and I cannot reproduce your problem (I used MITgcm/verification_other.git and not the CVS MITgcm_contrib/verification_other, which appears to be out of date). I grepped the code for ?m_boxmean_theta? and only found this:<br>
> <br>
> (base) bkli04l006::build (master)> grep m_boxmean_theta *.f<br>
> ad_input_code_ad.f: $'m_boxmean_theta') then<br>
> ad_input_code_ad.f: $'m_boxmean_theta') then<br>
> ad_input_code_ad.f: $'m_boxmean_theta') then<br>
> ad_input_code_ad.f: $'m_boxmean_theta') then<br>
> ad_input_code.f: if (gencost_barfile(kgen)(1:15).EQ.'m_boxmean_theta') then<br>
> ad_taf_output.f: $'m_boxmean_theta') then<br>
> ad_taf_output.f: $'m_boxmean_theta') then<br>
> ad_taf_output.f: $'m_boxmean_theta') then<br>
> ad_taf_output.f: $'m_boxmean_theta') then<br>
> ecco_check.f: & (gencost_barfile(k)(1:15).EQ.'m_boxmean_theta').OR.<br>
> ecco_phys.f: if (gencost_barfile(kgen)(1:15).EQ.'m_boxmean_theta') then<br>
> <br>
> (and I made sure that there?s this is really just m_boxmean_theta). Where in your code (which routine) does the model try to read adm_boxmean_theta?<br>
> <br>
> Martin<br>
> > On 21. Sep 2020, at 11:00, Dan Jones <<a href="mailto:dcjones.work@gmail.com" target="_blank">dcjones.work@gmail.com</a>> wrote:<br>
> > <br>
> > Hello. <br>
> > <br>
> > Apologies for the cross-posting - I've posted this as a GitHub issue, but I thought I should put it here as well.<br>
> > <br>
> > I am trying to build and test the global_oce_cs32 verification_other exercise using the code in the input_ad.sens directory. The forward case compiles and runs without error. The adjoint case (built using TAF) compiles without error, but at runtime I receive the following error in STDOUT:<br>
> > <br>
> > (PID.TID 0000.0001) MDS_READVEC_LOC: open file: south30_maskT <br>
> > (PID.TID 0000.0001) MDS_RD_REC_RL: iRec,Dim = 9 1<br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: filename: adm_boxmean_theta.0000000000.data<br>
> > (PID.TID 0000.0001) MDS_READ_FIELD: File does not exist<br>
> > <br>
> > and this error in STDERR:<br>
> > <br>
> > (PID.TID 0000.0001) *** ERROR *** MDS_READ_FIELD: filename: adm_boxmean_theta.0000000000.data<br>
> > (PID.TID 0000.0001) *** ERROR *** MDS_READ_FIELD: File does not exist<br>
> > <br>
> > My MITgcm source code is up-to-date with the master. I am running on <a href="http://archer.ac.uk" rel="noreferrer" target="_blank">archer.ac.uk</a> in parallel mode using 24 cores.<br>
> > <br>
> > What should I try here? I haven't run into this error before using other adjoint setups, at least not that I can recall. Should I just create an empty "dummy" file to start with? Thanks in advance for any help/guidance.<br>
> > <br>
> > Best regards,<br>
> > Dan<br>
> > <br>
> > --------------------------------------------------------------<br>
> > Dr Dan Jones / British Antarctic Survey<br>
> > <a href="http://danjonesocean.com" rel="noreferrer" target="_blank">danjonesocean.com</a> / @DanJonesOcean<br>
> > --------------------------------------------------------------<br>
> > _______________________________________________<br>
> > MITgcm-support mailing list<br>
> > <a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
> > <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> <br>
> <br>
> <br>
> ------------------------------<br>
> <br>
> Subject: Digest Footer<br>
> <br>
> _______________________________________________<br>
> MITgcm-support mailing list<br>
> <a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
> <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> <br>
> <br>
> ------------------------------<br>
> <br>
> End of MITgcm-support Digest, Vol 207, Issue 10<br>
> ***********************************************<br>
> _______________________________________________<br>
> MITgcm-support mailing list<br>
> <a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
> <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
<br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
MITgcm-support mailing list<br>
<a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
<a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
<br>
<br>
------------------------------<br>
<br>
End of MITgcm-support Digest, Vol 207, Issue 12<br>
***********************************************<br>
</blockquote></div></div></div></div></div>