[MITgcm-devel] restoring tapenade-tangent linear for STREAMICE: proposed solution

Tue Dec 23 15:05:26 EST 2025

Ok JM thanks for this. That means that for almost 2 months the experiment was running on Baudelaire with the 29 July tapenade version, which I think is what I have (and I will find out, I hadn't realised the trace gives the tapenade date, that is useful). So I will try testreport with 69i soon as I can..

Best
Dan

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Jean-Michel Campin <jmc at mit.edu>
Sent: Tuesday, December 23, 2025 4:00:42 PM
To: Daniel Goldberg <Dan.Goldberg at ed.ac.uk>
Cc: mitgcm-devel at mitgcm.org <mitgcm-devel at mitgcm.org>
Subject: Re: [MITgcm-devel] restoring tapenade-tangent linear for STREAMICE: proposed solution

Hi Dan,

The way I got Tapenade was to follow the instruction from MITgcm manual, section 7.6.5.,
so I don't specify a specific commit.
The tar file is always tapenade_3_16.tar but when I use it and the version change
then it reports a different time, so I believe that:
> > TangLin generated by Tapenade 3.16 (develop) - 29 Jul 2025
does identify a particular tapenade version.

And regarding the daily tests on baudelaire, it started on Aug 18, 2025, using tar file
version from - 21 Mar 2023, and run this way until Sep 29, 2025.
After that, starting on Sep 30, 2025, I switch to a newer version and I am still using
the same version which is from - 29 Jul 2025.

Cheers,
Jean-Michel

On Tue, Dec 23, 2025 at 02:29:09PM +0000, Daniel Goldberg wrote:
> Hi Jean Michel
>
> I believe that the develop version is subject to change and the only way to be sure is to use a specific commit. I recall that you were checking PR 927 you needed to update your version of tapenade on Baudelaire, even though (I think) it remained at 3.16.. but I could be wrong.
>
> Nevertheless I will try testreport with the command below when I get a chance, with checkpoint 39i, and the tapenade version I used to test PR 927, and let you know what I see..
>
> Thanks
> Dan
>
>
>
> Get Outlook for iOS<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637531926%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GfKvfcxWneI4j1imx1%2Fep%2FsKn7XjQa462bVuTPAWQvY%3D&reserved=0<https://aka.ms/o0ukef>>
> ________________________________
> From: Jean-Michel Campin <jmc at mit.edu>
> Sent: Tuesday, December 23, 2025 2:02:27 PM
> To: Daniel Goldberg <Dan.Goldberg at ed.ac.uk>
> Cc: jmc at mit.edu <jmc at mit.edu>
> Subject: Re: [MITgcm-devel] restoring tapenade-tangent linear for STREAMICE: proposed solution
>
> Hi Dan,
>
> The tapenade version is written in the summary.txt, and has not changed for these daily tests
> (because, for practical reasons, I am not currently updating or downloading the latest one):
> > TangLin generated by Tapenade 3.16 (develop) - 29 Jul 2025
>
> Cheers,
> Jean-Michel
>
> On Tue, Dec 23, 2025 at 06:09:52AM +0000, Daniel Goldberg wrote:
> > Hi j-m thanks. This is very strange, I could not reproduce this with a previous commit. I wonder if it has to do with the tapenade version.
> >
> > Did you just now run the test below with 69i, or are you looking at the logs? If this is in the logs, I?m just wondering if you are able to reproduce it now. I will try with 69i when I get a chance.
> >
> > The command, just to be sure, is
> > ./testreport -t halfpipe_streamice -tap -tlm
> > Correct?
> >
> > Thanks!
> > Dan
> >
> > Get Outlook for iOS<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637566870%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=teuLAi9M50%2BVQ%2F4DBoPB%2BEt7vTy82I4RfYFypBL1sTw%3D&reserved=0<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637596795%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=rpbiM%2Bgu4hXLDNOjNhYb8yJtEACbTwFjLG6IERyH%2BTg%3D&reserved=0<https://aka.ms/o0ukef>>>
> > ________________________________
> > From: Jean-Michel Campin <jmc at mit.edu>
> > Sent: Monday, December 22, 2025 5:22 PM
> > To: Daniel Goldberg <Dan.Goldberg at ed.ac.uk>
> > Subject: Re: Fw: [MITgcm-devel] restoring tapenade-tangent linear for STREAMICE: proposed solution
> >
> > Hi Dan,
> >
> > Before PR #927 got merged (on Nov 21), "halfpipe_streamice" TLM test with Tapenade was
> > run every day on baudelaire, see the last one from Nov 21:
> > https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmitgcm.org%2Ftesting%2Fresults%2F2025_11%2Ftr_baudelaire_20251121_6%2Fsummary.txt&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637624039%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=VNEWFck%2FmCrUr%2FK%2BlQb9faL6sHaaBlsy8q4JrwNFNeg%3D&reserved=0<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmitgcm.org%2Ftesting%2Fresults%2F2025_11%2Ftr_baudelaire_20251121_6%2Fsummary.txt&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637647699%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=AQhY539kHvGjuXFJrdDsj195g2JLZO8Jj9AACljQgYY%3D&reserved=0><https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmitgcm.org%2Ftesting%2Fresults%2F2025_11%2Ftr_baudelaire_20251121_6%2Fsummary.txt&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637671455%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=vSogakJKSkw4jHRazdnTRD9bUXrvFmpS0pIR03Q9u1I%3D&reserved=0><http://mitgcm.org/testing/results/2025_11/tr_baudelaire_20251121_6/summary.txt>
> >
> > To see what the grdchk looks like, I can go back do the last tag before Nov 21,
> > which was checkpoint69i, done on Nov 14 (and no changes related to halfpipe_streamice
> > between c69i and Nov 21):
> >  > git checkout checkpoint69i
> > And what I get from the reference output (in halfpipe_streamice/results/) is:
> > Tap-TL: grdchk  summary  :  RMS of    4 ratios =  7.1444934467587E-06
> > Tap-AD: grdchk  summary  :  RMS of    4 ratios =  9.3393363837944E-06
> > TAF-AD: grdchk  summary  :  RMS of    4 ratios =  1.2875200645822E-05
> > OAD-AD: grdchk  summary  :  RMS of    4 ratios =  7.3161247348914E-06
> >
> > So the agreement was good for Tapenade TL.
> >
> > Now on the current master branch, the ref. "output_tap_tlm.txt" has been removed but
> > from PR #927 discussion, it was reported that the grdchk RMS was 4.7, so much larger.
> >
> > Cheers,
> > Jean-Michel
> >
> > On Tue, Dec 16, 2025 at 05:12:27PM +0000, Daniel Goldberg wrote:
> > > hi J-M ? I am working on this right now because like you i was not happy about losing the TLM for streamice either.
> > >
> > > i am in the middle of trying several things including what i suggest below, but i am still not getting good answers.
> > >
> > > However, when i check out an old commit, i am still not able to see an accurate TLM. Im not sure if I ever did, though i cannot recall. You said that before my PR for tapenade/streamice was merged, you were able to see an accurate TLM test (i.e. the grad-res errors were small). Do you remember which commit?
> > >
> > > Also are you still able to see accurate values with an old commit? Im worried it might be a change in the tapenade version!
> > >
> > > NB Im currently using 3a00e4dea944ee4a8c4a579882340f83925af1f0.
> > >
> > > Many thanks
> > > Dan
> > >
> > > ________________________________
> > > From: Daniel Goldberg <Dan.Goldberg at ed.ac.uk>
> > > Sent: Monday, December 15, 2025 09:51
> > > To: MITgcm-devel at mitgcm.org <mitgcm-devel at mitgcm.org>
> > > Subject: Re: [MITgcm-devel] restoring tapenade-tangent linear for STREAMICE: proposed solution
> > >
> > > thanks martin, i think this is a reasonable idea then.
> > >
> > > from what i have seen, if tapenade is passed a subroutine, it does not care if there is a flow file entry or not. one difference is that the flow file for tapenade does not tell the tool which manual subroutines to use. one has to make sure they are defined somewhere (e.g. stubs_tap_adj.F).
> > >
> > > thanks
> > > dan
> > > ________________________________
> > > From: MITgcm-devel <mitgcm-devel-bounces at mitgcm.org> on behalf of Martin Losch <Martin.Losch at awi.de>
> > > Sent: Monday, December 15, 2025 13:54
> > > To: MITgcm-devel <MITgcm-devel at mitgcm.org>
> > > Subject: Re: [MITgcm-devel] restoring tapenade-tangent linear for STREAMICE: proposed solution
> > >
> > > Hi Dan,
> > >
> > > there is a similar problem for verification/bottom_ctrl_5x5 where we test the manual AD routine of cg2d (cg2d_mad.F), which does not work for the TLM (there is none) and hence the TLM gradients are totally wrong. An optional model_tlm_diff.list could be useful in this context, too. I am not sure how TAF handles that because when flow directives describing the interfact are given, then usually TAF ignores the corresponding files.
> > >
> > > Martin
> > >
> > > On 15. Dec 2025, at 13:46, Daniel Goldberg <Dan.Goldberg at ed.ac.uk> wrote:
> > >
> > > Dear J-M and all:
> > >
> > > Upon the merging of 927<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMITgcm%2FMITgcm%2Fpull%2F927&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637693947%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=vqmbQtS04WBuXmq%2FH79eC1zn%2Fu2fBo6jQaTyOe7jOXQ%3D&reserved=0<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMITgcm%2FMITgcm%2Fpull%2F927&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637717076%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=zJjf%2Fe25N5eFycWFb8LO6OScTOoF7ZGzBl6Av6jHN2Q%3D&reserved=0<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMITgcm%2FMITgcm%2Fpull%2F927&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637742029%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=W9%2BIyiUfL0S3hvGHW9tvXVgafp4LoOFVtVeRhhJShuQ%3D&reserved=0<https://github.com/MITgcm/MITgcm/pull/927>>>>(which was a good thing!), the tangent linear test of STREAMICE under Tapenade no longer worked, whereas it seemed to be giving accurate values. I am writing this email to run a proposed solution by the team, rather than submitting a PR unannounced, to get a feel for whether it would be welcomed, and looked at.
> > >
> > > The reason is that streamice_cg_solve.f is no longer passed to Tapenade, instead using a "manual" approach as with cg2d.f. This is necessary for large problems where the ad-generated adjoint code would be very slow and/or memory intensive and would preclude external library solvers. These issues would not apply as much to the TLM ? but no "manual" solution exists.
> > >
> > > My idea for a solution, is to pass streamice_cg_solve.f through tapenade only when the TLM is compiled. I propose to do this by modifying genmake2 to look for "supplemental" files *_tlm_diff.list (in addition to *_ad_diff.list). A list of S/Rs called $(TLM_FILES) (or something) will be created and passed as an argument to tapenade, only for the TLM call. The *_tlm_diff.list files (e.g. streamice_tlm_diff.list) files will be optional, and only need contain files that are passed to the TLM but not adjoint. I think this is justified as in general the set of functions that the TLM can parse easily is larger than that of the adjoint.
> > >
> > > All the best
> > > Dan
> > >
> > >
> > >
> > > --
> > > Daniel Goldberg, PhD (he/him)
> > > Reader in Glaciology
> > > School of Geosciences, University of Edinburgh
> > > Geography Building, Drummond Street, Edinburgh EH8 9XP
> > > https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdngoldberg.github.io%2F&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637767305%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=TIDAFz2WhZ%2BrMbZ04JHd4FR5Yj84JDeTWcAYQqbv9FE%3D&reserved=0<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdngoldberg.github.io%2F&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637790354%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=idE%2Ftx6RejBMsHRl7FliGlEwRqiCxHs4lI6RvlalJgU%3D&reserved=0><https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdngoldberg.github.io%2F&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637809540%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=E2DSp1oMQ%2B%2B65WPkPpO071xgU6H006T6eYW19%2FCvMxQ%3D&reserved=0><https://dngoldberg.github.io/>
> > > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th? ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336. _______________________________________________
> > > MITgcm-devel mailing list
> > > MITgcm-devel at mitgcm.org<mailto:MITgcm-devel at mitgcm.org>
> > > https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.mitgcm.org%2Fmailman%2Flistinfo%2Fmitgcm-devel&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637827504%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6n9iQv0JbaQI1KfM8AUK1IZlDq9ALg6JQelMawutWMA%3D&reserved=0<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.mitgcm.org%2Fmailman%2Flistinfo%2Fmitgcm-devel&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637845837%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=4hgl02iwnY95oB098cK3Zj5IcySQF%2BT12O1DMXuUXak%3D&reserved=0><https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmailman.mitgcm.org%2Fmailman%2Flistinfo%2Fmitgcm-devel&data=05%7C02%7C%7C95ace8eeeb0a4e8e136008de423c7704%7C2e9f06b016694589878910a06934dc61%7C0%7C0%7C639021024637865355%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=gtZ81OW7u5zGcygRJr%2B0uwav3ISgxb0oK7dvPzGf7z8%3D&reserved=0><http://mailman.mitgcm.org/mailman/listinfo/mitgcm-devel>
> > >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.mitgcm.org/pipermail/mitgcm-devel/attachments/20251223/918f3a53/attachment-0001.html>