[MITgcm-devel] Makefile for adjoint

[Jean-Michel Campin]

Hi Martin,

I am forwarding this to "ecco-devel at mit.edu" in case someone there
wants to make some comments.
I don't have time to look at this right now (among other, fixing some 
broken experiment ?) but will try sometime next week.

Cheers,
Jean-Michel

On Wed, Dec 10, 2014 at 10:01:10AM +0100, Martin Losch wrote:
> Hi again,
> 
> after the overwhelming replies to this post (-: I checked in my genmake2 in /u/gcmpack/MITgcm_contrib/mlosch/tools (for those of you, who have mail-filter that diverts cvs-messages directly into the trash), with some explanation of what I did. Please give this genmake2 a try. In a simple test using this genmake2 speeds up compiling (from scatch) with “make -j8 adall” on my 4core linux box moderately from 4m18sec to 2m44sec (but I have to admit that my proximity to TAF (it’s really much faster within Germany/Europe) makes the wait-time shorter anyway). There are portability issues, which I would like to solve, please have look at the README (also attached here) and let me know if you have any ideas about the issues.
> 
> Martin
> 
> Content of README:
> This version of genmake2 handles the new type of output of TAF v2,
> i.e. the fact that the forward code that TAF thinks relevant is
> returned together with the derivative code, in a different way than
> the original genmake2: Instead of concatenating all fortran sources
> and sending one monolithic ad_input_code.f to TAF, the sources are
> sent to TAF directly: staf [options] $(FLOWFILES) $(AD_FILES). This
> makes it faster to compile the returned sources because you can use
> the "parallel" capabilities of (g)make (with option -j n); also you
> can modify individual sources (that have been returned by TAF) without
> having to recompile a large ad_taf_output.f making debugging a little
> easier.
> 
> This version of genmake2 works with both testreport -adm and testreport -tlm. 
> 
> genmake2_patch.diff patches the "official" genmake2 until hopefully
> this version makes into the main repository.  Usage:
> patch -p1 your_genmake2 genmake2_patch.diff
> 
> In order to make this work I had to modify the target ad_taf_output.f
> (and similarly ftl_taf_output.f) and introduce new targets:
> 
> - remove_comments applies \$(TOOLSDIR)/remove_comments_sed to all
> $(AD_FILES) will remove the comments, this may be annoying, if you are
> used to working with the *.f files and need to comments for
> orientation within the file, but I think that the *.f files can be
> generated so easily, that this does not matter much. This target is
> really only needed if you have a slow uplink and want to sent smaller
> files to staf
> 
> - adj_sed and ftl_sed apply $(TOOLSDIR)/adjoint_sed to
>  $(AD_FILES:.$FS=_ad/tl.$FS) 
> 
> - adjobjfiles and ftlobjfiles force compiling the taf-generated files
>  after the $(TAF) command
> 
> The TAMC related target are not touched, but I did not test the result
> 
> Issues:
> >>>> The Makefile still generates ad_taf_output.f and ftl_taf_output.f to
> be used for svd_taf (although I am not sure, if this will work,
> because I don't know how it is supposed to work). <<<<
> 
> >>>> The targets adonlyfwd, adtrick will probably not work, but do we
> still need them? <<<<
> 
> >>>> The $(EXE_AD/FTL) targets now depends on ad/ftl_taf_output.$FS
> (not .o anymore), which is generated after making adjobjfiles. This is
> a bit of a hack and potentially fragile, but I couldn't think of
> anything else to trick "make" into compiling in two separate
> steps. <<<<
> 
> >>>> Another weakness is the "inplace" use of sed, (sed -i), which
> will not work with some sed implementations (POSIX?). Alternatively
> one could modify the .F.$FS and .$FS.o rules like this:
> 
> .F.$FS:
> 	sed -f \$(TOOLSDIR)/remove_comments_sed \$< > \$<.tmp && mv \$<.tmp \$<
> 	\$(CPPCMD) > \$@
> .$FS.o:
> 	sed -f \$(TOOLSDIR)/adjoint_sed \$< > \$<.tmp && mv \$<.tmp \$<
> 	$(FC) $(FFLAGS) $(FOPTIM) -c $<
> 
> but these would then be applied to all files and not only
> the $(AD_FILES). Any suggestion how to improve this is very welcome. <<<<
> 
> 
> On Dec 8, 2014, at 3:51 PM, Martin Losch <Martin.Losch at awi.de> wrote:
> 
> > Hi there,
> > 
> > since TAF v2 the forward code is replicated in ad_taf_output.f, which makes this file very large and very slow to compile. Not so much of a problem if you just want to have something that runs, but annoying if you want to debug the adjoint model (as I am doing unsuccessfully). It appears to be pretty straightforward to skip the cat-everything-into-ad_input_code.f and call TAF with taf -options $(FLOWFILES) $(AD_FILES) instead. As a result you get back $(AD_FILES:.f=_ad.f) that contain the adjoint code and the forward code of each routine in $(AD_FILES), that you can compiler much faster (with make -j adall or similar), also I can make changes in individual _ad.f and don’t have to recompile a huge file for minutes. This procedure requires only a few changes in genmake2, and I can debug the code much faster.
> > 
> > The problem starts with the dependencies within the makefile, i.e. because there are no longer any monolithic files like ad_input_code.f and ad_taf_output.f, I find it hard to tell “make” when taf should be called again etc. With this I probably would need some help.
> > 
> > Q1: Is there any general interest in changing the way taf is called and in having faster compiles? If not, I’ll just keep my genmake2 version and fiddle with it for debugging purposes. If yes, then
> > Q2: would anyone be willing to help me with the makefile/genmake? I would check in a “genmake3” (or maybe heretically a “genmake”) to work on, possibly in the contrib-area.
> > 
> > Martin
> > 
> > 
> > _______________________________________________
> > MITgcm-devel mailing list
> > MITgcm-devel at mitgcm.org
> > http://mitgcm.org/mailman/listinfo/mitgcm-devel
> 
> 
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel