[MITgcm-devel] multiple ptracers with divided adjoint
Jean-Michel Campin
jmc at ocean.mit.edu
Sat Nov 24 10:46:09 EST 2012
Hi Matt,
I think Greg came in to allow your 1rst email (from gmail) to go through.
This explains the delay you saw.
Cheers,
Jean-Michel
On Sat, Nov 24, 2012 at 07:34:18AM -0800, Matthew Mazloff wrote:
> Hi Patrick,
>
> Very sorry -- I sent this first from gmail and wasn't sure it went through. So I sent it again from ucsd.edu. The gmail one did go through, but with a delay. Hence the delayed second email…lessoned learned
>
> Thank you very much for the explanation and suggestion, I will give this a try
>
> Thanks!
> Matt
>
>
>
>
>
> On Nov 24, 2012, at 6:02 AM, Patrick Heimbach <heimbach at MIT.EDU> wrote:
>
> >
> > Hi Matt,
> >
> > not sure if you saw my previous email, so again:
> >
> > I haven't found the time to deal with the -v1 vs. -v2 issue.
> > It needs changes mainly in genmake2.
> > I had spoken to Ralf, he had send fixes based on a very old version of genmake (not genmake2)
> > that needed work - which I haven't yet looked into.
> > v2 inlines original code into TAF-generated code as a way to deal with modules.
> >
> > Again, as a quick fix to your problem, you could try (and it should work)
> > either to add to your staf option variable AD_TAF_FLAGS the flag '-f90'
> > or just use genmake2 flag '-adof $ROOTDIR/tools/adjoint_options/adjoint_f95'
> >
> > (TAF 2.3.1 onward becomes more strict regarding f77 versus f90 statements;
> > your directives are not f77 compatible, that's why TAF 2.3.1 chokes).
> >
> > p.
> >
> > On Nov 23, 2012, at 11:28 PM, Matthew Mazloff <mmazloff at gmail.com> wrote:
> >
> >> Hi Patrick,
> >>
> >> So I need to run with multiple tracers again (for DIC adjoint). That we are using -V1 means this fix is now unfixed. This is not a big deal as I can hardcode it, but made we wonder what the status was with TAF. Is anyone working with Ralf to remedy this V1 and V2 issue? What is that status? (I did attempt to compile without -v1 but had no success.)
> >>
> >> Thanks
> >> Matt
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Apr 12, 2011, at 3:03 PM, Matthew Mazloff <mmazloff at ucsd.EDU> wrote:
> >>
> >>> Hello,
> >>>
> >>> This thread has been resolved. Ralf installed a new TAF version (2.1.6) that generates correct calls to adwrite/adread in case of subarrays.
> >>>
> >>> I tested
> >>>
> >>> CADJ STORE ptracer(:,:,:,:,:,1),
> >>> CADJ & ptracer(:,:,:,:,:,2)
> >>> CADJ & = tapelev3, key = ilev_3
> >>>
> >>> and
> >>>
> >>> CADJ STORE ptracer(:,:,:,:,:,:)
> >>> CADJ & = tapelev3, key = ilev_3
> >>>
> >>> and also
> >>>
> >>> CADJ STORE ptracer(:,:,:,:,:,1:2),
> >>> CADJ & ptracer(:,:,:,:,:,3:4)
> >>> CADJ & = tapelev3, key = ilev_3
> >>>
> >>>
> >>> and all looks great.
> >>>
> >>>
> >>> Being able to store subarrays lets us run with many many tracers at once and not worry about I/O issues.
> >>>
> >>>
> >>> Thanks for the help,
> >>> Matt
> >>>
> >>>
> >>>
> >>>
> >>>>
> >>>> On Apr 5, 2011, at 3:47 PM, Matthew Mazloff wrote:
> >>>>
> >>>>> Hi Patrick,
> >>>>>
> >>>>> This still is not working.
> >>>>>
> >>>>> The issue is, for example,
> >>>>>
> >>>>> CADJ STORE ptracer(:,:,:,:,:,1),
> >>>>> CADJ & ptracer(:,:,:,:,:,2)
> >>>>> CADJ & = tapelev3, key = ilev_3
> >>>>>
> >>>>>
> >>>>> becomes
> >>>>>
> >>>>> call adwrite( mythid,
> >>>>> & 'tapelev3_18_the_main_loop_ptracer',33,26,18,
> >>>>> $ptracer,8,(1+snx+olx-(1-olx))*(1+sny+oly-(1-oly))*nr*nsx*nsy*
> >>>>> $ptracers_num,ilev_3 )
> >>>>> call adwrite( mythid,
> >>>>> & 'tapelev3_19_the_main_loop_ptracer',33,26,19,
> >>>>> $ptracer,8,(1+snx+olx-(1-olx))*(1+sny+oly-(1-oly))*nr*nsx*nsy*
> >>>>> $ptracers_num,ilev_3 )
> >>>>>
> >>>>> and not the desired:
> >>>>>
> >>>>> call adwrite( mythid,
> >>>>> & 'tapelev3_18_the_main_loop_ptracer',33,26,18,
> >>>>> $ptracer(:,:,:,:,:,1),8,(1+snx+olx-(1-olx))*(1+sny+oly-(1-oly))*nr*nsx*nsy*
> >>>>> $1,ilev_3 )
> >>>>> call adwrite( mythid,
> >>>>> & 'tapelev3_19_the_main_loop_ptracer',33,26,19,
> >>>>> $ptracer(:,:,:,:,:,2),8,(1+snx+olx-(1-olx))*(1+sny+oly-(1-oly))*nr*nsx*nsy*
> >>>>> $1,ilev_3 )
> >>>>>
> >>>>> so basically its trying to store the whole field ptracers_num times instead of part of it.
> >>>>>
> >>>>> Let me know if you have any advice...perhaps I missed something
> >>>>>
> >>>>> Thanks
> >>>>> Matt
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mar 24, 2011, at 5:23 AM, Patrick Heimbach wrote:
> >>>>>
> >>>>>>
> >>>>>> Hi Matt,
> >>>>>>
> >>>>>> I've updated the ptracers_ad_check_lev?_dir.h
> >>>>>> with working syntax (suggested by Ralf).
> >>>>>> Please update and set
> >>>>>> #define AUTODIFF_PTRACERS_SPLIT_FILES
> >>>>>> in ECCO_CPPOPTIONS.h
> >>>>>> I've tested it in tutorial_tracer_adjsens
> >>>>>>
> >>>>>> I'm also appending what Ralf wrote back.
> >>>>>>
> >>>>>> Cheers
> >>>>>> -Patrick
> >>>>>>
> >>>>>> ---
> >>>>>>
> >>>>>> Hi Patrick,
> >>>>>>
> >>>>>> The assumption is that a store-directive for a variable provides all
> >>>>>> parts of that variable which are required.
> >>>>>> So several store-directives for different parts won't work.
> >>>>>> A solution is to store different parts of that array in one directive:
> >>>>>>
> >>>>>> CADJ STORE ptracer(:,:,:,:,:,1),ptracer(:,:,:,:,:,2),ptracer(:,:,:,:,:,3)
> >>>>>> CADJ & = tapelev2_ptr, key = ilev_2
> >>>>>>
> >>>>>> or
> >>>>>>
> >>>>>> CADJ STORE ptracer(:,:,:,:,:,1:5),ptracer(:,:,:,:,:,6:nTracer)
> >>>>>> CADJ & = tapelev2_ptr, key = ilev_2
> >>>>>>
> >>>>>> Another solution would be to modify ADOPEN/ADREAD/ADWRITE/ADCLOSE to write several
> >>>>>> files in case of to large files.
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Ralf
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mar 18, 2011, at 9:00 PM, Matthew Mazloff wrote:
> >>>>>>
> >>>>>>> Hi Patrick,
> >>>>>>>
> >>>>>>> I tried this (and several other similar things). Taf just keeps dropping the store commands...they will be in ad_taf_input.f but gone from d_taf_input_ad.f and ad_taf_output.f
> >>>>>>>
> >>>>>>> Not sure why it won't store it....if you have any ideas please let me know
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>> Matt
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mar 14, 2011, at 10:19 PM, Patrick Heimbach wrote:
> >>>>>>>
> >>>>>>>> Hi Matt,
> >>>>>>>>
> >>>>>>>> ok.
> >>>>>>>> It ought to be possible to remedy this
> >>>>>>>> by having a loop over all tracers in
> >>>>>>>> ptracers_ad_check_levN_dir.h ( N> 1 )
> >>>>>>>> something like
> >>>>>>>>
> >>>>>>>> DO iTracer = 1, PTRACERS_num
> >>>>>>>> CADJ STORE ptracer(:,:,:,:,:,iTracer)
> >>>>>>>> CADJ & = tapelev2, key = ilev_2
> >>>>>>>> etc...
> >>>>>>>> ENDDO
> >>>>>>>>
> >>>>>>>> That should generate a separate file for each tracer.
> >>>>>>>>
> >>>>>>>> -p.
> >>>>>>>>
> >>>>>>>> On Mar 14, 2011, at 10:57 PM, Matthew Mazloff wrote:
> >>>>>>>>
> >>>>>>>>> Hi Patrick,
> >>>>>>>>>
> >>>>>>>>> Just to follow up on this issue -- the problem has to do with the size of the tapelevel files -- so its machine limitations, not the code.
> >>>>>>>>>
> >>>>>>>>> For my setup I can run with up to PTRACERS_num = 6 -- then at PTRACERS_num = 7 the outer tape level files get too large and the model poofs.
> >>>>>>>>>
> >>>>>>>>> -Matt
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Mar 7, 2011, at 7:41 PM, Matthew Mazloff wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Patrick,
> >>>>>>>>>>
> >>>>>>>>>> Well it poofed in the outer-most tapelevels for PTRACERS_num = 11.
> >>>>>>>>>> But actually, it poofed somewhere else for PTRACERS_num = 2.
> >>>>>>>>>>
> >>>>>>>>>> I am going to re-run the PTRACERS_num = 2 case to make sure I know
> >>>>>>>>>> exactly where it poofed
> >>>>>>>>>>
> >>>>>>>>>> I haven't tried it non-DIVA
> >>>>>>>>>>
> >>>>>>>>>> Thanks!
> >>>>>>>>>> -Matt
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Mar 7, 2011, at 5:17 PM, Patrick Heimbach wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Matt,
> >>>>>>>>>>> does it also poof in same setup, but non-DIVA?
> >>>>>>>>>>> Also, does it crash writing tapes or the DIVA snapshots?
> >>>>>>>>>>> Also, does it crash writing outer-most tapelevels or intermediate?
> >>>>>>>>>>> -p.
> >>>>>>>>>>>
> >>>>>>>>>>> On Mar 7, 2011, at 7:29 PM, Matthew Mazloff wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hello,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I tried to run the DIVA with multiple tracers but it crashes while
> >>>>>>>>>>>> writing the tapelevels. With PTRACERS_num = 1 everything is fine
> >>>>>>>>>>>> --- I switch to PTRACERS_num = 2 and poof.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'll look into whats going on later -- but wanted to check if
> >>>>>>>>>>>> anyone was already aware of this problem.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks
> >>>>>>>>>>>> Matt
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>> MITgcm-devel mailing list
> >>>>>>>>>>>> MITgcm-devel at mitgcm.org
> >>>>>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>>>>>>>>>
> >>>>>>>>>>> ---
> >>>>>>>>>>> Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
> >>>>>>>>>>> MIT | EAPS 54-1518 | 77 Massachusetts Ave | Cambridge MA 02139 USA
> >>>>>>>>>>> FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> _______________________________________________
> >>>>>>>>>>> MITgcm-devel mailing list
> >>>>>>>>>>> MITgcm-devel at mitgcm.org
> >>>>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> MITgcm-devel mailing list
> >>>>>>>>>> MITgcm-devel at mitgcm.org
> >>>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> _______________________________________________
> >>>>>>>>> MITgcm-devel mailing list
> >>>>>>>>> MITgcm-devel at mitgcm.org
> >>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>> Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
> >>>>>>>> MIT | EAPS 54-1518 | 77 Massachusetts Ave | Cambridge MA 02139 USA
> >>>>>>>> FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> MITgcm-devel mailing list
> >>>>>>>> MITgcm-devel at mitgcm.org
> >>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>>>>>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> MITgcm-devel mailing list
> >>>>>>> MITgcm-devel at mitgcm.org
> >>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>>>>
> >>>>>> ---
> >>>>>> Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
> >>>>>> MIT | EAPS 54-1518 | 77 Massachusetts Ave | Cambridge MA 02139 USA
> >>>>>> FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> MITgcm-devel mailing list
> >>>>>> MITgcm-devel at mitgcm.org
> >>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> MITgcm-devel mailing list
> >>>>> MITgcm-devel at mitgcm.org
> >>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>>
> >>>> ---
> >>>> Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
> >>>> MIT | EAPS 54-1518 | 77 Massachusetts Ave | Cambridge MA 02139 USA
> >>>> FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach
> >>>>
> >>>>
> >>>
> >>> _______________________________________________
> >>> MITgcm-devel mailing list
> >>> MITgcm-devel at mitgcm.org
> >>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>
> >> _______________________________________________
> >> MITgcm-devel mailing list
> >> MITgcm-devel at mitgcm.org
> >> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >
> >
> > ---
> > Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
> > MIT | EAPS 54-1420 | 77 Massachusetts Ave | Cambridge MA 02139 USA
> > FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach
> >
> > _______________________________________________
> > MITgcm-devel mailing list
> > MITgcm-devel at mitgcm.org
> > http://mitgcm.org/mailman/listinfo/mitgcm-devel
>
>
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel
More information about the MITgcm-devel
mailing list