[MITgcm-support] diagnosing problems with the adjoint

David Ferreira dfer at mit.edu
Wed Aug 12 14:31:07 EDT 2009


> (1) my North Atlantic setup (for debugging purposes running for 1 
> month at timestep of 3600 sec)
> problem: adxx values are on the order of 10^16 and optim.x crashes
>
> (2) a clean checkout of tutorial_global_oce_optim with only 2 changes 
> (synchronous time stepping at 1800 sec and 1 year execution)
> problem: with a current version of the GCM and tutorial, cost 
> reduction fails at iteration 1
> [David was able to see cost reductions over 10+ iterations when he ran 
> this 1 year ago and I've tested with a March version of the GCM and 
> had better results too]
With the adjoint, the fact things work with one set-up is no guaranty 
for another set-up (that's
actually also true for non-adjoint stuff)
So, is it worth spending time getting the tutorial to work for you ?
I understand your goal is to get (1) to work and it is very different 
from the tutorial set-up, no ?

Anyway, FYI, the testreport is done with:
./testreport -adm -of ../tools/build_options/{the opt. file you want} -t 
tutorial_global_oce_optim
in the verification directory.

david

> For (1) I am trying Matt's advice on turning off packages.  I could 
> also move to a faster machine and run for 1 year if that might help.
> For (2) I can try running with asynchronous time-stepping, though I 
> thought it was inadvisable for the adjoint









> at Aug 12 , 1:30 PM, Patrick Heimbach wrote:
>
>>
>> Holly,
>>
>> I am a bit confused now on what works in your setup and what doesn't.
>> Maybe rather than sensitivities blowing up, there's a bug in the code(?)
>>
>> David's suggestion of starting from a clean tutorial setup is a good 
>> one.
>> Also indicate over which time span you run (20 timesteps, a year, 10 
>> years?).
>> Finally, try doing a tangent linear test, to see whether there are 
>> problems
>> with store directives.
>>
>> Sorry for being at the wrong coast right now.
>>
>> -p.
>>
>> On Aug 12, 2009, at 1:17 PM, Holly Dail wrote:
>>
>>> I don't know how to check the testreport, but I did run the 
>>> tutorial.  I had to make two changes -- I switched to synchronous 
>>> time stepping (1800 sec) based on Patrick's advice, and I set the 
>>> tutorial to run for a year.
>>>
>>> The adjoint doesn't blow up, but I am still struggling with some 
>>> inconsistencies.
>>> - if I use a March 2009 version of the GCM, optim.x ends with iter0 
>>> at a cost of 14.66 and iter1 at a cost of 12.04; optim.x stops based 
>>> on maximal number of iterations reached which seems to be in line 
>>> with what you got when you developed this tutorial.
>>>
>>> - if I use a current version of the GCM, optim.x ends with iter0 
>>> again at a cost of 14.66, but iter1 fails to reduce cost with a 
>>> message 'the search direction is not a descent one'
>>>
>>> I did recompile and retest the March version this week to make sure 
>>> this was the case even with fresh compiles of both, and indeed it 
>>> is.  Not sure what could have changed in the GCM; I couldn't find 
>>> anything significant in the tutorial code or config, lsopt, optim, 
>>> or any of the packages I thought to check.
>>>
>>> Thanks,
>>> Holly
>>>
>>> On Aug 12, 2009, at Aug 12 , 12:49 PM, David Ferreira wrote:
>>>
>>>> Holy,
>>>> Just to be sure: does the testreport of tutorial_global_oce_optim 
>>>> run fine  for you ?
>>>> david
>>>>
>>>>
>>>> Holly Dail wrote:
>>>>> Thanks for the advice Matt.
>>>>>
>>>>> I'm not using the divided adjoint, but I'll try the 
>>>>> autodiff_inadmode_set.F approach.
>>>>>
>>>>> Here are the viscosities / diffusivities (chosen to be almost 
>>>>> exactly that used in ECCO):
>>>>> viscAz=1.E-3,
>>>>> viscAh=1.E4,
>>>>> diffKhT=100.,
>>>>> diffKzT=2.E-5,
>>>>> diffKhS=100.,
>>>>> diffKzS=1.E-5,
>>>>>
>>>>> I used your advection scheme based on your earlier advice, but 
>>>>> haven't tried
>>>>>> multiDimAdvection=.FALSE.,
>>>>> Will try that too.
>>>>>
>>>>> My time step is 3600 - again same as ECCO.
>>>>>
>>>>> Thanks -
>>>>> Holly
>>>>>
>>>>>
>>>>> On Aug 12, 2009, at Aug 12 , 11:42 AM, Matthew Mazloff wrote:
>>>>>
>>>>>> Hi Holly,
>>>>>>
>>>>>> Your adjoint is definitely blowing up (how many timesteps is your 
>>>>>> grad check....its blowing up fast).   Try turning off packages 
>>>>>> when you run the adjoint and see if that helps.  Are you using 
>>>>>> the divided adjoint?  If so you can just change some things to 
>>>>>> false in data.pkg when its about to start.  Turn off KPP and 
>>>>>> GMREDI and packages of that nature.  If you are not using the 
>>>>>> divided adjoint then you have to use autodiff_inadmode_set.F to 
>>>>>> turn these things off.  In this file just set
>>>>>>    usePtracers  = .FALSE.
>>>>>>    useKPP = .FALSE.
>>>>>>    useGMREDI = .FALSE.
>>>>>>    useSEAICE = .FALSE.
>>>>>>
>>>>>> Then try again
>>>>>>
>>>>>> -Matt
>>>>>>
>>>>>> ps> out of curiosity, what viscosity and diffusivity are you 
>>>>>> trying to run the adjoint with?
>>>>>>
>>>>>> Oh, and also some of the advection schemes may not be stable.  I 
>>>>>> am using
>>>>>> multiDimAdvection=.FALSE.,
>>>>>> tempAdvScheme=30,
>>>>>> saltAdvScheme=30,
>>>>>>
>>>>>>
>>>>>> pps> of course the real expert is just upstairs from you -- bug 
>>>>>> him :o)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Aug 12, 2009, at 8:14 AM, Holly Dail wrote:
>>>>>>
>>>>>>> Hello all -
>>>>>>>
>>>>>>> I'd like to use optimization with a regional North Atlantic 
>>>>>>> setup.  As a first case, I started with the approach laid out in 
>>>>>>> tutorial_global_oce_optim --
>>>>>>> - cost based on (1) divergence of annual mean surface 
>>>>>>> temperatures in the model from climatology and (2) reasonable 
>>>>>>> magnitude of control vector
>>>>>>> - control is a time-mean heat flux correction (2-d field)
>>>>>>>
>>>>>>> My sensitivities are astronomical (i.e. adxx = 10^16), the 
>>>>>>> gradient check seems to fail (as shown below, finite difference 
>>>>>>> gradients seem okay, adjoint gradients not so much), and optim.x 
>>>>>>> fails with message 'the linesearch failed'.
>>>>>>>
>>>>>>> (PID.TID 0000.0001) grdchk output:            
>>>>>>> procId               I        ITIL  EPOS        
>>>>>>> JTILEPOS           LAYER            X(I)      X(I)+/-EPS
>>>>>>> (PID.TID 0000.0001) grdchk output:            FC             
>>>>>>> FC1             FC2 FC1-FC2/(2*EPS)    ADJ GRAD(FC)   1-FDGRD/ADGRD
>>>>>>> (PID.TID 0000.0001) grdchk output:                 
>>>>>>> 0               1             56              35               1 
>>>>>>> 0.000000000D+00 -.100000000D+00
>>>>>>> (PID.TID 0000.0001) grdchk output:                   
>>>>>>> 0.261232434D+02 0.261232444     D+02 0.261232340D+02 
>>>>>>> 0.523051129D-04 -.115313924+108 0.100000000D+01
>>>>>>>
>>>>>>> I suppose this may mean the adjoint is blowing up?  I've tried 
>>>>>>> reducing my time step and increasing viscosity and I checked 
>>>>>>> that my climatology & error fields are defined at all wet 
>>>>>>> points; are there other fixes folks have had success with?  Also 
>>>>>>> if you have scripts that you use to diagnose your optimization 
>>>>>>> runs that would be really appreciated.
>>>>>>>
>>>>>>> Thanks -
>>>>>>> Holly
>>>>>>> _______________________________________________
>>>>>>> MITgcm-support mailing list
>>>>>>> MITgcm-support at mitgcm.org
>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>>>
>>>>>> _______________________________________________
>>>>>> MITgcm-support mailing list
>>>>>> MITgcm-support at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>>
>>>>> _______________________________________________
>>>>> MITgcm-support mailing list
>>>>> MITgcm-support at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>
>> ---
>> Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
>> MIT | EAPS 54-1518 | 77 Massachusetts Ave | Cambridge MA 02139 USA
>> FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach
>>
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support




More information about the MITgcm-support mailing list