[MITgcm-support] diagnosing problems with the adjoint

Holly Dail hdail at MIT.EDU
Thu Aug 13 20:56:42 EDT 2009


I believe I have gotten both of these setups working now thanks to the  
advice from the list -- thanks!  Thought I'd report on which fixes  
worked in case others face the same ...

tutorial_global_oce_optim:
	I can get a reduction in cost over at least 2 iterations with a  
clean, current (Aug. 10) checkout of the GCM with the following  
changes: nTimeSteps = 360 in data and lastinterval = 31104000 in  
data.cost; these changes are to be expected as the tutorial as checked  
in to the CVS must be very fast for testing purposes.  Cost only goes  
down for 1 iteration if one uses synchronous time stepping at 1800  
sec.  I had thought asynchronous time-stepping was problematic for the  
adjoint, but it seems to work in this case.

my north atlantic setup:
	I now have an adjoint that does not blow up and can get a reduction  
in cost.  I followed Matt's advice to use autodiff_inadmode_set_ad.F  
and autodiff_inadmode_unset_ad.F to turn off packages KPP, Ptracers,  
GMRedi, and Sea Ice for the adjoint stage and I set  
multiDimAdvection=.FALSE. in the data file.  I had thought having  
different packages in forward and adjoint modes was supposed to be  
problematic in terms of interpreting the results, but regardless I'm  
really grateful to have a version that works.

I'm still interested in any scripts you may have that I could build  
off of for diagnosing problems with the adjoint in time; sending them  
off list might make sense.

Thanks to all for the help -
Holly


On Aug 12, 2009, at Aug 12 , 1:57 PM, Holly Dail wrote:

> The confusion may be coming from the fact that the discussion was of  
> two independent setups --
>
> (1) my North Atlantic setup (for debugging purposes running for 1  
> month at timestep of 3600 sec)
> problem: adxx values are on the order of 10^16 and optim.x crashes
>
> (2) a clean checkout of tutorial_global_oce_optim with only 2  
> changes (synchronous time stepping at 1800 sec and 1 year execution)
> problem: with a current version of the GCM and tutorial, cost  
> reduction fails at iteration 1
> [David was able to see cost reductions over 10+ iterations when he  
> ran this 1 year ago and I've tested with a March version of the GCM  
> and had better results too]
>
> For (1) I am trying Matt's advice on turning off packages.  I could  
> also move to a faster machine and run for 1 year if that might help.
> For (2) I can try running with asynchronous time-stepping, though I  
> thought it was inadvisable for the adjoint
>
> The west coast is always the right coast, which means I'm on the  
> wrong coast!
> Holly
>
>
> On Aug 12, 2009, at Aug 12 , 1:30 PM, Patrick Heimbach wrote:
>
>>
>> Holly,
>>
>> I am a bit confused now on what works in your setup and what doesn't.
>> Maybe rather than sensitivities blowing up, there's a bug in the  
>> code(?)
>>
>> David's suggestion of starting from a clean tutorial setup is a  
>> good one.
>> Also indicate over which time span you run (20 timesteps, a year,  
>> 10 years?).
>> Finally, try doing a tangent linear test, to see whether there are  
>> problems
>> with store directives.
>>
>> Sorry for being at the wrong coast right now.
>>
>> -p.
>>
>> On Aug 12, 2009, at 1:17 PM, Holly Dail wrote:
>>
>>> I don't know how to check the testreport, but I did run the  
>>> tutorial.  I had to make two changes -- I switched to synchronous  
>>> time stepping (1800 sec) based on Patrick's advice, and I set the  
>>> tutorial to run for a year.
>>>
>>> The adjoint doesn't blow up, but I am still struggling with some  
>>> inconsistencies.
>>> - if I use a March 2009 version of the GCM, optim.x ends with  
>>> iter0 at a cost of 14.66 and iter1 at a cost of 12.04; optim.x  
>>> stops based on maximal number of iterations reached which seems to  
>>> be in line with what you got when you developed this tutorial.
>>>
>>> - if I use a current version of the GCM, optim.x ends with iter0  
>>> again at a cost of 14.66, but iter1 fails to reduce cost with a  
>>> message 'the search direction is not a descent one'
>>>
>>> I did recompile and retest the March version this week to make  
>>> sure this was the case even with fresh compiles of both, and  
>>> indeed it is.  Not sure what could have changed in the GCM; I  
>>> couldn't find anything significant in the tutorial code or config,  
>>> lsopt, optim, or any of the packages I thought to check.
>>>
>>> Thanks,
>>> Holly
>>>
>>> On Aug 12, 2009, at Aug 12 , 12:49 PM, David Ferreira wrote:
>>>
>>>> Holy,
>>>> Just to be sure: does the testreport of tutorial_global_oce_optim  
>>>> run fine  for you ?
>>>> david
>>>>
>>>>
>>>> Holly Dail wrote:
>>>>> Thanks for the advice Matt.
>>>>>
>>>>> I'm not using the divided adjoint, but I'll try the  
>>>>> autodiff_inadmode_set.F approach.
>>>>>
>>>>> Here are the viscosities / diffusivities (chosen to be almost  
>>>>> exactly that used in ECCO):
>>>>> viscAz=1.E-3,
>>>>> viscAh=1.E4,
>>>>> diffKhT=100.,
>>>>> diffKzT=2.E-5,
>>>>> diffKhS=100.,
>>>>> diffKzS=1.E-5,
>>>>>
>>>>> I used your advection scheme based on your earlier advice, but  
>>>>> haven't tried
>>>>>> multiDimAdvection=.FALSE.,
>>>>> Will try that too.
>>>>>
>>>>> My time step is 3600 - again same as ECCO.
>>>>>
>>>>> Thanks -
>>>>> Holly
>>>>>
>>>>>
>>>>> On Aug 12, 2009, at Aug 12 , 11:42 AM, Matthew Mazloff wrote:
>>>>>
>>>>>> Hi Holly,
>>>>>>
>>>>>> Your adjoint is definitely blowing up (how many timesteps is  
>>>>>> your grad check....its blowing up fast).   Try turning off  
>>>>>> packages when you run the adjoint and see if that helps.  Are  
>>>>>> you using the divided adjoint?  If so you can just change some  
>>>>>> things to false in data.pkg when its about to start.  Turn off  
>>>>>> KPP and GMREDI and packages of that nature.  If you are not  
>>>>>> using the divided adjoint then you have to use  
>>>>>> autodiff_inadmode_set.F to turn these things off.  In this file  
>>>>>> just set
>>>>>>   usePtracers  = .FALSE.
>>>>>>   useKPP = .FALSE.
>>>>>>   useGMREDI = .FALSE.
>>>>>>   useSEAICE = .FALSE.
>>>>>>
>>>>>> Then try again
>>>>>>
>>>>>> -Matt
>>>>>>
>>>>>> ps> out of curiosity, what viscosity and diffusivity are you  
>>>>>> trying to run the adjoint with?
>>>>>>
>>>>>> Oh, and also some of the advection schemes may not be stable.   
>>>>>> I am using
>>>>>> multiDimAdvection=.FALSE.,
>>>>>> tempAdvScheme=30,
>>>>>> saltAdvScheme=30,
>>>>>>
>>>>>>
>>>>>> pps> of course the real expert is just upstairs from you -- bug  
>>>>>> him :o)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Aug 12, 2009, at 8:14 AM, Holly Dail wrote:
>>>>>>
>>>>>>> Hello all -
>>>>>>>
>>>>>>> I'd like to use optimization with a regional North Atlantic  
>>>>>>> setup.  As a first case, I started with the approach laid out  
>>>>>>> in tutorial_global_oce_optim --
>>>>>>> - cost based on (1) divergence of annual mean surface  
>>>>>>> temperatures in the model from climatology and (2) reasonable  
>>>>>>> magnitude of control vector
>>>>>>> - control is a time-mean heat flux correction (2-d field)
>>>>>>>
>>>>>>> My sensitivities are astronomical (i.e. adxx = 10^16), the  
>>>>>>> gradient check seems to fail (as shown below, finite  
>>>>>>> difference gradients seem okay, adjoint gradients not so  
>>>>>>> much), and optim.x fails with message 'the linesearch failed'.
>>>>>>>
>>>>>>> (PID.TID 0000.0001) grdchk output:             
>>>>>>> procId               I        ITIL  EPOS         
>>>>>>> JTILEPOS           LAYER            X(I)      X(I)+/-EPS
>>>>>>> (PID.TID 0000.0001) grdchk output:            FC              
>>>>>>> FC1             FC2 FC1-FC2/(2*EPS)    ADJ GRAD(FC)   1-FDGRD/ 
>>>>>>> ADGRD
>>>>>>> (PID.TID 0000.0001) grdchk output:                  
>>>>>>> 0               1             56              35                
>>>>>>> 1 0.000000000D+00 -.100000000D+00
>>>>>>> (PID.TID 0000.0001) grdchk output:                    
>>>>>>> 0.261232434D+02 0.261232444     D+02 0.261232340D+02  
>>>>>>> 0.523051129D-04 -.115313924+108 0.100000000D+01
>>>>>>>
>>>>>>> I suppose this may mean the adjoint is blowing up?  I've tried  
>>>>>>> reducing my time step and increasing viscosity and I checked  
>>>>>>> that my climatology & error fields are defined at all wet  
>>>>>>> points; are there other fixes folks have had success with?   
>>>>>>> Also if you have scripts that you use to diagnose your  
>>>>>>> optimization runs that would be really appreciated.
>>>>>>>
>>>>>>> Thanks -
>>>>>>> Holly
>>>>>>> _______________________________________________
>>>>>>> MITgcm-support mailing list
>>>>>>> MITgcm-support at mitgcm.org
>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>>>
>>>>>> _______________________________________________
>>>>>> MITgcm-support mailing list
>>>>>> MITgcm-support at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>>
>>>>> _______________________________________________
>>>>> MITgcm-support mailing list
>>>>> MITgcm-support at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>
>> ---
>> Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
>> MIT | EAPS 54-1518 | 77 Massachusetts Ave | Cambridge MA 02139 USA
>> FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach
>>
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support




More information about the MITgcm-support mailing list