[MITgcm-support] diagnosing problems with the adjoint
Patrick Heimbach
heimbach at MIT.EDU
Wed Aug 12 13:30:50 EDT 2009
Holly,
I am a bit confused now on what works in your setup and what doesn't.
Maybe rather than sensitivities blowing up, there's a bug in the code(?)
David's suggestion of starting from a clean tutorial setup is a good
one.
Also indicate over which time span you run (20 timesteps, a year, 10
years?).
Finally, try doing a tangent linear test, to see whether there are
problems
with store directives.
Sorry for being at the wrong coast right now.
-p.
On Aug 12, 2009, at 1:17 PM, Holly Dail wrote:
> I don't know how to check the testreport, but I did run the
> tutorial. I had to make two changes -- I switched to synchronous
> time stepping (1800 sec) based on Patrick's advice, and I set the
> tutorial to run for a year.
>
> The adjoint doesn't blow up, but I am still struggling with some
> inconsistencies.
> - if I use a March 2009 version of the GCM, optim.x ends with iter0
> at a cost of 14.66 and iter1 at a cost of 12.04; optim.x stops
> based on maximal number of iterations reached which seems to be in
> line with what you got when you developed this tutorial.
>
> - if I use a current version of the GCM, optim.x ends with iter0
> again at a cost of 14.66, but iter1 fails to reduce cost with a
> message 'the search direction is not a descent one'
>
> I did recompile and retest the March version this week to make sure
> this was the case even with fresh compiles of both, and indeed it
> is. Not sure what could have changed in the GCM; I couldn't find
> anything significant in the tutorial code or config, lsopt, optim,
> or any of the packages I thought to check.
>
> Thanks,
> Holly
>
> On Aug 12, 2009, at Aug 12 , 12:49 PM, David Ferreira wrote:
>
>> Holy,
>> Just to be sure: does the testreport of tutorial_global_oce_optim
>> run fine for you ?
>> david
>>
>>
>> Holly Dail wrote:
>>> Thanks for the advice Matt.
>>>
>>> I'm not using the divided adjoint, but I'll try the
>>> autodiff_inadmode_set.F approach.
>>>
>>> Here are the viscosities / diffusivities (chosen to be almost
>>> exactly that used in ECCO):
>>> viscAz=1.E-3,
>>> viscAh=1.E4,
>>> diffKhT=100.,
>>> diffKzT=2.E-5,
>>> diffKhS=100.,
>>> diffKzS=1.E-5,
>>>
>>> I used your advection scheme based on your earlier advice, but
>>> haven't tried
>>>> multiDimAdvection=.FALSE.,
>>> Will try that too.
>>>
>>> My time step is 3600 - again same as ECCO.
>>>
>>> Thanks -
>>> Holly
>>>
>>>
>>> On Aug 12, 2009, at Aug 12 , 11:42 AM, Matthew Mazloff wrote:
>>>
>>>> Hi Holly,
>>>>
>>>> Your adjoint is definitely blowing up (how many timesteps is
>>>> your grad check....its blowing up fast). Try turning off
>>>> packages when you run the adjoint and see if that helps. Are
>>>> you using the divided adjoint? If so you can just change some
>>>> things to false in data.pkg when its about to start. Turn off
>>>> KPP and GMREDI and packages of that nature. If you are not
>>>> using the divided adjoint then you have to use
>>>> autodiff_inadmode_set.F to turn these things off. In this file
>>>> just set
>>>> usePtracers = .FALSE.
>>>> useKPP = .FALSE.
>>>> useGMREDI = .FALSE.
>>>> useSEAICE = .FALSE.
>>>>
>>>> Then try again
>>>>
>>>> -Matt
>>>>
>>>> ps> out of curiosity, what viscosity and diffusivity are you
>>>> trying to run the adjoint with?
>>>>
>>>> Oh, and also some of the advection schemes may not be stable. I
>>>> am using
>>>> multiDimAdvection=.FALSE.,
>>>> tempAdvScheme=30,
>>>> saltAdvScheme=30,
>>>>
>>>>
>>>> pps> of course the real expert is just upstairs from you -- bug
>>>> him :o)
>>>>
>>>>
>>>>
>>>>
>>>> On Aug 12, 2009, at 8:14 AM, Holly Dail wrote:
>>>>
>>>>> Hello all -
>>>>>
>>>>> I'd like to use optimization with a regional North Atlantic
>>>>> setup. As a first case, I started with the approach laid out
>>>>> in tutorial_global_oce_optim --
>>>>> - cost based on (1) divergence of annual mean surface
>>>>> temperatures in the model from climatology and (2) reasonable
>>>>> magnitude of control vector
>>>>> - control is a time-mean heat flux correction (2-d field)
>>>>>
>>>>> My sensitivities are astronomical (i.e. adxx = 10^16), the
>>>>> gradient check seems to fail (as shown below, finite difference
>>>>> gradients seem okay, adjoint gradients not so much), and
>>>>> optim.x fails with message 'the linesearch failed'.
>>>>>
>>>>> (PID.TID 0000.0001) grdchk output:
>>>>> procId I ITIL EPOS
>>>>> JTILEPOS LAYER X(I) X(I)+/-EPS
>>>>> (PID.TID 0000.0001) grdchk output: FC
>>>>> FC1 FC2 FC1-FC2/(2*EPS) ADJ GRAD(FC) 1-FDGRD/
>>>>> ADGRD
>>>>> (PID.TID 0000.0001) grdchk output:
>>>>> 0 1 56 35
>>>>> 1 0.000000000D+00 -.100000000D+00
>>>>> (PID.TID 0000.0001) grdchk output:
>>>>> 0.261232434D+02 0.261232444 D+02 0.261232340D+02
>>>>> 0.523051129D-04 -.115313924+108 0.100000000D+01
>>>>>
>>>>> I suppose this may mean the adjoint is blowing up? I've tried
>>>>> reducing my time step and increasing viscosity and I checked
>>>>> that my climatology & error fields are defined at all wet
>>>>> points; are there other fixes folks have had success with?
>>>>> Also if you have scripts that you use to diagnose your
>>>>> optimization runs that would be really appreciated.
>>>>>
>>>>> Thanks -
>>>>> Holly
>>>>> _______________________________________________
>>>>> MITgcm-support mailing list
>>>>> MITgcm-support at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support
---
Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
MIT | EAPS 54-1518 | 77 Massachusetts Ave | Cambridge MA 02139 USA
FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach
More information about the MITgcm-support
mailing list