[MITgcm-support] diagnosing problems with the adjoint

[Holly Dail]

Hello all -

I'd like to use optimization with a regional North Atlantic setup.  As  
a first case, I started with the approach laid out in  
tutorial_global_oce_optim --
- cost based on (1) divergence of annual mean surface temperatures in  
the model from climatology and (2) reasonable magnitude of control  
vector
- control is a time-mean heat flux correction (2-d field)

My sensitivities are astronomical (i.e. adxx = 10^16), the gradient  
check seems to fail (as shown below, finite difference gradients seem  
okay, adjoint gradients not so much), and optim.x fails with message  
'the linesearch failed'.

(PID.TID 0000.0001) grdchk output:            procId                
I        ITIL  EPOS        JTILEPOS           LAYER             
X(I)      X(I)+/-EPS
(PID.TID 0000.0001) grdchk output:            FC              
FC1             FC2 FC1-FC2/(2*EPS)    ADJ GRAD(FC)   1-FDGRD/ADGRD
(PID.TID 0000.0001) grdchk output:                 0                
1             56              35               1 0.000000000D+00 -. 
100000000D+00
(PID.TID 0000.0001) grdchk output:                   0.261232434D+02  
0.261232444     D+02 0.261232340D+02 0.523051129D-04 -.115313924+108  
0.100000000D+01

I suppose this may mean the adjoint is blowing up?  I've tried  
reducing my time step and increasing viscosity and I checked that my  
climatology & error fields are defined at all wet points; are there  
other fixes folks have had success with?  Also if you have scripts  
that you use to diagnose your optimization runs that would be really  
appreciated.

Thanks -
Holly