[MITgcm-devel] lab_sea default adjoint verif. "broken" since April '11

[Patrick Heimbach]

Hi Martin,

On Oct 13, 2011, at 11:53 AM, Martin Losch wrote:

> Hi Patrick,
> 
> not that I have any idea what's going on, but can you exclude that there is something wrong with the grdchk package? I have had instances (which I did not follow up on) where the cost function values differed with respect to the expected value (so FC1 was very different from FC giving a with fd-gradient, whereas FC1 and FC2 gave reasonalbe fd-gradients with good agreement with the adjoing gradient). If that is useful, I could find these examples again (they had to do with xx_shifwflux as far as I remember).

I had similar instances of problematic FD's, but this is way back,
and we mostly traced it back to incomplete re-initialization in the f.d. gradient checks.
It's very unlikely the source of error here.
As I said, just setting SIsalFRAC = 0. seems to remove the problems.

> Its also interesting that these jumps in FD-gradients come in steps, the biggest being between r1.42  and r1.43, another one between r1.40 and r1.41 (where one FD-gradient goes back to normal an another jumps up) and r1.39 and r1.40;

Yes, agreed.
But main problem is that this should have been noticed in the first place before committing r1.43.

> Martin
> 
> PS. ceterum censeo that the adjoint of LSR is fishy.

Yes, but the EVP solver is fishy too.
It's a another chapter altogether.



> On Oct 13, 2011, at 12:03 PM, Patrick Heimbach wrote:
> 
>> 
>> Hi there,
>> 
>> recently, Jean-Michel added a test on the finite-difference gradient to the testreport. 
>> Since April we had a very good example why it would have been useful, 
>> but didn't notice (since the test existed yet):
>> 
>> Before a series of changes made to pkg/seaice
>> the adjoint vs. finite-difference gradient accuracy was about 10E-4
>> (revision 1.39 of output_adm.txt)
>> after those changes that accuracy deteriorated to 10E+7
>> (revision 1.43 of output_adm.txt)
>> 
>> Interestingly, what's changed was not really the adjoint,
>> but the finite difference, which increased by orders of magnitude.
>> Reducing epsilon, or the number of timesteps (from 4 to 2) seems to
>> reduce the problem somewhat, but points to an extremely sensitive
>> threshold behavior that kicks in quickly.
>> A tangent linear test still shows the problem.
>> Another way to remove this issue is to set new parameter SIsalFRAC = 0.
>> 
>> Also interesting is that output_adm.evp.txt does not show this problem.
>> But a direct implication of solvers as culprits is not obvious at all.
>> The results for output_adm.seaice.txt in global_ocean.cs32x15/
>> seem to be better-behaved as well.
>> 
>> One question is whether lab_sea is notoriously ill-designed
>> for the tests that we'd like to run (we've had problems in the past).
>> But it still doesn't explain
>> * why use of LSR vs. EVP show the drastically different behavior
>> (EVP seemingly well behaved, and LSR also badly behaved in TLM mode);
>> again, it's unlikely due to the solvers per se;
>> * why the large sensitivity to SIsalFRAC;
>> 
>> Looking at alternatives, the obvious choice seems to be 1D_ocean_ice_column/
>> But looking at output_adm.txt there shows adj. vs. f.d. gradient accuracies
>> of E+0 to E+4 i.e. those results can't be taken serious either.
>> 
>> -Patrick
>> 
>> ---
>> Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
>> MIT | EAPS 54-1420 | 77 Massachusetts Ave | Cambridge MA 02139 USA
>> FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach
>> 
>> 
>> 
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> 
> 
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel

---
Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
MIT | EAPS 54-1420 | 77 Massachusetts Ave | Cambridge MA 02139 USA
FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach