[MITgcm-devel] updating AD verification outputs

Patrick Heimbach heimbach at MIT.EDU
Sat Mar 9 18:35:20 EST 2013 

Hi Jean-Michel,

indeed, looks like a change.
After very quick look, here's what seems to happen:

If you look in the previous reference results obtained with 2.3.8
and grep for "ref_cost_function' in both adm and tlm output
you will see that they differ (something that we've missed,
and should include in testing once we decide to do
adm and tlm test routinely side-by-side):
(PID.TID 0000.0001)  ADM  ref_cost_function      = -6.43687129344575E+11
(PID.TID 0000.0001)  ADM  ref_cost_function      = -6.43687129344575E+11
(PID.TID 0000.0001)  ADM  ref_cost_function      = -6.43687129344575E+11
(PID.TID 0000.0001)  ADM  ref_cost_function      = -6.43687129344575E+11
vs.
(PID.TID 0000.0001)  TLM  ref_cost_function      = -6.43687129344283E+11
(PID.TID 0000.0001)  TLM  ref_cost_function      = -6.43687129344283E+11
(PID.TID 0000.0001)  TLM  ref_cost_function      = -6.43687129344283E+11
(PID.TID 0000.0001)  TLM  ref_cost_function      = -6.43687129344283E+11

That means that there might be something not quite right in the TLM code for 2.3.8
(sometimes due to incomplete initialization, but in this case
probably due to something else).

The cost functions for 2.3.10 appear to be consistent for adm and tlm:
(PID.TID 0000.0001)  ADM  ref_cost_function      = -6.43687129344575E+11
(PID.TID 0000.0001)  ADM  ref_cost_function      = -6.43687129344575E+11
(PID.TID 0000.0001)  ADM  ref_cost_function      = -6.43687129344575E+11
(PID.TID 0000.0001)  ADM  ref_cost_function      = -6.43687129344575E+11
vs.
(PID.TID 0000.0001)  TLM  ref_cost_function      = -6.43687129344575E+11
(PID.TID 0000.0001)  TLM  ref_cost_function      = -6.43687129344575E+11
(PID.TID 0000.0001)  TLM  ref_cost_function      = -6.43687129344575E+11
(PID.TID 0000.0001)  TLM  ref_cost_function      = -6.43687129344575E+11

But I need to double-check more carefully, might have missed something
(I can't remember whether we really do re-compute the "reference" every time;
we should, but need to double-check the code - later).

p.

On Mar 9, 2013, at 5:52 PM, Jean-Michel Campin <jmc at ocean.mit.edu> wrote:

> Hi Patrick,
> 
> I updated the output (AD & TLM). But I noticed something strange with
> the TLM output of experiment tutorial_global_oce_biogeo:
> with TAF 2.3.8:
>> Y Y Y Y 16>16<16 pass  tutorial_global_oce_biogeo
> and with TAF 2.3.10:
>> Y Y Y Y 12>14<16 pass  tutorial_global_oce_biogeo
> so the cost function is not identical.
> In fact, there are some (small) differences that show up at the 
> 2nd iteration (cg2d: Sum(rhs)).
> I would have thought that the cost function and forward run 
> would not depend on the code that TAF produces.
> 
> Cheers,
> Jean-Michel
> 
> On Sat, Mar 09, 2013 at 12:07:50PM -0500, Jean-Michel Campin wrote:
>> Hi Patrick,
>> 
>> Getting also some small differences in tlm test, with 2 exp
>> falling below 13:
>> Y Y Y Y 16>12<16 FAIL  1D_ocean_ice_column
>> Y Y Y Y 16>12<16 FAIL  tutorial_tracer_adjsens.som81
>> (was all 16 last Sunday with TAF 2.3.8)
>> 
>> Did take a short look at the diffences between the code TAF 2.3.10
>> generates compared with 2.3.8, and I am seeing many more
>> paranthesis, in multiple multiplication expression, 
>> (a*b)*c versus a*b*c before, 
>> and also sum of more than 2 terms: (a+b)+c versus a+b+c before.
>> This can explain the small differences (we are still getting the same 
>> 1rst 12 digits for any experiment).
>> 
>> Will hardly see any changes in gradient-check (since these diff 
>> are too small).
>> 
>> Probably going to update the output (ad & tlm) that are matching by
>> less than 13 digits.
>> 
>> Cheers,
>> Jean-Michel
>> 
>> On Sat, Mar 09, 2013 at 10:06:55AM -0500, Patrick Heimbach wrote:
>>> 
>>> Hi Jean-Michel,
>>> 
>>> before updating, let's first check the TLM to see if there are similar changes,
>>> and what the gradient checks indicate in terms of actual accuracy changes.
>>> 
>>> May have to send a bug report to Fastopt, which I am not sure I can do
>>> since I am being blocked from their taf-user at fastopt.de list.
>>> 
>>> Cheers
>>> -Patrick
>>> 
>>> On Mar 9, 2013, at 9:58 AM, Jean-Michel Campin <jmc at ocean.mit.edu> wrote:
>>> 
>>>> Hi Patrick,
>>>> 
>>>> Looks like the recent update of TAF version (from 2.3.8 until yesterday
>>>> to 2.3.10 today) change some output at the level of machine truncation:
>>>> 
>>>> On baudelaire with gfortran (no mpi):
>>>> Y Y Y Y 16>12<16 FAIL  1D_ocean_ice_column
>>>> Y Y Y Y 16>12<16 FAIL  bottom_ctrl_5x5
>>>> Y Y Y Y 16>12<16 FAIL  lab_sea.noseaicedyn
>>>> Y Y Y Y 16>12<16 FAIL  offline_exf_seaice
>>>> 
>>>> whereas yesterday the same 4 experiments:
>>>> Y Y Y Y 16>13<16 pass  1D_ocean_ice_column
>>>> Y Y Y Y 16>16<16 pass  bottom_ctrl_5x5
>>>> Y Y Y Y 16>16<16 pass  lab_sea.noseaicedyn
>>>> Y Y Y Y 16>16<16 pass  offline_exf_seaice
>>>> 
>>>> Should we update these 4 experiments AD output ?
>>>> 
>>>> Did not check yet the TLM output but will do later today.
>>>> 
>>>> Cheers,
>>>> Jean-Michel
>>>> 
>>>> _______________________________________________
>>>> MITgcm-devel mailing list
>>>> MITgcm-devel at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>> 
>>> 
>>> ---
>>> Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
>>> MIT | EAPS 54-1420 | 77 Massachusetts Ave | Cambridge MA 02139 USA
>>> FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach
>>> 
>> 
>> 
>> 
>>> _______________________________________________
>>> MITgcm-devel mailing list
>>> MITgcm-devel at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>> 
>> 
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> 
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel


---
Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
MIT | EAPS 54-1420 | 77 Massachusetts Ave | Cambridge MA 02139 USA
FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1588 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-devel/attachments/20130309/2378491c/attachment.p7s>

More information about the MITgcm-devel mailing list