[MITgcm-devel] [MITgcm-cvs] MITgcm/pkg/seaice CVS Commit

Patrick Heimbach heimbach at MIT.EDU
Tue Jul 19 21:39:44 EDT 2011 

Hi Gael,

not sure I understand your email correctly,
but here's an attempt at replying
(I actually thought everything was already sorted out):

On Jul 19, 2011, at 6:18 PM, Gael Forget wrote:

> Martin, Jean-Michel and Patrick,
> 
> I gave this issue another look. My basic conclusion is that Martin found a bug.
> 
> By saving multiple times the whole tice array to tape (in md_seaice_growth) 
> we ended up with the post solve4temp values in the tape (except for the last bi/bj).
> So ad_seaice_growth, expecting the pre solve4temp values in the tape, was wrong. 

Yes, correct, but in fixing, he used wrong tapes and keys.

> After Patrick's fixes of the bugs Martin added in r137, I believe all is correct in r139 at least
> regarding tice, so I think an update of the global_ocean.cs32x15/seaice result is in order.

Yes, that's correct.

> I encourage you to double check though.

Should be fixed now, but one *perhaps* issue is that once we go
into AD mode (i.e. after MD... loop is completed)
we have an extra call to do_oceanic_phys
Ideally, that call would be omitted through appropriate store dirs.
If/when do_oceanic_phys can get fairly expensive
(i.e. a significant fraction of the timestep)
that slows down the adjoint further.

> With regard to the other recomputations in global_ocean.cs32x15 (solve_pentadiagonal, 
> solve_tridiagonal, gad_implicit_r, freeze_interior, thsice, advect, seaice_advdiff, and 
> seaice_lsr), I was under the impression that (except may be for the thsice and 
> freeze_interior ones) they were nothing new and they were benign. A thorough 
> examination would certainly be of value though.

I think, these are all OK.
Some of them are actually more difficult to deal with
(repeated calls to seaice_advdiff, seaice_lsr)
so storing should be avoided if possible.

Cheers
-p.

> Cheers,
> Gael
> 
> On Jul 19, 2011, at 3:50 AM, Martin Losch wrote:
> 
>> Hi Jean-Michel, Patrick
>> 
>> so I guess that we can live with this, especially, since the change was about (redundant) bi/bj-loops? 
>> 
>> I do not know about the taf_ad.log. There are a lot of recomputation warnings related to vertical implicit advection and thsice, both I have little experience with. But they are not affected by my changes.
>> 
>> Martin
>> 
>> On Jul 18, 2011, at 4:59 PM, Jean-Michel Campin wrote:
>> 
>>> Hi,
>>> 
>>> The latest testreport With the 2nd fix (from today, tapes and keys):
>>> global_ocean.cs32x15.seaice has 12 matching digits (10 before)
>>> lab_sea is back at 16 (13 before)
>>> and lab_sea.evp is passing with 13 digits (12 before) 
>>> All three were passing with 16 digits last Friday.
>>> 
>>> Cheers,
>>> Jean-Michel
>>> 
>>> On Mon, Jul 18, 2011 at 09:46:07AM -0400, Patrick Heimbach wrote:
>>>> 
>>>> Hi there,
>>>> 
>>>> no, it didn't completely fix the problem, i.e.
>>>> still deviations on baudelaire.
>>>> I haven't had time to look into this in more detail,
>>>> just tried to do the obvious fixes between airport gates.
>>>> 
>>>> There were two bugs:
>>>> 1. line length (fix on Sat.)
>>>> 2. wrong tapes and keys (fixed today)
>>>> 
>>>> The second one can conceivably change the result for tests that have bi/bj > 1
>>>> 
>>>> There are likely more problems.
>>>> Looks like the taf_ad.log hasn't been looked at in a while...
>>>> 
>>>> Cheers
>>>> -p.
>>>> 
>>>> On Jul 18, 2011, at 9:32 AM, Jean-Michel Campin wrote:
>>>> 
>>>>> Hi Martin,
>>>>> 
>>>>> I am going to re-run the standard test. Will give an update soon.
>>>>> Cheers,
>>>>> Jean-Michel
>>>>> 
>>>>> On Mon, Jul 18, 2011 at 03:21:29PM +0200, Martin Losch wrote:
>>>>>> Hi there,
>>>>>> did Patrick's second fix fix the problem (I forgot to adjust the keys when I changed the store directives, still not clear to my why it did work for me)? I am now getting 12 digits for global_ocean.cs32x15.seaice and 13 for lab_sea.evp on my machine with seaice_growth.F v1.139. The change to v1.136 that makes this difference is exactly the change of the storing. 
>>>>>> 
>>>>>> Did I introduce a bug, or do we need to update the output files?
>>>>>> 
>>>>>> Martin
>>>>>> 
>>>>>> On Jul 18, 2011, at 9:49 AM, Martin Losch wrote:
>>>>>> 
>>>>>>> Hi Jean-Michel,
>>>>>>> 
>>>>>>> I guess the TAF error was fixed by Patrick (it ran all OK for me, I don't understand what happened), lines were too long (thanks, Patrick).
>>>>>>> 
>>>>>>> I do not know why the gradient of global_ocean.cs32x15.seaice changed, again, for my platform it was all OK. Only three experiments seem to be affected: lab_sea, lab_sea.evp (which I assume is very unstable) and global_ocean.cs32x15.seaice (also on faulks). I changed a little bit of storage: instead of
>>>>>>> CADJ STORE tice, ....
>>>>>>> we now have
>>>>>>> CADJ STORE tice(:,:,bi,bj)
>>>>>>> (and similar for tices) in analogy to all other STORE directives. This means that in the adjoint code we no longer have code that copies 4D arrays within the bi/bj-loops. I can imagine that can have an effect on the numerical precision (although it shouldn't). In Ian's code (FENTY_AREA_EXPANSION_CONTRACTION) I added 3 lines:
>>>>>>> d_AREAbyATM(I,J) = 0. _d 0
>>>>>>> d_AREAbyICE(I,J) = 0. _d 0
>>>>>>> d_AREAbyOCN(I,J) = 0. _d 0
>>>>>>> but that part of the code is not tested in global_ocean.cs32x15.seaice nor lab_sea, as far as I can see. The other stuff I did (contracted two #ifdfe's into one) is not tested either. 
>>>>>>> 
>>>>>>> I'll have a closer look later.
>>>>>>> 
>>>>>>> Martin
>>>>>>> 
>>>>>>> 
>>>>>>> On mitgcm.org/testing/results I see on beaudelaire for gfortran:
>>>>>>>> run: ./testreport -adm -a jmc at mitgcm.org -match 13 -of=../tools/build_options/linux_amd64_gfortran+mpi_generic -MPI 6 -command 'mpirun -np TR_NPROC ./mitgcmuv_ad'
>>>>>>> 
>>>>>>>> Fri Jul 15 02:47:14 EDT 2011
>>>>>>>> Y Y Y Y 16>12<FAIL  global_ocean.cs32x15.seaice
>>>>>>>> Y Y Y Y 16>16<pass  lab_sea
>>>>>>>> Y Y Y Y 16>12<FAIL  lab_sea.evp
>>>>>>> and
>>>>>>>> Sun Jul 17 02:46:03 EDT 2011
>>>>>>> 
>>>>>>>> Y Y Y Y 16>10<FAIL  global_ocean.cs32x15.seaice
>>>>>>>> Y Y Y Y 16>13<pass  lab_sea
>>>>>>>> Y Y Y Y 16>12<FAIL  lab_sea.evp
>>>>>>> 
>>>>>>> for 
>>>>>>>> Fri Jul 15 01:03:19 EDT 2011
>>>>>>>> run: ./testreport -adm -a jmc at mitgcm.org -match 13 -of=../tools/build_options/linux_amd64_gfortran
>>>>>>>> Y Y Y Y 16>16<pass  global_ocean.cs32x15.seaice
>>>>>>>> Y Y Y Y 16>16<pass  lab_sea
>>>>>>>> Y Y Y Y 16>16<pass  lab_sea.evp
>>>>>>> and
>>>>>>>> Sun Jul 17 01:03:23 EDT 2011
>>>>>>>> Y Y Y Y 16>10<FAIL  global_ocean.cs32x15.seaice
>>>>>>>> Y Y Y Y 16>13<pass  lab_sea
>>>>>>>> Y Y Y Y 16>12<FAIL  lab_sea.evp
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Jul 16, 2011, at 2:42 PM, Jean-Michel Campin wrote:
>>>>>>> 
>>>>>>> 
>>>>>>>> Hi Martin,
>>>>>>>> 
>>>>>>>> Looks like there is a problem in the changes you made in seaice_growth.F:
>>>>>>>> TAF returns an error for lab_sea experiment (all last night AD lab_sea tests 
>>>>>>>> I loooked at failed). 
>>>>>>>> 
>>>>>>>> The gradient of global_ocean.cs32x15.seaice also changed (from 16.digits
>>>>>>>> to only 10 on ref. platform=baudelaire+gfortran) so that it's now
>>>>>>>> returnning a "FAIL". Was it expected ? Do we need to update
>>>>>>>> output_adm.seaice.txt ?
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Jean-Michel
>>>>>>>> 
>>>>>>>> On Fri, Jul 15, 2011 at 09:01:07AM -0400, Martin Losch wrote:
>>>>>>>>> Update of /u/gcmpack/MITgcm/pkg/seaice
>>>>>>>>> In directory forge:/tmp/cvs-serv6974/pkg/seaice
>>>>>>>>> 
>>>>>>>>> Modified Files:
>>>>>>>>> 	seaice_growth.F 
>>>>>>>>> Log Message:
>>>>>>>>> - fix recomputation with FENTY_AREA_EXPANSION_CONTRACTION code,
>>>>>>>>> remove then obsolete STORE directives
>>>>>>>>> - fix STORE directives for tice and tices, so that only the
>>>>>>>>> appropriate part of the fields are stored (and not the entire
>>>>>>>>> field)
>>>>>>>>> - cosmetic changes: adjust CPP flags and indentation for better
>>>>>>>>> legibilty
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> MITgcm-cvs mailing list
>>>>>>>>> MITgcm-cvs at mitgcm.org
>>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-cvs
>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> MITgcm-devel mailing list
>>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>>> 
>>>>>>> Martin Losch
>>>>>>> Martin.Losch at awi.de
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> MITgcm-devel mailing list
>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> MITgcm-devel mailing list
>>>>>> MITgcm-devel at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>> 
>>>>> _______________________________________________
>>>>> MITgcm-devel mailing list
>>>>> MITgcm-devel at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>> 
>>>> ---
>>>> Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
>>>> MIT | EAPS 54-1420 | 77 Massachusetts Ave | Cambridge MA 02139 USA
>>>> FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> MITgcm-devel mailing list
>>>> MITgcm-devel at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>> 
>>> _______________________________________________
>>> MITgcm-devel mailing list
>>> MITgcm-devel at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>> 
>> 
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> 
> 
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel

---
Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
MIT | EAPS 54-1420 | 77 Massachusetts Ave | Cambridge MA 02139 USA
FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach

More information about the MITgcm-devel mailing list