[MITgcm-devel] seaice

Jean-Michel Campin jmc at ocean.mit.edu
Thu Mar 26 12:17:22 EDT 2009 

Martin,

When I run lab_sea.salt_plume (exactly the standard test, without MPI)
and add debugLevel=2, it stop immediatly at the 1 iteration
with the error message:
> u-iteration did not converge
and I get:
> U lsr iters, error =      4  0.00000000000000E+00
in the standard output.

If I do the same with MPI on 2CPU, it finishes the 12.iters normally.

Does this tell you something ?

Jean-Michel

On Tue, Mar 24, 2009 at 03:05:05PM -0400, Jean-Michel Campin wrote:
> Martin,
> 
> I've started to look at the differences between
> 2-cpu and 1-cpu runs for the lab_sea.salt_plume exp.
> After the 1rst iteration UICE at i=11 (=tiles junction)
> and j=6 is different by 0.24 m/s (this is larger any ice velocity
> where there is ice). This confirms that there is a problem.
> Will need to dig into the code now.
> 
> Jean-Michel
> 
> On Tue, Mar 24, 2009 at 03:50:05PM +0100, Martin Losch wrote:
> > Hi Jean-Michel,
> >
> > the failing adjoint experiments are "explainable", as I said before, and 
> > as far as I know the gradients are not broken, I just did not update the 
> > results files (because I cannot run the adjoint model from here). You can 
> > call me up (now) and complain directly, so that you don't have to be so 
> > polite (o;
> >
> > What really worries me is the MPI problem.
> >
> > Martin
> >
> > On Mar 24, 2009, at 3:37 PM, Jean-Michel Campin wrote:
> >
> >> Hi Martin,
> >>
> >> On Tue, Mar 24, 2009 at 03:03:29PM +0100, Martin Losch wrote:
> >>> Me again,
> >>>
> >>> restart: after rereading your emails I am not sure what has happened:
> >>> does the restart that was originally broken, work now (e.g. did I
> >>> accidentally fix that part without knowing)?
> >> Well, I though you fix it (pass with pgf), but when I tried with
> >> gfortran, it's not passing the 2+2=4 test. In conclusion, it's not
> >> really fixed, but before it was not passing neither, so no worry.
> >>
> >> I am currently trying to figure out the AD test (lab_sea &  
> >> offline_exf_seaice).
> >> One thing that make it more difficult is that the same day
> >> (Wed March 18) you checked-in changes + a tag + other changes
> >> but no (AD) automatic test was run in between, so ...
> >>
> >> Will have a look at this mpi stuff later.
> >>
> >> Jean-Michel
> >>
> >>>
> >>> g77+mpi: I have no idea, why my changes should affect the runs with  
> >>> MPI,
> >>> because:
> >>> 1. the general structure of the code is the same as before. I did not
> >>> add or remove exchanges, global sums, etc
> >>> 2. the array boundaries did not change, and even if they did and I am
> >>> now using halo-points that are not properly defined, it would also  
> >>> be a
> >>> problem with non-mpi runs, right?, plus I should see effects along  
> >>> the
> >>> edges, which I don't.
> >>> 3. There are new 2D-fields k1/2AtC/Z, which are always zero along the
> >>> edges for non-spherical grids (for I=1-Olx, and sNx+Olx, etc), for
> >>> spherical grids (such as lab_sea), they should be defined correctly
> >>> everywhere. Also they used values are I=0,sNx+1,J=0,sNy+1, so that  
> >>> there
> >>> shouldn't be any problems except of Olx/y=1 (which won't work for the
> >>> seaice model anyway).
> >>>
> >>> I have no clue ...
> >>>
> >>> Martin
> >>>
> >>>
> >>>
> >>> On Mar 24, 2009, at 8:58 AM, Martin Losch wrote:
> >>>
> >>>> Hi Jean-Michel,
> >>>> what a mess. As far as I can see:
> >>>> - offline_exf_seaice fails on Mar19. I think it had not been updated
> >>>> after the seaice_lsr changes, so that is expected. I can't do it and
> >>>> Patrick probably forgot
> >>>> - on Mar19 (later that day) I change defaults to SEAICEadvSalt/
> >>>> Age=.true. This should affect the adjoint of lab_sea (because of the
> >>>> salt tracer), so that is OK, too.
> >>>> - g77+mpi is clearly happening with the seaice_lsr changes, so  
> >>>> that is
> >>>> a problem. I'll have a look at this later today, and maybe we can  
> >>>> talk
> >>>> when you come in (or later in the evening, I had planned to leave  
> >>>> early
> >>>> today to help Heike in the garden). Probably domain boundaries? but
> >>>> they should also affect the non-mpi runs, right?
> >>>> - restart: no idea.
> >>>>
> >>>> Martin
> >>>>
> >>>> On Mar 24, 2009, at 1:40 AM, Jean-Michel Campin wrote:
> >>>>
> >>>>> Martin,
> >>>>>
> >>>>> I added SEAICE_clipVelocities=.TRUE., and this has no effect
> >>>>> on ouput_adm.txt for lab_sea & lab_sea.noseaicedyn
> >>>>> It means that the AD ouput have changed but for an other reason.
> >>>>> Do you know what could it be ?
> >>>>>
> >>>>> And regarding the other things:
> >>>>> - First point is fixed (gfortran stuff).
> >>>>> - Second: with gfortran, seaice_obcs still does not pass the 2+2=4
> >>>>> test
> >>>>> - Third: waiting for suggestions.
> >>>>>
> >>>>> Jean-Michel
> >>>>>
> >>>>> On Mon, Mar 23, 2009 at 06:06:25PM -0400, Jean-Michel Campin wrote:
> >>>>>> Hi Martin,
> >>>>>>
> >>>>>> I am back, and will have a look at this.
> >>>>>>
> >>>>>> I've also noticed that:
> >>>>>> 1) the gfortran lab_sea ad-test fails to compile (but was OK
> >>>>>> before).
> >>>>>> 2) the restart for seaice_obcs is now passing ! do you remember
> >>>>>> fixing something wrong that would improve the restart ?
> >>>>>> 3) lab_sea.lsr & lab_sea.salt_plume are failing (only 3 & 0  
> >>>>>> matching
> >>>>>> digits) with g77+mpi (e.g., on the aces cluster) but were OK  
> >>>>>> before.
> >>>>>> It looks like a problem. Any suggestion is welcome.
> >>>>>>
> >>>>>> Jean-Michel
> >>>>>>
> >>>>>> On Mon, Mar 23, 2009 at 09:21:32AM +0100, Martin Losch wrote:
> >>>>>>> my last default changes (SEAICE_clipVelocities=.false.) broke two
> >>>>>>> adjoint lab_sea experiments again:
> >>>>>>> Y Y Y Y 5> 4<FAIL lab_sea
> >>>>>>> Y Y Y Y 16>16<pass lab_sea.noseaice
> >>>>>>> Y Y Y Y 7> 5<FAIL lab_sea.noseaicedyn
> >>>>>>> Further, the already broken adjoint offline_exf_seaice is also
> >>>>>>> affected.
> >>>>>>> Y Y Y Y 3> 2<FAIL offline_exf_seaice
> >>>>>>>
> >>>>>>> As I don't have access to TAF, it's practically impossible for me
> >>>>>>> to fix
> >>>>>>> this, so could someone with access please do it, either
> >>>>>>> - by resetting the flag (SEAICE_clipVelocities=.true.) in the
> >>>>>>> correct
> >>>>>>> data.seaice file (which one is it?)
> >>>>>>> - or by updating the output.txt files
> >>>>>>> If the first option is chosen, it should noted in data.seaice
> >>>>>>> that this
> >>>>>>> is not a recommended parameter setting.
> >>>>>>> for offline_exf_seaice, it's necessary to update output.txt  
> >>>>>>> anyway.
> >>>>>>> Martin
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> MITgcm-devel mailing list
> >>>>>>> MITgcm-devel at mitgcm.org
> >>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>>>> _______________________________________________
> >>>>>> MITgcm-devel mailing list
> >>>>>> MITgcm-devel at mitgcm.org
> >>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>>> _______________________________________________
> >>>>> MITgcm-devel mailing list
> >>>>> MITgcm-devel at mitgcm.org
> >>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>>
> >>>> _______________________________________________
> >>>> MITgcm-devel mailing list
> >>>> MITgcm-devel at mitgcm.org
> >>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>
> >>> _______________________________________________
> >>> MITgcm-devel mailing list
> >>> MITgcm-devel at mitgcm.org
> >>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >> _______________________________________________
> >> MITgcm-devel mailing list
> >> MITgcm-devel at mitgcm.org
> >> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >
> > _______________________________________________
> > MITgcm-devel mailing list
> > MITgcm-devel at mitgcm.org
> > http://mitgcm.org/mailman/listinfo/mitgcm-devel
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel

More information about the MITgcm-devel mailing list