[MITgcm-devel] seaice
Martin Losch
Martin.Losch at awi.de
Fri Mar 27 03:53:24 EDT 2009
Dear Jean-Michel,
I think, I owe you a few drinks! What a stupid mistake and what a nice
catch; I have no idea how that could happen. Thanks.
Martin
On Mar 26, 2009, at 11:39 PM, Jean-Michel Campin wrote:
> Hi,
>
> I think the bug is fixed.
>
> And to find more rapidly this kind of problem, I am going to change
> the lab_sea experiment(s), from 2x1 tiles to 2x2 tiles
> (+ regenerate all the output).
>
> Jean-Michel
>
> On Thu, Mar 26, 2009 at 05:40:16PM +0100, Martin Losch wrote:
>> My stop statement may be ridiculous (I stop, when the error S1 is
>> exactly zero; when the update is getting larger between two
>> iterations,
>> S1 is set to 0, and in the next iteration, the model continues w ith
>> "bad" results. In the next time step the problem usually goes away,
>> but
>> not always. I guess this constraint is a bit hard for the first time
>> step? On the other hand the solver comes back with UICE>30m/s!),
>> but at
>> least it tells me that either the _GLOBAL_MAX_R8 or the
>> EXCH_UV_3D_RL are
>> funny. Those two routines are the only differences between 1CPU and
>> 2,
>> right? Is it possible that EXCH_UV_3D_RL somehow mixes UICE and
>> VICE, and
>> that maybe VICE is not properly initialized?
>>
>> Martin
>>
>> On Mar 26, 2009, at 5:17 PM, Jean-Michel Campin wrote:
>>
>>> Martin,
>>>
>>> When I run lab_sea.salt_plume (exactly the standard test, without
>>> MPI)
>>> and add debugLevel=2, it stop immediatly at the 1 iteration
>>> with the error message:
>>>> u-iteration did not converge
>>> and I get:
>>>> U lsr iters, error = 4 0.00000000000000E+00
>>> in the standard output.
>>>
>>> If I do the same with MPI on 2CPU, it finishes the 12.iters
>>> normally.
>>>
>>> Does this tell you something ?
>>>
>>> Jean-Michel
>>>
>>> On Tue, Mar 24, 2009 at 03:05:05PM -0400, Jean-Michel Campin wrote:
>>>> Martin,
>>>>
>>>> I've started to look at the differences between
>>>> 2-cpu and 1-cpu runs for the lab_sea.salt_plume exp.
>>>> After the 1rst iteration UICE at i=11 (=tiles junction)
>>>> and j=6 is different by 0.24 m/s (this is larger any ice velocity
>>>> where there is ice). This confirms that there is a problem.
>>>> Will need to dig into the code now.
>>>>
>>>> Jean-Michel
>>>>
>>>> On Tue, Mar 24, 2009 at 03:50:05PM +0100, Martin Losch wrote:
>>>>> Hi Jean-Michel,
>>>>>
>>>>> the failing adjoint experiments are "explainable", as I said
>>>>> before, and
>>>>> as far as I know the gradients are not broken, I just did not
>>>>> update the
>>>>> results files (because I cannot run the adjoint model from here).
>>>>> You can
>>>>> call me up (now) and complain directly, so that you don't have to
>>>>> be so
>>>>> polite (o;
>>>>>
>>>>> What really worries me is the MPI problem.
>>>>>
>>>>> Martin
>>>>>
>>>>> On Mar 24, 2009, at 3:37 PM, Jean-Michel Campin wrote:
>>>>>
>>>>>> Hi Martin,
>>>>>>
>>>>>> On Tue, Mar 24, 2009 at 03:03:29PM +0100, Martin Losch wrote:
>>>>>>> Me again,
>>>>>>>
>>>>>>> restart: after rereading your emails I am not sure what has
>>>>>>> happened:
>>>>>>> does the restart that was originally broken, work now (e.g.
>>>>>>> did I
>>>>>>> accidentally fix that part without knowing)?
>>>>>> Well, I though you fix it (pass with pgf), but when I tried with
>>>>>> gfortran, it's not passing the 2+2=4 test. In conclusion, it's
>>>>>> not
>>>>>> really fixed, but before it was not passing neither, so no worry.
>>>>>>
>>>>>> I am currently trying to figure out the AD test (lab_sea &
>>>>>> offline_exf_seaice).
>>>>>> One thing that make it more difficult is that the same day
>>>>>> (Wed March 18) you checked-in changes + a tag + other changes
>>>>>> but no (AD) automatic test was run in between, so ...
>>>>>>
>>>>>> Will have a look at this mpi stuff later.
>>>>>>
>>>>>> Jean-Michel
>>>>>>
>>>>>>>
>>>>>>> g77+mpi: I have no idea, why my changes should affect the runs
>>>>>>> with
>>>>>>> MPI,
>>>>>>> because:
>>>>>>> 1. the general structure of the code is the same as before. I
>>>>>>> did not
>>>>>>> add or remove exchanges, global sums, etc
>>>>>>> 2. the array boundaries did not change, and even if they did
>>>>>>> and I am
>>>>>>> now using halo-points that are not properly defined, it would
>>>>>>> also
>>>>>>> be a
>>>>>>> problem with non-mpi runs, right?, plus I should see effects
>>>>>>> along
>>>>>>> the
>>>>>>> edges, which I don't.
>>>>>>> 3. There are new 2D-fields k1/2AtC/Z, which are always zero
>>>>>>> along the
>>>>>>> edges for non-spherical grids (for I=1-Olx, and sNx+Olx, etc),
>>>>>>> for
>>>>>>> spherical grids (such as lab_sea), they should be defined
>>>>>>> correctly
>>>>>>> everywhere. Also they used values are I=0,sNx+1,J=0,sNy+1, so
>>>>>>> that
>>>>>>> there
>>>>>>> shouldn't be any problems except of Olx/y=1 (which won't work
>>>>>>> for the
>>>>>>> seaice model anyway).
>>>>>>>
>>>>>>> I have no clue ...
>>>>>>>
>>>>>>> Martin
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mar 24, 2009, at 8:58 AM, Martin Losch wrote:
>>>>>>>
>>>>>>>> Hi Jean-Michel,
>>>>>>>> what a mess. As far as I can see:
>>>>>>>> - offline_exf_seaice fails on Mar19. I think it had not been
>>>>>>>> updated
>>>>>>>> after the seaice_lsr changes, so that is expected. I can't do
>>>>>>>> it and
>>>>>>>> Patrick probably forgot
>>>>>>>> - on Mar19 (later that day) I change defaults to SEAICEadvSalt/
>>>>>>>> Age=.true. This should affect the adjoint of lab_sea (because
>>>>>>>> of the
>>>>>>>> salt tracer), so that is OK, too.
>>>>>>>> - g77+mpi is clearly happening with the seaice_lsr changes, so
>>>>>>>> that is
>>>>>>>> a problem. I'll have a look at this later today, and maybe we
>>>>>>>> can
>>>>>>>> talk
>>>>>>>> when you come in (or later in the evening, I had planned to
>>>>>>>> leave
>>>>>>>> early
>>>>>>>> today to help Heike in the garden). Probably domain
>>>>>>>> boundaries? but
>>>>>>>> they should also affect the non-mpi runs, right?
>>>>>>>> - restart: no idea.
>>>>>>>>
>>>>>>>> Martin
>>>>>>>>
>>>>>>>> On Mar 24, 2009, at 1:40 AM, Jean-Michel Campin wrote:
>>>>>>>>
>>>>>>>>> Martin,
>>>>>>>>>
>>>>>>>>> I added SEAICE_clipVelocities=.TRUE., and this has no effect
>>>>>>>>> on ouput_adm.txt for lab_sea & lab_sea.noseaicedyn
>>>>>>>>> It means that the AD ouput have changed but for an other
>>>>>>>>> reason.
>>>>>>>>> Do you know what could it be ?
>>>>>>>>>
>>>>>>>>> And regarding the other things:
>>>>>>>>> - First point is fixed (gfortran stuff).
>>>>>>>>> - Second: with gfortran, seaice_obcs still does not pass
>>>>>>>>> the 2+2=4
>>>>>>>>> test
>>>>>>>>> - Third: waiting for suggestions.
>>>>>>>>>
>>>>>>>>> Jean-Michel
>>>>>>>>>
>>>>>>>>> On Mon, Mar 23, 2009 at 06:06:25PM -0400, Jean-Michel
>>>>>>>>> Campin wrote:
>>>>>>>>>> Hi Martin,
>>>>>>>>>>
>>>>>>>>>> I am back, and will have a look at this.
>>>>>>>>>>
>>>>>>>>>> I've also noticed that:
>>>>>>>>>> 1) the gfortran lab_sea ad-test fails to compile (but was OK
>>>>>>>>>> before).
>>>>>>>>>> 2) the restart for seaice_obcs is now passing ! do you
>>>>>>>>>> remember
>>>>>>>>>> fixing something wrong that would improve the restart ?
>>>>>>>>>> 3) lab_sea.lsr & lab_sea.salt_plume are failing (only 3 & 0
>>>>>>>>>> matching
>>>>>>>>>> digits) with g77+mpi (e.g., on the aces cluster) but were OK
>>>>>>>>>> before.
>>>>>>>>>> It looks like a problem. Any suggestion is welcome.
>>>>>>>>>>
>>>>>>>>>> Jean-Michel
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 23, 2009 at 09:21:32AM +0100, Martin Losch wrote:
>>>>>>>>>>> my last default changes (SEAICE_clipVelocities=.false.)
>>>>>>>>>>> broke two
>>>>>>>>>>> adjoint lab_sea experiments again:
>>>>>>>>>>> Y Y Y Y 5> 4<FAIL lab_sea
>>>>>>>>>>> Y Y Y Y 16>16<pass lab_sea.noseaice
>>>>>>>>>>> Y Y Y Y 7> 5<FAIL lab_sea.noseaicedyn
>>>>>>>>>>> Further, the already broken adjoint offline_exf_seaice is
>>>>>>>>>>> also
>>>>>>>>>>> affected.
>>>>>>>>>>> Y Y Y Y 3> 2<FAIL offline_exf_seaice
>>>>>>>>>>>
>>>>>>>>>>> As I don't have access to TAF, it's practically
>>>>>>>>>>> impossible for me
>>>>>>>>>>> to fix
>>>>>>>>>>> this, so could someone with access please do it, either
>>>>>>>>>>> - by resetting the flag (SEAICE_clipVelocities=.true.) in
>>>>>>>>>>> the
>>>>>>>>>>> correct
>>>>>>>>>>> data.seaice file (which one is it?)
>>>>>>>>>>> - or by updating the output.txt files
>>>>>>>>>>> If the first option is chosen, it should noted in
>>>>>>>>>>> data.seaice
>>>>>>>>>>> that this
>>>>>>>>>>> is not a recommended parameter setting.
>>>>>>>>>>> for offline_exf_seaice, it's necessary to update output.txt
>>>>>>>>>>> anyway.
>>>>>>>>>>> Martin
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> MITgcm-devel mailing list
>>>>>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>>>>>> _______________________________________________
>>>>>>>>>> MITgcm-devel mailing list
>>>>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>>>>> _______________________________________________
>>>>>>>>> MITgcm-devel mailing list
>>>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> MITgcm-devel mailing list
>>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> MITgcm-devel mailing list
>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>> _______________________________________________
>>>>>> MITgcm-devel mailing list
>>>>>> MITgcm-devel at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>
>>>>> _______________________________________________
>>>>> MITgcm-devel mailing list
>>>>> MITgcm-devel at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>> _______________________________________________
>>>> MITgcm-devel mailing list
>>>> MITgcm-devel at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>> _______________________________________________
>>> MITgcm-devel mailing list
>>> MITgcm-devel at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel
More information about the MITgcm-devel
mailing list