[MITgcm-devel] seaice
Jean-Michel Campin
jmc at ocean.mit.edu
Thu Mar 26 18:39:52 EDT 2009
Hi,
I think the bug is fixed.
And to find more rapidly this kind of problem, I am going to change
the lab_sea experiment(s), from 2x1 tiles to 2x2 tiles
(+ regenerate all the output).
Jean-Michel
On Thu, Mar 26, 2009 at 05:40:16PM +0100, Martin Losch wrote:
> My stop statement may be ridiculous (I stop, when the error S1 is
> exactly zero; when the update is getting larger between two iterations,
> S1 is set to 0, and in the next iteration, the model continues w ith
> "bad" results. In the next time step the problem usually goes away, but
> not always. I guess this constraint is a bit hard for the first time
> step? On the other hand the solver comes back with UICE>30m/s!), but at
> least it tells me that either the _GLOBAL_MAX_R8 or the EXCH_UV_3D_RL are
> funny. Those two routines are the only differences between 1CPU and 2,
> right? Is it possible that EXCH_UV_3D_RL somehow mixes UICE and VICE, and
> that maybe VICE is not properly initialized?
>
> Martin
>
> On Mar 26, 2009, at 5:17 PM, Jean-Michel Campin wrote:
>
>> Martin,
>>
>> When I run lab_sea.salt_plume (exactly the standard test, without MPI)
>> and add debugLevel=2, it stop immediatly at the 1 iteration
>> with the error message:
>>> u-iteration did not converge
>> and I get:
>>> U lsr iters, error = 4 0.00000000000000E+00
>> in the standard output.
>>
>> If I do the same with MPI on 2CPU, it finishes the 12.iters normally.
>>
>> Does this tell you something ?
>>
>> Jean-Michel
>>
>> On Tue, Mar 24, 2009 at 03:05:05PM -0400, Jean-Michel Campin wrote:
>>> Martin,
>>>
>>> I've started to look at the differences between
>>> 2-cpu and 1-cpu runs for the lab_sea.salt_plume exp.
>>> After the 1rst iteration UICE at i=11 (=tiles junction)
>>> and j=6 is different by 0.24 m/s (this is larger any ice velocity
>>> where there is ice). This confirms that there is a problem.
>>> Will need to dig into the code now.
>>>
>>> Jean-Michel
>>>
>>> On Tue, Mar 24, 2009 at 03:50:05PM +0100, Martin Losch wrote:
>>>> Hi Jean-Michel,
>>>>
>>>> the failing adjoint experiments are "explainable", as I said
>>>> before, and
>>>> as far as I know the gradients are not broken, I just did not
>>>> update the
>>>> results files (because I cannot run the adjoint model from here).
>>>> You can
>>>> call me up (now) and complain directly, so that you don't have to
>>>> be so
>>>> polite (o;
>>>>
>>>> What really worries me is the MPI problem.
>>>>
>>>> Martin
>>>>
>>>> On Mar 24, 2009, at 3:37 PM, Jean-Michel Campin wrote:
>>>>
>>>>> Hi Martin,
>>>>>
>>>>> On Tue, Mar 24, 2009 at 03:03:29PM +0100, Martin Losch wrote:
>>>>>> Me again,
>>>>>>
>>>>>> restart: after rereading your emails I am not sure what has
>>>>>> happened:
>>>>>> does the restart that was originally broken, work now (e.g. did I
>>>>>> accidentally fix that part without knowing)?
>>>>> Well, I though you fix it (pass with pgf), but when I tried with
>>>>> gfortran, it's not passing the 2+2=4 test. In conclusion, it's not
>>>>> really fixed, but before it was not passing neither, so no worry.
>>>>>
>>>>> I am currently trying to figure out the AD test (lab_sea &
>>>>> offline_exf_seaice).
>>>>> One thing that make it more difficult is that the same day
>>>>> (Wed March 18) you checked-in changes + a tag + other changes
>>>>> but no (AD) automatic test was run in between, so ...
>>>>>
>>>>> Will have a look at this mpi stuff later.
>>>>>
>>>>> Jean-Michel
>>>>>
>>>>>>
>>>>>> g77+mpi: I have no idea, why my changes should affect the runs
>>>>>> with
>>>>>> MPI,
>>>>>> because:
>>>>>> 1. the general structure of the code is the same as before. I
>>>>>> did not
>>>>>> add or remove exchanges, global sums, etc
>>>>>> 2. the array boundaries did not change, and even if they did
>>>>>> and I am
>>>>>> now using halo-points that are not properly defined, it would also
>>>>>> be a
>>>>>> problem with non-mpi runs, right?, plus I should see effects along
>>>>>> the
>>>>>> edges, which I don't.
>>>>>> 3. There are new 2D-fields k1/2AtC/Z, which are always zero
>>>>>> along the
>>>>>> edges for non-spherical grids (for I=1-Olx, and sNx+Olx, etc), for
>>>>>> spherical grids (such as lab_sea), they should be defined
>>>>>> correctly
>>>>>> everywhere. Also they used values are I=0,sNx+1,J=0,sNy+1, so that
>>>>>> there
>>>>>> shouldn't be any problems except of Olx/y=1 (which won't work
>>>>>> for the
>>>>>> seaice model anyway).
>>>>>>
>>>>>> I have no clue ...
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mar 24, 2009, at 8:58 AM, Martin Losch wrote:
>>>>>>
>>>>>>> Hi Jean-Michel,
>>>>>>> what a mess. As far as I can see:
>>>>>>> - offline_exf_seaice fails on Mar19. I think it had not been
>>>>>>> updated
>>>>>>> after the seaice_lsr changes, so that is expected. I can't do
>>>>>>> it and
>>>>>>> Patrick probably forgot
>>>>>>> - on Mar19 (later that day) I change defaults to SEAICEadvSalt/
>>>>>>> Age=.true. This should affect the adjoint of lab_sea (because
>>>>>>> of the
>>>>>>> salt tracer), so that is OK, too.
>>>>>>> - g77+mpi is clearly happening with the seaice_lsr changes, so
>>>>>>> that is
>>>>>>> a problem. I'll have a look at this later today, and maybe we can
>>>>>>> talk
>>>>>>> when you come in (or later in the evening, I had planned to leave
>>>>>>> early
>>>>>>> today to help Heike in the garden). Probably domain
>>>>>>> boundaries? but
>>>>>>> they should also affect the non-mpi runs, right?
>>>>>>> - restart: no idea.
>>>>>>>
>>>>>>> Martin
>>>>>>>
>>>>>>> On Mar 24, 2009, at 1:40 AM, Jean-Michel Campin wrote:
>>>>>>>
>>>>>>>> Martin,
>>>>>>>>
>>>>>>>> I added SEAICE_clipVelocities=.TRUE., and this has no effect
>>>>>>>> on ouput_adm.txt for lab_sea & lab_sea.noseaicedyn
>>>>>>>> It means that the AD ouput have changed but for an other reason.
>>>>>>>> Do you know what could it be ?
>>>>>>>>
>>>>>>>> And regarding the other things:
>>>>>>>> - First point is fixed (gfortran stuff).
>>>>>>>> - Second: with gfortran, seaice_obcs still does not pass
>>>>>>>> the 2+2=4
>>>>>>>> test
>>>>>>>> - Third: waiting for suggestions.
>>>>>>>>
>>>>>>>> Jean-Michel
>>>>>>>>
>>>>>>>> On Mon, Mar 23, 2009 at 06:06:25PM -0400, Jean-Michel
>>>>>>>> Campin wrote:
>>>>>>>>> Hi Martin,
>>>>>>>>>
>>>>>>>>> I am back, and will have a look at this.
>>>>>>>>>
>>>>>>>>> I've also noticed that:
>>>>>>>>> 1) the gfortran lab_sea ad-test fails to compile (but was OK
>>>>>>>>> before).
>>>>>>>>> 2) the restart for seaice_obcs is now passing ! do you remember
>>>>>>>>> fixing something wrong that would improve the restart ?
>>>>>>>>> 3) lab_sea.lsr & lab_sea.salt_plume are failing (only 3 & 0
>>>>>>>>> matching
>>>>>>>>> digits) with g77+mpi (e.g., on the aces cluster) but were OK
>>>>>>>>> before.
>>>>>>>>> It looks like a problem. Any suggestion is welcome.
>>>>>>>>>
>>>>>>>>> Jean-Michel
>>>>>>>>>
>>>>>>>>> On Mon, Mar 23, 2009 at 09:21:32AM +0100, Martin Losch wrote:
>>>>>>>>>> my last default changes (SEAICE_clipVelocities=.false.)
>>>>>>>>>> broke two
>>>>>>>>>> adjoint lab_sea experiments again:
>>>>>>>>>> Y Y Y Y 5> 4<FAIL lab_sea
>>>>>>>>>> Y Y Y Y 16>16<pass lab_sea.noseaice
>>>>>>>>>> Y Y Y Y 7> 5<FAIL lab_sea.noseaicedyn
>>>>>>>>>> Further, the already broken adjoint offline_exf_seaice is also
>>>>>>>>>> affected.
>>>>>>>>>> Y Y Y Y 3> 2<FAIL offline_exf_seaice
>>>>>>>>>>
>>>>>>>>>> As I don't have access to TAF, it's practically
>>>>>>>>>> impossible for me
>>>>>>>>>> to fix
>>>>>>>>>> this, so could someone with access please do it, either
>>>>>>>>>> - by resetting the flag (SEAICE_clipVelocities=.true.) in the
>>>>>>>>>> correct
>>>>>>>>>> data.seaice file (which one is it?)
>>>>>>>>>> - or by updating the output.txt files
>>>>>>>>>> If the first option is chosen, it should noted in data.seaice
>>>>>>>>>> that this
>>>>>>>>>> is not a recommended parameter setting.
>>>>>>>>>> for offline_exf_seaice, it's necessary to update output.txt
>>>>>>>>>> anyway.
>>>>>>>>>> Martin
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> MITgcm-devel mailing list
>>>>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>>>>> _______________________________________________
>>>>>>>>> MITgcm-devel mailing list
>>>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>>>> _______________________________________________
>>>>>>>> MITgcm-devel mailing list
>>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> MITgcm-devel mailing list
>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>>
>>>>>> _______________________________________________
>>>>>> MITgcm-devel mailing list
>>>>>> MITgcm-devel at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>> _______________________________________________
>>>>> MITgcm-devel mailing list
>>>>> MITgcm-devel at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>
>>>> _______________________________________________
>>>> MITgcm-devel mailing list
>>>> MITgcm-devel at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>> _______________________________________________
>>> MITgcm-devel mailing list
>>> MITgcm-devel at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel
More information about the MITgcm-devel
mailing list