[MITgcm-devel] seaice

Martin Losch Martin.Losch at awi.de
Thu Mar 26 12:40:16 EDT 2009 

My stop statement may be ridiculous (I stop, when the error S1 is  
exactly zero; when the update is getting larger between two  
iterations, S1 is set to 0, and in the next iteration, the model  
continues w ith "bad" results. In the next time step the problem  
usually goes away, but not always. I guess this constraint is a bit  
hard for the first time step? On the other hand the solver comes back  
with UICE>30m/s!), but at least it tells me that either the  
_GLOBAL_MAX_R8 or the EXCH_UV_3D_RL are funny. Those two routines are  
the only differences between 1CPU and 2, right? Is it possible that  
EXCH_UV_3D_RL somehow mixes UICE and VICE, and that maybe VICE is not  
properly initialized?

Martin

On Mar 26, 2009, at 5:17 PM, Jean-Michel Campin wrote:

> Martin,
>
> When I run lab_sea.salt_plume (exactly the standard test, without MPI)
> and add debugLevel=2, it stop immediatly at the 1 iteration
> with the error message:
>> u-iteration did not converge
> and I get:
>> U lsr iters, error =      4  0.00000000000000E+00
> in the standard output.
>
> If I do the same with MPI on 2CPU, it finishes the 12.iters normally.
>
> Does this tell you something ?
>
> Jean-Michel
>
> On Tue, Mar 24, 2009 at 03:05:05PM -0400, Jean-Michel Campin wrote:
>> Martin,
>>
>> I've started to look at the differences between
>> 2-cpu and 1-cpu runs for the lab_sea.salt_plume exp.
>> After the 1rst iteration UICE at i=11 (=tiles junction)
>> and j=6 is different by 0.24 m/s (this is larger any ice velocity
>> where there is ice). This confirms that there is a problem.
>> Will need to dig into the code now.
>>
>> Jean-Michel
>>
>> On Tue, Mar 24, 2009 at 03:50:05PM +0100, Martin Losch wrote:
>>> Hi Jean-Michel,
>>>
>>> the failing adjoint experiments are "explainable", as I said  
>>> before, and
>>> as far as I know the gradients are not broken, I just did not  
>>> update the
>>> results files (because I cannot run the adjoint model from here).  
>>> You can
>>> call me up (now) and complain directly, so that you don't have to  
>>> be so
>>> polite (o;
>>>
>>> What really worries me is the MPI problem.
>>>
>>> Martin
>>>
>>> On Mar 24, 2009, at 3:37 PM, Jean-Michel Campin wrote:
>>>
>>>> Hi Martin,
>>>>
>>>> On Tue, Mar 24, 2009 at 03:03:29PM +0100, Martin Losch wrote:
>>>>> Me again,
>>>>>
>>>>> restart: after rereading your emails I am not sure what has  
>>>>> happened:
>>>>> does the restart that was originally broken, work now (e.g. did I
>>>>> accidentally fix that part without knowing)?
>>>> Well, I though you fix it (pass with pgf), but when I tried with
>>>> gfortran, it's not passing the 2+2=4 test. In conclusion, it's not
>>>> really fixed, but before it was not passing neither, so no worry.
>>>>
>>>> I am currently trying to figure out the AD test (lab_sea &
>>>> offline_exf_seaice).
>>>> One thing that make it more difficult is that the same day
>>>> (Wed March 18) you checked-in changes + a tag + other changes
>>>> but no (AD) automatic test was run in between, so ...
>>>>
>>>> Will have a look at this mpi stuff later.
>>>>
>>>> Jean-Michel
>>>>
>>>>>
>>>>> g77+mpi: I have no idea, why my changes should affect the runs  
>>>>> with
>>>>> MPI,
>>>>> because:
>>>>> 1. the general structure of the code is the same as before. I  
>>>>> did not
>>>>> add or remove exchanges, global sums, etc
>>>>> 2. the array boundaries did not change, and even if they did and  
>>>>> I am
>>>>> now using halo-points that are not properly defined, it would also
>>>>> be a
>>>>> problem with non-mpi runs, right?, plus I should see effects along
>>>>> the
>>>>> edges, which I don't.
>>>>> 3. There are new 2D-fields k1/2AtC/Z, which are always zero  
>>>>> along the
>>>>> edges for non-spherical grids (for I=1-Olx, and sNx+Olx, etc), for
>>>>> spherical grids (such as lab_sea), they should be defined  
>>>>> correctly
>>>>> everywhere. Also they used values are I=0,sNx+1,J=0,sNy+1, so that
>>>>> there
>>>>> shouldn't be any problems except of Olx/y=1 (which won't work  
>>>>> for the
>>>>> seaice model anyway).
>>>>>
>>>>> I have no clue ...
>>>>>
>>>>> Martin
>>>>>
>>>>>
>>>>>
>>>>> On Mar 24, 2009, at 8:58 AM, Martin Losch wrote:
>>>>>
>>>>>> Hi Jean-Michel,
>>>>>> what a mess. As far as I can see:
>>>>>> - offline_exf_seaice fails on Mar19. I think it had not been  
>>>>>> updated
>>>>>> after the seaice_lsr changes, so that is expected. I can't do  
>>>>>> it and
>>>>>> Patrick probably forgot
>>>>>> - on Mar19 (later that day) I change defaults to SEAICEadvSalt/
>>>>>> Age=.true. This should affect the adjoint of lab_sea (because  
>>>>>> of the
>>>>>> salt tracer), so that is OK, too.
>>>>>> - g77+mpi is clearly happening with the seaice_lsr changes, so
>>>>>> that is
>>>>>> a problem. I'll have a look at this later today, and maybe we can
>>>>>> talk
>>>>>> when you come in (or later in the evening, I had planned to leave
>>>>>> early
>>>>>> today to help Heike in the garden). Probably domain boundaries?  
>>>>>> but
>>>>>> they should also affect the non-mpi runs, right?
>>>>>> - restart: no idea.
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>> On Mar 24, 2009, at 1:40 AM, Jean-Michel Campin wrote:
>>>>>>
>>>>>>> Martin,
>>>>>>>
>>>>>>> I added SEAICE_clipVelocities=.TRUE., and this has no effect
>>>>>>> on ouput_adm.txt for lab_sea & lab_sea.noseaicedyn
>>>>>>> It means that the AD ouput have changed but for an other reason.
>>>>>>> Do you know what could it be ?
>>>>>>>
>>>>>>> And regarding the other things:
>>>>>>> - First point is fixed (gfortran stuff).
>>>>>>> - Second: with gfortran, seaice_obcs still does not pass the  
>>>>>>> 2+2=4
>>>>>>> test
>>>>>>> - Third: waiting for suggestions.
>>>>>>>
>>>>>>> Jean-Michel
>>>>>>>
>>>>>>> On Mon, Mar 23, 2009 at 06:06:25PM -0400, Jean-Michel Campin  
>>>>>>> wrote:
>>>>>>>> Hi Martin,
>>>>>>>>
>>>>>>>> I am back, and will have a look at this.
>>>>>>>>
>>>>>>>> I've also noticed that:
>>>>>>>> 1) the gfortran lab_sea ad-test fails to compile (but was OK
>>>>>>>> before).
>>>>>>>> 2) the restart for seaice_obcs is now passing ! do you remember
>>>>>>>> fixing something wrong that would improve the restart ?
>>>>>>>> 3) lab_sea.lsr & lab_sea.salt_plume are failing (only 3 & 0
>>>>>>>> matching
>>>>>>>> digits) with g77+mpi (e.g., on the aces cluster) but were OK
>>>>>>>> before.
>>>>>>>> It looks like a problem. Any suggestion is welcome.
>>>>>>>>
>>>>>>>> Jean-Michel
>>>>>>>>
>>>>>>>> On Mon, Mar 23, 2009 at 09:21:32AM +0100, Martin Losch wrote:
>>>>>>>>> my last default changes (SEAICE_clipVelocities=.false.)  
>>>>>>>>> broke two
>>>>>>>>> adjoint lab_sea experiments again:
>>>>>>>>> Y Y Y Y 5> 4<FAIL lab_sea
>>>>>>>>> Y Y Y Y 16>16<pass lab_sea.noseaice
>>>>>>>>> Y Y Y Y 7> 5<FAIL lab_sea.noseaicedyn
>>>>>>>>> Further, the already broken adjoint offline_exf_seaice is also
>>>>>>>>> affected.
>>>>>>>>> Y Y Y Y 3> 2<FAIL offline_exf_seaice
>>>>>>>>>
>>>>>>>>> As I don't have access to TAF, it's practically impossible  
>>>>>>>>> for me
>>>>>>>>> to fix
>>>>>>>>> this, so could someone with access please do it, either
>>>>>>>>> - by resetting the flag (SEAICE_clipVelocities=.true.) in the
>>>>>>>>> correct
>>>>>>>>> data.seaice file (which one is it?)
>>>>>>>>> - or by updating the output.txt files
>>>>>>>>> If the first option is chosen, it should noted in data.seaice
>>>>>>>>> that this
>>>>>>>>> is not a recommended parameter setting.
>>>>>>>>> for offline_exf_seaice, it's necessary to update output.txt
>>>>>>>>> anyway.
>>>>>>>>> Martin
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> MITgcm-devel mailing list
>>>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>>>> _______________________________________________
>>>>>>>> MITgcm-devel mailing list
>>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>>> _______________________________________________
>>>>>>> MITgcm-devel mailing list
>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>>
>>>>>> _______________________________________________
>>>>>> MITgcm-devel mailing list
>>>>>> MITgcm-devel at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>
>>>>> _______________________________________________
>>>>> MITgcm-devel mailing list
>>>>> MITgcm-devel at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>> _______________________________________________
>>>> MITgcm-devel mailing list
>>>> MITgcm-devel at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>
>>> _______________________________________________
>>> MITgcm-devel mailing list
>>> MITgcm-devel at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel

More information about the MITgcm-devel mailing list