[MITgcm-support] problems at tile bdys?

Jody Klymak jklymak at uvic.ca
Mon Jun 17 14:54:10 EDT 2013 

OK, I chased this down some more.  In order to get the instability I tried a few things:

1) turned off obcs, removed bathy
2) set dx=const - this had no instabilty
3) re-introduced variable dx on tiles that were 47x16 in size (64x2 tiles for a 3008x32 domain).  My variable dx is constant in the middle and then linearly increasing around the middle. The instability appears on the interface between the first tile where dx is variable to the right of the region where it is constant.
4) Changed the tile size to 48x16 (3072x32) and there are no instabilities.

So, the instability is a combination of using variable dx and odd-sized dimensioned tiles.  Very odd case.  No idea where the problem is in the code.  Its a growing velocity instability that eventually gives NaNs.

Checkpoint: c63o.

I don't particularly need this fixed, I'll just stick with even-sized tiles.  But perplexing nonetheless.

Thanks,   Jody

On Jun 17, 2013, at  8:16 AM, Jody Klymak <jklymak at uvic.ca> wrote:

> 
> Hi Jean-Michel,
> 
> Here is mine.  
> 
> In the meantime I'll work on a more minimal example.  
> 
> While I use obcs, note that my obcs boundary conditions are basically zero forcing for the first 1200 time steps of the model, so this shouldn't cause a problem.  And, of course, the exact same set up works with 64 cores instead of 128.
> 
> Thanks,   Jody
> 
> <tileErr.tgz>
> 
> On Jun 17, 2013, at  5:41 AM, Jean-Michel Campin <jmc at ocean.mit.edu> wrote:
> 
>> Hi Yuan,
>> 
>> Could you proceed with the same recommendation (sending tar file of dir with 
>> modified/customized source file and all data* files for this set-up)
>> or is there too much source-code involved ?
>> 
>> If we are lucky, it could be the same Pb and then will be easier to identify 
>> with 2 set-up (instead of just 1). Otherwise, might have 2 separate Pb to fix ...
>> 
>> Cheers,
>> Jean-Michel
>> 
>> On Sun, Jun 16, 2013 at 09:35:08PM -0700, Yuan Lian wrote:
>>> Hi Jean-Michel,
>>> 
>>> I found the similar behavior of MITgcm when there are multiple tiles in x direction. The issue is particularly obvious when active tracers are enabled, e.g. water cycle, where I can see the periodical oscillation of tracers including temperature at the boundaries between the tiles. The problem is less apparent in y direction. I am wondering if it has something to do with exchange routines in x direction. The simulations are done in 3d channel. Thanks.
>>> 
>>> Best,
>>> Yuan 
>>> 
>>> Jean-Michel Campin <jmc at ocean.mit.edu> wrote:
>>> 
>>>> Hi Jody,
>>>> 
>>>> Could you send (in a tar file) the directory that contains all the 
>>>> modified/customized files you are using (the argument of "genmake2
>>>> -mods" command)
>>>> as well as all the parameter files (all data* files) for this set-up ?
>>>> Also, if you are not using the latest version of MITgcm code, which
>>>> version 
>>>> (checkpointXYz ? or time of the latest "cvs update") are you using ?
>>>> 
>>>> It's possible that we can find the problem like this; otherwise, will
>>>> have 
>>>> to try to reproduce it (and in this case will also need the binary
>>>> input files).
>>>> 
>>>> Cheers,
>>>> Jean-Michel
>>>> 
>>>> On Sat, Jun 15, 2013 at 11:03:30PM -0700, Jody Klymak wrote:
>>>>> 
>>>>> Hi All,
>>>>> 
>>>>> I ran a simulation in a 2-D channel but three-D in setup, and all ran
>>>> fine.  Well except it stayed 2-D because there was nothing to cause
>>>> there to be spanwise motion.  The simulations are nonhydrostatic.  
>>>>> 
>>>>> So, I added a very small amount of three d by a few different
>>>> methods, and everytime, no matter how I set up the three dimensionality
>>>> (bottom roughness, initial velocities, surface displacements) there
>>>> would an instability at the same spot in x.  It turns out it is a tile
>>>> boundary in the middle of the domain.  
>>>>> 
>>>>> I'm using advection scheme 33, and chnaged OLx and OLy to 5, but
>>>> still the same problem.  I tried a few other advection schemes, with
>>>> the same issue (it happens in a hundred or so timesteps, so its easy to
>>>> debug, at least).  I also tried upping the iterations on the CG3d
>>>> solver, but that didn't help at all.
>>>>> 
>>>>> I've now gone back to the same setup I started with, but changed by
>>>> tile layout (half as many tiles), and the instability is gone.  
>>>>> 
>>>>> Has anybody run into this?  Any suggestions for a fix or what to look
>>>> for?  (I guess the stupid fix is not to use that tile configuration,
>>>> but that seems flakey).
>>>>> 
>>>>> Thanks,   Jody
>>>>> 
>>>>> 
>>>>> --
>>>>> Jody Klymak    
>>>>> http://web.uvic.ca/~jklymak/
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> MITgcm-support mailing list
>>>>> MITgcm-support at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>> 
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>> 
>>> -- 
>>> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>> 
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>> 
>> 
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
> 
> --
> Jody Klymak    
> http://web.uvic.ca/~jklymak/
> 
> 
> 
> 
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support

--
Jody Klymak    
http://web.uvic.ca/~jklymak/

More information about the MITgcm-support mailing list