[MITgcm-support] problems at tile bdys?
Jody Klymak
jklymak at uvic.ca
Mon Jun 17 20:29:43 EDT 2013
Hi Jean-Michel
Thanks for your thoughts on this.
On Jun 17, 2013, at 15:44 PM, Jean-Michel Campin <jmc at ocean.mit.edu> wrote:
> I had a quick look at the files you sent, but did not see
> anything obviously problematic.
> It's not a small size set-up, but if the problem is there without OBCS,
> might be possible to put the binary input files somewhere so that
> we can try to reproduce the problem.
The problem persists in an 8x2 tile set of runs, so I've attached that, binary files and all. Let me know if you need my compile options too.
If I run with nx=48, then I do not get an instability. If I run with nx=47, I do (see attached PDFs). If I run with exch2, the instability is different, but still there.
Again, not dying to see this fixed, but it is an odd glitch.
Cheers, Jody
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Nx48.pdf
Type: application/pdf
Size: 15286 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20130617/0c6916a9/attachment-0003.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Nx47.pdf
Type: application/pdf
Size: 13802 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20130617/0c6916a9/attachment-0004.pdf>
-------------- next part --------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Nx47Exch2.pdf
Type: application/pdf
Size: 14917 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20130617/0c6916a9/attachment-0005.pdf>
-------------- next part --------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tileErr2.tgz
Type: application/octet-stream
Size: 145077 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20130617/0c6916a9/attachment-0001.obj>
-------------- next part --------------
> One thing you could try, would be to test with pkg/exch2:
> Just need to compile with it (add "exch2" to packages.conf),
> there is no run-time parameter to turn it on (compiled => used)
> and see if it changes something.
>
> Thanks,
> Jean-Michel
>
> On Mon, Jun 17, 2013 at 11:54:10AM -0700, Jody Klymak wrote:
>>
>> OK, I chased this down some more. In order to get the instability I tried a few things:
>>
>> 1) turned off obcs, removed bathy
>> 2) set dx=const - this had no instabilty
>> 3) re-introduced variable dx on tiles that were 47x16 in size (64x2 tiles for a 3008x32 domain). My variable dx is constant in the middle and then linearly increasing around the middle. The instability appears on the interface between the first tile where dx is variable to the right of the region where it is constant.
>> 4) Changed the tile size to 48x16 (3072x32) and there are no instabilities.
>>
>> So, the instability is a combination of using variable dx and odd-sized dimensioned tiles. Very odd case. No idea where the problem is in the code. Its a growing velocity instability that eventually gives NaNs.
>>
>> Checkpoint: c63o.
>>
>> I don't particularly need this fixed, I'll just stick with even-sized tiles. But perplexing nonetheless.
>>
>> Thanks, Jody
>>
>> On Jun 17, 2013, at 8:16 AM, Jody Klymak <jklymak at uvic.ca> wrote:
>>
>>>
>>> Hi Jean-Michel,
>>>
>>> Here is mine.
>>>
>>> In the meantime I'll work on a more minimal example.
>>>
>>> While I use obcs, note that my obcs boundary conditions are basically zero forcing for the first 1200 time steps of the model, so this shouldn't cause a problem. And, of course, the exact same set up works with 64 cores instead of 128.
>>>
>>> Thanks, Jody
>>>
>>> <tileErr.tgz>
>>>
>>> On Jun 17, 2013, at 5:41 AM, Jean-Michel Campin <jmc at ocean.mit.edu> wrote:
>>>
>>>> Hi Yuan,
>>>>
>>>> Could you proceed with the same recommendation (sending tar file of dir with
>>>> modified/customized source file and all data* files for this set-up)
>>>> or is there too much source-code involved ?
>>>>
>>>> If we are lucky, it could be the same Pb and then will be easier to identify
>>>> with 2 set-up (instead of just 1). Otherwise, might have 2 separate Pb to fix ...
>>>>
>>>> Cheers,
>>>> Jean-Michel
>>>>
>>>> On Sun, Jun 16, 2013 at 09:35:08PM -0700, Yuan Lian wrote:
>>>>> Hi Jean-Michel,
>>>>>
>>>>> I found the similar behavior of MITgcm when there are multiple tiles in x direction. The issue is particularly obvious when active tracers are enabled, e.g. water cycle, where I can see the periodical oscillation of tracers including temperature at the boundaries between the tiles. The problem is less apparent in y direction. I am wondering if it has something to do with exchange routines in x direction. The simulations are done in 3d channel. Thanks.
>>>>>
>>>>> Best,
>>>>> Yuan
>>>>>
>>>>> Jean-Michel Campin <jmc at ocean.mit.edu> wrote:
>>>>>
>>>>>> Hi Jody,
>>>>>>
>>>>>> Could you send (in a tar file) the directory that contains all the
>>>>>> modified/customized files you are using (the argument of "genmake2
>>>>>> -mods" command)
>>>>>> as well as all the parameter files (all data* files) for this set-up ?
>>>>>> Also, if you are not using the latest version of MITgcm code, which
>>>>>> version
>>>>>> (checkpointXYz ? or time of the latest "cvs update") are you using ?
>>>>>>
>>>>>> It's possible that we can find the problem like this; otherwise, will
>>>>>> have
>>>>>> to try to reproduce it (and in this case will also need the binary
>>>>>> input files).
>>>>>>
>>>>>> Cheers,
>>>>>> Jean-Michel
>>>>>>
>>>>>> On Sat, Jun 15, 2013 at 11:03:30PM -0700, Jody Klymak wrote:
>>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I ran a simulation in a 2-D channel but three-D in setup, and all ran
>>>>>> fine. Well except it stayed 2-D because there was nothing to cause
>>>>>> there to be spanwise motion. The simulations are nonhydrostatic.
>>>>>>>
>>>>>>> So, I added a very small amount of three d by a few different
>>>>>> methods, and everytime, no matter how I set up the three dimensionality
>>>>>> (bottom roughness, initial velocities, surface displacements) there
>>>>>> would an instability at the same spot in x. It turns out it is a tile
>>>>>> boundary in the middle of the domain.
>>>>>>>
>>>>>>> I'm using advection scheme 33, and chnaged OLx and OLy to 5, but
>>>>>> still the same problem. I tried a few other advection schemes, with
>>>>>> the same issue (it happens in a hundred or so timesteps, so its easy to
>>>>>> debug, at least). I also tried upping the iterations on the CG3d
>>>>>> solver, but that didn't help at all.
>>>>>>>
>>>>>>> I've now gone back to the same setup I started with, but changed by
>>>>>> tile layout (half as many tiles), and the instability is gone.
>>>>>>>
>>>>>>> Has anybody run into this? Any suggestions for a fix or what to look
>>>>>> for? (I guess the stupid fix is not to use that tile configuration,
>>>>>> but that seems flakey).
>>>>>>>
>>>>>>> Thanks, Jody
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Jody Klymak
>>>>>>> http://web.uvic.ca/~jklymak/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> MITgcm-support mailing list
>>>>>>> MITgcm-support at mitgcm.org
>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>>>
>>>>>> _______________________________________________
>>>>>> MITgcm-support mailing list
>>>>>> MITgcm-support at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>>
>>>>> --
>>>>> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>>>>
>>>>> _______________________________________________
>>>>> MITgcm-support mailing list
>>>>> MITgcm-support at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>
>>>>
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>
>>> --
>>> Jody Klymak
>>> http://web.uvic.ca/~jklymak/
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>
>> --
>> Jody Klymak
>> http://web.uvic.ca/~jklymak/
>>
>>
>>
>>
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support
--
Jody Klymak
http://web.uvic.ca/~jklymak/
More information about the MITgcm-support
mailing list