[MITgcm-devel] Re: TARGET_NEC_SX in ini_curvilinear_grid.F

Martin Losch Martin.Losch at awi.de
Mon Feb 16 03:42:26 EST 2009 

Hi Jean-Michel,
both you suggestion in this email and the fix that you checked in  
work, so I guess now this part is OK. Thanks.

However, I have another problem (that I almost forgot about). In my  
2cpu configuration (nPx=1,nPy=2,nSx=1,nSy=1) without EXCH2 it appears  
that only tile001.mitgrid is read:
> sx8::test> grep tile00* STDOUT.*
> STDOUT.0000:(PID.TID 0000.0001) tile:   1 ; Read from file  
> tile001.mitgrid
> STDOUT.0001:(PID.TID 0001.0001) tile:   1 ; Read from file  
> tile001.mitgrid
with the obvious consequences for the grid. (np2=1 and nsy=2 does not  
work either). It looks like the code is set up to read only cubed  
sphere faces (or domains with tiles in X only). Maybe it's a good idea  
to catch that case (#ifndef OLD_GRID_IO and if (sny*npy>1) then STOP)?

Martin



On Feb 13, 2009, at 5:25 PM, Jean-Michel Campin wrote:

> Hi Martin,
>
> Could you try to modify this part of ini_curvilinear_grid.F :
> 395       CALL EXCH_Z_3D_RS( yG, 1, myThid )
> 396 #ifdef ALLOW_EXCH2
> 397 # ifndef ALLOW_AUTODIFF_TAMC
> 398       CALL EXCH2_UV_BGRID_3D_RS( dxV, dyU, .FALSE., 1, myThid)
> 399 # endif
> 400 #endif
> 401       CALL EXCH_Z_3D_RS( rAz, 1, myThid )
>
> In the following way :
> 395       CALL EXCH_Z_3D_RS( yG, 1, myThid )
> 396       IF ( .NOT.useCubedSphereExchange) THEN
> 397         CALL EXCH_XY_RS( dxV, myThid )
> 398         CALL EXCH_XY_RS( dyU, myThid )
> 399       ELSE
> 400 #ifdef ALLOW_EXCH2
> 401 # ifndef ALLOW_AUTODIFF_TAMC
> 402       CALL EXCH2_UV_BGRID_3D_RS( dxV, dyU, .FALSE., 1, myThid)
> 403 # endif
> 404 #endif
> 405       ENDIF
> 406       CALL EXCH_Z_3D_RS( rAz, 1, myThid )
>
> And see if it fix the problem.
> In the mean time, I will test a EXCH_UV_BGRID_3D (not EXCH2)
> and add it to eesupp/src.
>
> Thanks,
> Jean-Michel
>
> On Fri, Feb 13, 2009 at 12:27:21PM +0100, Martin Losch wrote:
>> Hi Jean-Michel,
>>
>> It's not so easy. I think the problem is appears only with  
>> OLD_GRID_IO
>> undefined, because in that case the values for dyU/dxV read from
>> tile*.mitgrid are not exchanged, only when EXCH2 is used. With
>> OLD_GRID_IO defined (which I normally use), there is an exchange of
>> dyU/dxV and corners are fixed afterwards.
>>
>> The problem shows up when I use a curvilinear grid, that is *NOT* a
>> cubed sphere, that's why I don't use EXCH2 (and yes, I do not have
>> useAreaViscLength=T in my standard experiments, with this parameter  
>> set
>> to true it works), *AND* when I have #undef OLD_GRID_IO (because  
>> somehow
>> I couldn't make the model read two files for two tiles, but that's a
>> different story)
>>
>> But when I make mistake 1 dyU and dxV are zero in the entire halos,  
>> then
>> recip_dyU/dxV are also zero and for the computation of L2rdt (and
>> similar terms) we use the inverse of recip_dyU/dxV. That's what  
>> causes
>> the problem.
>>
>> I think, there should be catch in ini_curvilinear grid, that warns  
>> about
>> not exchangin dxV/dyU, or this should be fixed, and then we can  
>> remove
>> the hack for SX8.
>>
>> Martin
>>
>> On Feb 13, 2009, at 5:28 AM, Jean-Michel Campin wrote:
>>
>>> I only realised now that I send this to support list instead of  
>>> devel.
>>> Sorry.
>>> Jean-Michel
>>>
>>> On Thu, Feb 12, 2009 at 04:05:56PM -0500, Jean-Michel Campin wrote:
>>>> Hi Martin,
>>>>
>>>> I am still thinking that if there is a problem in mom_calc_visc.F
>>>> it should be fixed, and don't see what is wrong with the exch S/R.
>>>>
>>>> I have options for 2 compilers (gfortran & ifort) that make the run
>>>> to stop when a division by zero is found.
>>>> So I could try to see where the problem is.
>>>>
>>>> Do you know which grid, tiling and parameter file "data"
>>>> cause this problem ? (I guess you are not using useAreaViscLength=T
>>>> ?)
>>>>
>>>> Thanks,
>>>> Jean-Michel
>>>>
>>>>
>>>> On Thu, Feb 12, 2009 at 06:12:37PM +0100, Martin Losch wrote:
>>>>> Hi Jean-Michel,
>>>>>
>>>>> the problem occurs only on curvilinear grids (that's why
>>>>> ini_curvilinear_grid), and it happens somewhere in mom_common/
>>>>> mom_calc_visc.F. Basically, L2rdt (and other similar terms) are
>>>>> zero at
>>>>> some points in the overlaps, I think the corners, so i,j=-2, or so
>>>>> and
>>>>> then, when there is a division by L2rdt, then the sxf90 compiler
>>>>> complains/code crashes. The problem goes away with my fix (and we
>>>>> talked
>>>>> about this some time ago, but obviously since then nothing has
>>>>> changed in
>>>>> terms of exchanges) and the actual value of the initialization
>>>>> does not
>>>>> have an impact on the solution at all.
>>>>>
>>>>> You should be able to reproduce this with any cubed sphere grid
>>>>> (although I only get crashes with sxf90).
>>>>>
>>>>> Martin
>>>>>
>>>>> On Feb 12, 2009, at 6:02 PM, Jean-Michel Campin wrote:
>>>>>
>>>>>> Hi Martin,
>>>>>>
>>>>>> I am little bit confused:
>>>>>> - we do have the proper exchange for dxV & dyU (In fact, I  
>>>>>> think we
>>>>>> now have the full familly of exch S/R), may be not the adjoint
>>>>>> version.
>>>>>> - If there is a division by zero somewhere in the code, I would
>>>>>> prefer
>>>>>> to fix it (for all platforms, not only for NEC_SX) rather than
>>>>>> leave with it.
>>>>>> Can you point where the problem was ?
>>>>>> Thanks,
>>>>>> Jean-Michel
>>>>>>
>>>>>> On Thu, Feb 12, 2009 at 02:51:56AM -0500, Martin Losch wrote:
>>>>>>> Update of /u/gcmpack/MITgcm/model/src
>>>>>>> In directory forge:/tmp/cvs-serv13873/model/src
>>>>>>>
>>>>>>> Modified Files:
>>>>>>> 	ini_curvilinear_grid.F
>>>>>>> Log Message:
>>>>>>> check my hack for SX8 compiler, because I keep stumbling over  
>>>>>>> it.
>>>>>>> (forgive me, Jean-Michel):
>>>>>>> terrible hack to make sure that recip_dxV and recip_dyU are  
>>>>>>> never
>>>>>>> zero,
>>>>>>> until we have a proper exchange for them. Most compilers do not
>>>>>>> seem
>>>>>>> to mind divisions by zero in the overlaps, as long as the
>>>>>>> results are
>>>>>>> not used anywhere, but sxf90 does.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> MITgcm-cvs mailing list
>>>>>>> MITgcm-cvs at mitgcm.org
>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-cvs
>>>>>> _______________________________________________
>>>>>> MITgcm-support mailing list
>>>>>> MITgcm-support at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>>
>>>>> _______________________________________________
>>>>> MITgcm-support mailing list
>>>>> MITgcm-support at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>> _______________________________________________
>>> MITgcm-devel mailing list
>>> MITgcm-devel at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel

More information about the MITgcm-devel mailing list