[MITgcm-support] OpenMP Model Code.

[John Stark]

Hi,

We are currently evaluating several systems we a considering as part of
a modest high performance computing purchase. I've been using a
72x152x19 ocean on a long/lat grid. If anyone is interested, I've put
the full code here:
http://www.soes.soton.ac.uk/research/groups/ocean_climate/jods/mit_bench_pub.tgz

I warn you that the Makefile has been hacked about a bit though! See
macros.[system] for some templates.

The basis code was MITGCMUV release1, patch8.

So far it's been run on:
SGI Altix (900MHz Itanium2, in single process, 2 way OpenMP and 2 and 4
way MPI)
An SGI Onyx 300 (12 R14k MIPS / 4Mb cache) with 1 process, 2 and 4
processes.
A sun workstation (Blade 2000)
Several Opteron systems with 2 and 4 processors per node and clock
speeds from 1.4GHz to 1.8GHz.
And some other Itanium systems.

Please let me know if you have any comments about performance on any of
the above systems.

John


Chris Hill wrote:
> 
> John,
> 
>  That's great. What hardware and system software setup are you using?
> Also what rev/checkpoint of the code are you working from?
> 
> Chris
> 
> > -----Original Message-----
> > From: mitgcm-support-bounces at mitgcm.org
> > [mailto:mitgcm-support-bounces at mitgcm.org] On Behalf Of John Stark
> > Sent: Tuesday, October 07, 2003 1:37 PM
> > To: mitgcm-support at mitgcm.org
> > Subject: [MITgcm-support] OpenMP Model Code.
> >
> >
> > Hi,
> >
> > Does anyone else use the threaded version of the model,
> > rather than MPI? I have recently set up the code to use
> > OpenMP directives instead of the processor specific ones
> > currently in the code. The code runs at the same speed as it
> > does using the shared-memory mpich anyway though :(.
> >
> > If anyone wants to do the same, I'd recommend the following
> > modifications to the code:
> >
> > 1. main.F , MAIN_PDIRECTIVES1.h, MAIN_PDIRECTIVES2.h
> >    Replace the loop  DO I=1,nThreads with an OMP parallel
> > section. This allows the use of OMP BARRIERS within the code
> > whereas a DO does not. The use of a loop is misleading since
> > the model never actually uses this as a loop anyway, it is
> > only a construct for setting up the threads
> > on some platforms (more detail below).
> >
> > 2. barrier.F , bar2.F
> >
> > The whole contents of the subroutines barrier and bar2 can
> > become C$OMP BARRIER
> >
> >
> > 3. Some architectures (Itanium and Opteron) seem to have low
> > stack limits, which cause seg. faults with a reasonable model
> > size. To get around this I had to create thread-private
> > common blocks in the following files: thermodynamics.F,
> > dynamics.F and impldiff.F. These routines use very large
> > local arrays (see below). This seems to be an issue
> > regardless of stack limits set with 'ulimit -s' or 'export
> > MP_STACKSIZE=...'
> >
> > 4. exch_rl.F and exch_rs.F need an additional _BARRIER at the
> > start to ensure all threads have finished writing the array
> > before it is exchanged.
> >
> > 5. ini_masks_etc.F
> >  The  _BEGIN_MASTER( myThid ) and _END_MASTER should be
> > commented out since barriers in the underlying mds...
> > routines will cause the code to become out of synch and crash
> > otherwise.
> >
> > 6. mdsio_writefield.F
> >  Needs a    _BARRIER and  _BEGIN_MASTER( mythid )  at the start and an
> > _END_MASTER at the end.
> >
> >
> > Some more detail...
> >
> >
> > [ At the top of main.F ]
> > #ifdef USE_OMP_THREADING
> > c external routine return thread number from openmp
> >       integer OMP_GET_THREAD_NUM
> >       external OMP_GET_THREAD_NUM
> > #endif
> >
> > [ main.F ]
> > REplace DO...  with myThid=1 to handle the no threads case.
> >
> > [ MAIN_PDIRECTIVES1.h ]
> > #ifdef USE_OMP_THREADING
> > C$OMP PARALLEL
> > C$OMP&  SHARED(nThreads) ,
> > C$OMP&  PRIVATE(I, myThid)
> > c    we could use the NUM_THREADS(nThreads) directive
> > C    but does not seem to be supported in all systems.
> >       call OMP_SET_NUM_THREADS(nThreads)
> >
> > c Model uses thread ids from 1,2... whereas openmp uses 0,1...
> >       myThid = OMP_GET_THREAD_NUM() + 1
> > #define CODE_IS_THREADED
> > #endif
> >
> > [ MAIN_PDIRECTIVES2.h ]
> > #ifdef USE_OMP_THREADING
> > C$OMP END PARALLEL
> > #endif
> >
> >
> > [thermodynamics.F:]
> > #ifdef LESS_STACK_VARS
> >       common / thermo_work_S / xA, yA, maskUp
> >       common / thermo_work_L / uTrans, vTrans, rTrans, fVerT, fVerS
> >      &     , fVerTr1, phiHyd, rhokm1, rhok, phiSurfX, phiSurfY,
> >      &     KappaRT, KappaRS, sigmaX, sigmaY, sigmaR
> > C$OMP THREADPRIVATE( /thermo_work_S/ , /thermo_work_L/)
> > #endif
> >
> > [dynamics.F:]
> > #ifdef LESS_STACK_VARS
> >       common / dynamics_rl / fVerU, fVerV, phiHyd, rhokm1, rhok,
> >      &     phiSurfX, phiSurfY, KappaRU, KappaRV
> > c$OMP THREADPRIVATE( / dynamics_rl / )
> > #endif
> >
> >
> > [impldiff.F:]
> > #ifdef LESS_STACK_VARS
> >       common / impldiff_rl / gYnm1 , a , b , c , bet , gam
> > C$OMP THREADPRIVATE( / impldiff_rl / ) #endif
> >