[MITgcm-support] Help with CG2D settings

[Christopher L. Wolfe]

Hi MITgcmers,

I'm trying to get a high resolution model to run efficiently on lots of 
processors (1024 sp3s). Not surprisingly, the most time consuming step 
is solving for the pressure. We're running in hydrostatic mode, we only 
need to solve for the barotropic pressure, but I'm having a little 
trouble figuring out what appropriate settings for the target residual 
and max iteration should be. If I take the default target residual 
(1.5e-7), the CG solver seems to require an unreasonably large (>1500) 
iterations to reduce the residual to the target value. Since the model 
has a flat bottom and a rigid lid, I wouldn't expect each step to make 
the barotropic velocity too divergent if it was divergentless to begin 
with. Thus, I wouldn't expect each step to require too many iterations 
to update the pressure at each step. However, I don't have a good feel 
for what these numbers "should" be or how many CG iterations per step 
is "unreasonable." Does anyone have any rules of thumb for picking the 
CG2D settings?

My data and GRID.h files follow. Thanks,
Christopher

data:
# ====================
# | Model parameters |
# ====================
#
# Continuous equation parameters
  &PARM01
  tRef=-9.9, -9.9, -9.9, -9.9, -9.9, -9.9, -9.9,
       -9.9, -9.9, -9.9, -9.9, -9.9, -9.9, -9.9,
       -9.9, -9.9, -9.9, -9.9, -9.9, -9.9
  sRef=20*35.,
# Prandtl number P = 25
  viscAh=10E-0,
  viscAz=2E-3,
  no_slip_sides=.TRUE.,
  no_slip_bottom=.FALSE.,
  viscA4=50.E7,
  diffK4T=0.E9,
  diffKhT=8E-5,
  diffKzT=8E-5,
  diffKhS=1.E0,
  diffKzS=1.E0,
  f0=-1.3683e-04,
  beta=3.4208e-11,
  tAlpha=2.E-4,
  sBeta =0.,
  gravity=10.,
  rhoConst=1000.,
  rhoNil=1000.,
  heatCapacity_Cp=3900.,
  rigidLid=.TRUE.,
  implicitFreeSurface=.FALSE.,
  saltAdvection=.FALSE.,
  saltForcing=.FALSE.,
  saltStepping=.FALSE.,
  eosType='LINEAR',
  nonHydrostatic=.FALSE.,
  momAdvection=.TRUE.,
  implicitViscosity=.TRUE.,
  implicitDiffusion=.FALSE.,
# ivdc_kappa=10.e0,
  readBinaryPrec=64,
  tempAdvScheme=33,
  staggerTimeStep=.TRUE.,
  bottomDragLinear=4.4E-4,
  debugLevel=-1,
  useJamartWetPoints=.TRUE.,
  useSingleCpuIO=.TRUE.,
# hFacMin=0.1,
  /

# Elliptic solver parameters
  &PARM02
  cg2dMaxIters=1500,
  cg2dTargetResidual=1.5E-7,
  cg2dChkResFreq=1,
  /

# Time stepping parameters
  &PARM03
  nIter0=0013050000,
# nTimeSteps=225000,
  nTimeSteps=1000,
  deltaT=500.,
  abEps=0.1,
  pChkptFreq=112.5E+06,
  chkptFreq=11.25E6,
  dumpFreq=112.5E6,
  monitorFreq=11.25E6,
  tauThetaClimRelax=764400.,
  cAdjFreq=0,
  /

# Gridding parameters
  &PARM04
  usingCartesianGrid=.TRUE.,
  usingSphericalPolarGrid=.FALSE.,
  dXspacing=4.4643E3,
  dYspacing=4.4643E3,
  delZ=  12.0505,  14.9159,  18.4328,  22.7337,  27.9692,
         34.3067,  41.9245,  51.0025,  61.7057,  74.1598,
         88.4181, 104.4189, 121.9376, 140.5412, 159.5568,
        178.0710, 194.9769, 209.0770, 219.2372, 224.5642
  /

# Input datasets
  &PARM05
  thetaClimFile='TempSurf.two.sin_y',
  bathyFile='topo_two.box',
  hydrogThetaFile=,
  hydrogSaltFile=,
  zonalWindFile='windx_two.weakeq',
  meridWindFile=,
  /

SIZE.h:
       INTEGER sNx
       INTEGER sNy
       INTEGER OLx
       INTEGER OLy
       INTEGER nSx
       INTEGER nSy
       INTEGER nPx
       INTEGER nPy
       INTEGER Nx
       INTEGER Ny
       INTEGER Nr
       PARAMETER (
      &           sNx =  56,
      &           sNy =  56,
      &           OLx =   3,
      &           OLy =   3,
      &           nSx =   1,
      &           nSy =   1,
      &           nPx =   8,
      &           nPy =   32,
      &           Nx  = sNx*nSx*nPx,
      &           Ny  = sNy*nSy*nPy,
      &           Nr  =  20)

       INTEGER MAX_OLX
       INTEGER MAX_OLY
       PARAMETER ( MAX_OLX = OLx,
      &            MAX_OLY = OLy )


-----------------------------------------------------------
Dr. Christopher L. Wolfe             		 858-534-4560
Physical Oceanography Research Division    OAR 357
Scripps Institution of Oceanography, UCSD  clwolfe at ucsd.edu
-----------------------------------------------------------