[MITgcm-support] MPI process mismatch and segmentation fault issues in MITgcm + PDAF workflow (dummy vs real data)
VADALIA TULSIBEN
24mtpos013 at ddu.ac.in
Sun Mar 29 07:53:03 EDT 2026
Dear MITgcm Support Team,
My name is Tulsi Vadalia, and I am currently pursuing my M.Tech from
Dharmsinh Desai University (DDU), Nadiad, Gujarat. As part of my research
work, I am working at Space Applications Centre (SAC), ISRO, Ahmedabad on a
data assimilation project involving MITgcm coupled with PDAF.
I would like to describe my workflow and the issues I am facing
step-by-step.
------------------------------
1. Initial Learning and MITgcm Standalone Runs
I first learned how to use MITgcm and successfully ran the
*tutorial_barotropic_gyre* case. After that, I ran MITgcm independently
(without PDAF), and it executed correctly on my HPC system.
------------------------------
2. MITgcm + PDAF with Dummy Data (Working Case)
Next, I integrated PDAF with MITgcm by adding the .F files from the
MITgcm–PDAF binding into the code folder. After compiling and running the
model with PDAF using dummy data, the system worked successfully without
any errors.
------------------------------
3. Transition to Real Data (MITgcm Only – Working)
I then moved to real data by adding a pickup file (.data) in the input
configuration. Running MITgcm without PDAF works correctly and generates
expected output files.
------------------------------
4. Issue with Real Data + PDAF
When running MITgcm with PDAF using real data, I observed the following
issues:
Segmentation Fault (According to me, bcz of Larger Domain)
To investigate further, I tested different domain decompositions:
*Case 1:*
sNx = 44, sNy = 30
nPx = 20, nPy = 20
Nr = 66
→ Nx = 880, Ny = 600
This configuration resulted in segmentation faults.
------------------------------
*Case 2 (Reduced Configuration):*
sNx = 55, sNy = 75
nPx = 16, nPy = 8
Even after reducing complexity, segmentation faults still occurred.
------------------------------
5. Current Issue: MPI Process Mismatch (EEBOOT_MINIMAL Error)
To reduce complexity and debug the issue, I modified the domain
decomposition in the file:
25km/code/SIZE.h
Current configuration:
-
sNx = 32, sNy = 32
-
nPx = 4, nPy = 2
→ Expected MPI processes = nPx × nPy = 8
The model input parameters are defined in the data file in the run
directory.
I then ran the model using:
mpirun -np 8 --hostfile $PBS_NODEFILE ./mitgcmuv
However, the model aborts with the following error:
EEBOOT_MINIMAL: No. of procs = 1 not equal to nPx*nPy = 8
EEDIE: earlier error in multi-proc/thread setting
PROGRAM MAIN: ends with fatal Error
No Output Generation (Single Process Run)
when i set nPx = 1, nPy = 1 ,
When running with a single MPI process:
mpirun -np 1 --hostfile $PBS_NODEFILE ./mitgcmuv
-
PDAF initializes correctly
-
Assimilation completes successfully
-
However, no .data output files are generated
(Reference: output logs attached)
but after when we increase processes its not running
------------------------------
*Observations:*
-
MITgcm detects only 1 process (numberOfProcs = 1)
-
Expected processes = 8 (from SIZE.h configuration)
-
PDAF initialization (INIT_PARALLEL_PDAF) is invoked before the failure
-
PDAF reports running with only 1 PE
(Reference: STDERR and STDOUT logs attached)
This issue appeared after modifying the SIZE.h configuration to reduce the
domain size, and I would like to understand why MITgcm is not recognizing
multiple MPI processes.
------------------------------
Summary of Issues
-
No output generation when running MITgcm + PDAF with real data (single
process)
-
Segmentation faults for larger domain configurations
-
MPI process mismatch (numberOfProcs = 1 instead of expected value) after
modifying SIZE.h
------------------------------
Questions
1.
Could the segmentation faults be related to domain decomposition limits,
tile sizes, or memory layout in MITgcm?
2.
Why is MITgcm detecting only 1 MPI process despite launching multiple
processes?
3.
Could PDAF communicator initialization interfere with MITgcm’s MPI setup?
4.
Is there a recommended workflow for coupling MITgcm with PDAF when
transitioning from dummy to real data?
------------------------------
If required, I can provide additional files for further investigation.
Please let me know if any specific files or details are needed, and I will
be happy to share them.
I would greatly appreciate any guidance or suggestions to resolve these
issues.
Thank you for your time and support.
Best regards,
Tulsi Vadalia
M.Tech (DDU, Nadiad)
Research Intern, SAC-ISRO, Ahmedabad
--
DISCLAIMER: The information transmitted is intended only for the person or
entity to which it is addressed and may contain confidential and/or
privileged material which is the intellectual property of Dharmsinh Desai
University (D.D.U., Nadiad). Any review, retransmission, dissemination or
other use of, or taking of any action in reliance upon this information by
persons or entities other than the intended recipient is strictly
prohibited. If you are not the intended recipient, or the employee, or
agent responsible for delivering the message to the intended recipient
and/or if you have received this in error, please contact the sender and
delete the material from the computer or device. Dharmsinh Desai University
does not take any liability or responsibility for any malicious
codes/software and/or viruses/Trojan horses that may have been picked up
during the transmission of this message. By opening and solely relying on
the contents or part thereof this message, and taking action thereof, the
recipient relieves the Dharmsinh Desai University of all the liabilities
including any damages done to the recipient's pc/laptop/peripherals and
other communication devices due to any reason.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20260329/582aa164/attachment-0001.html>
-------------- next part --------------
1. Initial Learning and MITgcm Standalone Runs
[shaileepatel at master2 code_06]$ pwd
/sachome2/shaileepatel/tulsi/MITgcm/verification/EXPS/t_b_g/code
[shaileepatel at master2 code]$ cat SIZE.h
CBOP
C !ROUTINE: SIZE.h
C !INTERFACE:
C include SIZE.h
C !DESCRIPTION: \bv
C *==========================================================*
C | SIZE.h Declare size of underlying computational grid.
C *==========================================================*
C | The design here supports a three-dimensional model grid
C | with indices I,J and K. The three-dimensional domain
C | is comprised of nPx*nSx blocks (or tiles) of size sNx
C | along the first (left-most index) axis, nPy*nSy blocks
C | of size sNy along the second axis and one block of size
C | Nr along the vertical (third) axis.
C | Blocks/tiles have overlap regions of size OLx and OLy
C | along the dimensions that are subdivided.
C *==========================================================*
C \ev
C
C Voodoo numbers controlling data layout:
C sNx :: Number of X points in tile.
C sNy :: Number of Y points in tile.
C OLx :: Tile overlap extent in X.
C OLy :: Tile overlap extent in Y.
C nSx :: Number of tiles per process in X.
C nSy :: Number of tiles per process in Y.
C nPx :: Number of processes to use in X.
C nPy :: Number of processes to use in Y.
C Nx :: Number of points in X for the full domain.
C Ny :: Number of points in Y for the full domain.
C Nr :: Number of points in vertical direction.
CEOP
INTEGER sNx
INTEGER sNy
INTEGER OLx
INTEGER OLy
INTEGER nSx
INTEGER nSy
INTEGER nPx
INTEGER nPy
INTEGER Nx
INTEGER Ny
INTEGER Nr
PARAMETER (
& sNx = 62,
& sNy = 62,
& OLx = 2,
& OLy = 2,
& nSx = 1,
& nSy = 1,
& nPx = 1,
& nPy = 1,
& Nx = sNx*nSx*nPx,
& Ny = sNy*nSy*nPy,
& Nr = 1)
C MAX_OLX :: Set to the maximum overlap region size of any array
C MAX_OLY that will be exchanged. Controls the sizing of exch
C routine buffers.
INTEGER MAX_OLX
INTEGER MAX_OLY
PARAMETER ( MAX_OLX = OLx,
& MAX_OLY = OLy )
[shaileepatel at master2 input]$ cat data
# Model parameters
# Continuous equation parameters
&PARM01
viscAh=4.E2,
f0=1.E-4,
beta=1.E-11,
rhoConst=1000.,
gBaro=9.81,
rigidLid=.FALSE.,
implicitFreeSurface=.TRUE.,
# momAdvection=.FALSE.,
tempStepping=.FALSE.,
saltStepping=.FALSE.,
/
# Elliptic solver parameters
&PARM02
cg2dTargetResidual=1.E-7,
cg2dMaxIters=1000,
/
# Time stepping parameters
&PARM03
nIter0=0,
nTimeSteps=25,
deltaT=1200.0,
pChkptFreq=31104000.0,
chkptFreq=15552000.0,
dumpFreq=86400.0,
monitorFreq=1200.,
monitorSelect=2,
pickupStrictlyMatch=.FALSE.,
#-for longer run (3.0 yr):
# nTimeSteps=77760,
# monitorFreq=864000.,
/
# Gridding parameters
&PARM04
usingCartesianGrid=.TRUE.,
delX=62*20.E3,
delY=62*20.E3,
xgOrigin=-20.E3,
ygOrigin=-20.E3,
delR=5000.,
/
# Input datasets
&PARM05
bathyFile='bathy.bin'
zonalWindFile='windx_cosy.bin',
meridWindFile=,
/
.................................................................................................................................................................
3.Transition to Real Data (MITgcm Only – Working)
[shaileepatel at master2 IO-10km]$ pwd
/sachome2/shaileepatel/tulsi/MITgcm/verification/IO-10km
[shaileepatel at master2 code]$ ls
CPP_OPTIONS.h DIAGNOSTICS_SIZE.h EXF_OPTIONS.h OBCS_OPTIONS.h
packages.conf PP SIZE.h
[shaileepatel at master2 code]$ cat SIZE.h
C $Header:
/u/gcmpack/MITgcm/verification/coupled_ucla/code/Attic/SIZE.h.56,v
1.1.2.1 2004/06/03 23:58:45 dimitri Exp $
C $Name: release1_coupled $
C
C /==========================================================\
C | SIZE.h Declare size of underlying computational grid. |
C |==========================================================|
C | The design here support a three-dimensional model grid |
C | with indices I,J and K. The three-dimensional domain |
C | is comprised of nPx*nSx blocks of size sNx along one axis|
C | nPy*nSy blocks of size sNy along another axis and one |
C | block of size Nz along the final axis. |
C | Blocks have overlap regions of size OLx and OLy along the|
C | dimensions that are subdivided. |
C \==========================================================/
C Voodoo numbers controlling data layout.
C sNx - No. X points in sub-grid.
C sNy - No. Y points in sub-grid.
C OLx - Overlap extent in X.
C OLy - Overlat extent in Y.
C nSx - No. sub-grids in X.
C nSy - No. sub-grids in Y.
C nPx - No. of processes to use in X.
C nPy - No. of processes to use in Y.
C Nx - No. points in X for the total domain.
C Ny - No. points in Y for the total domain.
C Nr - No. points in Z for full process domain.
INTEGER sNx
INTEGER sNy
INTEGER OLx
INTEGER OLy
INTEGER nSx
INTEGER nSy
INTEGER nPx
INTEGER nPy
INTEGER Nx
INTEGER Ny
INTEGER Nr
PARAMETER (
& sNx = 44,
& sNy = 30,
& OLx = 3,
& OLy = 3,
& nSx = 1,
& nSy = 1,
& nPx = 20,
& nPy = 20,
& Nx = sNx*nSx*nPx,
& Ny = sNy*nSy*nPy,
& Nr = 66)
C MAX_OLX - Set to the maximum overlap region size of any array
C MAX_OLY that will be exchanged. Controls the sizing of exch
C routine buufers.
INTEGER MAX_OLX
INTEGER MAX_OLY
PARAMETER ( MAX_OLX = OLx,
& MAX_OLY = OLy )
[shaileepatel at master2 IO-10km]$ ls exp/
AngleCS.data mxld.0006834240.data
pickup.0006845904.meta salt.0006858000.meta STDERR.0199 STDOUT.0001
STDOUT.0203 T.0006858000.meta uvel.0006846336.meta
AngleCS.meta mxld.0006834240.meta
pickup.0006846336.data salt.data STDERR.0200 STDOUT.0002
STDOUT.0204 temp.0006831648.data uvel.0006846768.data
AngleSN.data mxld.0006834672.data
pickup.0006846336.meta salt.nc STDERR.0201 STDOUT.0003
STDOUT.0205 temp.0006831648.meta uvel.0006846768.meta
AngleSN.meta mxld.0006834672.meta
pickup.0006846768.data STDERR.0000 STDERR.0202 STDOUT.0004
STDOUT.0206 temp.0006832080.data uvel.0006847200.data
available_diagnostics.log mxld.0006835104.data
pickup.0006846768.meta STDERR.0001 STDERR.0203 STDOUT.0005
STDOUT.0207 temp.0006832080.meta uvel.0006847200.meta
CTL mxld.0006835104.meta
pickup.0006847200.data STDERR.0002 STDERR.0204 STDOUT.0006
STDOUT.0208 temp.0006832512.data uvel.0006847632.data
data mxld.0006835536.data
pickup.0006847200.meta STDERR.0003 STDERR.0205 STDOUT.0007
STDOUT.0209 temp.0006832512.meta uvel.0006847632.meta
data.cal mxld.0006835536.meta
pickup.0006847632.data STDERR.0004 STDERR.0206 STDOUT.0008
STDOUT.0210 temp.0006832944.data uvel.0006848064.data
data.diagnostics mxld.0006835968.data
pickup.0006847632.meta STDERR.0005 STDERR.0207 STDOUT.0009
STDOUT.0211 temp.0006832944.meta uvel.0006848064.meta
data.exf mxld.0006835968.meta
pickup.0006848064.data STDERR.0006 STDERR.0208 STDOUT.0010
STDOUT.0212 temp.0006833376.data uvel.0006848496.data
data.exf.master mxld.0006836400.data
pickup.0006848064.meta STDERR.0007 STDERR.0209 STDOUT.0011
STDOUT.0213 temp.0006833376.meta uvel.0006848496.meta
data.kpp mxld.0006836400.meta
pickup.0006848496.data STDERR.0008 STDERR.0210 STDOUT.0012
STDOUT.0214 temp.0006833808.data uvel.0006848928.data
data.mnc mxld.0006836832.data
pickup.0006848496.meta STDERR.0009 STDERR.0211 STDOUT.0013
STDOUT.0215 temp.0006833808.meta uvel.0006848928.meta
data.obcs mxld.0006836832.meta
pickup.0006848928.data STDERR.0010 STDERR.0212 STDOUT.0014
STDOUT.0216 temp.0006834240.data uvel.0006849360.data
data.pkg mxld.0006837264.data
pickup.0006848928.meta STDERR.0011 STDERR.0213 STDOUT.0015
STDOUT.0217 temp.0006834240.meta uvel.0006849360.meta
data.temp mxld.0006837264.meta
pickup.0006849360.data STDERR.0012 STDERR.0214 STDOUT.0016
STDOUT.0218 temp.0006834672.data uvel.0006849792.data
Depth.data mxld.0006837696.data
pickup.0006849360.meta STDERR.0013 STDERR.0215 STDOUT.0017
STDOUT.0219 temp.0006834672.meta uvel.0006849792.meta
Depth.meta mxld.0006837696.meta
pickup.0006849792.data STDERR.0014 STDERR.0216 STDOUT.0018
STDOUT.0220 temp.0006835104.data uvel.0006850224.data
DRC.data mxld.0006838128.data
pickup.0006849792.meta STDERR.0015 STDERR.0217 STDOUT.0019
STDOUT.0221 temp.0006835104.meta uvel.0006850224.meta
DRC.meta mxld.0006838128.meta
pickup.0006850224.data STDERR.0016 STDERR.0218 STDOUT.0020
STDOUT.0222 temp.0006835536.data uvel.0006850656.data
DRF.data mxld.0006838560.data
pickup.0006850224.meta STDERR.0017 STDERR.0219 STDOUT.0021
STDOUT.0223 temp.0006835536.meta uvel.0006850656.meta
DRF.meta mxld.0006838560.meta
pickup.0006850656.data STDERR.0018 STDERR.0220 STDOUT.0022
STDOUT.0224 temp.0006835968.data uvel.0006851088.data
DXC.data mxld.0006838992.data
pickup.0006850656.meta STDERR.0019 STDERR.0221 STDOUT.0023
STDOUT.0225 temp.0006835968.meta uvel.0006851088.meta
DXC.meta mxld.0006838992.meta
pickup.0006851088.data STDERR.0020 STDERR.0222 STDOUT.0024
STDOUT.0226 temp.0006836400.data uvel.0006851520.data
DXF.data mxld.0006839424.data
pickup.0006851088.meta STDERR.0021 STDERR.0223 STDOUT.0025
STDOUT.0227 temp.0006836400.meta uvel.0006851520.meta
DXF.meta mxld.0006839424.meta
pickup.0006851520.data STDERR.0022 STDERR.0224 STDOUT.0026
STDOUT.0228 temp.0006836832.data uvel.0006851952.data
DXG.data mxld.0006839856.data
pickup.0006851520.meta STDERR.0023 STDERR.0225 STDOUT.0027
STDOUT.0229 temp.0006836832.meta uvel.0006851952.meta
DXG.meta mxld.0006839856.meta
pickup.0006851952.data STDERR.0024 STDERR.0226 STDOUT.0028
STDOUT.0230 temp.0006837264.data uvel.0006852384.data
DXV.data mxld.0006840288.data
pickup.0006851952.meta STDERR.0025 STDERR.0227 STDOUT.0029
STDOUT.0231 temp.0006837264.meta uvel.0006852384.meta
DXV.meta mxld.0006840288.meta
pickup.0006852384.data STDERR.0026 STDERR.0228 STDOUT.0030
STDOUT.0232 temp.0006837696.data uvel.0006852816.data
DYC.data mxld.0006840720.data
pickup.0006852384.meta STDERR.0027 STDERR.0229 STDOUT.0031
STDOUT.0233 temp.0006837696.meta uvel.0006852816.meta
DYC.meta mxld.0006840720.meta
pickup.0006852816.data STDERR.0028 STDERR.0230 STDOUT.0032
STDOUT.0234 temp.0006838128.data uvel.0006853248.data
DYF.data mxld.0006841152.data
pickup.0006852816.meta STDERR.0029 STDERR.0231 STDOUT.0033
STDOUT.0235 temp.0006838128.meta uvel.0006853248.meta
DYF.meta mxld.0006841152.meta
pickup.0006853248.data STDERR.0030 STDERR.0232 STDOUT.0034
STDOUT.0236 temp.0006838560.data uvel.0006853680.data
DYG.data mxld.0006841584.data
pickup.0006853248.meta STDERR.0031 STDERR.0233 STDOUT.0035
STDOUT.0237 temp.0006838560.meta uvel.0006853680.meta
DYG.meta mxld.0006841584.meta
pickup.0006853680.data STDERR.0032 STDERR.0234 STDOUT.0036
STDOUT.0238 temp.0006838992.data uvel.0006854112.data
DYU.data mxld.0006842016.data
pickup.0006853680.meta STDERR.0033 STDERR.0235 STDOUT.0037
STDOUT.0239 temp.0006838992.meta uvel.0006854112.meta
DYU.meta mxld.0006842016.meta
pickup.0006854112.data STDERR.0034 STDERR.0236 STDOUT.0038
STDOUT.0240 temp.0006839424.data uvel.0006854544.data
eedata mxld.0006842448.data
pickup.0006854112.meta STDERR.0035 STDERR.0237 STDOUT.0039
STDOUT.0241 temp.0006839424.meta uvel.0006854544.meta
Eta.0006828192.data mxld.0006842448.meta
pickup.0006854544.data STDERR.0036 STDERR.0238 STDOUT.0040
STDOUT.0242 temp.0006839856.data uvel.0006854976.data
Eta.0006828192.meta mxld.0006842880.data
pickup.0006854544.meta STDERR.0037 STDERR.0239 STDOUT.0041
STDOUT.0243 temp.0006839856.meta uvel.0006854976.meta
Eta.0006831216.data mxld.0006842880.meta
pickup.0006854976.data STDERR.0038 STDERR.0240 STDOUT.0042
STDOUT.0244 temp.0006840288.data uvel.0006855408.data
Eta.0006831216.meta mxld.0006843312.data
pickup.0006854976.meta STDERR.0039 STDERR.0241 STDOUT.0043
STDOUT.0245 temp.0006840288.meta uvel.0006855408.meta
ETA.0006831648.data mxld.0006843312.meta
pickup.0006855408.data STDERR.0040 STDERR.0242 STDOUT.0044
STDOUT.0246 temp.0006840720.data uvel.0006855840.data
ETA.0006831648.meta mxld.0006843744.data
pickup.0006855408.meta STDERR.0041 STDERR.0243 STDOUT.0045
STDOUT.0247 temp.0006840720.meta uvel.0006855840.meta
ETA.0006832080.data mxld.0006843744.meta
pickup.0006855840.data STDERR.0042 STDERR.0244 STDOUT.0046
STDOUT.0248 temp.0006841152.data uvel.0006856272.data
ETA.0006832080.meta mxld.0006844176.data
pickup.0006855840.meta STDERR.0043 STDERR.0245 STDOUT.0047
STDOUT.0249 temp.0006841152.meta uvel.0006856272.meta
ETA.0006832512.data mxld.0006844176.meta
pickup.0006856272.data STDERR.0044 STDERR.0246 STDOUT.0048
STDOUT.0250 temp.0006841584.data uvel.0006856704.data
ETA.0006832512.meta mxld.0006844608.data
pickup.0006856272.meta STDERR.0045 STDERR.0247 STDOUT.0049
STDOUT.0251 temp.0006841584.meta uvel.0006856704.meta
ETA.0006832944.data mxld.0006844608.meta
pickup.0006856704.data STDERR.0046 STDERR.0248 STDOUT.0050
STDOUT.0252 temp.0006842016.data uvel.0006857136.data
ETA.0006832944.meta mxld.0006845040.data
pickup.0006856704.meta STDERR.0047 STDERR.0249 STDOUT.0051
STDOUT.0253 temp.0006842016.meta uvel.0006857136.meta
ETA.0006833376.data mxld.0006845040.meta
pickup.0006857136.data STDERR.0048 STDERR.0250 STDOUT.0052
STDOUT.0254 temp.0006842448.data uvel.0006857568.data
ETA.0006833376.meta mxld.0006845472.data
pickup.0006857136.meta STDERR.0049 STDERR.0251 STDOUT.0053
STDOUT.0255 temp.0006842448.meta uvel.0006857568.meta
ETA.0006833808.data mxld.0006845472.meta
pickup.0006857568.data STDERR.0050 STDERR.0252 STDOUT.0054
STDOUT.0256 temp.0006842880.data uvel.0006858000.data
ETA.0006833808.meta mxld.0006845904.data
pickup.0006857568.meta STDERR.0051 STDERR.0253 STDOUT.0055
STDOUT.0257 temp.0006842880.meta uvel.0006858000.meta
ETA.0006834240.data mxld.0006845904.meta
pickup.0006858000.data STDERR.0052 STDERR.0254 STDOUT.0056
STDOUT.0258 temp.0006843312.data V.0006828192.data
ETA.0006834240.meta mxld.0006846336.data
pickup.0006858000.meta STDERR.0053 STDERR.0255 STDOUT.0057
STDOUT.0259 temp.0006843312.meta V.0006828192.meta
ETA.0006834672.data mxld.0006846336.meta RAC.data
STDERR.0054 STDERR.0256 STDOUT.0058 STDOUT.0260
temp.0006843744.data V.0006831216.data
ETA.0006834672.meta mxld.0006846768.data RAC.meta
STDERR.0055 STDERR.0257 STDOUT.0059 STDOUT.0261
temp.0006843744.meta V.0006831216.meta
ETA.0006835104.data mxld.0006846768.meta RAS.data
STDERR.0056 STDERR.0258 STDOUT.0060 STDOUT.0262
temp.0006844176.data V.0006858000.data
ETA.0006835104.meta mxld.0006847200.data RAS.meta
STDERR.0057 STDERR.0259 STDOUT.0061 STDOUT.0263
temp.0006844176.meta V.0006858000.meta
ETA.0006835536.data mxld.0006847200.meta RAW.data
STDERR.0058 STDERR.0260 STDOUT.0062 STDOUT.0264
temp.0006844608.data vvel.0006831648.data
ETA.0006835536.meta mxld.0006847632.data RAW.meta
STDERR.0059 STDERR.0261 STDOUT.0063 STDOUT.0265
temp.0006844608.meta vvel.0006831648.meta
ETA.0006835968.data mxld.0006847632.meta RAZ.data
STDERR.0060 STDERR.0262 STDOUT.0064 STDOUT.0266
temp.0006845040.data vvel.0006832080.data
ETA.0006835968.meta mxld.0006848064.data RAZ.meta
STDERR.0061 STDERR.0263 STDOUT.0065 STDOUT.0267
temp.0006845040.meta vvel.0006832080.meta
ETA.0006836400.data mxld.0006848064.meta RC.data
STDERR.0062 STDERR.0264 STDOUT.0066 STDOUT.0268
temp.0006845472.data vvel.0006832512.data
ETA.0006836400.meta mxld.0006848496.data RC.meta
STDERR.0063 STDERR.0265 STDOUT.0067 STDOUT.0269
temp.0006845472.meta vvel.0006832512.meta
ETA.0006836832.data mxld.0006848496.meta reference.nc
STDERR.0064 STDERR.0266 STDOUT.0068 STDOUT.0270
temp.0006845904.data vvel.0006832944.data
ETA.0006836832.meta mxld.0006848928.data RF.data
STDERR.0065 STDERR.0267 STDOUT.0069 STDOUT.0271
temp.0006845904.meta vvel.0006832944.meta
ETA.0006837264.data mxld.0006848928.meta RF.meta
STDERR.0066 STDERR.0268 STDOUT.0070 STDOUT.0272
temp.0006846336.data vvel.0006833376.data
ETA.0006837264.meta mxld.0006849360.data RhoRef.data
STDERR.0067 STDERR.0269 STDOUT.0071 STDOUT.0273
temp.0006846336.meta vvel.0006833376.meta
ETA.0006837696.data mxld.0006849360.meta RhoRef.meta
STDERR.0068 STDERR.0270 STDOUT.0072 STDOUT.0274
temp.0006846768.data vvel.0006833808.data
ETA.0006837696.meta mxld.0006849792.data S.0006828192.data
STDERR.0069 STDERR.0271 STDOUT.0073 STDOUT.0275
temp.0006846768.meta vvel.0006833808.meta
ETA.0006838128.data mxld.0006849792.meta S.0006828192.meta
STDERR.0070 STDERR.0272 STDOUT.0074 STDOUT.0276
temp.0006847200.data vvel.0006834240.data
ETA.0006838128.meta mxld.0006850224.data S.0006831216.data
STDERR.0071 STDERR.0273 STDOUT.0075 STDOUT.0277
temp.0006847200.meta vvel.0006834240.meta
ETA.0006838560.data mxld.0006850224.meta S.0006831216.meta
STDERR.0072 STDERR.0274 STDOUT.0076 STDOUT.0278
temp.0006847632.data vvel.0006834672.data
ETA.0006838560.meta mxld.0006850656.data S.0006858000.data
STDERR.0073 STDERR.0275 STDOUT.0077 STDOUT.0279
temp.0006847632.meta vvel.0006834672.meta
ETA.0006838992.data mxld.0006850656.meta S.0006858000.meta
STDERR.0074 STDERR.0276 STDOUT.0078 STDOUT.0280
temp.0006848064.data vvel.0006835104.data
ETA.0006838992.meta mxld.0006851088.data safe
STDERR.0075 STDERR.0277 STDOUT.0079 STDOUT.0281
temp.0006848064.meta vvel.0006835104.meta
ETA.0006839424.data mxld.0006851088.meta
salt.0006831648.data STDERR.0076 STDERR.0278 STDOUT.0080
STDOUT.0282 temp.0006848496.data vvel.0006835536.data
ETA.0006839424.meta mxld.0006851520.data
salt.0006831648.meta STDERR.0077 STDERR.0279 STDOUT.0081
STDOUT.0283 temp.0006848496.meta vvel.0006835536.meta
ETA.0006839856.data mxld.0006851520.meta
salt.0006832080.data STDERR.0078 STDERR.0280 STDOUT.0082
STDOUT.0284 temp.0006848928.data vvel.0006835968.data
ETA.0006839856.meta mxld.0006851952.data
salt.0006832080.meta STDERR.0079 STDERR.0281 STDOUT.0083
STDOUT.0285 temp.0006848928.meta vvel.0006835968.meta
ETA.0006840288.data mxld.0006851952.meta
salt.0006832512.data STDERR.0080 STDERR.0282 STDOUT.0084
STDOUT.0286 temp.0006849360.data vvel.0006836400.data
ETA.0006840288.meta mxld.0006852384.data
salt.0006832512.meta STDERR.0081 STDERR.0283 STDOUT.0085
STDOUT.0287 temp.0006849360.meta vvel.0006836400.meta
ETA.0006840720.data mxld.0006852384.meta
salt.0006832944.data STDERR.0082 STDERR.0284 STDOUT.0086
STDOUT.0288 temp.0006849792.data vvel.0006836832.data
ETA.0006840720.meta mxld.0006852816.data
salt.0006832944.meta STDERR.0083 STDERR.0285 STDOUT.0087
STDOUT.0289 temp.0006849792.meta vvel.0006836832.meta
ETA.0006841152.data mxld.0006852816.meta
salt.0006833376.data STDERR.0084 STDERR.0286 STDOUT.0088
STDOUT.0290 temp.0006850224.data vvel.0006837264.data
ETA.0006841152.meta mxld.0006853248.data
salt.0006833376.meta STDERR.0085 STDERR.0287 STDOUT.0089
STDOUT.0291 temp.0006850224.meta vvel.0006837264.meta
ETA.0006841584.data mxld.0006853248.meta
salt.0006833808.data STDERR.0086 STDERR.0288 STDOUT.0090
STDOUT.0292 temp.0006850656.data vvel.0006837696.data
ETA.0006841584.meta mxld.0006853680.data
salt.0006833808.meta STDERR.0087 STDERR.0289 STDOUT.0091
STDOUT.0293 temp.0006850656.meta vvel.0006837696.meta
ETA.0006842016.data mxld.0006853680.meta
salt.0006834240.data STDERR.0088 STDERR.0290 STDOUT.0092
STDOUT.0294 temp.0006851088.data vvel.0006838128.data
ETA.0006842016.meta mxld.0006854112.data
salt.0006834240.meta STDERR.0089 STDERR.0291 STDOUT.0093
STDOUT.0295 temp.0006851088.meta vvel.0006838128.meta
ETA.0006842448.data mxld.0006854112.meta
salt.0006834672.data STDERR.0090 STDERR.0292 STDOUT.0094
STDOUT.0296 temp.0006851520.data vvel.0006838560.data
ETA.0006842448.meta mxld.0006854544.data
salt.0006834672.meta STDERR.0091 STDERR.0293 STDOUT.0095
STDOUT.0297 temp.0006851520.meta vvel.0006838560.meta
ETA.0006842880.data mxld.0006854544.meta
salt.0006835104.data STDERR.0092 STDERR.0294 STDOUT.0096
STDOUT.0298 temp.0006851952.data vvel.0006838992.data
ETA.0006842880.meta mxld.0006854976.data
salt.0006835104.meta STDERR.0093 STDERR.0295 STDOUT.0097
STDOUT.0299 temp.0006851952.meta vvel.0006838992.meta
ETA.0006843312.data mxld.0006854976.meta
salt.0006835536.data STDERR.0094 STDERR.0296 STDOUT.0098
STDOUT.0300 temp.0006852384.data vvel.0006839424.data
ETA.0006843312.meta mxld.0006855408.data
salt.0006835536.meta STDERR.0095 STDERR.0297 STDOUT.0099
STDOUT.0301 temp.0006852384.meta vvel.0006839424.meta
ETA.0006843744.data mxld.0006855408.meta
salt.0006835968.data STDERR.0096 STDERR.0298 STDOUT.0100
STDOUT.0302 temp.0006852816.data vvel.0006839856.data
ETA.0006843744.meta mxld.0006855840.data
salt.0006835968.meta STDERR.0097 STDERR.0299 STDOUT.0101
STDOUT.0303 temp.0006852816.meta vvel.0006839856.meta
ETA.0006844176.data mxld.0006855840.meta
salt.0006836400.data STDERR.0098 STDERR.0300 STDOUT.0102
STDOUT.0304 temp.0006853248.data vvel.0006840288.data
ETA.0006844176.meta mxld.0006856272.data
salt.0006836400.meta STDERR.0099 STDERR.0301 STDOUT.0103
STDOUT.0305 temp.0006853248.meta vvel.0006840288.meta
ETA.0006844608.data mxld.0006856272.meta
salt.0006836832.data STDERR.0100 STDERR.0302 STDOUT.0104
STDOUT.0306 temp.0006853680.data vvel.0006840720.data
ETA.0006844608.meta mxld.0006856704.data
salt.0006836832.meta STDERR.0101 STDERR.0303 STDOUT.0105
STDOUT.0307 temp.0006853680.meta vvel.0006840720.meta
ETA.0006845040.data mxld.0006856704.meta
salt.0006837264.data STDERR.0102 STDERR.0304 STDOUT.0106
STDOUT.0308 temp.0006854112.data vvel.0006841152.data
ETA.0006845040.meta mxld.0006857136.data
salt.0006837264.meta STDERR.0103 STDERR.0305 STDOUT.0107
STDOUT.0309 temp.0006854112.meta vvel.0006841152.meta
ETA.0006845472.data mxld.0006857136.meta
salt.0006837696.data STDERR.0104 STDERR.0306 STDOUT.0108
STDOUT.0310 temp.0006854544.data vvel.0006841584.data
ETA.0006845472.meta mxld.0006857568.data
salt.0006837696.meta STDERR.0105 STDERR.0307 STDOUT.0109
STDOUT.0311 temp.0006854544.meta vvel.0006841584.meta
ETA.0006845904.data mxld.0006857568.meta
salt.0006838128.data STDERR.0106 STDERR.0308 STDOUT.0110
STDOUT.0312 temp.0006854976.data vvel.0006842016.data
ETA.0006845904.meta mxld.0006858000.data
salt.0006838128.meta STDERR.0107 STDERR.0309 STDOUT.0111
STDOUT.0313 temp.0006854976.meta vvel.0006842016.meta
ETA.0006846336.data mxld.0006858000.meta
salt.0006838560.data STDERR.0108 STDERR.0310 STDOUT.0112
STDOUT.0314 temp.0006855408.data vvel.0006842448.data
ETA.0006846336.meta PH.0006858000.data
salt.0006838560.meta STDERR.0109 STDERR.0311 STDOUT.0113
STDOUT.0315 temp.0006855408.meta vvel.0006842448.meta
ETA.0006846768.data PH.0006858000.meta
salt.0006838992.data STDERR.0110 STDERR.0312 STDOUT.0114
STDOUT.0316 temp.0006855840.data vvel.0006842880.data
ETA.0006846768.meta PHL.0006858000.data
salt.0006838992.meta STDERR.0111 STDERR.0313 STDOUT.0115
STDOUT.0317 temp.0006855840.meta vvel.0006842880.meta
ETA.0006847200.data PHL.0006858000.meta
salt.0006839424.data STDERR.0112 STDERR.0314 STDOUT.0116
STDOUT.0318 temp.0006856272.data vvel.0006843312.data
ETA.0006847200.meta PHrefC.data
salt.0006839424.meta STDERR.0113 STDERR.0315 STDOUT.0117
STDOUT.0319 temp.0006856272.meta vvel.0006843312.meta
ETA.0006847632.data PHrefC.meta
salt.0006839856.data STDERR.0114 STDERR.0316 STDOUT.0118
STDOUT.0320 temp.0006856704.data vvel.0006843744.data
ETA.0006847632.meta PHrefF.data
salt.0006839856.meta STDERR.0115 STDERR.0317 STDOUT.0119
STDOUT.0321 temp.0006856704.meta vvel.0006843744.meta
ETA.0006848064.data PHrefF.meta
salt.0006840288.data STDERR.0116 STDERR.0318 STDOUT.0120
STDOUT.0322 temp.0006857136.data vvel.0006844176.data
ETA.0006848064.meta pickup.0006828192.data
salt.0006840288.meta STDERR.0117 STDERR.0319 STDOUT.0121
STDOUT.0323 temp.0006857136.meta vvel.0006844176.meta
ETA.0006848496.data pickup.0006828624.data
salt.0006840720.data STDERR.0118 STDERR.0320 STDOUT.0122
STDOUT.0324 temp.0006857568.data vvel.0006844608.data
ETA.0006848496.meta pickup.0006828624.meta
salt.0006840720.meta STDERR.0119 STDERR.0321 STDOUT.0123
STDOUT.0325 temp.0006857568.meta vvel.0006844608.meta
ETA.0006848928.data pickup.0006829056.data
salt.0006841152.data STDERR.0120 STDERR.0322 STDOUT.0124
STDOUT.0326 temp.0006858000.data vvel.0006845040.data
ETA.0006848928.meta pickup.0006829056.meta
salt.0006841152.meta STDERR.0121 STDERR.0323 STDOUT.0125
STDOUT.0327 temp.0006858000.meta vvel.0006845040.meta
ETA.0006849360.data pickup.0006829488.data
salt.0006841584.data STDERR.0122 STDERR.0324 STDOUT.0126
STDOUT.0328 temp.data vvel.0006845472.data
ETA.0006849360.meta pickup.0006829488.meta
salt.0006841584.meta STDERR.0123 STDERR.0325 STDOUT.0127
STDOUT.0329 temp.nc vvel.0006845472.meta
ETA.0006849792.data pickup.0006829920.data
salt.0006842016.data STDERR.0124 STDERR.0326 STDOUT.0128
STDOUT.0330 U.0006828192.data vvel.0006845904.data
ETA.0006849792.meta pickup.0006829920.meta
salt.0006842016.meta STDERR.0125 STDERR.0327 STDOUT.0129
STDOUT.0331 U.0006828192.meta vvel.0006845904.meta
ETA.0006850224.data pickup.0006830352.data
salt.0006842448.data STDERR.0126 STDERR.0328 STDOUT.0130
STDOUT.0332 U.0006831216.data vvel.0006846336.data
ETA.0006850224.meta pickup.0006830352.meta
salt.0006842448.meta STDERR.0127 STDERR.0329 STDOUT.0131
STDOUT.0333 U.0006831216.meta vvel.0006846336.meta
ETA.0006850656.data pickup.0006830784.data
salt.0006842880.data STDERR.0128 STDERR.0330 STDOUT.0132
STDOUT.0334 U.0006858000.data vvel.0006846768.data
ETA.0006850656.meta pickup.0006830784.meta
salt.0006842880.meta STDERR.0129 STDERR.0331 STDOUT.0133
STDOUT.0335 U.0006858000.meta vvel.0006846768.meta
ETA.0006851088.data pickup.0006831216.data
salt.0006843312.data STDERR.0130 STDERR.0332 STDOUT.0134
STDOUT.0336 uvel.0006831648.data vvel.0006847200.data
ETA.0006851088.meta pickup.0006831216.meta
salt.0006843312.meta STDERR.0131 STDERR.0333 STDOUT.0135
STDOUT.0337 uvel.0006831648.meta vvel.0006847200.meta
ETA.0006851520.data pickup.0006831648.data
salt.0006843744.data STDERR.0132 STDERR.0334 STDOUT.0136
STDOUT.0338 uvel.0006832080.data vvel.0006847632.data
ETA.0006851520.meta pickup.0006831648.meta
salt.0006843744.meta STDERR.0133 STDERR.0335 STDOUT.0137
STDOUT.0339 uvel.0006832080.meta vvel.0006847632.meta
ETA.0006851952.data pickup.0006832080.data
salt.0006844176.data STDERR.0134 STDERR.0336 STDOUT.0138
STDOUT.0340 uvel.0006832512.data vvel.0006848064.data
ETA.0006851952.meta pickup.0006832080.meta
salt.0006844176.meta STDERR.0135 STDERR.0337 STDOUT.0139
STDOUT.0341 uvel.0006832512.meta vvel.0006848064.meta
ETA.0006852384.data pickup.0006832512.data
salt.0006844608.data STDERR.0136 STDERR.0338 STDOUT.0140
STDOUT.0342 uvel.0006832944.data vvel.0006848496.data
ETA.0006852384.meta pickup.0006832512.meta
salt.0006844608.meta STDERR.0137 STDERR.0339 STDOUT.0141
STDOUT.0343 uvel.0006832944.meta vvel.0006848496.meta
ETA.0006852816.data pickup.0006832944.data
salt.0006845040.data STDERR.0138 STDERR.0340 STDOUT.0142
STDOUT.0344 uvel.0006833376.data vvel.0006848928.data
ETA.0006852816.meta pickup.0006832944.meta
salt.0006845040.meta STDERR.0139 STDERR.0341 STDOUT.0143
STDOUT.0345 uvel.0006833376.meta vvel.0006848928.meta
ETA.0006853248.data pickup.0006833376.data
salt.0006845472.data STDERR.0140 STDERR.0342 STDOUT.0144
STDOUT.0346 uvel.0006833808.data vvel.0006849360.data
ETA.0006853248.meta pickup.0006833376.meta
salt.0006845472.meta STDERR.0141 STDERR.0343 STDOUT.0145
STDOUT.0347 uvel.0006833808.meta vvel.0006849360.meta
ETA.0006853680.data pickup.0006833808.data
salt.0006845904.data STDERR.0142 STDERR.0344 STDOUT.0146
STDOUT.0348 uvel.0006834240.data vvel.0006849792.data
ETA.0006853680.meta pickup.0006833808.meta
salt.0006845904.meta STDERR.0143 STDERR.0345 STDOUT.0147
STDOUT.0349 uvel.0006834240.meta vvel.0006849792.meta
ETA.0006854112.data pickup.0006834240.data
salt.0006846336.data STDERR.0144 STDERR.0346 STDOUT.0148
STDOUT.0350 uvel.0006834672.data vvel.0006850224.data
ETA.0006854112.meta pickup.0006834240.meta
salt.0006846336.meta STDERR.0145 STDERR.0347 STDOUT.0149
STDOUT.0351 uvel.0006834672.meta vvel.0006850224.meta
ETA.0006854544.data pickup.0006834672.data
salt.0006846768.data STDERR.0146 STDERR.0348 STDOUT.0150
STDOUT.0352 uvel.0006835104.data vvel.0006850656.data
ETA.0006854544.meta pickup.0006834672.meta
salt.0006846768.meta STDERR.0147 STDERR.0349 STDOUT.0151
STDOUT.0353 uvel.0006835104.meta vvel.0006850656.meta
ETA.0006854976.data pickup.0006835104.data
salt.0006847200.data STDERR.0148 STDERR.0350 STDOUT.0152
STDOUT.0354 uvel.0006835536.data vvel.0006851088.data
ETA.0006854976.meta pickup.0006835104.meta
salt.0006847200.meta STDERR.0149 STDERR.0351 STDOUT.0153
STDOUT.0355 uvel.0006835536.meta vvel.0006851088.meta
ETA.0006855408.data pickup.0006835536.data
salt.0006847632.data STDERR.0150 STDERR.0352 STDOUT.0154
STDOUT.0356 uvel.0006835968.data vvel.0006851520.data
ETA.0006855408.meta pickup.0006835536.meta
salt.0006847632.meta STDERR.0151 STDERR.0353 STDOUT.0155
STDOUT.0357 uvel.0006835968.meta vvel.0006851520.meta
ETA.0006855840.data pickup.0006835968.data
salt.0006848064.data STDERR.0152 STDERR.0354 STDOUT.0156
STDOUT.0358 uvel.0006836400.data vvel.0006851952.data
ETA.0006855840.meta pickup.0006835968.meta
salt.0006848064.meta STDERR.0153 STDERR.0355 STDOUT.0157
STDOUT.0359 uvel.0006836400.meta vvel.0006851952.meta
ETA.0006856272.data pickup.0006836400.data
salt.0006848496.data STDERR.0154 STDERR.0356 STDOUT.0158
STDOUT.0360 uvel.0006836832.data vvel.0006852384.data
ETA.0006856272.meta pickup.0006836400.meta
salt.0006848496.meta STDERR.0155 STDERR.0357 STDOUT.0159
STDOUT.0361 uvel.0006836832.meta vvel.0006852384.meta
ETA.0006856704.data pickup.0006836832.data
salt.0006848928.data STDERR.0156 STDERR.0358 STDOUT.0160
STDOUT.0362 uvel.0006837264.data vvel.0006852816.data
ETA.0006856704.meta pickup.0006836832.meta
salt.0006848928.meta STDERR.0157 STDERR.0359 STDOUT.0161
STDOUT.0363 uvel.0006837264.meta vvel.0006852816.meta
ETA.0006857136.data pickup.0006837264.data
salt.0006849360.data STDERR.0158 STDERR.0360 STDOUT.0162
STDOUT.0364 uvel.0006837696.data vvel.0006853248.data
ETA.0006857136.meta pickup.0006837264.meta
salt.0006849360.meta STDERR.0159 STDERR.0361 STDOUT.0163
STDOUT.0365 uvel.0006837696.meta vvel.0006853248.meta
ETA.0006857568.data pickup.0006837696.data
salt.0006849792.data STDERR.0160 STDERR.0362 STDOUT.0164
STDOUT.0366 uvel.0006838128.data vvel.0006853680.data
ETA.0006857568.meta pickup.0006837696.meta
salt.0006849792.meta STDERR.0161 STDERR.0363 STDOUT.0165
STDOUT.0367 uvel.0006838128.meta vvel.0006853680.meta
Eta.0006858000.data pickup.0006838128.data
salt.0006850224.data STDERR.0162 STDERR.0364 STDOUT.0166
STDOUT.0368 uvel.0006838560.data vvel.0006854112.data
ETA.0006858000.data pickup.0006838128.meta
salt.0006850224.meta STDERR.0163 STDERR.0365 STDOUT.0167
STDOUT.0369 uvel.0006838560.meta vvel.0006854112.meta
Eta.0006858000.meta pickup.0006838560.data
salt.0006850656.data STDERR.0164 STDERR.0366 STDOUT.0168
STDOUT.0370 uvel.0006838992.data vvel.0006854544.data
ETA.0006858000.meta pickup.0006838560.meta
salt.0006850656.meta STDERR.0165 STDERR.0367 STDOUT.0169
STDOUT.0371 uvel.0006838992.meta vvel.0006854544.meta
eta1.nc pickup.0006838992.data
salt.0006851088.data STDERR.0166 STDERR.0368 STDOUT.0170
STDOUT.0372 uvel.0006839424.data vvel.0006854976.data
ETA.ctl pickup.0006838992.meta
salt.0006851088.meta STDERR.0167 STDERR.0369 STDOUT.0171
STDOUT.0373 uvel.0006839424.meta vvel.0006854976.meta
eta.data pickup.0006839424.data
salt.0006851520.data STDERR.0168 STDERR.0370 STDOUT.0172
STDOUT.0374 uvel.0006839856.data vvel.0006855408.data
etamean pickup.0006839424.meta
salt.0006851520.meta STDERR.0169 STDERR.0371 STDOUT.0173
STDOUT.0375 uvel.0006839856.meta vvel.0006855408.meta
glue.sh pickup.0006839856.data
salt.0006851952.data STDERR.0170 STDERR.0372 STDOUT.0174
STDOUT.0376 uvel.0006840288.data vvel.0006855840.data
hFacC.data pickup.0006839856.meta
salt.0006851952.meta STDERR.0171 STDERR.0373 STDOUT.0175
STDOUT.0377 uvel.0006840288.meta vvel.0006855840.meta
hFacC.meta pickup.0006840288.data
salt.0006852384.data STDERR.0172 STDERR.0374 STDOUT.0176
STDOUT.0378 uvel.0006840720.data vvel.0006856272.data
hFacS.data pickup.0006840288.meta
salt.0006852384.meta STDERR.0173 STDERR.0375 STDOUT.0177
STDOUT.0379 uvel.0006840720.meta vvel.0006856272.meta
hFacS.meta pickup.0006840720.data
salt.0006852816.data STDERR.0174 STDERR.0376 STDOUT.0178
STDOUT.0380 uvel.0006841152.data vvel.0006856704.data
hFacW.data pickup.0006840720.meta
salt.0006852816.meta STDERR.0175 STDERR.0377 STDOUT.0179
STDOUT.0381 uvel.0006841152.meta vvel.0006856704.meta
hFacW.meta pickup.0006841152.data
salt.0006853248.data STDERR.0176 STDERR.0378 STDOUT.0180
STDOUT.0382 uvel.0006841584.data vvel.0006857136.data
IO_10Km.e13207 pickup.0006841152.meta
salt.0006853248.meta STDERR.0177 STDERR.0379 STDOUT.0181
STDOUT.0383 uvel.0006841584.meta vvel.0006857136.meta
IO_10Km.o13207 pickup.0006841584.data
salt.0006853680.data STDERR.0178 STDERR.0380 STDOUT.0182
STDOUT.0384 uvel.0006842016.data vvel.0006857568.data
maskInC.data pickup.0006841584.meta
salt.0006853680.meta STDERR.0179 STDERR.0381 STDOUT.0183
STDOUT.0385 uvel.0006842016.meta vvel.0006857568.meta
maskInC.meta pickup.0006842016.data
salt.0006854112.data STDERR.0180 STDERR.0382 STDOUT.0184
STDOUT.0386 uvel.0006842448.data vvel.0006858000.data
maskInS.data pickup.0006842016.meta
salt.0006854112.meta STDERR.0181 STDERR.0383 STDOUT.0185
STDOUT.0387 uvel.0006842448.meta vvel.0006858000.meta
maskInS.meta pickup.0006842448.data
salt.0006854544.data STDERR.0182 STDERR.0384 STDOUT.0186
STDOUT.0388 uvel.0006842880.data W.0006828192.data
maskInW.data pickup.0006842448.meta
salt.0006854544.meta STDERR.0183 STDERR.0385 STDOUT.0187
STDOUT.0389 uvel.0006842880.meta W.0006828192.meta
maskInW.meta pickup.0006842880.data
salt.0006854976.data STDERR.0184 STDERR.0386 STDOUT.0188
STDOUT.0390 uvel.0006843312.data W.0006831216.data
mitgcmuv pickup.0006842880.meta
salt.0006854976.meta STDERR.0185 STDERR.0387 STDOUT.0189
STDOUT.0391 uvel.0006843312.meta W.0006831216.meta
mit-opr140.sh pickup.0006843312.data
salt.0006855408.data STDERR.0186 STDERR.0388 STDOUT.0190
STDOUT.0392 uvel.0006843744.data W.0006858000.data
mxld.0006831648.data pickup.0006843312.meta
salt.0006855408.meta STDERR.0187 STDERR.0389 STDOUT.0191
STDOUT.0393 uvel.0006843744.meta W.0006858000.meta
mxld.0006831648.meta pickup.0006843744.data
salt.0006855840.data STDERR.0188 STDERR.0390 STDOUT.0192
STDOUT.0394 uvel.0006844176.data XC.data
mxld.0006832080.data pickup.0006843744.meta
salt.0006855840.meta STDERR.0189 STDERR.0391 STDOUT.0193
STDOUT.0395 uvel.0006844176.meta XC.meta
mxld.0006832080.meta pickup.0006844176.data
salt.0006856272.data STDERR.0190 STDERR.0392 STDOUT.0194
STDOUT.0396 uvel.0006844608.data XG.data
mxld.0006832512.data pickup.0006844176.meta
salt.0006856272.meta STDERR.0191 STDERR.0393 STDOUT.0195
STDOUT.0397 uvel.0006844608.meta XG.meta
mxld.0006832512.meta pickup.0006844608.data
salt.0006856704.data STDERR.0192 STDERR.0394 STDOUT.0196
STDOUT.0398 uvel.0006845040.data YC.data
mxld.0006832944.data pickup.0006844608.meta
salt.0006856704.meta STDERR.0193 STDERR.0395 STDOUT.0197
STDOUT.0399 uvel.0006845040.meta YC.meta
mxld.0006832944.meta pickup.0006845040.data
salt.0006857136.data STDERR.0194 STDERR.0396 STDOUT.0198
T.0006828192.data uvel.0006845472.data YG.data
mxld.0006833376.data pickup.0006845040.meta
salt.0006857136.meta STDERR.0195 STDERR.0397 STDOUT.0199
T.0006828192.meta uvel.0006845472.meta YG.meta
mxld.0006833376.meta pickup.0006845472.data
salt.0006857568.data STDERR.0196 STDERR.0398 STDOUT.0200
T.0006831216.data uvel.0006845904.data
mxld.0006833808.data pickup.0006845472.meta
salt.0006857568.meta STDERR.0197 STDERR.0399 STDOUT.0201
T.0006831216.meta uvel.0006845904.meta
mxld.0006833808.meta pickup.0006845904.data
salt.0006858000.data STDERR.0198 STDOUT.0000 STDOUT.0202
T.0006858000.data uvel.0006846336.data
[shaileepatel at master2 exp]$ cat data
# ====================
# | Model parameters |
# ====================
#
# Continuous equation parameters
&PARM01
tRef = 15*23.,10*22.,21.,1*20.,19.,2*18.,17.,2*16.,15.,14.,13.,
12.,11.,2*9.,8.,7.,2*6.,2*5.,3*4.,3*3.,4*2.,10*1.,
sRef = 66*34.5,
viscAr= 1.77e-5,
no_slip_sides=.false.,
no_slip_bottom=.TRUE.,
diffKhT=0.,
diffK4T=0.,
diffKrT=1.51e-5,
diffKhS=0.,
diffK4S=0.,
diffKrS=1.51e-5,
tAlpha=2.E-4,
sBeta =7.4E-4,
gravity=9.81,
eosType='JMD95Z',
readBinaryPrec=32,
writeBinaryPrec=32,
useSingleCPUio=.TRUE.
globalFiles=.TRUE.
hFacMinDr=50.,
hFacMin=0.3,
hFacInf=0.1,
hFacSup=5.,
# z* - begin
# select_rStar=2,
# nonlinFreeSurf=4,
# * - end
implicitDiffusion=.true.,
implicitViscosity=.true.,
allowFreezing=.true.,
viscAh=0.,
viscAhGrid=0.,
viscC2Leith=0.,
viscA4=0.,
useFullLeith=.TRUE.
viscC4Leith=2.0,
viscC4Leithd=2.0,
useAnisotropicViscAGridMax=.TRUE.,
bottomDragQuadratic = 0.002,
tempAdvScheme=33,
saltAdvScheme=33,
StaggerTimeStep=.true.,
multiDimAdvection=.false.,
rigidLid=.FALSE.,
implicitFreeSurface=.TRUE.,
exactConserv=.TRUE.,
balanceEmPmR=.TRUE.,
convertFW2Salt=-1,
debuglevel=1,
&
# Elliptic solver parameters
&PARM02
cg2dMaxIters=1000,
cg2dTargetResidual=1.E-6,
&
# Time stepping parameters
&PARM03
startTime=1366243200,
endTime=1371600000,
abEps=0.1,
deltaT=200,
cAdjFreq=0.,
pChkptFreq=86400.,
# chkptFreq =86400.,
# dumpfreq = 21600.,
# tavefreq = 21600.,
#monitorFreq=86400.,
forcing_In_AB = .FALSE.,
pickupStrictlyMatch = .FALSE.,
&
# Gridding parameters
&PARM04
usingCartesianGrid=.FALSE.,
usingSphericalPolarGrid=.TRUE.,
delR =
15*1.,10*3.,6*5.,4*10.,11.13,14.46,19.82,26.94,35.48,44.99,55.00,65.01,74.52,83.06,90.18,95.54,98.87,4*100.00,111.42,143.93,192.60,250.00,307.40,356.07,388.58,7*400.00,
ygOrigin= -29.95,
xgOrigin= 28.05,
delY= 600*0.1,
delX= 880*0.1,
&
# Input datasets
&PARM05
bathyFile='../init/bathy_bs.bin'
hydrogThetaFile='../init/temp_Jan2006.grd',
hydrogSaltFile='../init/salt_Jan2006.grd',
#uVelInitFile='../init/u.grd.bin',
#vVelInitFile='../init/v.grd.bin',
#hydrogThetaFile='../init/OCT2016/tt-smooth.bin',
#hydrogSaltFile='../init/OCT2016/ss-smooth.bin',
#uVelInitFile='../init/OCT2016/uu-smooth.bin',
#vVelInitFile='../init/OCT2016/vv-smooth.bin',
#pSurfInitFile='init/ssh.grd.bin',
checkIniTemp=.true.,
checkIniSalt=.true.,
&
[shaileepatel at master2 exp]$ cat IO_10Km.e13207
NORMAL END
NORMAL END
NORMAL END
NORMAL END
[shaileepatel at master2 exp]$ cat mit-opr140.sh
#!/bin/bash
#PBS -l nodes=7:ppn=64,walltime=120:00:00
#PBS -q special2
#PBS -r n
#PBS -N IO_10Km
export I_MPI_HYDRA_BOOTSTRAP=rsh
export I_MPI_HYDRA_BOOTSTRAP_EXEC=pbs_tmrsh
cd /sachome2/shaileepatel/tulsi/MITgcm/verification/IO-10km/exp
#-----------------------------------------------------------------------------------------------#
#------------------------CHECKING PICKUP FILE
DURATION---------------------------#
#-----------------------------------------------------------------------------------------------#
mpirun -np 400 --hostfile $PBS_NODEFILE ./mitgcmuv
[shaileepatel at master2 exp]$ cat STDERR.0399
(PID.TID 0399.0001) ** Warning ** MNC_READPARMS: ignores "data.mnc" file since
(PID.TID 0399.0001) ** Warning ** MNC_READPARMS: useMNC= F (set from
"data.pkg")
(PID.TID 0399.0001) ** WARNING ** INI_MODEL_IO: use tiled-files to
write sections (for OBCS)
...........................................................................................................................................................................................................
4.Issue with Real Data + PDAF (Segmentation Fault)
[shaileepatel at master2 code_n2]$ pwd
/sachome2/shaileepatel/tulsi/MITgcm/verification/dupe1/code_n2
[shaileepatel at master2 dupe1]$ ls code_n2
assimilate_pdaf.F DIAGNOSTICS_SIZE.h get_taskid_pdaf.F
init_n_domains_pdaf.F init_pdaf_parse.F mod_parallel_pdaf.F
packages.conf SIZE.h
callback_obs_pdafomi.F distribute_state_pdaf.F init_dim_l_pdaf.F
init_parallel_pdaf.F mod_assimilation.F
next_observation_pdaf.F parser.F
collect_state_pdaf.F EXF_OPTIONS.h
init_dim_state_p_pdaf.F init_pdaf.F mod_mitgcm.F
OBCS_OPTIONS.h PP
CPP_OPTIONS.h finalize_pdaf.F init_ens_pdaf.F
init_pdaf_info.F mod_parallel_model.F obs_etan_pdafomi.F
prepoststep_ens_pdaf.F
[shaileepatel at master2 code_n2]$ cat SIZE.h
C $Header:
/u/gcmpack/MITgcm/verification/coupled_ucla/code/Attic/SIZE.h.56,v
1.1.2.1 2004/06/03 23:58:45 dimitri Exp $
C $Name: release1_coupled $
C
C /==========================================================\
C | SIZE.h Declare size of underlying computational grid. |
C |==========================================================|
C | The design here support a three-dimensional model grid |
C | with indices I,J and K. The three-dimensional domain |
C | is comprised of nPx*nSx blocks of size sNx along one axis|
C | nPy*nSy blocks of size sNy along another axis and one |
C | block of size Nz along the final axis. |
C | Blocks have overlap regions of size OLx and OLy along the|
C | dimensions that are subdivided. |
C \==========================================================/
C Voodoo numbers controlling data layout.
C sNx - No. X points in sub-grid.
C sNy - No. Y points in sub-grid.
C OLx - Overlap extent in X.
C OLy - Overlat extent in Y.
C nSx - No. sub-grids in X.
C nSy - No. sub-grids in Y.
C nPx - No. of processes to use in X.
C nPy - No. of processes to use in Y.
C Nx - No. points in X for the total domain.
C Ny - No. points in Y for the total domain.
C Nr - No. points in Z for full process domain.
INTEGER sNx
INTEGER sNy
INTEGER OLx
INTEGER OLy
INTEGER nSx
INTEGER nSy
INTEGER nPx
INTEGER nPy
INTEGER Nx
INTEGER Ny
INTEGER Nr
PARAMETER (
& sNx = 55,
& sNy = 75,
& OLx = 3,
& OLy = 3,
& nSx = 1,
& nSy = 1,
& nPx = 16,
& nPy = 8,
& Nx = sNx*nSx*nPx,
& Ny = sNy*nSy*nPy,
& Nr = 66)
C MAX_OLX - Set to the maximum overlap region size of any array
C MAX_OLY that will be exchanged. Controls the sizing of exch
C routine buufers.
INTEGER MAX_OLX
INTEGER MAX_OLY
PARAMETER ( MAX_OLX = OLx,
& MAX_OLY = OLy )
[shaileepatel at master2 run_n2]$ pwd
/sachome2/shaileepatel/tulsi/MITgcm/verification/dupe1/run_n2
[shaileepatel at master2 run_n2]$ cat IO_10Km.e17899
[node15:1101366:0:1101366] Caught signal 11 (Segmentation fault:
invalid permissions for mapped object at address 0x1515fc021000)
[node15:1101386:0:1101386] Caught signal 11 (Segmentation fault:
invalid permissions for mapped object at address 0x146648021000)
==== backtrace (tid:1101386) ====
==== backtrace (tid:1101366) ====
0 /lib64/libucs.so.0(ucs_handle_error+0x2dc) [0x15160794313c]
1 /lib64/libucs.so.0(+0x2c31c) [0x15160794331c]
2 /lib64/libucs.so.0(+0x2c4ea) [0x1516079434ea]
3 ./mitgcmuv() [0x40c9c8]
4 ./mitgcmuv() [0x7ff3a1]
5 ./mitgcmuv() [0x7f4228]
6 ./mitgcmuv() [0x4087df]
7 ./mitgcmuv() [0x781fd7]
8 ./mitgcmuv() [0x782370]
9 ./mitgcmuv() [0x699f4d]
10 ./mitgcmuv() [0x403f62]
11 /lib64/libc.so.6(__libc_start_main+0xe5) [0x1518602127e5]
12 ./mitgcmuv() [0x403e6e]
=================================
0 /lib64/libucs.so.0(ucs_handle_error+0x2dc) [0x14665396013c]
1 /lib64/libucs.so.0(+0x2c31c) [0x14665396031c]
2 /lib64/libucs.so.0(+0x2c4ea) [0x1466539604ea]
3 ./mitgcmuv() [0x40c9c8]
4 ./mitgcmuv() [0x7ff3a1]
5 ./mitgcmuv() [0x7f4228]
6 ./mitgcmuv() [0x4087df]
7 ./mitgcmuv() [0x781fd7]
8 ./mitgcmuv() [0x782370]
[node15:1101405:0:1101405] Caught signal 11 (Segmentation fault:
invalid permissions for mapped object at address 0x154104021000)
9 ./mitgcmuv() [0x699f4d]
10 ./mitgcmuv() [0x403f62]
11 /lib64/libc.so.6(__libc_start_main+0xe5) [0x1468ac22f7e5]
12 ./mitgcmuv() [0x403e6e]
=================================
[node15:1101344:0:1101344] Caught signal 11 (Segmentation fault:
invalid permissions for mapped object at address 0x1528f4021000)
[node16:257502:0:257502] Caught signal 11 (Segmentation fault: invalid
permissions for mapped object at address 0x14c550021000)
==== backtrace (tid:1101405) ====
0 /lib64/libucs.so.0(ucs_handle_error+0x2dc) [0x15410fca613c]
1 /lib64/libucs.so.0(+0x2c31c) [0x15410fca631c]
2 /lib64/libucs.so.0(+0x2c4ea) [0x15410fca64ea]
3 ./mitgcmuv() [0x40c9c8]
4 ./mitgcmuv() [0x7ff3a1]
5 ./mitgcmuv() [0x7f4228]
6 ./mitgcmuv() [0x4087df]
7 ./mitgcmuv() [0x781fd7]
8 ./mitgcmuv() [0x782370]
9 ./mitgcmuv() [0x699f4d]
10 ./mitgcmuv() [0x403f62]
11 /lib64/libc.so.6(__libc_start_main+0xe5) [0x1543685757e5]
12 ./mitgcmuv() [0x403e6e]
=================================
[node16:257523:0:257523] Caught signal 11 (Segmentation fault: invalid
permissions for mapped object at address 0x14dc20021000)
[node16:257547:0:257547] Caught signal 11 (Segmentation fault: invalid
permissions for mapped object at address 0x149890021000)
[node16:257499:0:257499] Caught signal 11 (Segmentation fault: invalid
permissions for mapped object at address 0x14a7f8021000)
[node16:257529:0:257529] Caught signal 11 (Segmentation fault: invalid
permissions for mapped object at address 0x1461b4021000)
[node16:257539:0:257539] Caught signal 11 (Segmentation fault: invalid
permissions for mapped object at address 0x15294c021000)
==== backtrace (tid:1101344) ====
0 /lib64/libucs.so.0(ucs_handle_error+0x2dc) [0x1528ffe5113c]
1 /lib64/libucs.so.0(+0x2c31c) [0x1528ffe5131c]
2 /lib64/libucs.so.0(+0x2c4ea) [0x1528ffe514ea]
==== backtrace (tid: 257523) ====
0 /lib64/libucs.so.0(ucs_handle_error+0x2dc) [0x14dc2bf1d13c]
1 /lib64/libucs.so.0(+0x2c31c) [0x14dc2bf1d31c]
2 /lib64/libucs.so.0(+0x2c4ea) [0x14dc2bf1d4ea]
3 ./mitgcmuv() [0x40c9c8]
4 ./mitgcmuv() [0x7ff3a1]
5 ./mitgcmuv() [0x7f4228]
6 ./mitgcmuv() [0x4087df]
7 ./mitgcmuv() [0x781fd7]
8 ./mitgcmuv() [0x782370]
9 ./mitgcmuv() [0x699f4d]
10 ./mitgcmuv() [0x403f62]
11 /lib64/libc.so.6(__libc_start_main+0xe5) [0x14de847ec7e5]
12 ./mitgcmuv() [0x403e6e]
=================================
3 ./mitgcmuv() [0x40c9c8]
[shaileepatel at master2 run_n2]$ cat IO_10Km.o17899
PARSER: dim_ens= 1
Initialize communicators for assimilation with PDAF
Configuration of parallelization:
world filter model couple filterPE
rank rank task rank task rank T/F
-----------------------------------------------------
0 0 1 0 1 0 T
28 28 1 28 29 0 T
1 1 1 1 2 0 T
57 57 1 57 58 0 T
5 5 1 5 6 0 T
7 7 1 7 8 0 T
11 11 1 11 12 0 T
33 33 1 33 34 0 T
112 112 1 112 113 0 T
117 117 1 117 118 0 T
118 118 1 118 119 0 T
116 116 1 116 117 0 T
87 87 1 87 88 0 T
80 80 1 80 81 0 T
19 19 1 19 20 0 T
23 23 1 23 24 0 T
101 101 1 101 102 0 T
40 40 1 40 41 0 T
25 25 1 25 26 0 T
36 36 1 36 37 0 T
84 84 1 84 85 0 T
65 65 1 65 66 0 T
90 90 1 90 91 0 T
67 67 1 67 68 0 T
37 37 1 37 38 0 T
30 30 1 30 31 0 T
74 74 1 74 75 0 T
75 75 1 75 76 0 T
120 120 1 120 121 0 T
2 2 1 2 3 0 T
22 22 1 22 23 0 T
99 99 1 99 100 0 T
111 111 1 111 112 0 T
51 51 1 51 52 0 T
17 17 1 17 18 0 T
104 104 1 104 105 0 T
18 18 1 18 19 0 T
71 71 1 71 72 0 T
45 45 1 45 46 0 T
127 127 1 127 128 0 T
56 56 1 56 57 0 T
73 73 1 73 74 0 T
42 42 1 42 43 0 T
53 53 1 53 54 0 T
76 76 1 76 77 0 T
113 113 1 113 114 0 T
72 72 1 72 73 0 T
110 110 1 110 111 0 T
52 52 1 52 53 0 T
14 14 1 14 15 0 T
10 10 1 10 11 0 T
29 29 1 29 30 0 T
55 55 1 55 56 0 T
15 15 1 15 16 0 T
49 49 1 49 50 0 T
54 54 1 54 55 0 T
83 83 1 83 84 0 T
26 26 1 26 27 0 T
69 69 1 69 70 0 T
70 70 1 70 71 0 T
89 89 1 89 90 0 T
78 78 1 78 79 0 T
107 107 1 107 108 0 T
20 20 1 20 21 0 T
86 86 1 86 87 0 T
9 9 1 9 10 0 T
77 77 1 77 78 0 T
31 31 1 31 32 0 T
8 8 1 8 9 0 T
13 13 1 13 14 0 T
108 108 1 108 109 0 T
27 27 1 27 28 0 T
32 32 1 32 33 0 T
93 93 1 93 94 0 T
59 59 1 59 60 0 T
85 85 1 85 86 0 T
38 38 1 38 39 0 T
91 91 1 91 92 0 T
16 16 1 16 17 0 T
61 61 1 61 62 0 T
48 48 1 48 49 0 T
105 105 1 105 106 0 T
39 39 1 39 40 0 T
103 103 1 103 104 0 T
21 21 1 21 22 0 T
41 41 1 41 42 0 T
81 81 1 81 82 0 T
97 97 1 97 98 0 T
109 109 1 109 110 0 T
43 43 1 43 44 0 T
115 115 1 115 116 0 T
124 124 1 124 125 0 T
24 24 1 24 25 0 T
58 58 1 58 59 0 T
92 92 1 92 93 0 T
126 126 1 126 127 0 T
63 63 1 63 64 0 T
121 121 1 121 122 0 T
6 6 1 6 7 0 T
35 35 1 35 36 0 T
46 46 1 46 47 0 T
47 47 1 47 48 0 T
50 50 1 50 51 0 T
79 79 1 79 80 0 T
96 96 1 96 97 0 T
95 95 1 95 96 0 T
106 106 1 106 107 0 T
125 125 1 125 126 0 T
34 34 1 34 35 0 T
3 3 1 3 4 0 T
119 119 1 119 120 0 T
123 123 1 123 124 0 T
60 60 1 60 61 0 T
100 100 1 100 101 0 T
4 4 1 4 5 0 T
68 68 1 68 69 0 T
12 12 1 12 13 0 T
62 62 1 62 63 0 T
88 88 1 88 89 0 T
102 102 1 102 103 0 T
44 44 1 44 45 0 T
64 64 1 64 65 0 T
66 66 1 66 67 0 T
82 82 1 82 83 0 T
94 94 1 94 95 0 T
98 98 1 98 99 0 T
114 114 1 114 115 0 T
122 122 1 122 123 0 T
PDAFuser: INITIALIZE PDAF - ONLINE MODE
PARSER: dim_ens= 1
PDAFuser: State dimension: 1093125
PDAFuser: Filter: LESTKF
PDAFuser: -- Standard mode
PDAFuser: ensemble size: 1
PDAFuser: Assimilation interval: 432
PDAFuser: forgetting factor: 1.00
PDAF ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PDAF +++ PDAF +++
PDAF +++ Parallel Data Assimilation Framework +++
PDAF +++ +++
PDAF +++ Version 3.0beta +++
PDAF +++ +++
PDAF +++ Please cite +++
PDAF +++ L. Nerger and W. Hiller, Computers and Geosciences +++
PDAF +++ 2013, 55, 110-118, doi:10.1016/j.cageo.2012.03.026 +++
PDAF +++ when publishing work resulting from using PDAF +++
PDAF +++ +++
PDAF +++ PDAF itself can also be cited as +++
PDAF +++ L. Nerger. Parallel Data Assimilation Framework +++
PDAF +++ (PDAF). Zenodo. 2024. doi:10.5281/zenodo.7861812 +++
PDAF ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PDAF: Initialize filter
PDAF +++++++++++++++++++++++++++++++++++++++++++++++++++++++
PDAF +++ Local Error Subspace Transform Kalman Filter +++
PDAF +++ (LESTKF) +++
PDAF +++ +++
PDAF +++ Domain-localized implementation of the ESTKF by +++
PDAF +++ Nerger et al., Mon. Wea. Rev. 140 (2012) 2335 +++
PDAF +++ doi:10.1175/MWR-D-11-00102.1 +++
PDAF +++++++++++++++++++++++++++++++++++++++++++++++++++++++
PDAF: Initialize Parallelization
PDAF Parallelization - Filter on model PEs:
PDAF Total number of PEs: 128
PDAF Number of parallel model tasks: 1
PDAF PEs for Filter: 128
PDAF # PEs per ensemble task and local ensemble sizes:
PDAF Task 1
PDAF #PEs 128
PDAF N 1
PDAF: Call ensemble initialization
PDAFuser: Initialize state ensemble
PDAFuser: --- read ensemble from files
PDAFuser: --- Ensemble size: 1
PDAF: Initialization completed
PDAF --- duration of PDAF initialization: 0.014 s
PDAF LESTKF configuration
PDAF ensemble size: 1
PDAF filter sub-type= 0
PDAF --> Standard LESTKF
PDAF param_int(5) type_forget= 0
PDAF --> Use fixed forgetting factor: 1.00
PDAF param_int(6) type_trans= 0
PDAF --> Transform ensemble with deterministic Omega (default)
PDAF param_int(7) type_sqrt= 0
PDAF --> symmetric square root (default)
PDAF param_int(8) observe_ens
PDAF --> 0: Apply H to ensemble mean to compute innovation
PDAF param_int(9) type_obs_init= 1
PDAF --> Initialize observations after PDAF prestep
PDAF param_real(1) forget= 1.000
PDAF ----------------------------------------------------------------
PDAF +++++ ASSIMILATION +++++
PDAF ----------------------------------------------------------------
PDAF Call pre-post routine at initial time
PDAFuser: Analize initial state ensemble
PDAFuser: --- compute ensemble mean
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 0 PID 1101343 RUNNING AT node15.localdomain
= KILLED BY SIGNAL: 9 (Killed)
===================================================================================
[shaileepatel at master2 run_n2]$ cat STDERR.0001.0127
(PID.TID 0127.0001) ** Warning ** MNC_READPARMS: ignores "data.mnc" file since
(PID.TID 0127.0001) ** Warning ** MNC_READPARMS: useMNC= F (set from
"data.pkg")
(PID.TID 0127.0001) ** WARNING ** INI_MODEL_IO: use tiled-files to
write sections (for OBCS)
.....................................................................................................................................................................................................................................................
5.Current Issue: MPI Process Mismatch (EEBOOT_MINIMAL Error) (here mitgcm + pdaf with real data )
[shaileepatel at master2 code]$ pwd
/sachome2/shaileepatel/tulsi/MITgcm/verification/25km/code
[shaileepatel at master2 code]$ ls
assimilate_pdaf.F DIAGNOSTICS_SIZE.h finalize_pdaf.F
init_n_domains_pdaf.F init_pdaf_parse.F mod_parallel_pdaf.F
packages.conf SIZE.h
callback_obs_pdafomi.F distribute_state_pdaf.F get_taskid_pdaf.F
init_parallel_pdaf.F mod_assimilation.F next_observation_pdaf.F
parser.F
collect_state_pdaf.F init_dim_l_pdaf.F init_pdaf.F
mod_mitgcm.F
OBCS_OPTIONS.h PP
CPP_OPTIONS.h EXF_OPTIONS.h init_ens_pdaf.F
init_pdaf_info.F mod_parallel_model.F obs_etan_pdafomi.F
prepoststep_ens_pdaf.F
[shaileepatel at master2 code]$ cat SIZE.h
C $Header:
/u/gcmpack/MITgcm/verification/coupled_ucla/code/Attic/SIZE.h.56,v
1.1.2.1 2004/06/03 23:58:45 dimitri Exp $
C $Name: release1_coupled $
C
C /==========================================================\
C | SIZE.h Declare size of underlying computational grid. |
C |==========================================================|
C | The design here support a three-dimensional model grid |
C | with indices I,J and K. The three-dimensional domain |
C | is comprised of nPx*nSx blocks of size sNx along one axis|
C | nPy*nSy blocks of size sNy along another axis and one |
C | block of size Nz along the final axis. |
C | Blocks have overlap regions of size OLx and OLy along the|
C | dimensions that are subdivided. |
C \==========================================================/
C Voodoo numbers controlling data layout.
C sNx - No. X points in sub-grid.
C sNy - No. Y points in sub-grid.
C OLx - Overlap extent in X.
C OLy - Overlat extent in Y.
C nSx - No. sub-grids in X.
C nSy - No. sub-grids in Y.
C nPx - No. of processes to use in X.
C nPy - No. of processes to use in Y.
C Nx - No. points in X for the total domain.
C Ny - No. points in Y for the total domain.
C Nr - No. points in Z for full process domain.
INTEGER sNx
INTEGER sNy
INTEGER OLx
INTEGER OLy
INTEGER nSx
INTEGER nSy
INTEGER nPx
INTEGER nPy
INTEGER Nx
INTEGER Ny
INTEGER Nr
PARAMETER (
& sNx = 32,
& sNy = 32,
& OLx = 3,
& OLy = 3,
& nSx = 1,
& nSy = 1,
& nPx = 4,
& nPy = 2,
& Nx = sNx*nSx*nPx,
& Ny = sNy*nSy*nPy,
& Nr = 66)
C MAX_OLX - Set to the maximum overlap region size of any array
C MAX_OLY that will be exchanged. Controls the sizing of exch
C routine buufers.
INTEGER MAX_OLX
INTEGER MAX_OLY
PARAMETER ( MAX_OLX = OLx,
& MAX_OLY = OLy )
[shaileepatel at master2 run1]$ pwd
/sachome2/shaileepatel/tulsi/MITgcm/verification/25km/run1
[shaileepatel at master2 run1]$ ls
data IO_10Km.e18866 IO_10Km.o18809 STDERR.0001.0000
data.cal IO_10Km.e19009 IO_10Km.o18811 STDERR.0002.0000
data.diagnostics IO_10Km.e19013 IO_10Km.o18866 STDERR.0003.0000
data.exf IO_10Km.e19014 IO_10Km.o19009 STDERR.0004.0000
data.exf.master IO_10Km.e19018 IO_10Km.o19013 STDERR.0005.0000
data.kpp IO_10Km.e19019 IO_10Km.o19014 STDERR.0006.0000
data.mnc IO_10Km.e19020 IO_10Km.o19018 STDERR.0007.0000
data.obcs IO_10Km.e19034 IO_10Km.o19019 STDERR.0008.0000
data.pkg IO_10Km.e19037 IO_10Km.o19020 STDOUT.0000.0000
data.temp IO_10Km.e19038 IO_10Km.o19034 STDOUT.0000.0001
eedata IO_10Km.e19063 IO_10Km.o19037 STDOUT.0001.0000
IO_10Km.e17940 IO_10Km.e19069 IO_10Km.o19038 STDOUT.0002.0000
IO_10Km.e18693 IO_10Km.e19264 IO_10Km.o19063 STDOUT.0003.0000
IO_10Km.e18702 IO_10Km.o17940 IO_10Km.o19069 STDOUT.0004.0000
IO_10Km.e18795 IO_10Km.o18693 IO_10Km.o19264 STDOUT.0005.0000
IO_10Km.e18798 IO_10Km.o18702 mitgcmuv STDOUT.0006.0000
IO_10Km.e18800 IO_10Km.o18795 mit-opr140.sh STDOUT.0007.0000
IO_10Km.e18807 IO_10Km.o18798 pdaf.nml STDOUT.0008.0000
IO_10Km.e18809 IO_10Km.o18800 STDERR.0000.0000 temp.ctl
IO_10Km.e18811 IO_10Km.o18807 STDERR.0000.0001
run for only , #mpirun -np 1 --hostfile $PBS_NODEFILE ./mitgcmuv
[shaileepatel at master2 run1]$ cat IO_10Km.e17940
** Warning ** MNC_READPARMS: ignores "data.mnc" file since
** Warning ** MNC_READPARMS: useMNC= F (set from "data.pkg")
** Warning ** CAL_READPARMS: ignores "data.cal" file since
** Warning ** CAL_READPARMS: useCAL= F (set from "data.pkg")
** Warning ** EXF_READPARMS: ignores "data.exf" file since
** Warning ** EXF_READPARMS: useEXF= F (set from "data.pkg")
** Warning ** OBCS_READPARMS: ignores "data.obcs" file since
** Warning ** OBCS_READPARMS: useOBCS= F (set from "data.pkg")
** Warning ** DIAGNOSTICS_READPARMS: ignores "data.diagnostics" file since
** Warning ** DIAGNOSTICS_READPARMS: useDiagnostics= F (set from "data.pkg")
NORMAL END
[shaileepatel at master2 run1]$ cat IO_10Km.o17940
PARSER: dim_ens= 64
Initialize communicators for assimilation with PDAF
!!! Resetting number of parallel ensemble tasks
to total number of PEs
Configuration of parallelization:
world filter model couple filterPE
rank rank task rank task rank T/F
-----------------------------------------------------
0 0 1 0 1 0 T
PDAFuser: INITIALIZE PDAF - ONLINE MODE
PARSER: dim_ens= 64
PDAFuser: State dimension: 2170880
PDAFuser: Filter: LESTKF
PDAFuser: -- Standard mode
PDAFuser: ensemble size: 64
PDAFuser: Assimilation interval: 432
PDAFuser: forgetting factor: 1.00
PDAF ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PDAF +++ PDAF +++
PDAF +++ Parallel Data Assimilation Framework +++
PDAF +++ +++
PDAF +++ Version 3.0beta +++
PDAF +++ +++
PDAF +++ Please cite +++
PDAF +++ L. Nerger and W. Hiller, Computers and Geosciences +++
PDAF +++ 2013, 55, 110-118, doi:10.1016/j.cageo.2012.03.026 +++
PDAF +++ when publishing work resulting from using PDAF +++
PDAF +++ +++
PDAF +++ PDAF itself can also be cited as +++
PDAF +++ L. Nerger. Parallel Data Assimilation Framework +++
PDAF +++ (PDAF). Zenodo. 2024. doi:10.5281/zenodo.7861812 +++
PDAF ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PDAF: Initialize filter
PDAF +++++++++++++++++++++++++++++++++++++++++++++++++++++++
PDAF +++ Local Error Subspace Transform Kalman Filter +++
PDAF +++ (LESTKF) +++
PDAF +++ +++
PDAF +++ Domain-localized implementation of the ESTKF by +++
PDAF +++ Nerger et al., Mon. Wea. Rev. 140 (2012) 2335 +++
PDAF +++ doi:10.1175/MWR-D-11-00102.1 +++
PDAF +++++++++++++++++++++++++++++++++++++++++++++++++++++++
PDAF: Initialize Parallelization
PDAF Parallelization - Filter on model PEs:
PDAF Total number of PEs: 1
PDAF Number of parallel model tasks: 1
PDAF PEs for Filter: 1
PDAF # PEs per ensemble task and local ensemble sizes:
PDAF Task 1
PDAF #PEs 1
PDAF N 64
PDAF: Call ensemble initialization
PDAFuser: Initialize state ensemble
PDAFuser: --- read ensemble from files
PDAFuser: --- Ensemble size: 64
PDAF: Initialization completed
PDAF --- duration of PDAF initialization: 0.196 s
PDAF LESTKF configuration
PDAF ensemble size: 64
PDAF filter sub-type= 0
PDAF --> Standard LESTKF
PDAF param_int(5) type_forget= 0
PDAF --> Use fixed forgetting factor: 1.00
PDAF param_int(6) type_trans= 0
PDAF --> Transform ensemble with deterministic Omega (default)
PDAF param_int(7) type_sqrt= 0
PDAF --> symmetric square root (default)
PDAF param_int(8) observe_ens
PDAF --> 0: Apply H to ensemble mean to compute innovation
PDAF param_int(9) type_obs_init= 1
PDAF --> Initialize observations after PDAF prestep
PDAF param_real(1) forget= 1.000
PDAF ----------------------------------------------------------------
PDAF +++++ ASSIMILATION +++++
PDAF ----------------------------------------------------------------
PDAF Call pre-post routine at initial time
PDAFuser: Analize initial state ensemble
PDAFuser: --- compute ensemble mean
PDAFuser: RMS errors according to sampled variance:
PDAFuser: etaN theta salt uvel vvel
PDAFuser: 1.8619E-01 1.8619E-01 1.8619E-01 1.8619E-01
1.8619E-01 RMSe-ini
PDAF --- duration of prestep: 6.171 s
PDAF Forecast -------------------------------------------------------
PDAF Evolve state ensemble
PDAF PDAF Timing information - call-back routines
PDAF ----------------------------------------------------
PDAF Initialize PDAF: 0.196 s
PDAF init_ens_pdaf: 0.196 s
PDAF Ensemble forecast: 0.000 s
PDAF MPI communication in PDAF: 0.000 s
PDAF distribute_state_pdaf: 0.011 s
PDAF collect_state_pdaf: 0.006 s
PDAF LESTKF analysis: 0.000 s
PDAF PDAF-internal operations: 0.000 s
PDAF init_n_domains_pdaf: 0.000 s
PDAF init_dim_obs_f_pdaf: 0.000 s
PDAF obs_op_f_pdaf: 0.000 s
PDAF init_dim_l_pdaf: 0.000 s
PDAF init_dim_obs_l_pdaf: 0.000 s
PDAF g2l_state_pdaf: 0.000 s
PDAF g2l_obs_pdaf: 0.000 s
PDAF init_obs_l_pdaf: 0.000 s
PDAF prodRinvA_l_pdaf: 0.000 s
PDAF l2g_state_pdaf: 0.000 s
PDAF prepoststep_pdaf: 6.171 s
PDAF PDAF Memory overview
PDAF ---------------------------------------------
PDAF Allocated memory (MiB)
PDAF state and A: 16.593 MiB (persistent)
PDAF ensemble array: 1060.000 MiB (persistent)
PDAF analysis step: 0.000 MiB (temporary)
PDAFuser: Assimilation with PDAF completed!
but not generated any output (.data) files .
but for , #mpirun -np 8 --hostfile $PBS_NODEFILE ./mitgcmuv
[shaileepatel at master2 run1]$ cat IO_10Km.o19264
===== RUNNING MITGCM =====
PARSER: dim_ens= 8
Initialize communicators for assimilation with PDAF
Configuration of parallelization:
world filter model couple filterPE
rank rank task rank task rank T/F
-----------------------------------------------------
0 0 1 0 1 0 T
3 4 0 1 3 F
6 7 0 1 6 F
1 2 0 1 1 F
2 3 0 1 2 F
4 5 0 1 4 F
5 6 0 1 5 F
7 8 0 1 7 F
[shaileepatel at master2 run1]$ cat IO_10Km.e19264
EEBOOT_MINIMAL: No. of procs= 1 not equal to nPx*nPy= 8
EEDIE: earlier error in multi-proc/thread setting
EEBOOT_MINIMAL: No. of procs= 1 not equal to nPx*nPy= 8
EEDIE: earlier error in multi-proc/thread setting
EEBOOT_MINIMAL: No. of procs= 1 not equal to nPx*nPy= 8
EEBOOT_MINIMAL: No. of procs= 1 not equal to nPx*nPy= 8
EEDIE: earlier error in multi-proc/thread setting
EEDIE: earlier error in multi-proc/thread setting
EEBOOT_MINIMAL: No. of procs= 1 not equal to nPx*nPy= 8
EEDIE: earlier error in multi-proc/thread setting
EEBOOT_MINIMAL: No. of procs= 1 not equal to nPx*nPy= 8
EEBOOT_MINIMAL: No. of procs= 1 not equal to nPx*nPy= 8
EEBOOT_MINIMAL: No. of procs= 1 not equal to nPx*nPy= 8
EEDIE: earlier error in multi-proc/thread setting
EEDIE: earlier error in multi-proc/thread setting
EEDIE: earlier error in multi-proc/thread setting
PROGRAM MAIN: ends with fatal Error
PROGRAM MAIN: ends with fatal Error
PROGRAM MAIN: ends with fatal Error
PROGRAM MAIN: ends with fatal Error
ABNORMAL END: PROGRAM MAIN
ABNORMAL END: PROGRAM MAIN
PROGRAM MAIN: ends with fatal Error
PROGRAM MAIN: ends with fatal Error
ABNORMAL END: PROGRAM MAIN
PROGRAM MAIN: ends with fatal Error
ABNORMAL END: PROGRAM MAIN
PROGRAM MAIN: ends with fatal Error
ABNORMAL END: PROGRAM MAIN
ABNORMAL END: PROGRAM MAIN
ABNORMAL END: PROGRAM MAIN
ABNORMAL END: PROGRAM MAIN
[shaileepatel at master2 run1]$ cat STDERR.0008.0000
(PID.TID 0000.0001) *** ERROR *** EEBOOT_MINIMAL: No. of procs= 1
not equal to nPx*nPy= 8
(PID.TID 0000.0001) *** ERROR *** EEDIE: earlier error in
multi-proc/thread setting
(PID.TID 0000.0001) *** ERROR *** PROGRAM MAIN: ends with fatal Error
[shaileepatel at master2 run1]$ cat STDOUT.0008.0000
PROGRAM MAIN: ends with fatal Error
[shaileepatel at master2 run1]$ cat data
# ====================
# | Model parameters |
# ====================
#
# Continuous equation parameters
&PARM01
tRef = 15*23.,10*22.,21.,1*20.,19.,2*18.,17.,2*16.,15.,14.,13.,
12.,11.,2*9.,8.,7.,2*6.,2*5.,3*4.,3*3.,4*2.,10*1.,
sRef = 66*34.5,
viscAr= 1.77e-5,
no_slip_sides=.false.,
no_slip_bottom=.TRUE.,
diffKhT=0.,
diffK4T=0.,
diffKrT=1.51e-5,
diffKhS=0.,
diffK4S=0.,
diffKrS=1.51e-5,
tAlpha=2.E-4,
sBeta =7.4E-4,
gravity=9.81,
eosType='JMD95Z',
readBinaryPrec=32,
writeBinaryPrec=32,
useSingleCPUio=.TRUE.
globalFiles=.TRUE.
hFacMinDr=50.,
hFacMin=0.3,
hFacInf=0.1,
hFacSup=5.,
# z* - begin
# select_rStar=2,
# nonlinFreeSurf=4,
# * - end
implicitDiffusion=.true.,
implicitViscosity=.true.,
allowFreezing=.true.,
viscAh=0.,
viscAhGrid=0.,
viscC2Leith=0.,
viscA4=0.,
useFullLeith=.TRUE.
viscC4Leith=2.0,
viscC4Leithd=2.0,
useAnisotropicViscAGridMax=.TRUE.,
bottomDragQuadratic = 0.002,
tempAdvScheme=33,
saltAdvScheme=33,
StaggerTimeStep=.true.,
multiDimAdvection=.false.,
rigidLid=.FALSE.,
implicitFreeSurface=.TRUE.,
exactConserv=.TRUE.,
balanceEmPmR=.TRUE.,
convertFW2Salt=-1,
debuglevel=1,
&
# Elliptic solver parameters
&PARM02
cg2dMaxIters=1000,
cg2dTargetResidual=1.E-6,
&
# Time stepping parameters
&PARM03
startTime=0,
endTime=172800,
abEps=0.1,
deltaT=200,
cAdjFreq=0.,
pChkptFreq=86400.,
# chkptFreq =86400.,
# dumpfreq = 21600.,
# tavefreq = 21600.,
#monitorFreq=86400.,
forcing_In_AB = .FALSE.,
pickupStrictlyMatch = .FALSE.,
&
# Gridding parameters
&PARM04
usingCartesianGrid=.FALSE.,
usingSphericalPolarGrid=.TRUE.,
delR =
15*1.,10*3.,6*5.,4*10.,11.13,14.46,19.82,26.94,35.48,44.99,55.00,65.01,74.52,83.06,90.18,95.54,98.87,4*100.00,111.42,143.93,192.60,250.00,307.40,356.07,388.58,7*400.00,
ygOrigin= 10.00,
xgOrigin= 50.00,
delY= 64*0.25,
delX= 128*0.25,
&
# Input datasets
&PARM05
bathyFile='../init/25_topo.grd'
#hydrogThetaFile='../init/temp_Jan2006.grd',
#hydrogSaltFile='../init/salt_Jan2006.grd',
#uVelInitFile='../init/u.grd.bin',
#vVelInitFile='../init/v.grd.bin',
#hydrogThetaFile='../init/OCT2016/tt-smooth.bin',
#hydrogSaltFile='../init/OCT2016/ss-smooth.bin',
#uVelInitFile='../init/OCT2016/uu-smooth.bin',
#vVelInitFile='../init/OCT2016/vv-smooth.bin',
#pSurfInitFile='init/ssh.grd.bin',
checkIniTemp=.true.,
checkIniSalt=.true.,
&
run script, mit-opr140.sh
[shaileepatel at master2 run1]$ cat mit-opr140.sh
#!/bin/bash
#PBS -l nodes=1:ppn=64,walltime=120:00:00
#PBS -q special2
#PBS -r n
#PBS -N IO_10Km
module load intel/2021.2.0
module load mpi/2021.2.0
export I_MPI_HYDRA_BOOTSTRAP=rsh
export I_MPI_HYDRA_BOOTSTRAP_EXEC=pbs_tmrsh
cd /sachome2/shaileepatel/tulsi/MITgcm/verification/25km/run1
#-----------------------------------------------------------------------------------------------#
#------------------------CHECKING PICKUP FILE
DURATION---------------------------#
#-----------------------------------------------------------------------------------------------#
#mpirun -np 8 --hostfile $PBS_NODEFILE ./mitgcmuv
More information about the MITgcm-support
mailing list