[MITgcm-support] cubeSphereExchange and MPI error?

Anthony Coletti ajcolett at geo.umass.edu
Tue Jul 8 08:45:40 EDT 2014 

Hi Jean-Michel,

Let’s start with the exch2 problem,

So I check the PACKAGES_CONFIG.h and it seems you are right..for some reason, exch2 is undef

Here is the copy of my PACKAGES_CONFIG.h file:

reated by convert_cpp_cmd2defines with the following command line arguments:
 -bPACKAGES_CONFIG_H Disabled packages: -UALLOW_ADMTLM -UALLOW_AIM_V23 -UALLOW_ATM2D -UALLOW_ATM_COMMON -UALLOW_A
TM_COMPON_INTERF -UALLOW_ATM_OCN_COUPLER -UALLOW_ATM_PHYS -UALLOW_AUTODIFF -UALLOW_BBL -UALLOW_BULK_FORCE -UALLOW
_CAL -UALLOW_CD_CODE -UALLOW_CFC -UALLOW_CHEAPAML -UALLOW_CHRONOS -UALLOW_COMPON_COMMUNIC -UALLOW_COST -UALLOW_CT
RL -UALLOW_DIAGNOSTICS -UALLOW_DIC -UALLOW_DOWN_SLOPE -UALLOW_EBM -UALLOW_ECCO -UALLOW_EMBED_FILES -UALLOW_EXCH2 
-UALLOW_EXF -UALLOW_FIZHI -UALLOW_FLT -UALLOW_FRAZIL -UALLOW_GCHEM -UALLOW_GGL90 -UALLOW_GMREDI -UALLOW_GRDCHK -U
ALLOW_GRIDALT -UALLOW_ICEFRONT -UALLOW_KPP -UALLOW_LAND -UALLOW_LAYERS -UALLOW_LONGSTEP -UALLOW_MATRIX -UALLOW_MN
C -UALLOW_MY82 -UALLOW_MYPACKAGE -UALLOW_OBCS -UALLOW_OCN_COMPON_INTERF -UALLOW_OFFLINE -UALLOW_OPENAD -UALLOW_OP
PS -UALLOW_PP81 -UALLOW_PROFILES -UALLOW_PTRACERS -UALLOW_RBCS -UALLOW_REGRID -UALLOW_RUNCLOCK -UALLOW_SALT_PLUME
 -UALLOW_SBO -UALLOW_SEAICE -UALLOW_SHAP_FILT -UALLOW_SHELFICE -UALLOW_SHOWFLOPS -UALLOW_SMOOTH -UALLOW_SPHERE -U
ALLOW_STREAMICE -UALLOW_THSICE -UALLOW_TIMEAVE -UALLOW_ZONAL_FILT   Enabled packages: -DALLOW_DEBUG -DALLOW_GENER
IC_ADVDIFF -DALLOW_MDSIO -DALLOW_MOM_COMMON -DALLOW_MOM_FLUXFORM -DALLOW_MOM_VECINV -DALLOW_MONITOR -DALLOW_RW
*/

#ifndef PACKAGES_CONFIG_H
#define PACKAGES_CONFIG_H
/*  Disabled packages:  */
#undef  ALLOW_ADMTLM
#undef  ALLOW_AIM_V23
#undef  ALLOW_ATM2D
#undef  ALLOW_ATM_COMMON
#undef  ALLOW_ATM_COMPON_INTERF
#undef  ALLOW_ATM_OCN_COUPLER
#undef  ALLOW_ATM_PHYS
#undef  ALLOW_AUTODIFF
#undef  ALLOW_BBL
#undef  ALLOW_BULK_FORCE
#undef  ALLOW_CAL
#undef  ALLOW_CD_CODE
#undef  ALLOW_CFC
#undef  ALLOW_CHEAPAML
#undef  ALLOW_CHRONOS
#undef  ALLOW_COMPON_COMMUNIC
#undef  ALLOW_COST
#undef  ALLOW_CTRL
#undef  ALLOW_DIAGNOSTICS
#undef  ALLOW_DIC
#undef  ALLOW_DOWN_SLOPE
#undef  ALLOW_EBM
#undef  ALLOW_ECCO
#undef  ALLOW_EMBED_FILES
#undef  ALLOW_EXCH2
#undef  ALLOW_EXF
#undef  ALLOW_FIZHI
#undef  ALLOW_FLT
#undef  ALLOW_FRAZIL
#undef  ALLOW_GCHEM
#undef  ALLOW_GGL90
#undef  ALLOW_GMREDI
#undef  ALLOW_GRDCHK
#undef  ALLOW_GRIDALT
#undef  ALLOW_ICEFRONT
#undef  ALLOW_KPP
#undef  ALLOW_LAND
#undef  ALLOW_LAYERS
#undef  ALLOW_LONGSTEP
#undef  ALLOW_MATRIX
#undef  ALLOW_MNC
#undef  ALLOW_MY82
#undef  ALLOW_MYPACKAGE
#undef  ALLOW_OBCS
#undef  ALLOW_OCN_COMPON_INTERF
#undef  ALLOW_OFFLINE
#undef  ALLOW_OPENAD
#undef  ALLOW_OPPS
#undef  ALLOW_PP81
#undef  ALLOW_PROFILES
#undef  ALLOW_PTRACERS
#undef  ALLOW_RBCS
#undef  ALLOW_REGRID
#undef  ALLOW_RUNCLOCK
#undef  ALLOW_SALT_PLUME
#undef  ALLOW_SBO
#undef  ALLOW_SEAICE
#undef  ALLOW_SHAP_FILT
#undef  ALLOW_SHELFICE
#undef  ALLOW_SHOWFLOPS
#undef  ALLOW_SMOOTH
#undef  ALLOW_SPHERE
#undef  ALLOW_STREAMICE
#undef  ALLOW_THSICE
#undef  ALLOW_TIMEAVE
#undef  ALLOW_ZONAL_FILT
/*   */
/*  Enabled packages:  */
#define ALLOW_DEBUG
#define ALLOW_GENERIC_ADVDIFF
#define ALLOW_MDSIO
#define ALLOW_MOM_COMMON
#define ALLOW_MOM_FLUXFORM
#define ALLOW_MOM_VECINV
#define ALLOW_MONITOR
#define ALLOW_RW
#endif /* PACKAGES_CONFIG_H */


I wonder why that would be considering I have the packages to install listed in my packages.config file.

This is the command I use for compiling the GCM which seems correct:
../../../tools/genmake2 -optfile=../../../tools/build_options/linux_amd64_gfortran -mpi


And here is some lines from genmake2:

===  Processing options files and arguments  ===
  getting local config information:  none found
Warning: ROOTDIR was not specified ; try using a local copy of MITgcm found at "../../.."
  getting OPTFILE information:
    using OPTFILE="../../../tools/build_options/linux_amd64_gfortran"
  getting AD_OPTFILE information:
    using AD_OPTFILE="../../../tools/adjoint_options/adjoint_default"
  check makedepend (local: 0, system: 0, 0)
  Turning on MPI cpp macros

===  Checking system libraries  ===
  Do we have the system() command using gfortran...  yes
  Do we have the fdate() command using gfortran...  yes
  Do we have the etime() command using gfortran...  no
  Can we call simple C routines (here, "cloc()") using gfortran...  yes
  Can we unlimit the stack size using gfortran...  yes
  Can we register a signal handler using gfortran...  yes
  Can we use stat() through C calls...  yes
  Can we create NetCDF-enabled binaries...  yes
  Can we create LAPACK-enabled binaries...  no
  Can we call FLUSH intrinsic subroutine...  yes

===  Setting defaults  ===
  Adding MODS directories: 
  Making source files in eesupp from templates
  Making source files in pkg/exch2 from templates
  Making source files in pkg/regrid from templates

===  Determining package settings  ===
  getting package dependency info from  ../../../pkg/pkg_depend
  getting package groups info from      ../../../pkg/pkg_groups
  checking list of packages to compile:
    before group expansion packages are: default_pkg_list
    replacing "default_pkg_list" with:  gfd
    replacing "gfd" with:  mom_common mom_fluxform mom_vecinv generic_advdiff debug mdsio rw monitor
    after group expansion packages are:  mom_common mom_fluxform mom_vecinv generic_advdiff debug mdsio rw monitor
  applying DISABLE settings
  applying ENABLE settings
    packages are:  debug generic_advdiff mdsio mom_common mom_fluxform mom_vecinv monitor rw
  applying package dependency rules
    packages are:  debug generic_advdiff mdsio mom_common mom_fluxform mom_vecinv monitor rw
  Adding STANDARDDIRS='eesupp model'
  Searching for *OPTIONS.h files in order to warn about the presence
    of "#define "-type statements that are no longer allowed:
    found CPP_EEOPTIONS="../../../eesupp/inc/CPP_EEOPTIONS.h"
    found CPP_OPTIONS="../../../model/inc/CPP_OPTIONS.h"
  Creating the list of files for the adjoint compiler.

===  Creating the Makefile  ===
  setting INCLUDES
  Determining the list of source and include files
  Writing makefile: Makefile
  Add the source list for AD code generation
  Making list of "exceptions" that need ".p" files
  Making list of NOOPTFILES
  Add rules for links
  Adding makedepend marker

FYI: I am building a simulation using the 32x6x32x15 MPI run.

The makefile looks okay…it seems to be making the dependencies.  I do have 2 or 3 sets of this warning:

gfortran -fconvert=big-endian -fimplicit-none -mcmodel=medium  -O0 -funroll-loops -c solve_tridiagonal.f
cat solve_uv_tridiago.F |  cpp -traditional -P -DWORDLENGTH=4 -DNML_TERMINATOR -DALLOW_USE_MPI -DALWAYS_USE_MPI -DALLOW_USE_MPI -DHAVE_SYSTEM -DHAVE_FDATE -DHAVE_CLOC -DHAVE_SETRLSTK -DHAVE_SIGREG -DHAVE_STAT -DHAVE_NETCDF -DHAVE_FLUSH  -I/opt/cray/netcdf-hdf5parallel/4.3.1/GNU/48/include -I/opt/cray/mpt/6.3.0/gni/mpich2-gnu/48/include | ../../../tools/set64bitConst.sh  > solve_uv_tridiago.f
gfortran -fconvert=big-endian -fimplicit-none -mcmodel=medium  -O0 -funroll-loops -c solve_uv_tridiago.f
solve_uv_tridiago.f:1076.72:

          DO bj=2,nSy                                                   
                                                                        1
Warning: DO loop at (1) will be executed zero times
solve_uv_tridiago.f:1103.24:

          DO bj=nSy-1,1,-1                                              
                        1
Warning: DO loop at (1) will be executed zero times


I am not sure if that means anything.


Attached is my ini_threading_environment.f (small f)


Thanks for your fruitful insight!
Anthony


Anthony J. Coletti
Climate System Research Center
Department of Geosciences
Morrill Building
611 N. Pleasant Street
233 Morrill Science Center
University of Massachusetts-Amherst
paleoclimate.org
Email: ajcolett at geo.umass.edu
http://blogs.umass.edu/ajcolett/
http://necsc.umass.edu/people/anthony-coletti

“For me, I am driven by two main philosophies: know more today about the world than I knew yesterday and lessen the  suffering of others. You'd be surprised how far that gets you.” ― Neil deGrasse Tyson




On Jul 7, 2014, at 11:30 PM, Jean-Michel Campin <jmc at ocean.mit.edu> wrote:

> PACKAGES_CONFIG.h

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140708/76b12ac1/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ini_threading_environment.f
Type: application/octet-stream
Size: 71203 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140708/76b12ac1/attachment-0001.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140708/76b12ac1/attachment-0003.htm>

More information about the MITgcm-support mailing list