[MITgcm-support] usingMPI, ALLOW_USE_MPI, and ALWAYS_USE_MPI: a suggestion

[Gus Correa]

Hello MITgcm Support

The STDOUT files always tell that "usingMPI" is .FALSE., even when MPI 
is activated
by genmake2 with the option --mpi:

1806-fats% grep usingMPI STDOUT.0001
(PID.TID 0001.0001) usingMPI =    F ; /* Flag used to control whether MPI is in use */

I think I found the reason and a simple fix for this, which I describe 
below.

The logical variable "usingMPI" is set to conflicting default values 
twice in the code.
It is set to .TRUE. in eeboot.F, and to .FALSE. in eeset_parms.F, 
respectively:

1808-fats% grep -n "usingMPI " ../../../eesupp/src/eeboot.F
100:      usingMPI = .TRUE.

1810-fats% grep -n "usingMPI " ../../../eesupp/src/eeset_parms.F
76:C     usingMPI is a flag which controls whether MPI message
81:C     If usingMPI is set to TRUE but % mpirun .... was not
95:      usingMPI                   = .FALSE.


The latter subroutine is called by the former, after the former sets 
usingMPI = .TRUE..
Hence, for the rest of the run, the value of usingMPI is .FALSE..
Even when the code is running with MPI
properly activated by the suitable --mpi option passed to genmake2,
usingMPI is .FALSE. for most of the run.
The only way to avoid this inconsistency is to set usingMPI=.TRUE. in
the namelist EEPARMS, in the input file eedata, which is read by the 
subroutines eeset_parms.
However, this introduces yet another control for MPI.

A quick and dirty fix to this problem is to comment out line 95 of 
eeset_parms.F:

1822-fats% grep -n "usingMPI " ../../../eesupp/src/eeset_parms.F
76:C     usingMPI is a flag which controls whether MPI message
81:C     If usingMPI is set to TRUE but % mpirun .... was not
95:C      usingMPI                   = .FALSE.


Apparently the only side effect of letting usingMPI to run as .FALSE., 
as it currently does,
is the aforementioned misleading message on the STDOUT files.
However, no problem happens because all additional occurrences of 
"usingMPI" in the code are
removed by the preprocessor flag  "ALWAYS_USE_MPI"  (which is also set 
by genmake2).
The removal is brought about by chunks of code like this:

#ifndef ALWAYS_USE_MPI
       IF ( usingMPI ) THEN
#endif


Nevertheless, a comment in eeboot.F tells that the intent of the code 
developers is to use
yet another preprocessor directive, called "ALLOW_USE_MPI",
to control the use of MPI:

C     Annoyingly there is no universal way to have the usingMPI
C     parameter work as one might expect. This is because, on some
C     systems I/O does not work until MPI_Init has been called. The
C     solution for now is that the parameter below may need to be changed
C     manually!
#ifdef ALLOW_USE_MPI
      usingMPI = .TRUE.
#endif


The redundancy of having two preprocessor flags and one Fortran logical 
variable to control
the same thing is a bit confusing, at least to me.

A simple suggestion to reduce the multiple controls for the same feature 
(MPI),
is to remove "usingMPI" from the namelist /EEPARMS/ declared in 
eeset_parms.F,
and to remove the code line 95 as above.
This would also require removing "usingMPI" from any "eedata" input 
files, where that namelist is read from.
However, this won't take much work, because there are only two examples 
in the
"verification" directory where the value of usingMPI is set in the 
eedata files!
These minor changes would let the preprocessor act alone to control the 
use of MPI
through the directive "ALLOW_USE_MPI" only.

To reduce even more the redundancy of controls on MPI,
maybe the second preprocessor directive, "ALWAYS_USE_MPI",
could also be completely removed.
This is not too difficult, but it will require editing a larger number 
of source files.

Thank you.

Gus Correa

PS - FYI, I am using checkpoint57y_post.

-- 
---------------------------------------------------------------------
Gustavo J. Ponce Correa - Email: gus at ldeo.columbia.edu
Lamont-Doherty Earth Observatory - Columbia University
P.O. Box 1000 [61 Route 9W] - Palisades, NY, 10964-8000 - USA
Oceanography Bldg., Rm. 103-D, ph. (845) 365-8911, fax (845) 365-8736
---------------------------------------------------------------------