[MITgcm-support] Fatal error in PMPI_Comm_rank: Invalid communicator, error stack

Wed Sep 12 14:50:59 EDT 2018

Hi Srinivasu,

In the commands you listed, you unload the OpenMPI module, but don’t load any other MPI modules. Are you certain that Intel MPI is available by default? The fact that `make depend` couldn’t find `mpif.h`, makes me suspect that you are trying to compile against an MPI that isn’t available.

Cheers,
Ed

________________________________
Edward Doddridge
Postdoctoral researcher
Earth, Atmospheric and Planetary Sciences
Massachusetts Institute of Technology

www.doddridge.me<http://www.doddridge.me>

On 11 Sep 2018, at 01:05, Srinivasu U <srinivasu.u at incois.gov.in<mailto:srinivasu.u at incois.gov.in>> wrote:

Thank you very much for your follow up response.
I tried to compile with all intel-MPI as follows(avoiding openMPI)

cd /moes/home/srinivas/MITgcm/verification/tutorial_barotropic_gyre/build
module unload netcdf/4.2.1
module load netcdf/3.6.3_new
module unload openmpi/1.8.4
export NETCDF_ROOT="/gpfs1/home/Libs/INTEL/NETCDF-3.6.3_new"
export MPI="true"
export MPI_INC_DIR="/gpfs1/opt/intel/icsxe/impi/4.1.3.048/intel64"

with MPI support and with netcdf support
../../../tools/genmake2 -mpi -enable=mnc -mods ../code -optfile ../../../tools/build_options/linux_amd64_ifort+impi
make depend

Now at make depend stage itself. I am getting a lot of following errror.
EESUPPORT.h:173: error: mpif.h: No such file or directory
In file included from bar2.F:24:

I attached linux_amd64_ifort+impi for your reference.

Srinivasu U
Scientist-C, CSG,
Indian National Center for Ocean Information Services(INCOIS),
Hyderabad-500090.
Telephone:04023886146
Mobile:9032325438.
________________________________
From: MITgcm-support <mitgcm-support-bounces at mitgcm.org<mailto:mitgcm-support-bounces at mitgcm.org>> on behalf of Ali Ramadhan <alir at mit.edu<mailto:alir at mit.edu>>
Sent: Friday, September 7, 2018 6:48 PM
To: mitgcm-support at mitgcm.org<mailto:mitgcm-support at mitgcm.org>
Subject: Re: [MITgcm-support] Fatal error in PMPI_Comm_rank: Invalid communicator, error stack

Hi Srinivasu,

Can you check to see if mpirun, mpiifort, and mpiicc are all coming from the same installation using the which command? Also, can you post your SIZE.h file? The number of processors specified with the -np flag must equal nPx*nPy from SIZE.h.

Also I think I just noticed a mistake in your original post. You should be linking to the mitgcmuv executable, not copying it. The commands

cd ../run
ln -s ../input/* .
cp ../build/mitgcmuv .

should be

cd ../run
ln -s ../input/* .
ln -s ../build/mitgcmuv .

Well, actually it should run fine if you just copy the executable but then if you change some code and recompile/remake you may forget to copy the new executable and run the old executable without realizing it.

Cheers,
Ali

On Fri, Sep 7, 2018 at 1:07 AM Srinivasu U <srinivasu.u at incois.gov.in<mailto:srinivasu.u at incois.gov.in>> wrote:
Hi Ali,

Thanks for your reply. I unloaded openmpi and run "which mpirun". As you said, it is showing a different mpirun path.

module load openmpi

which mpirun

/gpfs1/home/Libs/INTEL/OPENMPI/openmpi-1.8.4/bin/mpirun

module unload openmpi

which mpirun

/gpfs1/opt/intel/icsxe/impi/4.1.3.048/intel64/bin/mpirun<http://4.1.3.048/intel64/bin/mpirun>

I tried to run with second mpirun, but getting the same error.
Same error persists which mpiexec as well.
Any more suggestions?

Srinivasu U
Scientist-C, CSG,
Indian National Center for Ocean Information Services(INCOIS),
Hyderabad-500090.
Telephone:04023886146
Mobile:9032325438.
________________________________
From: MITgcm-support <mitgcm-support-bounces at mitgcm.org<mailto:mitgcm-support-bounces at mitgcm.org>> on behalf of Ali Ramadhan <alir at mit.edu<mailto:alir at mit.edu>>
Sent: Thursday, September 6, 2018 6:37 PM
To: mitgcm-support at mitgcm.org<mailto:mitgcm-support at mitgcm.org>
Subject: Re: [MITgcm-support] Fatal error in PMPI_Comm_rank: Invalid communicator, error stack

Hi Srinivas,

I think I had a similar problem a few weeks ago. I notice you are correctly using the " linux_amd64_ifort+impi" build option which should compile using the mpiifort and mpiicc compilers, but then use mpirun from OpenMPI.

What worked for me was using the mpirun provided by the intel compiler (impi) which you presumably have loaded for compilation. If you "module unload openmpi" do you still have an mpirun in your PATH? You can run "which mpirun" to check
 where it is located, hopefully from the Intel compiler.

I should also mention that Intel's mpirun and mpiexec are slightly different so if mpirun still does not work, maybe try mpiexec.

Let us know if this helps or not!

Cheers,
Ali

On Thu, Sep 6, 2018 at 6:23 AM Srinivasu U <srinivasu.u at incois.gov.in<mailto:srinivasu.u at incois.gov.in>> wrote:
Dear MITgcm support,

I am trying to run a tutorial example of tutorial_barotropic_gyre

I built it as follows

cd verification/tutorial_barotropic_gyre/build
module unload netcdf/4.2.1
module load netcdf/3.6.3_new
module load openmpi/1.8.4
export NETCDF_ROOT=”/gpfs1/home/Libs/INTEL/NETCDF-3.6.3_new”
export MPI_INC_DIR="/gpfs1/home/Libs/INTEL/OPENMPI/openmpi-1.8.4/include”
../../../tools/genmake2 -enable=mnc -mods ../code -optfile ../../../tools/build_options/linux_amd64_ifort+impi -mpi
make depend
make

cd ../run
ln -s ../input/* .
cp ../build/mitgcmuv .

But while running the model as follows
mpirun -np 4 ./mitgcmuv
I get the following error

Fatal error in PMPI_Comm_rank: Invalid communicator, error stack:
PMPI_Comm_rank(121): MPI_Comm_rank(comm=0x0, rank=0xeea1a4) failed
PMPI_Comm_rank(73).: Invalid communicator

Thanks in Advance.

Srinivasu U
Scientist-C, CSG,
Indian National Center for Ocean Information Services(INCOIS),
Hyderabad-500090.
Telephone:04023886146
Mobile:9032325438.
_______________________________________________
MITgcm-support mailing list
MITgcm-support at mitgcm.org<mailto:MITgcm-support at mitgcm.org>
http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
_______________________________________________
MITgcm-support mailing list
MITgcm-support at mitgcm.org<mailto:MITgcm-support at mitgcm.org>
http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
<linux_amd64_ifort+impi>_______________________________________________
MITgcm-support mailing list
MITgcm-support at mitgcm.org<mailto:MITgcm-support at mitgcm.org>
http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20180912/c5f9acea/attachment-0001.html>