[MITgcm-support] Fatal error in PMPI_Comm_rank: Invalid communicator, error stack

Tue Sep 11 01:05:00 EDT 2018

Thank you very much for your follow up response.
I tried to compile with all intel-MPI as follows(avoiding openMPI)

cd /moes/home/srinivas/MITgcm/verification/tutorial_barotropic_gyre/build
module unload netcdf/4.2.1
module load netcdf/3.6.3_new
module unload openmpi/1.8.4
export NETCDF_ROOT="/gpfs1/home/Libs/INTEL/NETCDF-3.6.3_new"
export MPI="true"
export MPI_INC_DIR="/gpfs1/opt/intel/icsxe/impi/4.1.3.048/intel64"

with MPI support and with netcdf support
../../../tools/genmake2 -mpi -enable=mnc -mods ../code -optfile ../../../tools/build_options/linux_amd64_ifort+impi
make depend

Now at make depend stage itself. I am getting a lot of following errror.
EESUPPORT.h:173: error: mpif.h: No such file or directory
In file included from bar2.F:24:

I attached linux_amd64_ifort+impi for your reference.

Srinivasu U
Scientist-C, CSG,
Indian National Center for Ocean Information Services(INCOIS),
Hyderabad-500090.
Telephone:04023886146
Mobile:9032325438.
________________________________
From: MITgcm-support <mitgcm-support-bounces at mitgcm.org> on behalf of Ali Ramadhan <alir at mit.edu>
Sent: Friday, September 7, 2018 6:48 PM
To: mitgcm-support at mitgcm.org
Subject: Re: [MITgcm-support] Fatal error in PMPI_Comm_rank: Invalid communicator, error stack

Hi Srinivasu,

Can you check to see if mpirun, mpiifort, and mpiicc are all coming from the same installation using the which command? Also, can you post your SIZE.h file? The number of processors specified with the -np flag must equal nPx*nPy from SIZE.h.

Also I think I just noticed a mistake in your original post. You should be linking to the mitgcmuv executable, not copying it. The commands

cd ../run
ln -s ../input/* .
cp ../build/mitgcmuv .

should be

cd ../run
ln -s ../input/* .
ln -s ../build/mitgcmuv .

Well, actually it should run fine if you just copy the executable but then if you change some code and recompile/remake you may forget to copy the new executable and run the old executable without realizing it.

Cheers,
Ali

On Fri, Sep 7, 2018 at 1:07 AM Srinivasu U <srinivasu.u at incois.gov.in<mailto:srinivasu.u at incois.gov.in>> wrote:

Hi Ali,

Thanks for your reply. I unloaded openmpi and run "which mpirun". As you said, it is showing a different mpirun path.

module load openmpi

which mpirun

/gpfs1/home/Libs/INTEL/OPENMPI/openmpi-1.8.4/bin/mpirun

module unload openmpi

which mpirun

/gpfs1/opt/intel/icsxe/impi/4.1.3.048/intel64/bin/mpirun<http://4.1.3.048/intel64/bin/mpirun>

I tried to run with second mpirun, but getting the same error.
Same error persists which mpiexec as well.
Any more suggestions?

Srinivasu U
Scientist-C, CSG,
Indian National Center for Ocean Information Services(INCOIS),
Hyderabad-500090.
Telephone:04023886146
Mobile:9032325438.
________________________________
From: MITgcm-support <mitgcm-support-bounces at mitgcm.org<mailto:mitgcm-support-bounces at mitgcm.org>> on behalf of Ali Ramadhan <alir at mit.edu<mailto:alir at mit.edu>>
Sent: Thursday, September 6, 2018 6:37 PM
To: mitgcm-support at mitgcm.org<mailto:mitgcm-support at mitgcm.org>
Subject: Re: [MITgcm-support] Fatal error in PMPI_Comm_rank: Invalid communicator, error stack

Hi Srinivas,

I think I had a similar problem a few weeks ago. I notice you are correctly using the " linux_amd64_ifort+impi" build option which should compile using the mpiifort and mpiicc compilers, but then use mpirun from OpenMPI.

What worked for me was using the mpirun provided by the intel compiler (impi) which you presumably have loaded for compilation. If you "module unload openmpi" do you still have an mpirun in your PATH? You can run "which mpirun" to check
 where it is located, hopefully from the Intel compiler.

I should also mention that Intel's mpirun and mpiexec are slightly different so if mpirun still does not work, maybe try mpiexec.

Let us know if this helps or not!

Cheers,
Ali

On Thu, Sep 6, 2018 at 6:23 AM Srinivasu U <srinivasu.u at incois.gov.in<mailto:srinivasu.u at incois.gov.in>> wrote:
Dear MITgcm support,

I am trying to run a tutorial example of tutorial_barotropic_gyre

I built it as follows

cd verification/tutorial_barotropic_gyre/build
module unload netcdf/4.2.1
module load netcdf/3.6.3_new
module load openmpi/1.8.4
export NETCDF_ROOT="/gpfs1/home/Libs/INTEL/NETCDF-3.6.3_new"
export MPI_INC_DIR="/gpfs1/home/Libs/INTEL/OPENMPI/openmpi-1.8.4/include"
../../../tools/genmake2 -enable=mnc -mods ../code -optfile ../../../tools/build_options/linux_amd64_ifort+impi -mpi
make depend
make

cd ../run
ln -s ../input/* .
cp ../build/mitgcmuv .

But while running the model as follows
mpirun -np 4 ./mitgcmuv
I get the following error

Fatal error in PMPI_Comm_rank: Invalid communicator, error stack:
PMPI_Comm_rank(121): MPI_Comm_rank(comm=0x0, rank=0xeea1a4) failed
PMPI_Comm_rank(73).: Invalid communicator

Thanks in Advance.

Srinivasu U
Scientist-C, CSG,
Indian National Center for Ocean Information Services(INCOIS),
Hyderabad-500090.
Telephone:04023886146
Mobile:9032325438.
_______________________________________________
MITgcm-support mailing list
MITgcm-support at mitgcm.org<mailto:MITgcm-support at mitgcm.org>
http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
_______________________________________________
MITgcm-support mailing list
MITgcm-support at mitgcm.org<mailto:MITgcm-support at mitgcm.org>
http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20180911/f93494e6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: linux_amd64_ifort+impi
Type: application/octet-stream
Size: 3790 bytes
Desc: linux_amd64_ifort+impi
URL: <http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20180911/f93494e6/attachment.obj>