[MITgcm-support] Fatal error in PMPI_Comm_rank: Invalid communicator, error stack

[Srinivasu U]

Hi Ali,


Thanks for your reply. I unloaded openmpi and run "which mpirun". As you said, it is showing a different mpirun path.


module load openmpi


which mpirun


/gpfs1/home/Libs/INTEL/OPENMPI/openmpi-1.8.4/bin/mpirun


module unload openmpi


which mpirun

/gpfs1/opt/intel/icsxe/impi/4.1.3.048/intel64/bin/mpirun

I tried to run with second mpirun, but getting the same error.
Same error persists which mpiexec as well.
Any more suggestions?

Srinivasu U
Scientist-C, CSG,
Indian National Center for Ocean Information Services(INCOIS),
Hyderabad-500090.
Telephone:04023886146
Mobile:9032325438.
________________________________
From: MITgcm-support <mitgcm-support-bounces at mitgcm.org> on behalf of Ali Ramadhan <alir at mit.edu>
Sent: Thursday, September 6, 2018 6:37 PM
To: mitgcm-support at mitgcm.org
Subject: Re: [MITgcm-support] Fatal error in PMPI_Comm_rank: Invalid communicator, error stack

Hi Srinivas,

I think I had a similar problem a few weeks ago. I notice you are correctly using the " linux_amd64_ifort+impi" build option which should compile using the mpiifort and mpiicc compilers, but then use mpirun from OpenMPI.

What worked for me was using the mpirun provided by the intel compiler (impi) which you presumably have loaded for compilation. If you "module unload openmpi" do you still have an mpirun in your PATH? You can run "which mpirun" to check
 where it is located, hopefully from the Intel compiler.

I should also mention that Intel's mpirun and mpiexec are slightly different so if mpirun still does not work, maybe try mpiexec.

Let us know if this helps or not!

Cheers,
Ali

On Thu, Sep 6, 2018 at 6:23 AM Srinivasu U <srinivasu.u at incois.gov.in<mailto:srinivasu.u at incois.gov.in>> wrote:
Dear MITgcm support,

I am trying to run a tutorial example of tutorial_barotropic_gyre

I built it as follows

cd verification/tutorial_barotropic_gyre/build
module unload netcdf/4.2.1
module load netcdf/3.6.3_new
module load openmpi/1.8.4
export NETCDF_ROOT="/gpfs1/home/Libs/INTEL/NETCDF-3.6.3_new"
export MPI_INC_DIR="/gpfs1/home/Libs/INTEL/OPENMPI/openmpi-1.8.4/include"
../../../tools/genmake2 -enable=mnc -mods ../code -optfile ../../../tools/build_options/linux_amd64_ifort+impi -mpi
make depend
make

cd ../run
ln -s ../input/* .
cp ../build/mitgcmuv .

But while running the model as follows
mpirun -np 4 ./mitgcmuv
I get the following error

Fatal error in PMPI_Comm_rank: Invalid communicator, error stack:
PMPI_Comm_rank(121): MPI_Comm_rank(comm=0x0, rank=0xeea1a4) failed
PMPI_Comm_rank(73).: Invalid communicator

Thanks in Advance.

Srinivasu U
Scientist-C, CSG,
Indian National Center for Ocean Information Services(INCOIS),
Hyderabad-500090.
Telephone:04023886146
Mobile:9032325438.
_______________________________________________
MITgcm-support mailing list
MITgcm-support at mitgcm.org<mailto:MITgcm-support at mitgcm.org>
http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20180907/9d18e5c5/attachment.html>