[MITgcm-support] Fatal error in PMPI_Comm_rank: Invalid communicator, error stack

[Ali Ramadhan]

Hi Srinivasu,

Can you check to see if mpirun, mpiifort, and mpiicc are all coming from
the same installation using the which command? Also, can you post your
SIZE.h file? The number of processors specified with the -np flag must
equal nPx*nPy from SIZE.h.

Also I think I just noticed a mistake in your original post. You should be
linking to the mitgcmuv executable, not copying it. The commands

cd ../run
ln -s ../input/* .
cp ../build/mitgcmuv .

should be

cd ../run
ln -s ../input/* .
ln -s ../build/mitgcmuv .

Well, actually it should run fine if you just copy the executable but then
if you change some code and recompile/remake you may forget to copy the new
executable and run the old executable without realizing it.

Cheers,
Ali

On Fri, Sep 7, 2018 at 1:07 AM Srinivasu U <srinivasu.u at incois.gov.in>
wrote:

> Hi Ali,
>
>
> Thanks for your reply. I unloaded openmpi and run "which mpirun". As you
> said, it is showing a different mpirun path.
>
>
> module load openmpi
>
>
> which mpirun
>
>
> /gpfs1/home/Libs/INTEL/OPENMPI/openmpi-1.8.4/bin/mpirun
>
>
> module unload openmpi
>
>
> which mpirun
>
> /gpfs1/opt/intel/icsxe/impi/4.1.3.048/intel64/bin/mpirun
>
> I tried to run with second mpirun, but getting the same error.
> Same error persists which mpiexec as well.
> Any more suggestions?
>
> *Srinivasu U*
> *Scientist-C, CSG,*
> *Indian National Center for Ocean Information Services(INCOIS),*
> *Hyderabad-500090.*
> *Telephone:04023886146*
> *Mobile:9032325438.*
> ------------------------------
> *From:** MITgcm-support <mitgcm-support-bounces at mitgcm.org
> <mitgcm-support-bounces at mitgcm.org>> on behalf of Ali Ramadhan
> <alir at mit.edu <alir at mit.edu>>*
> *Sent:** Thursday, September 6, 2018 6:37 PM*
> *To:** mitgcm-support at mitgcm.org <mitgcm-support at mitgcm.org>*
> *Subject:** Re: [MITgcm-support] Fatal error in PMPI_Comm_rank: Invalid
> communicator, error stack*
>
> *Hi Srinivas,*
>
> *I think I had a similar problem a few weeks ago. I notice you are
> correctly using the " linux_amd64_ifort+impi" build option which should
> compile using the mpiifort and mpiicc compilers, but then use mpirun from
> OpenMPI.*
>
> *What worked for me was using the mpirun provided by the intel compiler
> (impi) which you presumably have loaded for compilation. If you **"module
> unload openmpi"** do you still have an mpirun in your PATH? You can run **"which
> mpirun"** to check*
> * where it is located, hopefully from the Intel compiler.*
>
> *I should also mention that Intel's mpirun and mpiexec are slightly
> different so if mpirun still does not work, maybe try mpiexec.*
>
> *Let us know if this helps or not!*
>
> *Cheers,*
> *Ali*
>
> *On Thu, Sep 6, 2018 at 6:23 AM Srinivasu U <**srinivasu.u at incois.gov.in*
> <srinivasu.u at incois.gov.in>*> wrote:*
>
>> *Dear MITgcm support,*
>>
>> *I am trying to run a tutorial example of tutorial_barotropic_gyre*
>>
>> *I built it as follows*
>>
>> *cd verification/tutorial_barotropic_gyre/build*
>> *module unload netcdf/4.2.1*
>> *module load netcdf/3.6.3_new*
>> *module load openmpi/1.8.4*
>> *export NETCDF_ROOT=”/gpfs1/home/Libs/INTEL/NETCDF-3.6.3_new”*
>> *export
>> MPI_INC_DIR="/gpfs1/home/Libs/INTEL/OPENMPI/openmpi-1.8.4/include”*
>> *../../../tools/genmake2 -enable=mnc -mods ../code -optfile
>> ../../../tools/build_options/linux_amd64_ifort+impi -mpi*
>> *make depend*
>> *make*
>>
>> *cd ../run*
>> *ln -s ../input/* .*
>> *cp ../build/mitgcmuv .*
>>
>> *But while running the model as follows*
>> *mpirun -np 4 ./mitgcmuv*
>> *I get the following error*
>>
>> *Fatal error in PMPI_Comm_rank: Invalid communicator, error stack:*
>> *PMPI_Comm_rank(121): MPI_Comm_rank(comm=0x0, rank=0xeea1a4) failed*
>> *PMPI_Comm_rank(73).: Invalid communicator*
>>
>> *Thanks in Advance.*
>>
>> *Srinivasu U*
>> *Scientist-C, CSG,*
>> *Indian National Center for Ocean Information Services(INCOIS),*
>> *Hyderabad-500090.*
>> *Telephone:04023886146*
>> *Mobile:9032325438.*
>> *_______________________________________________*
>> *MITgcm-support mailing list*
>> *MITgcm-support at mitgcm.org* <MITgcm-support at mitgcm.org>
>> *http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support*
>> <http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support>
>>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20180907/f1f38688/attachment-0001.html>