<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none"><!--P{margin-top:0;margin-bottom:0;} .ms-cui-menu {background-color:#ffffff;border:1px rgb(171, 171, 171) solid;font-family:"Segoe UI WPC", "Segoe UI", Tahoma, "Microsoft Sans Serif", Verdana, sans-serif;font-size:11pt;color:rgb(51, 51, 51);} .ms-cui-menusection-title {display:none;} .ms-cui-ctl {vertical-align:text-top;text-decoration:none;color:rgb(51, 51, 51);} .ms-cui-ctl-on {background-color:rgb(232, 239, 244);opacity: 0.8;} .ms-cui-img-cont-float {display:inline-block;margin-top:2px} .ms-cui-smenu-inner {padding-top:0px;} .ms-owa-paste-option-icon {margin: 2px 4px 0px 4px;vertical-align:sub;padding-bottom: 2px;display:inline-block;} .ms-rtePasteFlyout-option:hover {background-color:rgb(232, 239, 244) !important;opacity:1 !important;} .ms-rtePasteFlyout-option {padding:8px 4px 8px 4px;outline:none;} .ms-cui-menusection {float:left; width:85px;height:24px;overflow:hidden}--></style>
</head>
<body>
<div style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Arial,Helvetica,sans-serif;">
<p></p>
<div>Thank you very much for your follow up response.<br>
</div>
<div>I tried to compile with all intel-MPI as follows(avoiding openMPI)<br>
</div>
<div><br>
</div>
<div>
<div>cd /moes/home/srinivas/MITgcm/verification/tutorial_barotropic_gyre/build<br>
</div>
<div>module unload netcdf/4.2.1</div>
<div>module load netcdf/3.6.3_new</div>
<div>module unload openmpi/1.8.4</div>
<div>export NETCDF_ROOT="/gpfs1/home/Libs/INTEL/NETCDF-3.6.3_new"</div>
<div>export MPI="true"</div>
<div>export MPI_INC_DIR="/gpfs1/opt/intel/icsxe/impi/4.1.3.048/intel64"</div>
<div><br>
</div>
<div>with MPI support and with netcdf support</div>
<div>../../../tools/genmake2 -mpi -enable=mnc -mods ../code -optfile ../../../tools/build_options/linux_amd64_ifort+impi <br>
</div>
<div>make depend<br>
</div>
<div><br>
</div>
Now at make depend stage itself. I am getting a lot of following errror.<br>
</div>
<div>EESUPPORT.h:173: error: mpif.h: No such file or directory</div>
<div>In file included from bar2.F:24:<br>
</div>
<p><br>
</p>
<p>I attached linux_amd64_ifort+impi for your reference.<br>
</p>
<div>
<p><br>
</p>
<div name="divtagdefaultwrapper" style="font-family: calibri, arial, helvetica, sans-serif; margin: 0px;">
<div class="BodyFragment"><font size="2">
<div class="PlainText"><strong><em>Srinivasu U</em></strong><br>
<strong><em>Scientist-C, CSG,</em></strong><br>
<strong><em>Indian National Center for Ocean Information Services(INCOIS),</em></strong><br>
<strong><em>Hyderabad-500090.</em></strong></div>
<div class="PlainText"><strong><em>Telephone:04023886146</em></strong><br>
<strong><em>Mobile:9032325438.</em></strong></div>
</font></div>
</div>
</div>
<div style="color: #282828;">
<hr tabindex="-1" style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size: 11pt;"><strong><em>From:</em></strong><strong><em> MITgcm-support <mitgcm-support-bounces@mitgcm.org> on behalf of Ali Ramadhan <alir@mit.edu></em></strong><br>
<strong></strong><strong><em>Sent:</em></strong><strong><em> Friday, September 7, 2018 6:48 PM</em></strong><br>
<strong></strong><strong><em>To:</em></strong><strong><em> mitgcm-support@mitgcm.org</em></strong><br>
<strong></strong><strong><em>Subject:</em></strong><strong><em> Re: [MITgcm-support] Fatal error in PMPI_Comm_rank: Invalid communicator, error stack</em></strong></font><strong>
</strong>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr"><strong><em>Hi Srinivasu,</em></strong><br>
<strong></strong><br>
<strong><em>Can you check to see if mpirun, mpiifort, and mpiicc are all coming from the same installation using the which command? Also, can you post your SIZE.h file? The number of processors specified with the -np flag must equal nPx*nPy from SIZE.h.</em></strong><br>
<strong></strong><br>
<strong><em>Also I think I just noticed a mistake in your original post. You should be linking to the mitgcmuv executable, not copying it. The commands</em></strong><br>
<strong></strong><br>
<strong><em>cd ../run</em></strong><br>
<strong><em>ln -s ../input/* .</em></strong><br>
<strong><em>cp ../build/mitgcmuv .</em></strong><br>
<strong></strong><br>
<strong><em>should be</em></strong><br>
<strong></strong><br>
<strong><em>cd ../run</em></strong><br>
<strong><em>ln -s ../input/* .</em></strong><br>
<strong><em>ln -s ../build/mitgcmuv .</em></strong><br>
<strong></strong><br>
<strong><em>Well, actually it should run fine if you just copy the executable but then if you change some code and recompile/remake you may forget to copy the new executable and run the old executable without realizing it.</em></strong><br>
<strong></strong><br>
<strong><em>Cheers,</em></strong><br>
<strong><em>Ali</em></strong><br>
<strong></strong></div>
</div>
<strong></strong><br>
<strong></strong>
<div class="gmail_quote">
<div dir="ltr"><strong><em>On Fri, Sep 7, 2018 at 1:07 AM Srinivasu U <</em></strong><a href="mailto:srinivasu.u@incois.gov.in"><strong><em>srinivasu.u@incois.gov.in</em></strong></a><strong><em>> wrote:</em></strong><br>
<strong></strong></div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;">
<div>
<div style="font-size: 12pt; color: #000000; background-color: #ffffff; font-family: arial, helvetica, sans-serif;">
<p><strong><em>Hi Ali,</em></strong><br>
<strong></strong></p>
<p><br>
<strong></strong></p>
<p><strong><em>Thanks for your reply. I unloaded openmpi and run "which mpirun". As you said, it is showing a different mpirun path.</em></strong><br>
<strong></strong></p>
<p><br>
<strong></strong></p>
<p><strong><em>module load openmpi</em></strong><br>
<strong></strong></p>
<p><br>
<strong></strong></p>
<p><strong><em>which mpirun</em></strong><br>
<strong></strong></p>
<p><br>
<strong></strong></p>
<p><strong><em>/gpfs1/home/Libs/INTEL/OPENMPI/openmpi-1.8.4/bin/mpirun</em></strong><br>
<strong></strong></p>
<p><br>
<strong></strong></p>
<p><strong><em>module unload openmpi</em></strong><br>
<strong></strong></p>
<p><br>
<strong></strong></p>
<div><strong><em>which mpirun</em></strong><br>
<strong></strong></div>
<div><br>
<strong></strong></div>
<div><strong><em>/gpfs1/opt/intel/icsxe/impi/</em></strong><a href="http://4.1.3.048/intel64/bin/mpirun" target="_blank"><strong><em>4.1.3.048/intel64/bin/mpirun</em></strong></a><br>
<strong></strong></div>
<div><br>
<strong></strong></div>
<div><strong><em>I tried to run with second mpirun, but getting the same error.</em></strong><br>
<strong></strong></div>
<div><strong><em>Same error persists which mpiexec as well.</em></strong><br>
<strong></strong></div>
<div><strong><em>Any more suggestions?</em></strong><br>
<strong></strong></div>
<div><br>
<strong></strong><span style="font-size: small; font-family: calibri, arial, helvetica, sans-serif;"><strong><em>Srinivasu U</em></strong></span><br>
<strong></strong></div>
<div>
<div name="divtagdefaultwrapper" style="font-family: calibri, arial, helvetica, sans-serif; margin: 0px;">
<div class="m_-6815878227422177730BodyFragment"><font size="2">
<div class="m_-6815878227422177730PlainText"><strong><em>Scientist-C, CSG,</em></strong><br>
<strong></strong><strong><em>Indian National Center for Ocean Information Services(INCOIS),</em></strong><br>
<strong></strong><strong><em>Hyderabad-500090.</em></strong></div>
<div class="m_-6815878227422177730PlainText"><strong><em>Telephone:04023886146</em></strong><br>
<strong></strong><strong><em>Mobile:9032325438.</em></strong></div>
</font></div>
</div>
</div>
<div style="color: #282828;">
<hr style="display: inline-block; width: 98%;">
<div id="m_-6815878227422177730divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size: 11pt;"><strong><em>From:</em></strong><strong><em> MITgcm-support <</em></strong><a href="mailto:mitgcm-support-bounces@mitgcm.org" target="_blank"><strong><em>mitgcm-support-bounces@mitgcm.org</em></strong></a><strong><em>>
on behalf of Ali Ramadhan <</em></strong><a href="mailto:alir@mit.edu" target="_blank"><strong><em>alir@mit.edu</em></strong></a><strong><em>></em></strong><br>
<strong></strong><strong><em>Sent:</em></strong><strong><em> Thursday, September 6, 2018 6:37 PM</em></strong><br>
<strong></strong><strong><em>To:</em></strong><strong> </strong><a href="mailto:mitgcm-support@mitgcm.org" target="_blank"><strong><em>mitgcm-support@mitgcm.org</em></strong></a><br>
<strong></strong><strong><em>Subject:</em></strong><strong><em> Re: [MITgcm-support] Fatal error in PMPI_Comm_rank: Invalid communicator, error stack</em></strong></font><strong>
</strong><strong></strong>
<div> </div>
</div>
<div>
<div dir="ltr">
<div><strong><em>Hi Srinivas,</em></strong></div>
<div><br>
<strong></strong></div>
<div><strong><em>I think I had a similar problem a few weeks ago. I notice you are correctly using the " linux_amd64_ifort+impi" build option which should compile using the mpiifort and mpiicc compilers, but then use mpirun from OpenMPI.</em></strong></div>
<div><br>
<strong></strong></div>
<div><strong><em>What worked for me was using the mpirun provided by the intel compiler (impi) which you presumably have loaded for compilation. If you
</em></strong><span style="font-family: monospace, monospace;"><strong><em>"module unload openmpi"</em></strong></span><strong><em> do you still have an mpirun in your PATH? You can run
</em></strong><span style="font-family: monospace, monospace;"><strong><em>"which mpirun"</em></strong></span><strong><em> to check</em></strong><br>
<strong></strong></div>
<div><strong><em> where it is located, hopefully from the Intel compiler.</em></strong></div>
<div><br>
<strong></strong></div>
<div><strong><em>I should also mention that Intel's mpirun and mpiexec are slightly different so if mpirun still does not work, maybe try mpiexec.</em></strong></div>
<div><br>
<strong></strong></div>
<div><strong><em>Let us know if this helps or not!</em></strong><br>
<strong></strong></div>
<div><br>
<strong></strong></div>
<div><strong><em>Cheers,</em></strong></div>
<div><strong><em>Ali</em></strong><br>
<strong></strong></div>
</div>
<strong></strong><br>
<strong></strong><strong></strong>
<div class="gmail_quote">
<div dir="ltr"><strong><em>On Thu, Sep 6, 2018 at 6:23 AM Srinivasu U <</em></strong><a href="mailto:srinivasu.u@incois.gov.in" target="_blank"><strong><em>srinivasu.u@incois.gov.in</em></strong></a><strong><em>> wrote:</em></strong><br>
<strong></strong></div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;">
<strong></strong><strong><em>Dear MITgcm support,</em></strong><br>
<strong></strong><br>
<strong></strong><strong><em>I am trying to run a tutorial example of tutorial_barotropic_gyre</em></strong><br>
<strong></strong><br>
<strong></strong><strong><em>I built it as follows</em></strong><br>
<strong></strong><br>
<strong></strong><strong><em>cd verification/tutorial_barotropic_gyre/build</em></strong><br>
<strong></strong><strong><em>module unload netcdf/4.2.1</em></strong><br>
<strong></strong><strong><em>module load netcdf/3.6.3_new</em></strong><br>
<strong></strong><strong><em>module load openmpi/1.8.4</em></strong><br>
<strong></strong><strong><em>export NETCDF_ROOT=”/gpfs1/home/Libs/INTEL/NETCDF-3.6.3_new”</em></strong><br>
<strong></strong><strong><em>export MPI_INC_DIR="/gpfs1/home/Libs/INTEL/OPENMPI/openmpi-1.8.4/include”</em></strong><br>
<strong></strong><strong><em>../../../tools/genmake2 -enable=mnc -mods ../code -optfile ../../../tools/build_options/linux_amd64_ifort+impi -mpi</em></strong><br>
<strong></strong><strong><em>make depend</em></strong><br>
<strong></strong><strong><em>make</em></strong><br>
<strong></strong><br>
<strong></strong><strong><em>cd ../run</em></strong><br>
<strong></strong><strong><em>ln -s ../input/* .</em></strong><br>
<strong></strong><strong><em>cp ../build/mitgcmuv .</em></strong><br>
<strong></strong><br>
<strong></strong><strong><em>But while running the model as follows</em></strong><br>
<strong></strong><strong><em>mpirun -np 4 ./mitgcmuv</em></strong><br>
<strong></strong><strong><em>I get the following error</em></strong><br>
<strong></strong><br>
<strong></strong><strong><em>Fatal error in PMPI_Comm_rank: Invalid communicator, error stack:</em></strong><br>
<strong></strong><strong><em>PMPI_Comm_rank(121): MPI_Comm_rank(comm=0x0, rank=0xeea1a4) failed</em></strong><br>
<strong></strong><strong><em>PMPI_Comm_rank(73).: Invalid communicator</em></strong><br>
<strong></strong><br>
<strong></strong><strong><em>Thanks in Advance.</em></strong><br>
<strong></strong><br>
<strong></strong><strong><em>Srinivasu U</em></strong><br>
<strong></strong><strong><em>Scientist-C, CSG,</em></strong><br>
<strong></strong><strong><em>Indian National Center for Ocean Information Services(INCOIS),</em></strong><br>
<strong></strong><strong><em>Hyderabad-500090.</em></strong><br>
<strong></strong><strong><em>Telephone:04023886146</em></strong><br>
<strong></strong><strong><em>Mobile:9032325438.</em></strong><br>
<strong></strong><strong><em>_______________________________________________</em></strong><br>
<strong></strong><strong><em>MITgcm-support mailing list</em></strong><br>
<strong></strong><a href="mailto:MITgcm-support@mitgcm.org" target="_blank"><strong><em>MITgcm-support@mitgcm.org</em></strong></a><br>
<strong></strong><a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank"><strong><em>http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</em></strong></a><br>
<strong></strong></blockquote>
</div>
</div>
</div>
</div>
</div>
<strong><em>_______________________________________________</em></strong><br>
<strong><em>MITgcm-support mailing list</em></strong><br>
<strong></strong><a href="mailto:MITgcm-support@mitgcm.org" target="_blank"><strong><em>MITgcm-support@mitgcm.org</em></strong></a><br>
<strong></strong><a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank"><strong><em>http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</em></strong></a><br>
<strong></strong></blockquote>
</div>
</div>
</div>
</div>
</body>
</html>