[MITgcm-support] Myrinet, PGI, & mac

Mon Nov 16 12:23:26 EST 2009

Hi Amato,

I think the problem was just related to the preprocessor that you  
specified. I don't know what pgcpp does in detail, but with the  
present options it certainly leaves the *.for files empty.

I changed the variable to
CPP='/usr/bin/cpp -traditional -P'
so that the model uses some gnu-cpp and things seem to work. I also  
changed you netcdf-paths to where the files are on your system (/ 
common/usr/local/libs and include)
and added a
DEFINES='-DWORDLENGTH=4'
You'll find a basic (but working) opt file in
~mlosch/MITgcm/tools/build_options/darwin_ia32_pgf95+mpich2-mx_trane
I tried it out in verification/tutorial_baroclinic_gyre/build

You'll need to figure out, which optimization you want, and which  
optimization you can afford without breaking the code (sometimes  
you'll need to specify a list of files that require lower  
optimization, NOOPTFILES='...')

If you want to go back to pgcpp, you'll need to find out what options  
it takes to just produce fortran code (and not tries to compile it).

M.

On Nov 16, 2009, at 5:03 PM, Amato Evan wrote:

> Hi Martin!
>
> Wow, what a laundry list!  Thanks so much for the help in  
> identifying my problems. Now I have a lot of info to take to PGI &  
> UVa support.  I'll get you an account on my cluster, and will start  
> tackling these issues one at a time.
>
>> I am no specialist for this, but what makes we wonder are warnings
>> like this:
>>> cat main.F | ../../../tools/set64bitConst.sh | pgcpp -DALLOW_USE_MPI
>>> -DALWAYS_USE_MPI -DHAVE_SYSTEM -DHAVE_FDATE -DHAVE_ETIME -DHAVE_CLOC
>>> -DHAVE_SETRLSTK -DHAVE_SIGREG -DHAVE_STAT -I/common/netcdf-4.0.1 >
>>> main.for
>>> pgcpp-Warning-No files to process
>>> mpif90   -c main.for
>>> PGF90-W-0006-Input file empty (main.for)
>> These warning seem to appear for each file that is processed. In the
>> end the main program is not available (as f90main complains, this is
>> not a MITgcm routine, only something that invokes the main program
>> "MAIN", which you'll find in "eesupp/scr/main.F").
> I've bumped into this before and people at PGI said that it could be  
> an issue with mpich2-mx drivers.  I'll ping them again about it.
>
>> You seem to have fundamental problem in the way the sources are
>> processed, so that already in the CPP step (.F -> .for) there is
>> nothing in either the *.F or the *.for files. Maybe you can verify
>> that by having a look at some *.for files. They should be the same as
>> the *.F files, except that all the CPP-flags have been evaluated and
>> the header-files are included. You probably need to specify special
>> flags for pgcpp? Or are symbolic links treated in a funny way on your
>> system? (I once had problems with symbolic links and MacOS in
>> different contexts).
> Yes, the .for files are all empty (.F ones are OK). I need to ID  
> some special CPP-flags here then.
>
>> Further, but I think that that may be unrelated, you're MPI paths
>> don't seem to show up in the make.log, probably because you overwrite
>> them in the optfile. It should look like this
>>
>> #- for MPI:
>> MPI_INC_DIR="/common/mpich2-mx-1.1.1p1..5/src/include"
>> #-- for NetCDF:
>> NETCDF_INCDIR="/common/netcdf-4.0.1"
>> # and then
>> INCLUDES="-I$NETCDF_INCDIR -I$MPI_INC_DIR"
>> # or
>> INCLUDEDIRS="$NETCDF_INCDIR $MPI_INC_DIR"
>> # this is only for netcdf
>> LIBS="-L/common/netcdf-4.0.1/libsrc"
> OK, thanks.
>
>>
>> Again, a problem to fix once your are able to get any executable: in
>> your genmake2, you get a message, that you cannot compile netcdf
>> files. After the genmake2 step (or make makefile) you can check in
>> genmake_warnings, what the specific error messages related to this  
>> are.
>
> Right, I figured I'd work on the netcdf stuff after I sorted out  
> this mpi/pgf mess.
>
> Thanks!
>
> -Amato
>
>>
>> Martin
>>
>> PS. If I can get access to that machine (I know that that's not  
>> always
>> possible, certainly not a my institute) I might be able to figure out
>> what's going myself.
>>
>> On Nov 16, 2009, at 4:01 AM, Amato Evan wrote:
>>
>>> Hi Martin,
>>>
>>> Firstly, thank you so much for the help!  I have posted my working
>>> optfile, and the output from "genmake" and "make" here:
>>> http://trane.evsc.virginia.edu/mitgcm/
>>>
>>> If you have a second, please let me know if anything is obviously
>>> amiss here. I am getting in touch with PGI support about compiling
>>> the MITgcm, so once I have my optfile (and flags) down I will
>>> definitely submit for inclusion in the repository.
>>>
>>> Sincerely,
>>> Amato
>>>
>>> On Nov 14, 2009, at 4:54 AM, Martin Losch wrote:
>>>
>>>> Amato,
>>>>
>>>> I would expect that with a working mpi-installation, you shouldn't
>>>> have these problems.
>>>> You are using mpif90 for compiling and linking, so that the library
>>>> paths to the mpi libs should already be included; the only thing  
>>>> that
>>>> you need is the path to the mpi-header files (because the makefile
>>>> first produces *.f files with $(CPP) and not $(FC), the INCLUDES is
>>>> passed to $(CPP)). I assume it's this: "/common/mpich2- 
>>>> mx-1.1.1p1..5/
>>>> src/include" (it should show up somewhere in mpif90). So your
>>>> settings
>>>> look good to me.
>>>> You seem to get as far as the link step (because the only error
>>>> message you report is "ld: ..."), so that I think it has to do with
>>>> the library paths. For more help you'd need to give more details,
>>>> e.g.
>>>> 1. the complete build options file
>>>> 2. output of genmake2 (and genmake_warnings, if available after
>>>> running genmake2)
>>>> 3. full error messages of the make step
>>>>
>>>> BTW: Once you have managed to get the model compiled and running,  
>>>> it
>>>> would be nice if you can share the successful build_options file,  
>>>> and
>>>> we can add it to the repository.
>>>>
>>>> Martin
>>>>
>>>> On Nov 13, 2009, at 10:33 PM, Amato Evan wrote:
>>>>
>>>>> Good afternoon!
>>>>>
>>>>> I'm new to HPC, MPI, PGI, MX, MPD, MITgcm... And just about any
>>>>> fancy acronym
>>>>> you can think of. Although I've gotten the MITgcm running on a
>>>>> macbook
>>>>> pro, trying to do this on a cluster is much more involved! I am
>>>>> trying
>>>>> to compile the verification/exp2 code on:
>>>>>
>>>>> -An eight node Xserve cluster with Myrinet MPI comms
>>>>> -mpich2-mx compiled with pgcc/pgf95/pgcpp
>>>>>
>>>>> I am pretty sure that mpicc etc is working since I can run IMB- 
>>>>> MPI1
>>>>> codes.  But trying to compile the MITgcm is a total disaster! I am
>>>>> getting "ld: symbol(s) not found" errors on "Make" although I
>>>>> think I
>>>>> am including paths to my mpi and pgi libs.  Can anyone with a
>>>>> similar
>>>>> setup post an optfile?
>>>>>
>>>>> Some of my (sloppy) settings are:
>>>>>
>>>>> CPP='pgcpp'
>>>>> FC='mpif90'
>>>>> CC='mpicc'
>>>>> LINK='mpif90'
>>>>> INCLUDES='-I/usr/local/include -I/common/usr/local/include -I/
>>>>> common/
>>>>> mpich2-mx-1.1.1p1..5/src/include'
>>>>> INCLUDEFILES='-I/usr/local/include -I/common/usr/local/include -I/
>>>>> common/mpich2-mx-1.1.1p1..5/src/include'
>>>>> LIBS='-L/usr/local/lib -L/common/mpich2-mx-1.1.1p1..5/src/binding/
>>>>> f90 -
>>>>> L/common/usr/local/lib'
>>>>>
>>>>> Thanks for the help!!!
>>>>>
>>>>> Sincerely,
>>>>> Amato
>>>>> _______________________________________________
>>>>> MITgcm-support mailing list
>>>>> MITgcm-support at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>
>>>>
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>
>>> ---
>>> Amato Evan
>>> Asst. Professor
>>> Dept. of Environmental Sciences
>>> University of Virginia
>>> 434.243.7711
>>> aevan at virginia.edu
>>> http://trane.evsc.virginia.edu
>>>
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
> ---
> Amato Evan
> Asst. Professor
> Dept. of Environmental Sciences
> University of Virginia
> 434.243.7711
> aevan at virginia.edu
> http://trane.evsc.virginia.edu
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support