[MITgcm-support] Myrinet, PGI, & mac

Amato Evan ate9c at virginia.edu
Mon Nov 16 12:30:13 EST 2009 

Hi Martin,

Wow, that's really great!  I really appreciate you getting "looking  
under the hood" for me.  I think there must be a flag for pgcpp that I  
need to set, and that should be pretty easy to track down (as with the  
optimization stuff too).

Thank you again so much for your help!

Sincerely,
Amato

On Nov 16, 2009, at 12:23 PM, Martin Losch wrote:

> Hi Amato,
>
> I think the problem was just related to the preprocessor that you
> specified. I don't know what pgcpp does in detail, but with the
> present options it certainly leaves the *.for files empty.
>
> I changed the variable to
> CPP='/usr/bin/cpp -traditional -P'
> so that the model uses some gnu-cpp and things seem to work. I also
> changed you netcdf-paths to where the files are on your system (/
> common/usr/local/libs and include)
> and added a
> DEFINES='-DWORDLENGTH=4'
> You'll find a basic (but working) opt file in
> ~mlosch/MITgcm/tools/build_options/darwin_ia32_pgf95+mpich2-mx_trane
> I tried it out in verification/tutorial_baroclinic_gyre/build
>
> You'll need to figure out, which optimization you want, and which
> optimization you can afford without breaking the code (sometimes
> you'll need to specify a list of files that require lower
> optimization, NOOPTFILES='...')
>
> If you want to go back to pgcpp, you'll need to find out what options
> it takes to just produce fortran code (and not tries to compile it).
>
> M.
>
> On Nov 16, 2009, at 5:03 PM, Amato Evan wrote:
>
>> Hi Martin!
>>
>> Wow, what a laundry list!  Thanks so much for the help in
>> identifying my problems. Now I have a lot of info to take to PGI &
>> UVa support.  I'll get you an account on my cluster, and will start
>> tackling these issues one at a time.
>>
>>> I am no specialist for this, but what makes we wonder are warnings
>>> like this:
>>>> cat main.F | ../../../tools/set64bitConst.sh | pgcpp - 
>>>> DALLOW_USE_MPI
>>>> -DALWAYS_USE_MPI -DHAVE_SYSTEM -DHAVE_FDATE -DHAVE_ETIME - 
>>>> DHAVE_CLOC
>>>> -DHAVE_SETRLSTK -DHAVE_SIGREG -DHAVE_STAT -I/common/netcdf-4.0.1 >
>>>> main.for
>>>> pgcpp-Warning-No files to process
>>>> mpif90   -c main.for
>>>> PGF90-W-0006-Input file empty (main.for)
>>> These warning seem to appear for each file that is processed. In the
>>> end the main program is not available (as f90main complains, this is
>>> not a MITgcm routine, only something that invokes the main program
>>> "MAIN", which you'll find in "eesupp/scr/main.F").
>> I've bumped into this before and people at PGI said that it could be
>> an issue with mpich2-mx drivers.  I'll ping them again about it.
>>
>>> You seem to have fundamental problem in the way the sources are
>>> processed, so that already in the CPP step (.F -> .for) there is
>>> nothing in either the *.F or the *.for files. Maybe you can verify
>>> that by having a look at some *.for files. They should be the same  
>>> as
>>> the *.F files, except that all the CPP-flags have been evaluated and
>>> the header-files are included. You probably need to specify special
>>> flags for pgcpp? Or are symbolic links treated in a funny way on  
>>> your
>>> system? (I once had problems with symbolic links and MacOS in
>>> different contexts).
>> Yes, the .for files are all empty (.F ones are OK). I need to ID
>> some special CPP-flags here then.
>>
>>> Further, but I think that that may be unrelated, you're MPI paths
>>> don't seem to show up in the make.log, probably because you  
>>> overwrite
>>> them in the optfile. It should look like this
>>>
>>> #- for MPI:
>>> MPI_INC_DIR="/common/mpich2-mx-1.1.1p1..5/src/include"
>>> #-- for NetCDF:
>>> NETCDF_INCDIR="/common/netcdf-4.0.1"
>>> # and then
>>> INCLUDES="-I$NETCDF_INCDIR -I$MPI_INC_DIR"
>>> # or
>>> INCLUDEDIRS="$NETCDF_INCDIR $MPI_INC_DIR"
>>> # this is only for netcdf
>>> LIBS="-L/common/netcdf-4.0.1/libsrc"
>> OK, thanks.
>>
>>>
>>> Again, a problem to fix once your are able to get any executable: in
>>> your genmake2, you get a message, that you cannot compile netcdf
>>> files. After the genmake2 step (or make makefile) you can check in
>>> genmake_warnings, what the specific error messages related to this
>>> are.
>>
>> Right, I figured I'd work on the netcdf stuff after I sorted out
>> this mpi/pgf mess.
>>
>> Thanks!
>>
>> -Amato
>>
>>>
>>> Martin
>>>
>>> PS. If I can get access to that machine (I know that that's not
>>> always
>>> possible, certainly not a my institute) I might be able to figure  
>>> out
>>> what's going myself.
>>>
>>> On Nov 16, 2009, at 4:01 AM, Amato Evan wrote:
>>>
>>>> Hi Martin,
>>>>
>>>> Firstly, thank you so much for the help!  I have posted my working
>>>> optfile, and the output from "genmake" and "make" here:
>>>> http://trane.evsc.virginia.edu/mitgcm/
>>>>
>>>> If you have a second, please let me know if anything is obviously
>>>> amiss here. I am getting in touch with PGI support about compiling
>>>> the MITgcm, so once I have my optfile (and flags) down I will
>>>> definitely submit for inclusion in the repository.
>>>>
>>>> Sincerely,
>>>> Amato
>>>>
>>>> On Nov 14, 2009, at 4:54 AM, Martin Losch wrote:
>>>>
>>>>> Amato,
>>>>>
>>>>> I would expect that with a working mpi-installation, you shouldn't
>>>>> have these problems.
>>>>> You are using mpif90 for compiling and linking, so that the  
>>>>> library
>>>>> paths to the mpi libs should already be included; the only thing
>>>>> that
>>>>> you need is the path to the mpi-header files (because the makefile
>>>>> first produces *.f files with $(CPP) and not $(FC), the INCLUDES  
>>>>> is
>>>>> passed to $(CPP)). I assume it's this: "/common/mpich2-
>>>>> mx-1.1.1p1..5/
>>>>> src/include" (it should show up somewhere in mpif90). So your
>>>>> settings
>>>>> look good to me.
>>>>> You seem to get as far as the link step (because the only error
>>>>> message you report is "ld: ..."), so that I think it has to do  
>>>>> with
>>>>> the library paths. For more help you'd need to give more details,
>>>>> e.g.
>>>>> 1. the complete build options file
>>>>> 2. output of genmake2 (and genmake_warnings, if available after
>>>>> running genmake2)
>>>>> 3. full error messages of the make step
>>>>>
>>>>> BTW: Once you have managed to get the model compiled and running,
>>>>> it
>>>>> would be nice if you can share the successful build_options file,
>>>>> and
>>>>> we can add it to the repository.
>>>>>
>>>>> Martin
>>>>>
>>>>> On Nov 13, 2009, at 10:33 PM, Amato Evan wrote:
>>>>>
>>>>>> Good afternoon!
>>>>>>
>>>>>> I'm new to HPC, MPI, PGI, MX, MPD, MITgcm... And just about any
>>>>>> fancy acronym
>>>>>> you can think of. Although I've gotten the MITgcm running on a
>>>>>> macbook
>>>>>> pro, trying to do this on a cluster is much more involved! I am
>>>>>> trying
>>>>>> to compile the verification/exp2 code on:
>>>>>>
>>>>>> -An eight node Xserve cluster with Myrinet MPI comms
>>>>>> -mpich2-mx compiled with pgcc/pgf95/pgcpp
>>>>>>
>>>>>> I am pretty sure that mpicc etc is working since I can run IMB-
>>>>>> MPI1
>>>>>> codes.  But trying to compile the MITgcm is a total disaster! I  
>>>>>> am
>>>>>> getting "ld: symbol(s) not found" errors on "Make" although I
>>>>>> think I
>>>>>> am including paths to my mpi and pgi libs.  Can anyone with a
>>>>>> similar
>>>>>> setup post an optfile?
>>>>>>
>>>>>> Some of my (sloppy) settings are:
>>>>>>
>>>>>> CPP='pgcpp'
>>>>>> FC='mpif90'
>>>>>> CC='mpicc'
>>>>>> LINK='mpif90'
>>>>>> INCLUDES='-I/usr/local/include -I/common/usr/local/include -I/
>>>>>> common/
>>>>>> mpich2-mx-1.1.1p1..5/src/include'
>>>>>> INCLUDEFILES='-I/usr/local/include -I/common/usr/local/include - 
>>>>>> I/
>>>>>> common/mpich2-mx-1.1.1p1..5/src/include'
>>>>>> LIBS='-L/usr/local/lib -L/common/mpich2-mx-1.1.1p1..5/src/ 
>>>>>> binding/
>>>>>> f90 -
>>>>>> L/common/usr/local/lib'
>>>>>>
>>>>>> Thanks for the help!!!
>>>>>>
>>>>>> Sincerely,
>>>>>> Amato
>>>>>> _______________________________________________
>>>>>> MITgcm-support mailing list
>>>>>> MITgcm-support at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> MITgcm-support mailing list
>>>>> MITgcm-support at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>
>>>> ---
>>>> Amato Evan
>>>> Asst. Professor
>>>> Dept. of Environmental Sciences
>>>> University of Virginia
>>>> 434.243.7711
>>>> aevan at virginia.edu
>>>> http://trane.evsc.virginia.edu
>>>>
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>
>>>
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>
>> ---
>> Amato Evan
>> Asst. Professor
>> Dept. of Environmental Sciences
>> University of Virginia
>> 434.243.7711
>> aevan at virginia.edu
>> http://trane.evsc.virginia.edu
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support

---
Amato Evan
Asst. Professor
Dept. of Environmental Sciences
University of Virginia
434.243.7711
aevan at virginia.edu
http://trane.evsc.virginia.edu

More information about the MITgcm-support mailing list