[MITgcm-support] MITgcm on Sun Fire cluster

Tue Jun 3 10:46:56 EDT 2008

Abbas,

   The Sun Studio 10 Fortran Programming Guide is at:
       http://docs.sun.com/app/docs/doc/819-0491
   The Studio 11 equivalent is at:
       http://docs.sun.com/app/docs/doc/819-3685-10

   Chapter 9 in each docs explains the compiler options.

   The optfile below indicates either high (O4) or minimal (O0) 
optimization depending on whether IEEE-754 compatibility is specified.

Choosing O4 optimization will make a huge difference in the run time 
performance.

stackvar tells the compiler to assign local variable space from the stack.

dalign forces real*8 variables to be aligned on 8byte boundaries.

xarch=native tells the compiler to optimize the code for the system you 
are compiling on, since you will run the app on the same system that you 
compiled on.

fsimple=2 allows the compiler to rearrange the sequence of floating 
point computations within a statement for optimum performance.

Regards,

Bill

Abbas Dorostkar wrote:
> Hi Bill and Martin,
> 
> I am using Sun HPC ClusterTools 6 software which supports SPARC and x64
> systems, Solaris 10, Sun Studio 10,11, Shared memory and TCP
> communication and the full MPI-2 specification. The GNU tools are also
> available. Which one is better to use?
> I am thinking Martin may be right about the domain size. I am making the
> domain size very big to see the result.
> By the way, I am not sure about the flags in optimization section of the
> optfile. Could anyone explain the flags to me or advise me where I can
> get some information. Here is the optimization part:
> 
> # optimization
> NOOPTFLAGS='-dalign -O2 -xarch=native'
> NOOPTFILES='solve_for_pressure.F'
> #NOOPTFILES='solve_for_pressure.F barrier.F different_multiple.F
> external_fields_load.F ini_vertic
> if test "x$IEEE" = x ; then
>     #  No need for IEEE-754
>     FFLAGS='-stackvar -e -u -xtypemap=real:64,double:64,integer:32'
>     FOPTIM='-dalign -O4 -xarch=native -fsimple=2'
>     CFLAGS='-dalign -xO4 -xarch=native'
> else
>     #  Try to follow IEEE-754
>     FFLAGS='-e -u -xtypemap=real:64,double:64,integer:32 -fsimple=0'
>     FOPTIM='-O0'
>     CFLAGS='-xO0'
> fi
> 
> 
> I have also attached the full optfile:
> 
> 
> #!/bin/bash
> #
> #  $Header:
> /u/gcmpack/MITgcm/tools/build_options/sunos_sun4u_mpf77+mpi_sunfire,v
> 1.4 2007/02/18 19:19:42 mlosch Exp $
> #
> # developed and tested on 
> # SunOS model 5.9 Generic_112233-11 sun4u sparc SUNW,Sun-Fire-15000
> # (hostname: model.awi-bremerhaven.de)
> #
> FC_NAMEMANGLE="#define FC_NAMEMANGLE(X) X ## _"
> S64='$(TOOLSDIR)/set64bitConst.sh'
> DEFINES='-DWORDLENGTH=4'
> 
> MPI='true'
> AWK='gawk'
> # if regular make does not work try gmake
> #MAKE='gmake'
> MAKEDEPEND='$(TOOLSDIR)/cyrus-imapd-makedepend/makedepend'
> #MAKEDEPEND='$(TOOLSDIR)/xmakedepend'
> # compiler
> FC='mpf77'
> LINK='mpf77'
> CPP='/usr/ccs/lib/cpp -P'
> # paths
> INCLUDES='-I/opt/SUNWhpc/include -I/usr/local/include'
> LIBS='-L/opt/SUNWhpc/lib -lmpi -lthread -lsocket -lnsl -L/usr/local/lib'
> # optimization
> NOOPTFLAGS='-dalign -O2 -xarch=native'
> NOOPTFILES='solve_for_pressure.F'
> #NOOPTFILES='solve_for_pressure.F barrier.F different_multiple.F
> external_fields_load.F ini_vertic
> if test "x$IEEE" = x ; then
>     #  No need for IEEE-754
>     FFLAGS='-stackvar -e -u -xtypemap=real:64,double:64,integer:32'
>     FOPTIM='-dalign -O4 -xarch=native -fsimple=2'
>     CFLAGS='-dalign -xO4 -xarch=native'
> else
>     #  Try to follow IEEE-754
>     FFLAGS='-e -u -xtypemap=real:64,double:64,integer:32 -fsimple=0'
>     FOPTIM='-O0'
>     CFLAGS='-xO0'
> fi
> 
> 
> Thanks so much
> Abbas
> 
> 
> 
> 
> 
> -----Original Message-----
> From: William.Aiken at sun.com [mailto:William.Aiken at sun.com] 
> Sent: June 2, 2008 12:17 PM
> To: Abbas Dorostkar
> Cc: mitgcm-support at mitgcm.org; William Aiken
> Subject: Re: [MITgcm-support] MITgcm on Sun Fire cluster
> 
> Abbas,
> 
>    Have you run other codes on both systems, or just MIT-gcm?  Certain 
> codes scale better than others. Your results indicate scaling up to 4 
> CPU's, then the overhead of MPI makes the performance worse. Another 
> factor may be NUMA on the UltraSPARC-IV system.
> 
> Your relative clock speed (2.5 GHz vs 1.5 GHz) would result in 1.6 
> speedup on the Intel system if other factors do not contribute.
> 
> Another factor is the compiler optimization used on each system.
> 
> Please check the optfile and see what the -O parameter is. You should 
> have -O4 or -O5 set to get the best optimization. What version of Sun 
> Studio are you using (or are you using gnu tools)?
> 
> Regards,
> 
> Bill
> 
> 
> Abbas Dorostkar wrote:
>> Hi all,
>>
>>  
>>
>> I have been running a test case -internal wave test case in 
>> nonhydrostatic mode- with MPI on a SunFire cluster. Unfortunately, the
> 
>> performance of the model on the cluster is terrible in comparison with
> 
>> the same run on my desktop computer.
>>
>> Here, I mentioned the wall clock times for both systems:
>>
>> On the cluster , Sun Fire 2500 server with 72 (2 MB on-chip L2 cache
> and 
>> 32 MB L3 cache)  dual-core UltraSPARC-IV+ 1.5 GHZ processors with 676
> GB 
>> of RAM:
>>
>> serial-- > 457 s
>>
>> np=2 -- > 353 s
>>
>> np=4 -- > 275 s
>>
>> np=6 -- > 315 s
>>
>>  
>>
>> On my Desktop (Intel core 2 Due - 2.4 GHZ - 4 MB cache - 1066 MHZ FSB
> ):
>> serial -- > 171 s
>>
>> np=2 --->  162 s
>>
>>  
>>
>> I use the sunos_sun4u_mpf77+mpi_sunfire optfile shipped with the
> MITgcm 
>> model (v 1.4 2007/02/18). I have to mention that I get following 
>> warnings while I compile the model on the cluster:
>>
>> make: Warning: Too many rules defined for target PACKAGES_CONFIG.h
>> make: Warning: Too many rules defined for target AD_CONFIG.h
>> make: Warning: Too many rules defined for target FC_NAMEMANGLE.h
>> make: Warning: Too many rules defined for target BUILD_INFO.h
>>
>>  
>>
>> Does anyone  know what is going wrong? Should I change something in
> the 
>> optfile (e.g adding some optimization flags)?
>>
>>  
>>
>> Your help is much appreciated!
>>
>>  
>>
>> Abbas
>>
> 
> ------------------------------------------------------------------------
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
> 
> 

-- 
William Aiken          Sun Microsystems, Inc.
Phone: 781.442.3312    ISV Engineering
FAX: 781.442.1520      45 Network Drive UBUR01-204
                        Burlington, Massachusetts 01803