[MITgcm-support] MITgcm on Sun Fire cluster

Tue Jun 3 10:38:11 EDT 2008

Hi Bill and Martin,

I am using Sun HPC ClusterTools 6 software which supports SPARC and x64
systems, Solaris 10, Sun Studio 10,11, Shared memory and TCP
communication and the full MPI-2 specification. The GNU tools are also
available. Which one is better to use?
I am thinking Martin may be right about the domain size. I am making the
domain size very big to see the result.
By the way, I am not sure about the flags in optimization section of the
optfile. Could anyone explain the flags to me or advise me where I can
get some information. Here is the optimization part:

# optimization
NOOPTFLAGS='-dalign -O2 -xarch=native'
NOOPTFILES='solve_for_pressure.F'
#NOOPTFILES='solve_for_pressure.F barrier.F different_multiple.F
external_fields_load.F ini_vertic
if test "x$IEEE" = x ; then
    #  No need for IEEE-754
    FFLAGS='-stackvar -e -u -xtypemap=real:64,double:64,integer:32'
    FOPTIM='-dalign -O4 -xarch=native -fsimple=2'
    CFLAGS='-dalign -xO4 -xarch=native'
else
    #  Try to follow IEEE-754
    FFLAGS='-e -u -xtypemap=real:64,double:64,integer:32 -fsimple=0'
    FOPTIM='-O0'
    CFLAGS='-xO0'
fi

I have also attached the full optfile:

#!/bin/bash
#
#  $Header:
/u/gcmpack/MITgcm/tools/build_options/sunos_sun4u_mpf77+mpi_sunfire,v
1.4 2007/02/18 19:19:42 mlosch Exp $
#
# developed and tested on 
# SunOS model 5.9 Generic_112233-11 sun4u sparc SUNW,Sun-Fire-15000
# (hostname: model.awi-bremerhaven.de)
#
FC_NAMEMANGLE="#define FC_NAMEMANGLE(X) X ## _"
S64='$(TOOLSDIR)/set64bitConst.sh'
DEFINES='-DWORDLENGTH=4'

MPI='true'
AWK='gawk'
# if regular make does not work try gmake
#MAKE='gmake'
MAKEDEPEND='$(TOOLSDIR)/cyrus-imapd-makedepend/makedepend'
#MAKEDEPEND='$(TOOLSDIR)/xmakedepend'
# compiler
FC='mpf77'
LINK='mpf77'
CPP='/usr/ccs/lib/cpp -P'
# paths
INCLUDES='-I/opt/SUNWhpc/include -I/usr/local/include'
LIBS='-L/opt/SUNWhpc/lib -lmpi -lthread -lsocket -lnsl -L/usr/local/lib'
# optimization
NOOPTFLAGS='-dalign -O2 -xarch=native'
NOOPTFILES='solve_for_pressure.F'
#NOOPTFILES='solve_for_pressure.F barrier.F different_multiple.F
external_fields_load.F ini_vertic
if test "x$IEEE" = x ; then
    #  No need for IEEE-754
    FFLAGS='-stackvar -e -u -xtypemap=real:64,double:64,integer:32'
    FOPTIM='-dalign -O4 -xarch=native -fsimple=2'
    CFLAGS='-dalign -xO4 -xarch=native'
else
    #  Try to follow IEEE-754
    FFLAGS='-e -u -xtypemap=real:64,double:64,integer:32 -fsimple=0'
    FOPTIM='-O0'
    CFLAGS='-xO0'
fi

Thanks so much
Abbas

-----Original Message-----
From: William.Aiken at sun.com [mailto:William.Aiken at sun.com] 
Sent: June 2, 2008 12:17 PM
To: Abbas Dorostkar
Cc: mitgcm-support at mitgcm.org; William Aiken
Subject: Re: [MITgcm-support] MITgcm on Sun Fire cluster

Abbas,

   Have you run other codes on both systems, or just MIT-gcm?  Certain 
codes scale better than others. Your results indicate scaling up to 4 
CPU's, then the overhead of MPI makes the performance worse. Another 
factor may be NUMA on the UltraSPARC-IV system.

Your relative clock speed (2.5 GHz vs 1.5 GHz) would result in 1.6 
speedup on the Intel system if other factors do not contribute.

Another factor is the compiler optimization used on each system.

Please check the optfile and see what the -O parameter is. You should 
have -O4 or -O5 set to get the best optimization. What version of Sun 
Studio are you using (or are you using gnu tools)?

Regards,

Bill

Abbas Dorostkar wrote:
> Hi all,
> 
>  
> 
> I have been running a test case -internal wave test case in 
> nonhydrostatic mode- with MPI on a SunFire cluster. Unfortunately, the

> performance of the model on the cluster is terrible in comparison with

> the same run on my desktop computer.
> 
> Here, I mentioned the wall clock times for both systems:
> 
> On the cluster , Sun Fire 2500 server with 72 (2 MB on-chip L2 cache
and 
> 32 MB L3 cache)  dual-core UltraSPARC-IV+ 1.5 GHZ processors with 676
GB 
> of RAM:
> 
> serial-- > 457 s
> 
> np=2 -- > 353 s
> 
> np=4 -- > 275 s
> 
> np=6 -- > 315 s
> 
>  
> 
> On my Desktop (Intel core 2 Due - 2.4 GHZ - 4 MB cache - 1066 MHZ FSB
):
> 
> serial -- > 171 s
> 
> np=2 --->  162 s
> 
>  
> 
> I use the sunos_sun4u_mpf77+mpi_sunfire optfile shipped with the
MITgcm 
> model (v 1.4 2007/02/18). I have to mention that I get following 
> warnings while I compile the model on the cluster:
> 
> make: Warning: Too many rules defined for target PACKAGES_CONFIG.h
> make: Warning: Too many rules defined for target AD_CONFIG.h
> make: Warning: Too many rules defined for target FC_NAMEMANGLE.h
> make: Warning: Too many rules defined for target BUILD_INFO.h
> 
>  
> 
> Does anyone  know what is going wrong? Should I change something in
the 
> optfile (e.g adding some optimization flags)?
> 
>  
> 
> Your help is much appreciated!
> 
>  
> 
> Abbas
> 

>
------------------------------------------------------------------------
> 
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support

-- 
William Aiken          Sun Microsystems, Inc.
Phone: 781.442.3312    ISV Engineering
FAX: 781.442.1520      45 Network Drive UBUR01-204
                        Burlington, Massachusetts 01803