[MITgcm-support] Changing the number of cores in ECCOv4

[Dan Jones]

Hi Gael,

I am running ECCOv4 on ARCHER (http://www.archer.ac.uk/). I am performing a
one year long adjoint sensitivity experiment (using TAF). Here's how the
walltime scales with the number of processes:

nprocs =  96, walltime = 15.6 hours
nprocs = 192, walltime = 7.8 hours
nprocs = 360, walltime = 3.8 hours

I used the default ECCOv4 nchecklev_1 = 4, nchecklev_2 = nchecklev_3 = 220
in tamc.h. (This is more than is required for a one year run, but these are
test runs for a longer 20-year experiment). I can get a very modest (~5%)
decrease in wall time by setting nchecklev_1 = 120 and nchecklev_2 =
nchecklev_3 = 39, but increasing the number of cores has a *much* bigger
effect.

Thanks,
Dan


On Mon, Dec 19, 2016 at 2:52 PM, <mitgcm-support-request at mitgcm.org> wrote:

> Send MITgcm-support mailing list submissions to
>         mitgcm-support at mitgcm.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mitgcm.org/mailman/listinfo/mitgcm-support
> or, via email, send a message with subject or body 'help' to
>         mitgcm-support-request at mitgcm.org
>
> You can reach the person managing the list at
>         mitgcm-support-owner at mitgcm.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of MITgcm-support digest..."
>
>
>    4. Re: Changing the number of cores in ECCOv4 (gael  forget)
>       (gael forget)
>
>
> ------------------------------
>
> Message: 4
> Date: Mon, 19 Dec 2016 09:51:17 -0500
> From: gael forget <gforget at mit.edu>
> To: mitgcm-support at mitgcm.org
> Subject: Re: [MITgcm-support] Changing the number of cores in ECCOv4
>         (gael   forget)
> Message-ID: <BA328748-A598-4F53-BCA0-E6E5D405D7A9 at mit.edu>
> Content-Type: text/plain; charset="windows-1252"
>
> Hi Dan,
> sounds correct. How did this work out? I would be curious to know how
> linearly the computation scales up
> on the machine you are using ? what kind of wall clock times did you get
> with 96, 192, and 360 cores?
> Cheers,
> Gael
>
> On Dec 17, 2016, at 2:14 AM, Dan Jones <dcjones.work at gmail.com> wrote:
>
> > Hi Gael,
> >
> > Great! That worked perfectly, thanks.
> >
> > For nprocs=384, I guess sNx=sNy=15, nPx=192, and nPy=2 in SIZE.h, and
> the "blanklist" line for 384 should be commented out in data.exch2.  (I've
> got a test run with this configuration sitting on the queue.)
> >
> > Thanks!
> > Dan
> >
> > On Thu, Dec 15, 2016 at 5:00 PM, <mitgcm-support-request at mitgcm.org>
> wrote:
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Thu, 15 Dec 2016 09:34:16 -0500
> > From: gael forget <gforget at mit.edu>
> > To: mitgcm-support at mitgcm.org
> > Subject: Re: [MITgcm-support] Changing the number of cores in ECCOv4
> > Message-ID: <6F89820D-9735-41AD-9790-98704EAC2510 at mit.edu>
> > Content-Type: text/plain; charset="windows-1252"
> >
> > Hi Dan,
> > you seem to be on the right track. Unless I forget something all you
> want to do is indeed
> > - change sNx & nPx jointly, e.g., to make them 15 & 192 in
> http://wwwcvs.mitgcm.org/viewvc/MITgcm/MITgcm_contrib/gael/
> verification/ECCO_v4_r2/code/SIZE.h?view=markup
> > - uncomment the 'blankList? line for nprocs=192 and comment out the one
> for nprocs=96 in http://wwwcvs.mitgcm.org/viewv
> c/MITgcm/MITgcm_contrib/gael/verification/ECCO_v4_r2/input/
> data.exch2?view=markup
> > Cheers,
> > Gael
> >
> >
> >
> >
> > On Dec 15, 2016, at 8:02 AM, Dan Jones <dcjones.work at gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I'm running ECCOv4, and I would like to double the number of cores.  I
> haven't used exch2 before, so I'm not sure how to do that.
> > >
> > > I've read this discussion thread:
> > > http://mitgcm.org/pipermail/mitgcm-support/2014-December/009660.html
> > >
> > > So I believe that I should set nPx=192 in SIZE.h, but I'm not sure
> what else needs to be changed.  I'm used to conserving the total number of
> grid points (by changing sNx and sNy), but I'm not sure how exch2 handles
> that.
> > >
> > > I'm also not sure what to do in data.exch2.  I've uncommented the
> 'blankList' for nprocs=192 and commented out the 'blankList' for
> nprocs=96.  Is there anything else?
> > >
> > > Thanks in advance, exch2 experts!
> > >
> > > -Dan
> > > _______________________________________________
> > > MITgcm-support mailing list
> > > MITgcm-support at mitgcm.org
> > > http://mitgcm.org/mailman/listinfo/mitgcm-support
> >
> >
> >
> > End of MITgcm-support Digest, Vol 162, Issue 11
> > ***********************************************
> >
> > _______________________________________________
> > MITgcm-support mailing list
> > MITgcm-support at mitgcm.org
> > http://mitgcm.org/mailman/listinfo/mitgcm-support
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/2016
> 1219/51807ade/attachment.htm>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: smime.p7s
> Type: application/pkcs7-signature
> Size: 1843 bytes
> Desc: not available
> URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/2016
> 1219/51807ade/attachment.p7s>
>
> ------------------------------
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
>
> End of MITgcm-support Digest, Vol 162, Issue 17
> ***********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20161221/b29ccee9/attachment.htm>