<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Palatino;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:-webkit-standard;
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-AU" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks Matt and Dimitris.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">48 cores is definitely a small number for a job like this, but I wasn’t pushing the memory limits – these cores all have plenty of memory (4GB per core). As for sharing the node, it’s a good thought, but each
node has 48 cores (which is why I went in multiples of 48). I also tried to keep the tiles as square as possible. They weren’t always perfect, but the 480 core run actually had slightly squarer tiles than the 384 core run and the 768 core run had perfectly
square tiles.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">“</span><i><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">Another possibility is that this is within the machine noise”<o:p></o:p></span></i></p>
<p class="MsoNormal"><span style="font-size:11.0pt">That’s certainly a possibility. I haven’t rerun all of the tests, but I reran a couple and the timings were very similar. The variation wasn’t enough to pull the curve below the ideal scaling curve. I don’t
think this is noise. It seemsto me that there is structure in the signal.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">Are you doing any I/O?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’m not doing much I/O. I set it to output a few monthly mean fields, but that was all. Using the timing breakdown in STDOUT, the time spent in “DO_THE_MODEL_IO” scales pretty well with the core count (see
attached).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Ed<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"-webkit-standard",serif;color:black;background:white;mso-fareast-language:EN-GB">Dimitris Menemenlis</span></b><span style="font-size:13.5pt;font-family:"-webkit-standard",serif;color:black;background:white;mso-fareast-language:EN-GB"> </span><span style="font-size:11.0pt;mso-fareast-language:EN-GB"><a href="mailto:mitgcm-support%40mitgcm.org?Subject=Re:%20Re%3A%20%5BMITgcm-support%5D%20%5BEXTERNAL%5D%20%20Better%20than%20expected%20HPC%20Scaling&In-Reply-To=%3C2BA4C263-D240-4F32-9233-D60B208EDD15%40jpl.nasa.gov%3E" title="[MITgcm-support] [EXTERNAL] Better than expected HPC Scaling"><span style="font-family:"-webkit-standard",serif;background:white">menemenlis
at jpl.nasa.gov</span><span style="font-family:"-webkit-standard",serif;color:black;background:white;text-decoration:none"> </span></a></span><span style="font-size:11.0pt;font-family:"-webkit-standard",serif;color:black;mso-fareast-language:EN-GB"><br>
<i><span style="background:white">Tue Oct 13 00:38:33 EDT 2020</span></i></span><span style="font-size:11.0pt;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">I like the theory of better cache utilization initially, from 48 to ~250 processors, then degrading @ higher processor
count because of communications (and to a lesser extent more grid cells and hence more computations in the overlap regions).<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">Are you doing any I/O?<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">><i> On Oct 12, 2020, at 9:24 PM, Matthew Mazloff <<a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support">mmazloff
at ucsd.edu</a>> wrote:<o:p></o:p></i></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">><i>
<o:p></o:p></i></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">><i> Hi Ed<o:p></o:p></i></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">><i>
<o:p></o:p></i></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">><i> It depends on the machine you are running on. But its only slightly better at 250 than 48, and 48 does seem a
bit low for a job of that size. Maybe you were pushing max memory? It is odd, but I suspect you are on a machine that has very fast interconnects and I/O and 250 is just more efficient.
<o:p></o:p></i></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">><i>
<o:p></o:p></i></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">><i> Another possibility is that this is within the machine noise. Or that you were sharing a node when you ran the
48 and 96 job. Or that your tiles were very rectangular for the 48 and 96 core jobs so you had more overlap.<o:p></o:p></i></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">><i>
<o:p></o:p></i></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">><i> Matt<o:p></o:p></i></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">><i>
<o:p></o:p></i></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Courier New";color:black;mso-fareast-language:EN-GB">><i>
</i><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Edward Doddridge <edward.doddridge@utas.edu.au><br>
<b>Date: </b>Tuesday, 13 October 2020 at 15:00<br>
<b>To: </b>"mitgcm-support@mitgcm.org" <mitgcm-support@mitgcm.org><br>
<b>Subject: </b>Better than expected HPC Scaling</span><span style="color:black;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-family:Palatino">Hi MITgcmers,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Palatino"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Palatino">As part of an HPC bid I need to provide some scaling information for MITgcm on their cluster. The test configuration is a reentrant channel 600x800x50 grid points, using just the ocean component and some
idealised forcing fields. As I increased the core count between 48 and 384 the model scaled better than the theoretical scaling (see attached figure). I’m not complaining that it ran faster, but I was surprised. Any thoughts about what would cause this sort
of behaviour? I wondered if it might be something to do with the tiles not fitting in the cache for the low core count simulations. The bid might be more convincing if I can give a plausible explanation for why the model scales better than ideal.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Palatino"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Palatino;color:black;mso-fareast-language:EN-GB">Cheers,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Palatino;color:black;mso-fareast-language:EN-GB">Ed</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:Palatino;color:black;mso-fareast-language:EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black;mso-fareast-language:EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-family:Palatino;color:black;mso-fareast-language:EN-GB"><img border="0" width="1373" height="1" style="width:14.302in;height:.0104in" id="Rectangle_x0020_1" src="cid:image001.png@01D6A2D2.A27644A0"></span><o:p></o:p></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:Palatino;color:black;mso-fareast-language:EN-GB">Edward Doddridge</span></b><o:p></o:p></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:Palatino;color:black;mso-fareast-language:EN-GB">Research Associate and Theme Leader</span></i><o:p></o:p></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:Palatino;color:black;mso-fareast-language:EN-GB">Australian Antarctic Program Partnership (AAPP)</span></i><o:p></o:p></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:Palatino;color:black;mso-fareast-language:EN-GB">Institute for Marine and Antarctic Studies (IMAS)</span></i><o:p></o:p></p>
<p class="MsoNormal"><i><span style="font-size:10.0pt;font-family:Palatino;color:black;mso-fareast-language:EN-GB">University of Tasmania (UTAS)</span></i><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Palatino;color:black"><a href="doddridge.me"><span style="color:#0563C1">doddridge.me</span></a></span><o:p></o:p></p>
</div>
<p style="font-size:10pt; line-height:10pt; font-family: Calibri,sans-serif;"><br>
<br>
University of Tasmania Electronic Communications Policy (December, 2014). <br>
This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained
in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.
</p>
</body>
</html>