<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><span class=""></span><span class="">Bonjour Jean-Michel, would you be aware of any possible issues with switching the hi-res LLC simulations to </span><span class="">SOLVE_DIAGONAL_LOWMEMORY ?</span><div class=""><span class="">Right now we use the default settings of:</span></div><div class=""><span class=""><br class=""></span></div><div class=""><div class=""><div class=""><div class="">C o Choices for implicit solver routines solve_*diagonal.F</div><div class="">C The following has low memory footprint, but not suitable for AD</div><div class="">#undef SOLVE_DIAGONAL_LOWMEMORY</div><div class="">C The following one suitable for AD but does not vectorize</div><div class="">#undef SOLVE_DIAGONAL_KINNER</div></div><div class=""><br class=""></div><div class="">But Dan notes that the low-memory variant is approximately twice as fast than the default settings for the llc_540 set-up.</div><div class="">I could not find any previous discussion of SOLVE_DIAGONAL_LOWMEMORY in mitgcm-support.</div><div class=""><br class=""></div><div class="">Merci, Dimitris</div><div class=""><br class=""></div></div><span class=""><br class=""><blockquote type="cite" class="">Begin forwarded message:<br class=""><br class="">From: "Kokron, Daniel S. (ARC-606.2)[InuTeq, LLC]" <<a href="mailto:daniel.s.kokron@nasa.gov" class="">daniel.s.kokron@nasa.gov</a>><br class="">Subject: Re: Performance analysis feedback<br class="">Date: January 7, 2021 at 10:49:23 AM PST<br class="">To: Dimitris Menemenlis <<a href="mailto:menemenlis@jpl.nasa.gov" class="">menemenlis@jpl.nasa.gov</a>><br class="">Cc: "Zhang, Hong (JPL-398K)[JPL Employee]" <<a href="mailto:hong.zhang@jpl.nasa.gov" class="">hong.zhang@jpl.nasa.gov</a>><br class=""><br class="">Kinner 147.07/186.45<br class="">Fall though 60.37/125.83<br class="">Low mem 26.1/67.9<br class=""> <br class="">See attached. Upper left is Kinner, upper right is fall through and lower middle is low mem. solve_tridiagonal() is highlighted in yellow.<br class="">Dan<br class=""> <br class="">From: "Kokron, Daniel S. (ARC-606.2)[InuTeq, LLC]" <<a href="mailto:daniel.s.kokron@nasa.gov" class="">daniel.s.kokron@nasa.gov</a>><br class="">Date: Thursday, January 7, 2021 at 10:05 AM<br class="">To: Dimitris Menemenlis <menemenlis@jpl.nasa.gov><br class="">Cc: "Zhang, Hong (JPL-398K)[JPL Employee]" <hong.zhang@jpl.nasa.gov><br class="">Subject: Re: Performance analysis feedback<br class=""> <br class="">Comparing profiles with and without Kinner, Kinner is definitely slower than the fall-through path. I did not try using the lowmem path.<br class=""> <br class="">There is a lot of load imbalance among the ranks so I’ve included min and max times (s) spent in the solve_tridiagonal() routine using the llc_540 case run on 767 ranks.<br class=""> <br class="">Kinner 147.07/186.45<br class="">Fall though 60.37/125.83<br class=""> <br class="">Fall though is activated with<br class="">#undef SOLVE_DIAGONAL_LOWMEMORY<br class="">#undef SOLVE_DIAGONAL_KINNER<br class=""> <br class=""> <br class="">From: Dimitris Menemenlis <menemenlis@jpl.nasa.gov><br class="">Date: Wednesday, January 6, 2021 at 7:57 PM<br class="">To: "Kokron, Daniel S. (ARC-606.2)[InuTeq, LLC]" <daniel.s.kokron@nasa.gov><br class="">Cc: "Zhang, Hong (JPL-398K)[JPL Employee]" <hong.zhang@jpl.nasa.gov><br class="">Subject: Re: Performance analysis feedback<br class=""> <br class="">actually, it looks like we are “not” using the low-memory option in any of our set-ups:<br class=""> <br class="">bash-3.2$ grep SOLVE_DIAGONAL_LOW */*/code*/*h */code*/*h <br class="">llc_540/tides_exp/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY<br class="">llc_90/tides_exps/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY<br class="">llc_1080/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY<br class="">llc_2160/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY<br class="">llc_270/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY<br class="">llc_270/code_ad/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY<br class="">llc_4320/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY<br class="">llc_540/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY<br class="">llc_90/code-async-noseaice/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY<br class="">llc_90/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY<br class=""> <br class="">which happens to be the default for unmodified MITgcm CPP_OPTIONS.h<br class=""> <br class="">bash-3.2$ grep SOLVE_DIAGONAL CPP_OPTIONS.h <br class="">#undef SOLVE_DIAGONAL_LOWMEMORY<br class="">#undef SOLVE_DIAGONAL_KINNER<br class=""> <br class="">what do you recommend?<br class=""> <br class="">Dimitris<br class=""> <br class=""> <br class=""><br class=""><br class=""><br class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite">On Jan 6, 2021, at 3:03 PM, Kokron, Daniel S. (ARC-606.2)[InuTeq, LLC] <daniel.s.kokron@nasa.gov> wrote:<br class=""> <br class="">Hong,<br class="">The Kinner tri-diagonal solver code path is showing up in my profiling of the llc_540 case. None of the other user cases I have is using this path. Does your investigation require using the Kinner path?<br class="">Dan<br class=""> <br class=""></blockquote> <br class=""><img style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="" apple-inline="yes" id="EC60744A-4FD0-4A71-A9F6-9CCE7A4A8EAF" src="cid:322342C7CF0023438A3986B9442842E3@namprd09.prod.outlook.com"><br class=""></blockquote></span><br class=""></div></body></html>