[MITgcm-support] SOLVE_DIAGONAL_LOWMEMORY

[Dimitris Menemenlis]

Bonjour Jean-Michel, would you be aware of any possible issues with switching the hi-res LLC simulations to SOLVE_DIAGONAL_LOWMEMORY ?
Right now we use the default settings of:

C o Choices for implicit solver routines solve_*diagonal.F
C   The following has low memory footprint, but not suitable for AD
#undef SOLVE_DIAGONAL_LOWMEMORY
C   The following one suitable for AD but does not vectorize
#undef SOLVE_DIAGONAL_KINNER

But Dan notes that the low-memory variant is approximately twice as fast than the default settings for the llc_540 set-up.
I could not find any previous discussion of SOLVE_DIAGONAL_LOWMEMORY in mitgcm-support.

Merci, Dimitris


> Begin forwarded message:
> 
> From: "Kokron, Daniel S. (ARC-606.2)[InuTeq, LLC]" <daniel.s.kokron at nasa.gov>
> Subject: Re: Performance analysis feedback
> Date: January 7, 2021 at 10:49:23 AM PST
> To: Dimitris Menemenlis <menemenlis at jpl.nasa.gov>
> Cc: "Zhang, Hong (JPL-398K)[JPL Employee]" <hong.zhang at jpl.nasa.gov>
> 
> Kinner                   147.07/186.45
> Fall though          60.37/125.83
> Low mem            26.1/67.9
>  
> See attached.  Upper left is Kinner, upper right is fall through and lower middle is low mem.  solve_tridiagonal() is highlighted in yellow.
> Dan
>  
> From: "Kokron, Daniel S. (ARC-606.2)[InuTeq, LLC]" <daniel.s.kokron at nasa.gov>
> Date: Thursday, January 7, 2021 at 10:05 AM
> To: Dimitris Menemenlis <menemenlis at jpl.nasa.gov>
> Cc: "Zhang, Hong (JPL-398K)[JPL Employee]" <hong.zhang at jpl.nasa.gov>
> Subject: Re: Performance analysis feedback
>  
> Comparing profiles with and without Kinner, Kinner is definitely slower than the fall-through path.  I did not try using the lowmem path.
>  
> There is a lot of load imbalance among the ranks so I’ve included min and max times (s) spent in the solve_tridiagonal() routine using the llc_540 case run on 767 ranks.
>  
> Kinner                   147.07/186.45
> Fall though          60.37/125.83
>  
> Fall though is activated with
> #undef SOLVE_DIAGONAL_LOWMEMORY
> #undef SOLVE_DIAGONAL_KINNER
>  
>  
> From: Dimitris Menemenlis <menemenlis at jpl.nasa.gov>
> Date: Wednesday, January 6, 2021 at 7:57 PM
> To: "Kokron, Daniel S. (ARC-606.2)[InuTeq, LLC]" <daniel.s.kokron at nasa.gov>
> Cc: "Zhang, Hong (JPL-398K)[JPL Employee]" <hong.zhang at jpl.nasa.gov>
> Subject: Re: Performance analysis feedback
>  
> actually, it looks like we are “not” using the low-memory option in any of our set-ups:
>  
> bash-3.2$ grep SOLVE_DIAGONAL_LOW */*/code*/*h */code*/*h 
> llc_540/tides_exp/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY
> llc_90/tides_exps/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY
> llc_1080/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY
> llc_2160/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY
> llc_270/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY
> llc_270/code_ad/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY
> llc_4320/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY
> llc_540/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY
> llc_90/code-async-noseaice/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY
> llc_90/code/CPP_OPTIONS.h:#undef SOLVE_DIAGONAL_LOWMEMORY
>  
> which happens to be the default for unmodified MITgcm CPP_OPTIONS.h
>  
> bash-3.2$ grep SOLVE_DIAGONAL CPP_OPTIONS.h 
> #undef SOLVE_DIAGONAL_LOWMEMORY
> #undef SOLVE_DIAGONAL_KINNER
>  
> what do you recommend?
>  
> Dimitris
>  
>  
> 
> 
> 
>> On Jan 6, 2021, at 3:03 PM, Kokron, Daniel S. (ARC-606.2)[InuTeq, LLC] <daniel.s.kokron at nasa.gov> wrote:
>>  
>> Hong,
>> The Kinner tri-diagonal solver code path is showing up in my profiling of the llc_540 case.  None of the other user cases I have is using this path.  Does your investigation require using the Kinner path?
>> Dan
>>  
>  
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20210107/78cf1b6f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LLC540_SolveTriDiagonal.png
Type: image/png
Size: 2513770 bytes
Desc: not available
URL: <http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20210107/78cf1b6f/attachment-0001.png>