[MITgcm-support] Memory issues with regional simulation

[Christopher L. P. Wolfe]

Hi Ruth,

Any chance you could increase the number of processors to reduce the memory usage per CPU? Another thing to try is an OpenMP/MPI hybrid approach where you reduce the number of processors per node but increase the number of threads per processes. Threads can be much more lightweight memory-wise than independent processes. In my experience OpenMP sometimes works and sometimes doesn’t: it works fine my local linux box but not on Stampede at the U Texas (if anyone’s got it working, let me know).

Really pairing down the number of packages you’re compiling in can also help. Also, if you’re not using Second Order Moments advection, you want to make sure it’s disabled in GAD_OPTIONS.h because it soaks up a lot of memory.

Good luck,
Christopher

On Aug 15, 2014, at 8:04 PM, Ruth Musgrave <rmusgrave at ucsd.edu> wrote:

> Hi everyone,
> 
> I’m trying to run a regional model with 1e8 grid points on 64 processors. I have 1Gb of RAM per processor. Sadly with the number of full 3D diagnostics I would like to output, 1Gb of RAM isn’t enough. I think the diagnostics package is allocating double precision arrays for each of my diagnostics, which is what is causing my problems.
> 
> I wondered if there is a simple way to reduce the precision of the qdiag array in the diagnostics package. I had a look at it and I can only see very complicated ways of doing this (i.e. I tried and was unsuccessful). Is there something easy that I’m missing?
> 
> As a last resort I wondered if I could just run the entire model in single precision by making some changes to CPP_EEMACROS.h. It seems a bit desperate, and I don’t really have a sense of how errors would accumulate over a 1e5 timestep simulation. I’d be very grateful to hear other people’s experience and opinions on doing this.
> 
> If anyone has any other suggestions on how to reduce memory usage by the model (other than reduce grid points!) I’d be very happy to hear them.
> 
> Thanks in advance,
> 
> Ruth
> 
> 
> 
> 
> 
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support