[MITgcm-support] MITgcm virtual memory usage

[Ed Hill]

On Thu, 21 Sep 2006 20:51:34 -0700 (PDT)
jdawe at eos.ubc.ca wrote:

> 
> > Yes, you can reduce the overall memory footprint by reducing the
> > size of the domain and/or by turning off packages or reducing
> > the size of the buffers within some packages.  The prime
> > candidates are timeave and diagnostics which have significant
> > work arrays.
> 
> Ok, I'll try turning things off in those packages.

Yes, turn off the unnecessary packages and scale back the size of
the work array in, for instance, diagnostics.  And then move to MPI
so it can scale across more nodes.


> > But, really, who cares what the virtual memory usage is?  What
> > difference does it make?  The main thing that matters is whether
> > your model fits in memory or not.  If it spills over into swap
> > then you have a huge performance hit.  But if you can hold the
> > entire problem in RAM (without digging into swap on a continual
> > basis) then what difference does it make whether you use a few
> > more MB of RAM or not?
> 
> It matters because I want to upscale this run to a cluster with
> slave nodes that have 2GB of RAM and no swap.  If I can get rid of
> the 60% of allocated ram that MITgcm doesn't use, I can double the
> vertical resolution of the model.  That matters to me.

I think you're missing the bigger picture.  VM OSes (eg. Linux) are
generally allowed to overcommit even when there is no swap device.
Therefor, you can have a program that allocates a larger chunk of
memory than exists and, so long as parts of that memory are never
actually paged in, then its fine--it will run.


> > Oh, and don't place too much faith in any numbers you get from
> > ps, top, or their ilk.  They're quite approximate.  See:
> >
> > http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html
> >
> > for a quick intro.
> 
> I am well aware of this.  For a array-intensive program like
> MITgcm, however, ps gives a fair indication of memory usage.
> Examining pmap's output, for instance, suggests that 1602 of the
> 1617 MB of MITgcm's process are unshared.

Sharing is only part of the story.  What matters more is whether the
pages are ever actually used.  If the pages get allocated but are
never touched then they can stay virtual and we're back to my
original contention which is: who cares what the virtual size is?

Also, if swap usage is not intense (that is, the pages are moved
in/out relatively infrequently), then you can create swap partitions
on network devices such as NFS or a network block device (NBD).  This
allows your VM to push the rarely-used resources out of RAM.  And it
will incur a negligible performance hit *IF* the swap usage is
sufficiently infrequent.  If you use the swap frequently then it will
of course be a huge bottleneck.

Ed

-- 
Edward H. Hill III, PhD  |  ed at eh3.com  |  http://eh3.com/