[MITgcm-support] release of xmitgcm v0.2.0

[Jody Klymak]

Hi Ryan,

Does this release fix the `memmap` issue where a huge `numpy.memmap` gets made for the mds store?  I just did a test on my heavily `ulimit` constrained machine, and it *seems* to have fixed it if I do chunking, but I wasn’t clear if you’ve adopted a new approach.

Right now, I can survive by opening some of the data using `xm_open_mdsdataset` and then saving right away to a netcdf file, and then doing everything in xarray. However, that is obviously suboptimal.

So I don’t run into memory issues is the new technique described somewhere?  The docs seem to say it just uses `memmap` still.  

Thanks so much, limitations and all it is still a very useful package.

Cheers,   Jody




> On 14 Feb 2017, at  11:28 AM, Ryan Abernathey <ryan.abernathey at gmail.com> wrote:
> 
> Hello MITgcm Folks,
> 
> I am happy to announce the release of xmitgcm v0.2.0.
> http://xmitgcm.readthedocs.io/en/latest/ <http://xmitgcm.readthedocs.io/en/latest/>
> https://github.com/xgcm/xmitgcm <https://github.com/xgcm/xmitgcm>
> 
> xmitgcm is a python package for reading MITgcm <http://mitgcm.org/public/r2_manual/latest/online_documents/node277.html> binary MDS files into xarray <http://xarray.pydata.org/> data structures. By storing data in dask <http://dask.pydata.org/> arrays, xmitgcm enables parallel, out-of-core <https://en.wikipedia.org/wiki/Out-of-core_algorithm> analysis of MITgcm output data.
> 
> This release contains the following feature enhancements:
> 
> Files are not read until the data are accessed. This helps overcome a common “too many open files issue” (#11 <https://github.com/xgcm/xmitgcm/issues/11>).
> A workaround for missing .meta files (#12 <https://github.com/xgcm/xmitgcm/issues/12>).
> Option for a separate grid_dir in case it is different from data_dir (#13 <https://github.com/xgcm/xmitgcm/issues/13>).
> Refactor of the way LLC data is read which allows for more efficient chunking and lower memory usage (#20 <https://github.com/xgcm/xmitgcm/issues/20>)
> Bug fix related to the handling of default_dtype parameter (#34 <https://github.com/xgcm/xmitgcm/pull/34>). By Guillaume Sérazin <https://github.com/serazing>.
> Support for older MITgcm versions that write a different lenght drC variable (#8 <https://github.com/xgcm/xmitgcm/issues/8>). By Liam Brannigan <https://github.com/braaannigan>.
> Support for cartesian curvilinear grids. By Andrea Cimatoribus <https://github.com/sambarluc>.
> Expanded and improved documentation.
> 
> I am very happy to see other MITgcm users using and contributing actively to xmitgcm.
> 
> One major goal for this release was to improve the reading of LLC4320 files on pleiades. xmitgcm is now very efficient at extracting sub-regions from the LLC simulation. If you are interested in this topic, I encourage you to check out the documentation and performance analysis here:
> http://xmitgcm.readthedocs.io/en/latest/performance.html <http://xmitgcm.readthedocs.io/en/latest/performance.html>
> 
> Installation instructions:
> http://xmitgcm.readthedocs.io/en/latest/ <http://xmitgcm.readthedocs.io/en/latest/>
> 
> Cheers,
> Ryan Abernathey
> 
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support

--
Jody Klymak    
http://web.uvic.ca/~jklymak/





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20170214/1a861940/attachment.htm>