[MITgcm-support] release of xmitgcm v0.2.0

[Ryan Abernathey]

Yes! It does fix it, by only opening files when their data is actually accessed. All calls to read data are now by default wrapped in dask.delayed objects. There is still plenty of room for benchmarking the various read options, of which there are now many permutations. Would love your help with this. I can provide some guidance off list. 

You are also correct that we neglected to update the docs to clarify this. 

-Ryan
 

> On Feb 14, 2017, at 5:38 PM, Jody Klymak <jklymak at uvic.ca> wrote:
> 
> Hi Ryan,
> 
> Does this release fix the `memmap` issue where a huge `numpy.memmap` gets made for the mds store?  I just did a test on my heavily `ulimit` constrained machine, and it *seems* to have fixed it if I do chunking, but I wasn’t clear if you’ve adopted a new approach.
> 
> Right now, I can survive by opening some of the data using `xm_open_mdsdataset` and then saving right away to a netcdf file, and then doing everything in xarray. However, that is obviously suboptimal.
> 
> So I don’t run into memory issues is the new technique described somewhere?  The docs seem to say it just uses `memmap` still.  
> 
> Thanks so much, limitations and all it is still a very useful package.
> 
> Cheers,   Jody
> 
> 
> 
> 
>> On 14 Feb 2017, at  11:28 AM, Ryan Abernathey <ryan.abernathey at gmail.com> wrote:
>> 
>> Hello MITgcm Folks,
>> 
>> I am happy to announce the release of xmitgcm v0.2.0.
>> http://xmitgcm.readthedocs.io/en/latest/
>> https://github.com/xgcm/xmitgcm
>> 
>> xmitgcm is a python package for reading MITgcm binary MDS files into xarray data structures. By storing data in dask arrays, xmitgcm enables parallel, out-of-core analysis of MITgcm output data.
>> 
>> This release contains the following feature enhancements:
>> 
>> Files are not read until the data are accessed. This helps overcome a common “too many open files issue” (#11).
>> A workaround for missing .meta files (#12).
>> Option for a separate grid_dir in case it is different from data_dir (#13).
>> Refactor of the way LLC data is read which allows for more efficient chunking and lower memory usage (#20)
>> Bug fix related to the handling of default_dtype parameter (#34). By Guillaume Sérazin.
>> Support for older MITgcm versions that write a different lenght drC variable (#8). By Liam Brannigan.
>> Support for cartesian curvilinear grids. By Andrea Cimatoribus.
>> Expanded and improved documentation.
>> 
>> I am very happy to see other MITgcm users using and contributing actively to xmitgcm.
>> 
>> One major goal for this release was to improve the reading of LLC4320 files on pleiades. xmitgcm is now very efficient at extracting sub-regions from the LLC simulation. If you are interested in this topic, I encourage you to check out the documentation and performance analysis here:
>> http://xmitgcm.readthedocs.io/en/latest/performance.html
>> 
>> Installation instructions:
>> http://xmitgcm.readthedocs.io/en/latest/
>> 
>> Cheers,
>> Ryan Abernathey
>> 
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
> 
> --
> Jody Klymak    
> http://web.uvic.ca/~jklymak/
> 
> 
> 
> 
> 
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20170214/35a35766/attachment-0001.htm>