[MITgcm-support] xmitgcm: python package for reading mitgcm mds files

Tue Oct 18 10:40:31 EDT 2016

Matt, very interesting to hear. Should be pretty easy for xmitgcm to deal
with tiled output...I will put it on the todo list.

Since the discussion is open, I should point out that xmitgcm is only
necessary because netCDF output is not widely used in MITgcm. We are really
an outlier compared to all other ocean models on this point.

On Tue, Oct 18, 2016 at 10:34 AM, Matthew Mazloff <mmazloff at ucsd.edu> wrote:

> Hi Ryan
>
> Just a note — hope not to distract on this great package — There are a lot
> of machines (e.g. stampede) and setups where one needs
>  useSingleCpuIO=.FALSE.
>
> This is primarily so each node can write to its local /tmp disk instead of
> the global file system. Makes for a huge performance increase.
>
> Also, if ones setup gets huge one node may not be able to receive all the
> data to I/O. It may just not have enough memory to hold the global array.
>
> Thanks for the package!
> Matt
>
>
> On Oct 18, 2016, at 7:27 AM, Ryan Abernathey <ryan.abernathey at gmail.com>
> wrote:
>
> Great question Liam. The package definitely assumes useSingleCpuIO=.TRUE.
> However, it would probably be pretty straightforward to accommodate
>  tile-based files. In fact, this would mesh quite well with the concept of
> dask "chunks" (http://dask.pydata.org/en/latest/array-creation.html#chunks)
> and could be implemented pretty far upstream.
>
> If this is something that would be useful to you, please open an issue:
> https://github.com/xgcm/xmitgcm/issues
>
> I am curious why anyone uses useSingleCpuIO=.FALSE. Are there any
> performance benefits? To me it just seems like a hassle dealing with all
> the files, and my models run faster with useSingleCpuIO=.TRUE. So it's a
> win-win.
>
> Note that Andrea also raised an issue related to reading diagnostics that
> use the `levels` option in data.diagnostics to select only certain vertical
> levels for output. (It doesn't work with xmitgcm...yet)
> https://github.com/xgcm/xmitgcm/issues/8
>
> -Ryan
>
>
>
>
> On Tue, Oct 18, 2016 at 10:10 AM, Liam Brannigan <
> liam.brannigan at misu.su.se> wrote:
>
>> Hi Ryan
>>
>> Thanks for developing this.  I was wondering how you produce the output
>> files for parallel runs  - do you have useSingleCpuIO=.TRUE. or do you glue
>> together the mds output to produce the files read by xmitgcm?
>>
>> Liam
>> ________________________________________
>> From: mitgcm-support-request at mitgcm.org <mitgcm-support-request at mitgcm
>> .org>
>> Sent: 15 October 2016 18:00
>> To: mitgcm-support at mitgcm.org
>> Subject: MITgcm-support Digest, Vol 160, Issue 7
>>
>> Send MITgcm-support mailing list submissions to
>>         mitgcm-support at mitgcm.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         http://mitgcm.org/mailman/listinfo/mitgcm-support
>> or, via email, send a message with subject or body 'help' to
>>         mitgcm-support-request at mitgcm.org
>>
>> You can reach the person managing the list at
>>         mitgcm-support-owner at mitgcm.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of MITgcm-support digest..."
>>
>>
>> Today's Topics:
>>
>>    1. xmitgcm: python package for reading mitgcm mds    files
>>       (Ryan Abernathey)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Sat, 15 Oct 2016 00:19:45 -0400
>> From: Ryan Abernathey <ryan.abernathey at gmail.com>
>> To: "mitgcm-support at mitgcm.org" <mitgcm-support at mitgcm.org>
>> Subject: [MITgcm-support] xmitgcm: python package for reading mitgcm
>>         mds     files
>> Message-ID:
>>         <CAJAZx5DG=vWDD-G9nyN+RqvekzUpxdZsZm3_dTTuM+61_QM6Hg at mail.
>> gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Dear Colleagues,
>>
>> I have released a first version of my python package called xmitgcm:
>>
>> xmitgcm is a python package for reading MITgcm <http://mitgcm.org/>
>> binary
>> MDS
>> <http://mitgcm.org/public/r2_manual/latest/online_documents/node277.html>
>> files
>> into xarray <http://xarray.pydata.org/> data structures. By storing data
>> in
>> dask <http://dask.readthedocs.org/> arrays, xmitgcm enables parallel,
>> out-of-core <https://en.wikipedia.org/wiki/Out-of-core_algorithm>
>> analysis
>> of MITgcm output data.
>>
>> For more information, please consult
>>
>>    - The online documentation: http://xmitgcm.readthedocs.io
>>    - The GitHub repository: https://github.com/xgcm/xmitgcm
>>
>> Among many features, the package includes basic support for reading llc
>> mds
>> data, which is something we have been lacking in python. xmitgcm does not
>> depend on any existing MITgcm scripts like rdmds.
>>
>> xmitgcm is not a complete analysis framework like gcmfaces. It is just a
>> tool for reading mds data. Nevertheless, once the data is in xarray form,
>> many powerful options become available.
>>
>> Early in its development, xmitgcm used to be called "xgcm". Those of you
>> who were using the earlier "xgcm" package should switch to this one. It
>> should be largely backwards compatible, but xmitgcm will continue to
>> receive udpates, bug fixes, and new features.
>>
>> I welcome your feedback on your experience with xmitgcm via email, or,
>> even
>> better, as github issues
>> https://github.com/xgcm/xmitgcm/issues
>> and pull requests
>> https://github.com/xgcm/xmitgcm/pulls
>>
>> Happy pythoning!
>>
>> -Ryan
>> http://rabernat.github.io/
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/2016
>> 1015/050b0b75/attachment.html>
>>
>> ------------------------------
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>
>>
>> End of MITgcm-support Digest, Vol 160, Issue 7
>> **********************************************
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
>
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20161018/f040860d/attachment-0001.htm>