[MITgcm-support] Converting mdsio output to netcdf, or reading mdsio in python

Jody Klymak jklymak at uvic.ca
Fri Oct 26 13:13:08 EDT 2012


Hi Oliver,

Well, isn't that clever! 

Sorry to not have read the documentation closely enough...

I guess I didn't grok it because I don't have the 'res_*' prefix on any of my directories.  Maybe this option needs to be more explicitly stated as serving the needs of directory organizations like the one I have.

i.e.: 

T = rdmds('*0/T',2880) # will read all files in 0000/T.0000002880.*, 0001/T.0000002880.*, etc.

I guess I was also mystified by "rec" in the documentation, and multiple-dimensional levels.  Is this just for diags and pickup?  Normal data don't have "records" do they?

I love the "region" option!

Thanks a lot, this will make a transition to python a *lot* easier for me.

Cheers,   Jody


On Oct 26, 2012, at  9:44 AM, Oliver Jahn <jahn at MIT.EDU> wrote:

> Hi Jody,
> 
> the python rdmds accepts shell wildcards:
> 
> a = rdmds('0*/T', 0)
> 
> It will then find the files (maybe more reliable with a prefix as in rdmds('res_*/T', 0)).
> 
> There's an example of this in the doc string!  ;)
> 
> Oliver
> 
> On 2012-10-26 12:13, Jody Klymak wrote:
>> Hi Oliver et al,
>> 
>> Thanks for this, it works great, except my MITgcm outputs files into directories by tile (or maybe cpu, they are the same for me):
>> 
>> 0000/T0000000000.001.001.data
>> 0001/T0000000000.001.002.data
>> 0002/T0000000000.001.003.data
>> etc
>> 
>> I already modified the matlab fcn to account for this.  Happy to do the same for the python one, but, I wonder why I'm the only one who gets their files organized in this fashion (which I like, btw).
>> 
>> Thanks,   Jody
>> 
>> On Oct 25, 2012, at  16:40 PM, Oliver Jahn <jahn at MIT.EDU> wrote:
>> 
>>> Hi Andrea,
>>> 
>>> there is one now, in MITgcm/utils/python/MITgcmutils
>>> 
>>> Either install it by running
>>> 
>>>  python setup.py install --user
>>> 
>>> (or wherever you like), or just put the directory on your PYTHONPATH.
>>> Either way, you should be able to
>>> 
>>>  from MITgcmutils import rdmds
>>> 
>>> and start reading mds files using rdmds.  It has a (hopefully reasonably useful) doc string.
>>> 
>>> There is also a script scripts/gluemncbig for gluing netcdf files, in case you go the other way.  It depends only on python and numpy.
>>> 
>>> Cheers,
>>> Oliver
>>> 
>>> 
>>> On 2012-10-25 12:13, cimatori wrote:
>>>> If there is a MITgcm python package, I will gladly shift to mds output
>>>> (which seems a lot faster).
>>>> 
>>>> Andrea Cimatoribus
>>>> Royal Netherlands Meteorological Institute
>>>> www.knmi.nl/~cimatori
>>>> 
>>>> On 10/25/2012 04:07 PM, Ryan Abernathey wrote:
>>>>> Hi Andrea,
>>>>> 
>>>>> You have 3 options:
>>>>> - Simply read the mds files into python using np.fromfile (they are
>>>>> just raw binary data)
>>>>> - Use Oliver Jahn's MITgcm python utils package, which contains a
>>>>> python mdsio and many other useful scripts (Oliver, is this publicly
>>>>> available yet?)
>>>>> - Upgrade your code to the latest version of layers, which now does
>>>>> output via the diagnostics package and therefore should support netcdf
>>>>> format. (Disclaimer: I have not tested netcdf output, but maybe Gael
>>>>> has?)
>>>>> 
>>>>> Personally, I actually prefer mds output. The main reason is that
>>>>> netcdf does not support singleCPUio and consequently produces a
>>>>> ridiculous amount of output files that then have to be glued together
>>>>> in a separate step.
>>>>> 
>>>>> Best,
>>>>> Ryan
>>>>> 
>>>>> 
>>>>> On Oct 25, 2012, at 6:26 AM, cimatori wrote:
>>>>> 
>>>>>> Hi everybody,
>>>>>> I have a technical issue which is probably rather common. I am using
>>>>>> layers package to compute the stream function in density, but layers
>>>>>> package can produce output only in the meta/data format.
>>>>>> Unfortunately, I have no matlab available, and all my scripts are
>>>>>> based on netcdf format. Is there a way convert meta/data to netcdf?
>>>>>> Is there an easy way to read meta/data files in python?
>>>>>> I do have access to octave (an open source version of matlab) if that
>>>>>> can help.
>>>>>> Many thanks,
>>>>>> 
>>>>>> --
>>>>>> 
>>>>>> Andrea Cimatoribus
>>>>>> Royal Netherlands Meteorological Institute
>>>>>> www.knmi.nl/~cimatori
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> MITgcm-support mailing list
>>>>>> MITgcm-support at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> MITgcm-support mailing list
>>>>> MITgcm-support at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>> 
>>>> 
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>> 
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>> 
>> --
>> Jody Klymak
>> http://web.uvic.ca/~jklymak/
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>> 
> 
> 
> -- 
> Oliver Jahn                                       tel:  +1 617 253 2454
> Earth System Initiative and                       fax:  +1 617 253 4464
> Dept. of Earth, Atmospheric and Planetary Sciences
> Massachusetts Institute of Technology
> 77 Massachusetts Ave., Bldg. 54-1510              skype:    oliver.jahn
> Cambridge, MA 02139-4307 USA                      email:   jahn at mit.edu
> 
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support

--
Jody Klymak    
http://web.uvic.ca/~jklymak/







More information about the MITgcm-support mailing list