[MITgcm-support] Converting mdsio output to netcdf, or reading mdsio in python
Oliver Jahn
jahn at MIT.EDU
Fri Oct 26 14:09:05 EDT 2012
Hi Jody,
thanks for the feedback. I hadn't taken too much time documenting the
more exotic features well. I've added an explanation of the wild-card
thing further up in the doc string and a few equivalent examples to
clarify how the "lev" option translates into array indexing. Hopefully
it will make it clearer. If you have more suggestions, let me know, or
feel free to improve it yourself.
You are right about the "rec" option. It is useful for pickups and
diagnostics output with multiple fields per file. If a file doesn't
have multiple records, you don't have to worry about it!
Cheers,
Oliver
On 2012-10-26 13:13, Jody Klymak wrote:
> Hi Oliver,
>
> Well, isn't that clever!
>
> Sorry to not have read the documentation closely enough...
>
> I guess I didn't grok it because I don't have the 'res_*' prefix on any of my directories. Maybe this option needs to be more explicitly stated as serving the needs of directory organizations like the one I have.
>
> i.e.:
>
> T = rdmds('*0/T',2880) # will read all files in 0000/T.0000002880.*, 0001/T.0000002880.*, etc.
>
> I guess I was also mystified by "rec" in the documentation, and multiple-dimensional levels. Is this just for diags and pickup? Normal data don't have "records" do they?
>
> I love the "region" option!
>
> Thanks a lot, this will make a transition to python a *lot* easier for me.
>
> Cheers, Jody
>
>
> On Oct 26, 2012, at 9:44 AM, Oliver Jahn <jahn at MIT.EDU> wrote:
>
>> Hi Jody,
>>
>> the python rdmds accepts shell wildcards:
>>
>> a = rdmds('0*/T', 0)
>>
>> It will then find the files (maybe more reliable with a prefix as in rdmds('res_*/T', 0)).
>>
>> There's an example of this in the doc string! ;)
>>
>> Oliver
>>
>> On 2012-10-26 12:13, Jody Klymak wrote:
>>> Hi Oliver et al,
>>>
>>> Thanks for this, it works great, except my MITgcm outputs files into directories by tile (or maybe cpu, they are the same for me):
>>>
>>> 0000/T0000000000.001.001.data
>>> 0001/T0000000000.001.002.data
>>> 0002/T0000000000.001.003.data
>>> etc
>>>
>>> I already modified the matlab fcn to account for this. Happy to do the same for the python one, but, I wonder why I'm the only one who gets their files organized in this fashion (which I like, btw).
>>>
>>> Thanks, Jody
>>>
>>> On Oct 25, 2012, at 16:40 PM, Oliver Jahn <jahn at MIT.EDU> wrote:
>>>
>>>> Hi Andrea,
>>>>
>>>> there is one now, in MITgcm/utils/python/MITgcmutils
>>>>
>>>> Either install it by running
>>>>
>>>> python setup.py install --user
>>>>
>>>> (or wherever you like), or just put the directory on your PYTHONPATH.
>>>> Either way, you should be able to
>>>>
>>>> from MITgcmutils import rdmds
>>>>
>>>> and start reading mds files using rdmds. It has a (hopefully reasonably useful) doc string.
>>>>
>>>> There is also a script scripts/gluemncbig for gluing netcdf files, in case you go the other way. It depends only on python and numpy.
>>>>
>>>> Cheers,
>>>> Oliver
>>>>
>>>>
>>>> On 2012-10-25 12:13, cimatori wrote:
>>>>> If there is a MITgcm python package, I will gladly shift to mds output
>>>>> (which seems a lot faster).
>>>>>
>>>>> Andrea Cimatoribus
>>>>> Royal Netherlands Meteorological Institute
>>>>> www.knmi.nl/~cimatori
>>>>>
>>>>> On 10/25/2012 04:07 PM, Ryan Abernathey wrote:
>>>>>> Hi Andrea,
>>>>>>
>>>>>> You have 3 options:
>>>>>> - Simply read the mds files into python using np.fromfile (they are
>>>>>> just raw binary data)
>>>>>> - Use Oliver Jahn's MITgcm python utils package, which contains a
>>>>>> python mdsio and many other useful scripts (Oliver, is this publicly
>>>>>> available yet?)
>>>>>> - Upgrade your code to the latest version of layers, which now does
>>>>>> output via the diagnostics package and therefore should support netcdf
>>>>>> format. (Disclaimer: I have not tested netcdf output, but maybe Gael
>>>>>> has?)
>>>>>>
>>>>>> Personally, I actually prefer mds output. The main reason is that
>>>>>> netcdf does not support singleCPUio and consequently produces a
>>>>>> ridiculous amount of output files that then have to be glued together
>>>>>> in a separate step.
>>>>>>
>>>>>> Best,
>>>>>> Ryan
>>>>>>
>>>>>>
>>>>>> On Oct 25, 2012, at 6:26 AM, cimatori wrote:
>>>>>>
>>>>>>> Hi everybody,
>>>>>>> I have a technical issue which is probably rather common. I am using
>>>>>>> layers package to compute the stream function in density, but layers
>>>>>>> package can produce output only in the meta/data format.
>>>>>>> Unfortunately, I have no matlab available, and all my scripts are
>>>>>>> based on netcdf format. Is there a way convert meta/data to netcdf?
>>>>>>> Is there an easy way to read meta/data files in python?
>>>>>>> I do have access to octave (an open source version of matlab) if that
>>>>>>> can help.
>>>>>>> Many thanks,
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Andrea Cimatoribus
>>>>>>> Royal Netherlands Meteorological Institute
>>>>>>> www.knmi.nl/~cimatori
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> MITgcm-support mailing list
>>>>>>> MITgcm-support at mitgcm.org
>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> MITgcm-support mailing list
>>>>>> MITgcm-support at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> MITgcm-support mailing list
>>>>> MITgcm-support at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>
>>> --
>>> Jody Klymak
>>> http://web.uvic.ca/~jklymak/
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>
>>
>>
>> --
>> Oliver Jahn tel: +1 617 253 2454
>> Earth System Initiative and fax: +1 617 253 4464
>> Dept. of Earth, Atmospheric and Planetary Sciences
>> Massachusetts Institute of Technology
>> 77 Massachusetts Ave., Bldg. 54-1510 skype: oliver.jahn
>> Cambridge, MA 02139-4307 USA email: jahn at mit.edu
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
> --
> Jody Klymak
> http://web.uvic.ca/~jklymak/
>
>
>
>
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
More information about the MITgcm-support
mailing list