[MITgcm-support] glue netcdf files efficiently

[Martin Losch]

Hi Yinghui,

thanks for pointing out the higher efficiency of the python script gluemncbig. I did a quick test (I have python 2.6 I think) on my 2D output files with 16 tiles (so a fairly small problem)

time gluemnc diags2D.*.t*.nc 
real  0m24.198s
user 0m5.916s
sys  0m9.177s

and with
time gluemncgib -o diags2D.nc diags2D.*.t*.nc
real 0m0.259s
user 0m0.188s
sys 0m0.068s

so gluemncbig is almost 100 times faster! 

Oliver, you should be much less modest about your great contribution!

Martin

On Jan 9, 2013, at 11:48 AM, 何映晖 <heyinghui1984 at gmail.com> wrote:

> Hi Oliver et al
> 
> The gluemncbig works fine with python 2.7.3 and numpy 1.6.2 and is very efficient. 
> 
> By the way, I also have tried to run it with python 3.3.0. Unfortunately, it is aborted with the following error message.
> 
> 
>   File "/home/Phoenix/MITgcm_c64b/utils/python/MITgcmutils/scripts/gluemncbig", line 1190
>     print 'Files to be read:'
>                             ^
> SyntaxError: invalid syntax
> 
> 
> So it seems that the syntax is not compatible with python 3.3.0.
> 
> The bash script gluemnc still can not work with small files. I guess the problem may be related with the syntax of different nco version. Because I had once successfully run it on system CentOS 5 until the code was ported on system CentOS 6. Since building the nco source code is complicated, I installed the nco 4.0.5 by .rpm file which is the only version I can find which is compatible with CentOS 6. And I can't remember the nco version I used before.
> 
> Since I didn't know the nco much, I haven't confirmed my supposition. And what more, I think the python script, gluemncbig, is more efficient than the script gluemnc I used before.
> 
> Thanks,
> Yinghui
> 
> 2013/1/8 Oliver Jahn <jahn at mit.edu>
> On 2013-01-08 08:09, 何映晖 wrote:
> Finally, I try to use the python script /gluemncbig/. Since I don't know
> 
> the python at all, it also doesn't work.
> 
> $ gluemncbig -o state.glob.nc <http://state.glob.nc> state.0000000000.t00*
> 
> Tiled dimensions: Yp1 Y X Xp1
> Record dimension: T
> Writing non-record variables
> Traceback (most recent call last):
>    File
> "/home/Phoenix/MITgcm_c64b/utils/python/MITgcmutils/scripts/gluemncbig",
> line 1336, in <module>
>      if progress and not verbose: sys.stderr.write('Writing {} records:
> '.format(nrec))
> ValueError: zero length field name in format
> 
> Any ideas why I get those errors? Considering the files which I want to
> combine are so large (about 50G), which method is more efficient?
> 
> Hi Yinghui
> 
> the python error is because you don't have python >= 2.7 or >= 3.2. I've just checked in a fix for this that should work on older python. This script should work as long as each individual variable (or time record) fits in memory.  Let me know how it goes!
> 
> Oliver
> 
> 
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support
> 
> 
> 
> -- 
> 何映晖
> 南海海洋研究所
> 中国科学院 _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support