[MITgcm-devel] gluemncbig and many tiles

[Martin Losch]

Oliver,

I am not sure I understand everything in gluemncbig, but I think, unless one want to try to change the number of allowed open files to infinity, one could try to change the order of the loops around. Instead of opening all files and then looping over the variables, I can imagine to loop over the files, then over the variables always reading the local bit and also writing it locally:

for fname in files0:
    nc = netcdf_file(fname, 'r', **readopts)
    […]
    for name,pos,irec in ncout.begins
        […]
        data = nc.read_recvar(name, irecin)
       ncout.write_recvar_sub(name, irec, iX, iY, data )
   
    nc.close()

where write_recvar_sub is something like the fortran nf_put_vars_real (which I don't know yet how to write, but assume you do).
That shouldn't be much slower that the current implementation. What do you think?

Martin

On Jan 14, 2013, at 10:29 AM, Martin Losch <Martin.Losch at awi.de> wrote:

> Hi Oliver,
> 
> I am trying to use the great gluemncbig script to glue many (1120) netcdf files. After fixing this re.compile issue (although this may not be the best fix, and maybe {3,5} is even better):
>    tilepatt = re.compile(r'(\.t[0-9]{3,4}\.nc)$')
> I am getting different error messages on different computers:
> 
> 1. On hicegate.hlrn.de (www.hlrn.de) with python 4.2
> 
> hicegate0:/gfs1/work/hbklosch/MITgcm/arctic_4km/run_jfnk_01120 $ gluemncbig -o diags2D.glob.nc diags2D.0*.t*.nc
> Tiled dimensions: Yp1 Y X Xp1
> Record dimension: T
> Writing non-record variables
> Traceback (most recent call last):
>  File "/home/h/hbklosch/bin/gluemncbig", line 1336, in ?
>    if progress and not verbose: sys.stderr.write('Writing {} records: '.format(nrec))
> AttributeError: 'str' object has no attribute 'format'
> 
> 2. On a local machine at AWI (uv100.awi.de) with python 2.6
> uv100:run_jfnk_1120> gluemncbig -o diags2D.glob.nc diags2D.0001971000.t*
> Tiled dimensions: Yp1 Y X Xp1
> Record dimension: T
> Traceback (most recent call last):
>  File "/uv/home1/mlosch/bin/gluemncbig", line 1299, in <module>
>    ncs[fname] = nc = netcdf_file(fname, 'r', **readopts)
>  File "/uv/home1/mlosch/bin/gluemncbig", line 260, in __init__
>    self.fp = open(self.filename, '%sb' % mode)
> IOError: [Errno 24] Too many open files: 'diags2D.0001971000.t900.nc'
> 
> I am absolutely new to python (started looking at python last Thursday), so my opinion on this is probably very unqualified, but I assume that the first error has to do with the python version, but that the second error can be fixed somehow, right? I didn't find any solution yet, but while I keep looking you probably know the answer right away.
> 
> A more general question: I am so excited by the things python can do now (with matplotlib can plot finite elements very easily and beautifully), that I want to use it more. Do you already have a set of scripts that would replace our matlab functions (I saw mds, mnc, but, e.g. plotting the cube etc.) that you are willing to share?
> 
> Martin
> 
> 
> 
> 
> 
> 
> --
> 
> Martin Losch
> Alfred Wegener Institute for Polar and Marine Research
> Postfach 120161, 27515 Bremerhaven, Germany;
> Tel./Fax: ++49(0471)4831-1872/1797
> 
> 
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel