[MITgcm-devel] gluemncbig and many tiles

[Oliver Jahn]

(back to list)

On 2013-01-14 08:39, Martin Losch wrote:
> I am looking forward to your "other" gluemnc script. If speed is an issue, one can also think of switching between the fast and the slower version depending on the number of tiles?
> Going only along one direction will alleviate the problem most likely for another couple of years (untile we have >1000 tiles in x (o:)

I've added an option --many to the script in cvs that avoids opening all 
files at once, instead opening only a row of tiles along x at a time. 
This should avoid the "open files" problem for a while and also uses 
less memory.  With limited testing I've found a slowdown of up to 
5-fold, but not in all cases.  In some cases, --many was even faster.

Cheers,
Oliver

> Martin
>
> On Jan 14, 2013, at 2:23 PM, Oliver Jahn <jahn at MIT.EDU> wrote:
>
>> Hi Martin,
>>
>> thanks for taking an interest in this!
>>
>> On 2013-01-14 04:29, Martin Losch wrote:
>>> Hi Oliver,
>>>
>>> I am trying to use the great gluemncbig script to glue many (1120)
>>> netcdf files. After fixing this re.compile issue (although this may not
>>> be the best fix, and maybe {3,5} is even better):
>>>      tilepatt = re.compile(r'(\.t[0-9]{3,4}\.nc)$')
>>> I am getting different error messages on different computers:
>>>
>>> 1. On hicegate.hlrn.de (www.hlrn.de) with python 4.2
>>
>> I assume this is python 2.4.  In this case, yes, it is a python issue. 2.4 isn't supported by current numpy (which is needed by gluemncbig) anymore, so I didn't think too much about it.  Are you sure this machine doesn't have a more recent python in a module or something?
>>
>>> A more general question: I am so excited by the things python can do now
>>> (with matplotlib can plot finite elements very easily and beautifully),
>>> that I want to use it more. Do you already have a set of scripts that
>>> would replace our matlab functions (I saw mds, mnc, but, e.g. plotting
>>> the cube etc.) that you are willing to share?
>>
>> I have a bunch of unorganized scripts that I'd be happy to share, and I would love to put together a more cleaned-up set of tools relating to MITgcm when I get around to it.  Any help on this would be greatly welcomed.
>>
>> Oliver
>>
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>
>
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>