[MITgcm-support] changing the number of tiles

Samar Khatiwala spk at ldeo.columbia.edu
Wed Mar 8 15:03:41 EST 2006 

Paola:

I haven't used joinmds, and perhaps I don't fully understand your  
problem, but it seems
like a simple matter of writing a little matlab script to read in  
each level at a time from the
tiled files, joint them, dump that level to file, and go to the next  
level. If your files are truly humongous,
you can also loop over the meridional direction. These are direct  
access binary files, so it should
be a simple matter to implement this without running out of memory.
This may help: http://www.ldeo.columbia.edu/~spk/Code/Matlab/Misc/ 
read_binary.m

Samar

On Mar 8, 2006, at 2:51 PM, Paola Cessi wrote:

> Thanks Stephanie. However that did not work for me. The model has  
> 30 levels, and the merging was only done for the 448x896 grid at  
> one level (I think). The total number of data should be 448x896x30.
>
> Is there a way to have joinmds merge all levels so that all the  
> data are merged? Below is the output from joinmds that I got.
>
> Notice that joinmds reads only a fraction of the pickup files,  
> which are 97542144 bytes long, not 401408 bytes. However, there was  
> an error warnign which I do not understand. Any help on how to fix  
> it? Or is there a new version of joinmds that you can send me?
>
> Thanks, paola
>
> ------------------------------------------------------------------
> bash-2.05b$ ~/MITgcm/utils/scripts/joinmds pickup 0000240000
> Name "main::nrecords_" used only once: possible typo at /spin/home/ 
> cessi/MITgcm/utils/scripts/joinmds line 182.
> There are 8 datasets.
> All existing meta files are self- and mutually consistent.
> Datasets are complete.
> Current machine's endianness: little-endian
> pickup.0000240000.001.001.data: 401408 bytes, okay, merged from .
> pickup.0000240000.001.002.data: 401408 bytes, okay, merged from .
> pickup.0000240000.001.003.data: 401408 bytes, okay, merged from .
> pickup.0000240000.001.004.data: 401408 bytes, okay, merged from .
> pickup.0000240000.001.005.data: 401408 bytes, okay, merged from .
> pickup.0000240000.001.006.data: 401408 bytes, okay, merged from .
> pickup.0000240000.001.007.data: 401408 bytes, okay, merged from .
> pickup.0000240000.001.008.data: 401408 bytes, okay, merged from .
> Global data (448x896) is in ./pickup.0000240000.data (endianness is  
> big-endian).
>
>
> On Tue, 7 Mar 2006, Stephanie Dutkiewicz wrote:
>
>>
>> Paola -
>> I usually join the files with the script:
>> utils/script/joinmds
>>
>>> joinmds pickup 0000003200
>>
>>
>> steph
>>
>> On Tue, 7 Mar 2006, Paola Cessi wrote:
>>
>>> Hi Jean-Michel, is the step of reassembling the pickup files into  
>>> a single file necessary? I am asking because I cannot do it in  
>>> Matlab, given the size of my files. Do you have an alternative  
>>> suggestion that is not as memory intensive? Thanks,
>>> paola
>>> On Wed, 1 Mar 2006, Jean-Michel Campin wrote:
>>>> Hi Paola,
>>>> The thing that I do generally, in this case, is to read
>>>> all the tile-files (in matlab, with rdmds), and then
>>>> I just write the output from rdmds to a binary file:
>>>> namfil='pickup.0000003200';
>>>> va=rdmds(namfil);
>>>> wrfile=[namfil,'.data'];
>>>> fid=fopen(wrfile,'w','b');
>>>> fwrite(fid,va,'real*8');
>>>> fclose(fid);
>>>> And there is no problem for the model to restart from a unique  
>>>> file,
>>>> i.e. a "global" file that contains all the tiles, whatever the
>>>> new domain decomposition is.
>>>> Jean-Michel
>>>> On Wed, Mar 01, 2006 at 07:46:46PM -0500, Baylor Fox-Kemper wrote:
>>>>> No interpickups isn't necessary if there is no interpolation or
>>>>> coarsening, but it will work in that case, too.
>>>>> What Ed is suggesting (I think) doesn't require netcdf.  However,
>>>>> someone not using netcdf should spell out exactly how this is  
>>>>> to be
>>>>> done.  I think usually one needs to keep the number of tiles  
>>>>> fixed,
>>>>> while changing the number of processors.  However, this process
>>>>> seemed messy to me, so that's why I wrote interpickups in the  
>>>>> first
>>>>> place.
>>>>> Will someone please explain how to do retiling, or at pickup
>>>>> conversions (without cubed sphere complications is fine) in the
>>>>> pickups for posterity?
>>>>>   -Baylor
>>>>> On Mar 1, 2006, at 5:49 PM, Paola Cessi wrote:
>>>>>> Is interpickups.m necessary if the number of gridpoints is the
>>>>>> same, and thus no interpolation is needed, but the tiling  
>>>>>> changes?
>>>>>> paola
>>>>>> On Wed, 1 Mar 2006, Baylor Fox-Kemper wrote:
>>>>>>> Hi Paola,
>>>>>>> There is a little program: utils/matlab/interpickups.m, that I
>>>>>>> wrote a while back (and I think works with the present  
>>>>>>> version of
>>>>>>> the code).  It takes a bunch of mnc input pickup files with one
>>>>>>> tiling and a bunch of mnc output pickup files with a different
>>>>>>> tiling, and interpolates between them in a simple way.  This way
>>>>>>> you can coarsen or fine-grain a set of pickup files to restart
>>>>>>> with a different tiling or even a different resolution.  If the
>>>>>>> number of gridpoints in the input and outputs is the same, it
>>>>>>> shouldn't change the data at all, it should just copy it
>>>>>>> appropriately from the input files to the outputs.
>>>>>>> The easy way to get the output pickups is to set up the model
>>>>>>> with the new gridding and run for one timestep outputting a
>>>>>>> pickup.  The routine will then overwrite the data in the files
>>>>>>> with the interpolated data from the set of input files.
>>>>>>> Cheers,
>>>>>>>   -Baylor
>>>>>>> On Mar 1, 2006, at 2:47 PM, Paola Cessi wrote:
>>>>>>>> Is it possible to restart a computation with 4*N tiles,  
>>>>>>>> using the
>>>>>>>> pickup files generated with N tiles, while keeping Nx and Ny
>>>>>>>> fixed? In other words, is it possible to change nPx and nPy  
>>>>>>>> in a
>>>>>>>> restart run, while keeping Nx and Ny the same? (without using
>>>>>>>> netcdf)
>>>>>>>> paola
>>>>>>>> --------------------------------------------------------------- 
>>>>>>>> -----
>>>>>>>> -----
>>>>>>>> Paola Cessi                            Tel: +1 858 534 0622
>>>>>>>> Scripps Institution of Oceanography    Fax: +1 858 534 8045
>>>>>>>> 9500 Gilman Drive
>>>>>>>> UCSD-0213                              e-mail: pcessi at ucsd.edu
>>>>>>>> La Jolla, CA 92093-0213
>>>>>>>> USA                      Web:  http://www-pord.ucsd.edu/~pcessi
>>>>>>>> _______________________________________________
>>>>>>>> MITgcm-support mailing list
>>>>>>>> MITgcm-support at mitgcm.org
>>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>>>> _______________________________________________
>>>>>>> MITgcm-support mailing list
>>>>>>> MITgcm-support at mitgcm.org
>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>>> _______________________________________________
>>>>>> MITgcm-support mailing list
>>>>>> MITgcm-support at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>>> _______________________________________________
>>>>> MITgcm-support mailing list
>>>>> MITgcm-support at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>>> _______________________________________________
>>>> MITgcm-support mailing list
>>>> MITgcm-support at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>> _______________________________________________
>>> MITgcm-support mailing list
>>> MITgcm-support at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-support
>>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support

More information about the MITgcm-support mailing list