[Mitgcm-support] Re: joinmds on Alpha

mitgcm-support at dev.mitgcm.org mitgcm-support at dev.mitgcm.org
Wed Jul 9 15:57:30 EDT 2003


Arne,

We run joinmds on Alpha's here. Are you trying merge the Cray files on the Alpha
or Alpha files on the Alpha? The big/little endian thing hasn't been a problem
before.
It sounds more like either a perl problem or a shell memory problem.

Try "unlimit" to remove all stacksizes etc....

What version of perl are you using?

perl -v

Try using the -d option (debugger) to see where it dies. 

I'll copy this to  Zhangfan (xing at pacific.jpl.nasa.gov) who wrote the script.

Alistair.



Arne Biastoch wrote:
> 
> Hi Alistair,
> 
> greetings from Germany. After almost two month to be back in Kiel I try
> to stand all MOM fetishists and to continue - at least from time to time
> - working with MITgcmUV.
> 
> My first runs on a Cray T3E were successful (though I had to modify some
> details in the MPI routines: MPI_REAL4 and MPI_REAL8 are not available
> and had to be changed to MI_REAL and MPI_DOUBLEPRECISION).
> 
> But I ran into a problem merging the local files to global ones on a DEC
> Alpha. Using the joinmds (actually an older one called joinmdsr) routine
> gave the following:
> 
>  > joinmdsr T 0000000144
> There are 25 datasets.
> All existing meta files are self- and mutually consistent.
> Datasets are complete.
> Current machine's endianness: little-endian
> T.0000000144.001.001.data: 453600 bytes, okay, merged from .
> T.0000000144.001.002.data: 453600 bytes, okay, merged from .
> Segmentation fault (core dumped)
> 
> I think the problem might be the mixing of big_endians (local files) and
>   little_endian (Alpha). Do you have idea how to modify the script to
> work on an Alpha? Or do you know of anyone using it on such architecture?
> 
> Please note that merging the files on a SUN works properly. So, it's not
> the files, it's the machine.
> 
> Ciao,
>         Arne
> 
> --
> 
> Dr. Arne Biastoch
> 
> Institute for Marine Research         phone: ++49 431 600-4013
> FB1 Ocean Circulation and Climate     fax  : ++49 431 600-1515
> Duesternbrooker Weg 20                email: abiastoch at ifm.uni-kiel.de
> 24105 Kiel, Germany
> 
> http://www.ifm.uni-kiel.de/to/data/pers/abiastoch.html
> 
>   --------------------------------------------------------------------------------
> #!/usr/local/bin/perl -w
> 
> # MITgcmUV DataSet joining utility.
> # Tested with perl 4.0 and newer.
> # Tested on Linux 2.0.27/I486, Irix 6.2/{IP22,IP25}
> # Zhangfan XING, xing at pacific.jpl.nasa.gov
> # Adapted to work with MDS I/O format by adcroft at mit.edu, 5/7/1999
> #
> # LOGS:
> # 980707, version 0.0.1, basically works
> # 980721, version 0.2.0, proper handling of data file's header and terminator
> #         for diff bytesex.
> # 990507, HACK'd by AJA. Needs to be properly merged with the original joinds
> 
> #------
> # usage
> #------
> sub usage {
>         print STDERR
>                 "\nUsage:$0 [-Ddir0 -Ddir1 ...] " .
>                 "prefix suffix [(little-endian|big-endian)]\n";
>         print STDERR "\nMITgcmUV DataSet joining utility, version 0.3.0\n";
>         print STDERR
>         "Check http://escher.jpl.nasa.gov:2000/tools/ for newer version.\n";
>         print STDERR "Report problem to xing\@pacific.jpl.nasa.gov\n\n";
>         exit 1;
> }
> 
> #------------------------------
> # product of a list of integers
> #------------------------------
> sub listprod {
>         local ($product) = 1;
>         local ($x);
>         foreach $x (@_) {
>                 $product *= $x;
>         }
>         $product;
> }
> 
> #----------------
> # @list1 + @list2
> #----------------
> sub lists_add {
>         local (*l1,*l2) = @_;
>         ($#l1 == $#l2) || return undef;
> 
>         local (@l);
>         for (local($i)=0;$i<=$#l1;$i++) {
>                 $l[$i]=$l1[$i]+$l2[$i];
>         }
>         @l;
> }
> 
> #-------------
> # pos to index
> # 0-based.
> #-------------
> sub pos2index {
> 
>         local ($pos, at dim) = @_;
>         local ($rightmost) = pop(@dim);
> 
>         local (@index,$d);
>         foreach $d (@dim) {
>                 push(@index,$pos%$d);
>                 $pos = int($pos/$d);
>         }
> 
>         # self-guarding
>         unless ($rightmost > $pos) {
>                 return undef;
>         }
> 
>         push(@index,$pos);
>         @index;
> }
> 
> #-------------
> # index to pos
> # 0-based.
> #-------------
> sub index2pos {
>         local (*index,*dim) = @_;
> 
>         return undef unless ($#index == $#dim);
> 
>         local ($pos) = $index[$#index];
>         for (local($i)=$#dim;$i>0;$i--) {
>                 $pos = $pos * $dim[$i-1] + $index[$i-1];
>         }
>         $pos;
> }
> 
> #-------------------------
> # check machine's bytesex.
> # returns "little-endian" or "big-endian"
> # or dies if unable to figure out
> #-------------------------
> sub mach_bytesex {
> 
>         local ($foo)  = pack("s2",1,2);
>         if ($foo eq "\1\0\2\0") {
>                 return "little-endian";
>         } elsif ($foo eq "\0\1\0\2") {
>                 return "big-endian";
>         } else {
>                 die "Your machine has a strange bytesex.\n".
>                 "Email your platform info to xing\@pacific.jpl.nasa.gov\n";
>         }
> }
> 
> #--------------------------------------------------
> # check bytesex of a fortran unformatted data file
> # current machine's bytesex is used as a reference.
> # returns: one of "little-endian", "big-endian", "undecidable" and "unknown"
> #--------------------------------------------------
> sub file_bytesex {
> 
>         # only if this platform's bytesex is either big- or little-endian
>         # otherwise dies. Hope this won't happen.
>         local($mach_bytesex) = &mach_bytesex();
> 
>         local ($file) = shift;
>         local (*FILE);
> 
>         open(FILE,$file) || die "$file: $!\n";
> 
>         local(@fstat) = stat(FILE);
>         local ($size) = $fstat[7] - 8;  # total data size in bytes
> 
>         local($hdr,$tmr) = ("","");
>         read(FILE,$hdr,4);
>         seek(FILE,-4,2);
>         read(FILE,$tmr,4);
>         close(FILE);
> 
>         # this part checks for self-consistency of Fortran unformatted file
>         ($hdr eq $tmr) || die "$file: not a Fortran unformatted data file.\n";
> 
>         local ($ori) = unpack("I",$hdr);
>         local ($rev) = unpack("I",join("",reverse(split(//,$hdr))));
> 
>         ($ori != $size && $rev != $size) &&
>                 return "unknown";
> 
>         ($ori == $size && $rev == $size) &&
>                 return "undecidable";
> 
>         local ($opposite) = ($mach_bytesex eq "little-endian") ?
>                                 "big-endian" : "little-endian";
> 
>         return ($ori == $size) ? $mach_bytesex : $opposite;
> 
> }
> 
> #--------------------------------
> # check meta info for one dataset
> #--------------------------------
> 
> sub check_meta {
> 
>         local ($ds,$dir) = @_;
>         local ($fmeta) = "$dir/$ds.meta";
> 
>         #~~~~~~~~~~~~~~~~
>         # check meta info
>         #~~~~~~~~~~~~~~~~
> 
>         undef $/;               # read to the end of file
>         open(MFILE,"<$fmeta") || die "$fmeta: $!\n";
>         $_=<MFILE>;
>         close(MFILE);
>         $/ = "\n";              # never mess up
> 
>         s/\([^)]*\)//g;         #rm (.*)
>         s/\/\/[^\n]*\n//g;      #rm comment lines
>         s/\/\*.*\*\///g;        #rm inline comments
>         s/\s+//g;               #rm white spaces
>         /nDims=\[(.+)\];dimList=\[(.+)\];format=\['(.+)'\];nrecords=\[(.+)\];timeStepNumber=\[(.+)\];/
>                 || die "$fmeta: meta file format error\n";
>         local ($nDims_,$dimList_,$format_,$nrecords_,$timeStepNumber_) = ($1,$2,$3,$4,$5);
> 
>         # add dimensions corresponding to records
>         local ($mydims) = join(",",$dimList_,$nrecords_,1,$nrecords_);
>         local ($myndims) = $nDims_+1;
>         $dimList_ = $mydims;
>         $nDims_ = $myndims;
> 
>         # check Identifier
>         (defined $timeStepNumber) || ($timeStepNumber = $timeStepNumber_);
>         ($timeStepNumber eq $timeStepNumber_) ||
>                 die "$fmeta: timeStepNumber $timeStepNumber_ inconsistent with other dataset\n";
> 
>         # check Number of dimensions
>         (defined $nDims) || ($nDims = $nDims_);
>         ($nDims eq $nDims_) ||
>                 die "$fmeta: nDims $nDims_ inconsistent with other dataset\n";
> 
>         # check Field format
>         (defined $format) || ($format = $format_);
>         ($format eq $format_) ||
>                 die "$fmeta: format $format_ inconsistent with other dataset\n";
> 
>         # check dimList
>         # calc dimesions and leading index of this subset
>         local (@dimList_) = split(/,/,$dimList_);
> 
>         ($nDims_*3 == $#dimList_+1) ||
>                 die "$fmeta: nDims and dimList conflicting\n";
> 
>         local (@Dim, at dim, at Index0) = ();
>         for (local($i)=0;$i<$nDims_;$i++) {
>                 push(@Dim,$dimList_[$i*3]);
>                 push(@dim,$dimList_[$i*3+2]-$dimList_[$i*3+1]+1);
>                 push(@Index0,$dimList_[$i*3+1]-1);
>         }
>         local ($Dim_) = join(",", at Dim);
>         local ($dim_) = join(",", at dim);
> 
>         (defined $Dim) || ($Dim = $Dim_);
>         ($Dim eq $Dim_) ||
>                 die "$fmeta: dimList Global inconsistent with other dataset\n";
> 
>         (defined $dim) || ($dim = $dim_);
>         ($dim eq $dim_) ||
>                 die "$fmeta: dimList Local inconsistent with other dataset\n";
> 
>         $ds_Index0{$ds} = join(",", @Index0);
> 
> #       print STDOUT "Okay $fmeta\n";
> }
> 
> #-------------------------------
> # check completeness of datasets
> # need to be more sophisticated
> #-------------------------------
> sub check_entirety {
> 
>         local (*Dim,*dim,*ds_Index0) = @_;
> 
>         local ($N) = &listprod(@Dim);
>         local ($n) = &listprod(@dim);
>         ($N) || return 0;               # against null dimension
>         ($n) || return 0;               # against null dimension
>         ($N%$n) && return 0;            # $N/$n must be a whole number
> 
>         local (@ds) = keys %ds_Index0;
>         ($#ds+1 == $N/$n) || return 0;  # Num of datasets must match subdomain
> 
>         1;
> }
> 
> #------------------
> # merge one dataset
> # assume @Dim, @dim and $bytes existing
> # assume $Byte_Reorder existing
> #------------------
> sub merge_data {
> 
>         local ($ds,$dir,*Index0) = @_;
>         local ($fdata) = "$dir/$ds.data";
> 
>         # data size of one subset in bytes as told by meta info
>         local ($size) = &listprod(@dim) * $bytes;
> 
>         open(DFILE, "<$fdata") || die "$fdata: $!\n";
> 
>         local ($raw) = "";
> #aja    sysread(DFILE,$raw,4);
>         # Swap header if bytesex is diff from machine's
>         local ($hdr);
>         if ($Byte_Reorder) {
>                 $hdr = unpack("I",join("",reverse(split(//,$raw))));
>         } else {
>                 $hdr = unpack("I",$raw);
>         }
> 
> #aja    ($size == $hdr) ||
> #aja            die "$fdata: $hdr bytes inconsistent with meta info\n";
> 
>         print STDOUT "$ds.data: $size bytes, okay, ";
> 
> #       seek(DFILE,4,0);        # rewind back to the beginning of data
> 
>         local ($data) = "";             # old perl (< 4.0) needs this to
>         sysread(DFILE,$data,$size);     # avoid warning by sysread()
>         local ($len_chunk) = $dim[0] * $bytes;
>         local ($num_chunk) = $size/$len_chunk;
> 
>         local ($pos, at index,$Pos, at Index);
>         for (local($i)=0;$i<$num_chunk;$i++) {
>                 $pos = $i * $dim[0];
>                 @index = &pos2index($pos, at dim);
>                 @Index = &lists_add(*index,*Index0);
>                 $Pos = &index2pos(*Index,*Dim);
> #aja            seek(FILE,$Pos*$bytes+4,0);
>                 seek(FILE,$Pos*$bytes,0);
>                 syswrite(FILE,$data,$len_chunk,$pos*$bytes);
>         }
> 
>         close(DFILE);
> 
>         print STDOUT "merged from $dir\n";
> }
> 
> #============
> # main script
> #============
> 
> #------------
> # parse @ARGV
> #............
> 
> ($#ARGV >= 1) || &usage();
> 
> undef @dirs;
> while (1) {
>         $x = shift(@ARGV);
>         unless ($x =~ /^-D(.+)$/) {
>                 unshift(@ARGV,$x);
>                 last;
>         }
>         push(@dirs,$1);
> }
> (@dirs) || push(@dirs,".");
> # @dirs is not empty after this line.
> #print STDOUT join(" ", at dirs), "\n";
> 
> ($#ARGV >= 1) || &usage();
> 
> # data set prefix and suffix
> $pref = shift(@ARGV);
> $suff = shift(@ARGV);
> 
> ($#ARGV >= 1) && &usage();
> undef $forced_bytesex;
> if (@ARGV) {
>         $forced_bytesex = shift(@ARGV);
>         $forced_bytesex =~ /^(little|big)-endian$/ || &usage();
> }
> #print STDOUT $forced_bytesex, "\n";
> 
> #--------------------------
> # obtain a list of datasets
> #..........................
> 
> # %ds_dir is a hash to store the directory that a dataset is in.
> # After this step, it is assured that, for a dataset $ds,
> # both $ds.meta and $ds.data exist in a unique dir $ds_dir{$ds}.
> 
> %ds_dir = ();
> foreach $dir (@dirs) {
>         opendir(DIR, $dir) || die "$dir: $!\n";
>         @fmeta = grep(/^$pref\.$suff\.\d+\.\d+\.meta$/, readdir(DIR));
>         closedir(DIR);
>         foreach $fmeta (@fmeta) {
>                 $ds = $fmeta; $ds =~ s/\.meta$//g;
>                 (defined $ds_dir{$ds}) &&
>                         die "$fmeta appears in two dirs: $ds_dir{$ds} & $dir\n";
>                 (-f "$dir/$ds.data") || die "In $dir, $ds.data missing\n";
>                 $ds_dir{$ds} = $dir;
>         }
> }
> 
> @ds = sort(keys %ds_dir);        # list of datasets
> (@ds) || die "No dataset found.\n";
> print STDOUT "There are ", $#ds+1, " datasets.\n";
> 
> #---------------------------------
> # check meta info for all datasets
> #.................................
> 
> undef $timeStepNumber;
> undef $nDims;
> undef $format;
> 
> undef $Dim;
> undef $dim;
> undef %ds_Index0;
> 
> #..............................................
> # check each meta file and set some global vars
> 
> foreach $ds (@ds) {
>         &check_meta($ds,$ds_dir{$ds});
> }
> print STDOUT "All existing meta files are self- and mutually consistent.\n";
> 
> #print join(" ",$timeStepNumber,$nDims,$format,$Dim,$dim), "\n";
> #foreach $ds (@ds) {
> #       $dir = $ds_dir{$ds};
> #       $Index0 = $ds_Index0{$ds};
> #       print "$ds\n";
> #       print "$Index0\n";
> #}
> 
> @Dim = split(/,/,$Dim);
> @dim = split(/,/,$dim);
> 
> #................................
> # check meta info in its entirety
> 
> &check_entirety(*Dim,*dim,*ds_Index0) ||
>         die "Datasets are not complete!\n";
> 
> print STDOUT "Datasets are complete.\n";
> 
> #...........
> # set $bytes
> 
> if ($format eq "float32") {
>         $bytes = 4;
> } elsif ($format eq "float64") {
>         $bytes = 8
> } else {
>         die "format '$format' unknown\n";
> }
> 
> #---------------------------
> # check and merge data files
> #...........................
> 
> #........................
> # check machine's bytesex
> # it dies if neither little- nor big-endian.
> 
> $Mach_Bytesex = &mach_bytesex();
> print STDOUT "Current machine's endianness: $Mach_Bytesex\n";
> 
> #...................
> # check file bytesex and resolve related issues
> #aja undef $File_Bytesex;
> #aja foreach $ds (@ds) {
> #aja    $fdata = "$ds.data";
> #aja    $file_bytesex = &file_bytesex($ds_dir{$ds}."/$fdata");
> #aja    ($file_bytesex eq "unknown") &&
> #aja            die "$fdata: endianness is neither little- nor big-endian.\n";
> #aja    print STDOUT "$fdata: $file_bytesex\n";
> #aja    unless ($File_Bytesex) {
> #aja            $File_Bytesex = $file_bytesex;
> #aja    } else {
> #aja            ($File_Bytesex eq $file_bytesex) ||
> #aja            die "Data files are mutually inconsistent in endianness\n";
> #aja    }
> #aja }
> $File_Bytesex = 'big-endian';
> 
> #------------------
> # set $Byte_Reorder, which controls swapping of bytes in
> # header and terminator of Fortran unformatted data files.
> #aja $Byte_Reorder = 0;
> $Byte_Reorder = 0;
> 
> # if machine and data file have the same bytesex, no need for swapping
> #aja ($File_Bytesex eq $Mach_Bytesex) && ($Byte_Reorder = 0);
> 
> # if we can't determine bytesex of data file, need forced one from @ARGV.
> if ($File_Bytesex eq "undecidable") {
>         # if no forced bytesex available, dies.
>         ($forced_bytesex) ||
>                 die "Endianness of data files is undecidable, " .
>                 "you have to give one at command line.\n";
>         ($forced_bytesex eq $Mach_Bytesex) && ($Byte_Reorder = 0);
>         print STDOUT "Endianness of data files is undecidable.\n";
>         print STDOUT "Data file header/tail will be treated as ";
>         print STDOUT "$forced_bytesex as you have instructed.\n";
> # otherwise

> } else {
> # give a warining, if swapping is needed.
> ($Byte_Reorder) &&
>         print STDOUT
>         "Please note: data files have different bytesex than machine!\n";
> }
> 
> #................
> # merge data sets
> 
> $Size = &listprod(@Dim) * $bytes;
> 
> $fout = "$pref.$suff.data";
> 
> open(FILE, ">$fout") || die "$fout: $!\n";
> 
> # prepare header and teminator. Do byte reordering if necessary
> $HdrTmr = pack("I",$Size);
> ($Byte_Reorder) && ($HdrTmr = join("",reverse(split(//,$HdrTmr))));
> 
> # write 4 byte header
> #aja syswrite(FILE,$HdrTmr,4);
> 
> # merge each dataset
> foreach $ds (@ds) {
>         $dir = $ds_dir{$ds};
>         @Index0 = split(/,/,$ds_Index0{$ds});
>         &merge_data($ds,$dir,*Index0);
> }
> 
> # write 4 byte terminator
> #aja seek(FILE,$Size+4,0);
> #aja syswrite(FILE,$HdrTmr,4);
> 
> close(FILE);
> 
> print STDOUT "Global data (" .
>         join("x", at Dim) .
>         ") is in ./$fout (endianness is $File_Bytesex).\n";
> 
> exit 0;



More information about the MITgcm-support mailing list