[MITgcm-support] mpi run with cpu more than 9999

Wed Mar 11 02:19:27 EDT 2020

Thanks Jean-Michel, you are right, I am using the version of c65x. I should
catch up and get updated. =. =

Cheers,
Daquan

On Tue, Mar 10, 2020 at 11:46 PM Jean-Michel Campin <jmc at mit.edu> wrote:

> Hi Daquan,
>
> From the code that you listed below, it seems that you are using an older
> version (older than Aug 10, 2017) of MITgcm.
> Might be useful to use a more recent version to run on large number of
> procs.
>
> Cheers,
> Jean-Michel
>
> On Tue, Mar 10, 2020 at 10:47:04PM +0300, Daquan Guo wrote:
> > Thanks very much Martin and Jean-Michel for your suggestions,
> >
> > To update, by simply changing it from I4.4 to I5.5 for the writing of
> files
> > STDERR*, STDOUT* and scratch* in few files related (I listed below), the
> > model managed to run. I have not tried the #define SINGLE_DISK_IO (in
> > CPP_EEOPTIONS.h), but it looks like a smarter solution, I will give it a
> > try.
> >
> > eeboot_minimal.F:       WRITE(myProcessStr,'(I5.5)') myProcId
> > eeboot_minimal.F:         WRITE(fNam,'(A,A)') 'STDERR.',
> myProcessStr(1:5)
> > eeboot_minimal.F:         WRITE(fNam,'(A,A)') 'STDOUT.',
> myProcessStr(1:5)
> >
> > eeset_parms.F:      WRITE(scratchFile1,'(A,I5.5)') 'scratch1.', myProcId
> > eeset_parms.F:      WRITE(scratchFile2,'(A,I5.5)') 'scratch2.', myProcId
> >
> > open_copy_data_file.F:      WRITE(scratchFile1,'(A,I5.5)') 'scratch1.',
> > myProcId
> > open_copy_data_file.F:      WRITE(scratchFile2,'(A,I5.5)') 'scratch2.',
> > myProcId
> >
> >
> >
> >
> >
> >
> > _____________________________
> > Daquan Guo
> > Post-doctoral Fellow
> > Physical Sciences and Engineering
> > King Abdullah University of Science and Technology (KAUST)
> > Bldg 1, Lv 4, 4700 KAUST, Thuwal 23955-6900, Jeddah, Saudi Arabia
> > Mobile: +966 541048507
> >
> >
> > On Tue, Mar 10, 2020 at 10:26 PM Jean-Michel Campin <jmc at mit.edu> wrote:
> >
> > > Hi Daquan,
> > >
> > > Regarding STDOUT & STDERR files, you are right, this need to be fixed.
> > > Until now, the only time MITgcm has been run using more than 10000
> procs
> > > was with  #define SINGLE_DISK_IO (in CPP_EEOPTIONS.h).
> > > You might want to give it a try ?
> > >
> > > But regarding "scratch" files, the ones that are used to copy any
> parameter
> > > file (eedata, data and all data.* ) should have nine digits
> (FMT_PROC_ID =
> > > 'I9.9')
> > > for the proc number so it should be OK.
> > > May be our scratch file problem is coming from a different place ?
> > >
> > > Cheers,
> > > Jean-Michel
> > >
> > > On Tue, Mar 10, 2020 at 04:59:52PM +0100, Martin Losch wrote:
> > > > Hi Daquan,
> > > >
> > > > I have no experience with so many processors. I assume that there
> will
> > > be plenty of problems with order 1e4 files open (depending on your file
> > > system), but you can fix the ???*****??? problem by changing the
> > > definitions in eesupp/src/eeboot_minimal.F:
> > > > Look for ???USE_PDAF??? to see how the names of STDERR and STDOUT are
> > > changed to have longer numbers and do something similar for the default
> > > case.
> > > >
> > > > Alternatively you can define SINGLE_DISK_IO, but then only process 0
> > > (0000) will write a STDOUT/STDERR pair.
> > > >
> > > > Martin
> > > >
> > > > > On 10. Mar 2020, at 12:52, Daquan Guo <Daquan.Guo at kaust.edu.sa>
> wrote:
> > > > >
> > > > > Dear mitgcm community and developers,
> > > > >
> > > > > I am running a case with 16650 cpus and facing a problem.
> > > > > It seems the files scratch.*, STDERR.* and STDOUT.* can not be
> written
> > > well if the number exceeds 9999, instead it generates one file named
> > > 'scratch.****', which can not be read and processed then the model
> crashed.
> > > > > I am wondering if anyone has experience on this and knows how to
> fix
> > > it?
> > > > > Thanks in advance.
> > > > >
> > > > > Best,
> > > > > Daquan
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > This message and its contents, including attachments are intended
> > > solely for the original recipient. If you are not the intended
> recipient or
> > > have received this message in error, please notify me immediately and
> > > delete this message from your computer system. Any unauthorized use or
> > > distribution is prohibited. Please consider the environment before
> printing
> > > this email._______________________________________________
> > > > > MITgcm-support mailing list
> > > > > MITgcm-support at mitgcm.org
> > > > > http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
> > > >
> > > > _______________________________________________
> > > > MITgcm-support mailing list
> > > > MITgcm-support at mitgcm.org
> > > > http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
> > > _______________________________________________
> > > MITgcm-support mailing list
> > > MITgcm-support at mitgcm.org
> > > http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
> > >
> >
> > --
> >
> > This message and its contents, including attachments are intended solely
> > for the original recipient. If you are not the intended recipient or
> have
> > received this message in error, please notify me immediately and delete
> > this message from your computer system. Any unauthorized use or
> > distribution is prohibited. Please consider the environment before
> printing
> > this email.
>
> > _______________________________________________
> > MITgcm-support mailing list
> > MITgcm-support at mitgcm.org
> > http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
>

-- 

This message and its contents, including attachments are intended solely 
for the original recipient. If you are not the intended recipient or have 
received this message in error, please notify me immediately and delete 
this message from your computer system. Any unauthorized use or 
distribution is prohibited. Please consider the environment before printing 
this email.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20200311/5edb5499/attachment-0001.html>