<div dir="ltr"><div>Thanks Jean-Michel, you are right, I am using the version of c65x. I should catch up and get updated. =. =<br></div><div><br></div><div>Cheers,</div><div>Daquan</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 10, 2020 at 11:46 PM Jean-Michel Campin <<a href="mailto:jmc@mit.edu">jmc@mit.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Daquan,<br>
<br>
>From the code that you listed below, it seems that you are using an older<br>
version (older than Aug 10, 2017) of MITgcm. <br>
Might be useful to use a more recent version to run on large number of procs.<br>
<br>
Cheers,<br>
Jean-Michel<br>
<br>
On Tue, Mar 10, 2020 at 10:47:04PM +0300, Daquan Guo wrote:<br>
> Thanks very much Martin and Jean-Michel for your suggestions,<br>
> <br>
> To update, by simply changing it from I4.4 to I5.5 for the writing of files<br>
> STDERR*, STDOUT* and scratch* in few files related (I listed below), the<br>
> model managed to run. I have not tried the #define SINGLE_DISK_IO (in<br>
> CPP_EEOPTIONS.h), but it looks like a smarter solution, I will give it a<br>
> try.<br>
> <br>
> eeboot_minimal.F: WRITE(myProcessStr,'(I5.5)') myProcId<br>
> eeboot_minimal.F: WRITE(fNam,'(A,A)') 'STDERR.', myProcessStr(1:5)<br>
> eeboot_minimal.F: WRITE(fNam,'(A,A)') 'STDOUT.', myProcessStr(1:5)<br>
> <br>
> eeset_parms.F: WRITE(scratchFile1,'(A,I5.5)') 'scratch1.', myProcId<br>
> eeset_parms.F: WRITE(scratchFile2,'(A,I5.5)') 'scratch2.', myProcId<br>
> <br>
> open_copy_data_file.F: WRITE(scratchFile1,'(A,I5.5)') 'scratch1.',<br>
> myProcId<br>
> open_copy_data_file.F: WRITE(scratchFile2,'(A,I5.5)') 'scratch2.',<br>
> myProcId<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> _____________________________<br>
> Daquan Guo<br>
> Post-doctoral Fellow<br>
> Physical Sciences and Engineering<br>
> King Abdullah University of Science and Technology (KAUST)<br>
> Bldg 1, Lv 4, 4700 KAUST, Thuwal 23955-6900, Jeddah, Saudi Arabia<br>
> Mobile: +966 541048507<br>
> <br>
> <br>
> On Tue, Mar 10, 2020 at 10:26 PM Jean-Michel Campin <<a href="mailto:jmc@mit.edu" target="_blank">jmc@mit.edu</a>> wrote:<br>
> <br>
> > Hi Daquan,<br>
> ><br>
> > Regarding STDOUT & STDERR files, you are right, this need to be fixed.<br>
> > Until now, the only time MITgcm has been run using more than 10000 procs<br>
> > was with #define SINGLE_DISK_IO (in CPP_EEOPTIONS.h).<br>
> > You might want to give it a try ?<br>
> ><br>
> > But regarding "scratch" files, the ones that are used to copy any parameter<br>
> > file (eedata, data and all data.* ) should have nine digits (FMT_PROC_ID =<br>
> > 'I9.9')<br>
> > for the proc number so it should be OK.<br>
> > May be our scratch file problem is coming from a different place ?<br>
> ><br>
> > Cheers,<br>
> > Jean-Michel<br>
> ><br>
> > On Tue, Mar 10, 2020 at 04:59:52PM +0100, Martin Losch wrote:<br>
> > > Hi Daquan,<br>
> > ><br>
> > > I have no experience with so many processors. I assume that there will<br>
> > be plenty of problems with order 1e4 files open (depending on your file<br>
> > system), but you can fix the ???*****??? problem by changing the<br>
> > definitions in eesupp/src/eeboot_minimal.F:<br>
> > > Look for ???USE_PDAF??? to see how the names of STDERR and STDOUT are<br>
> > changed to have longer numbers and do something similar for the default<br>
> > case.<br>
> > ><br>
> > > Alternatively you can define SINGLE_DISK_IO, but then only process 0<br>
> > (0000) will write a STDOUT/STDERR pair.<br>
> > ><br>
> > > Martin<br>
> > ><br>
> > > > On 10. Mar 2020, at 12:52, Daquan Guo <<a href="mailto:Daquan.Guo@kaust.edu.sa" target="_blank">Daquan.Guo@kaust.edu.sa</a>> wrote:<br>
> > > ><br>
> > > > Dear mitgcm community and developers,<br>
> > > ><br>
> > > > I am running a case with 16650 cpus and facing a problem.<br>
> > > > It seems the files scratch.*, STDERR.* and STDOUT.* can not be written<br>
> > well if the number exceeds 9999, instead it generates one file named<br>
> > 'scratch.****', which can not be read and processed then the model crashed.<br>
> > > > I am wondering if anyone has experience on this and knows how to fix<br>
> > it?<br>
> > > > Thanks in advance.<br>
> > > ><br>
> > > > Best,<br>
> > > > Daquan<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > This message and its contents, including attachments are intended<br>
> > solely for the original recipient. If you are not the intended recipient or<br>
> > have received this message in error, please notify me immediately and<br>
> > delete this message from your computer system. Any unauthorized use or<br>
> > distribution is prohibited. Please consider the environment before printing<br>
> > this email._______________________________________________<br>
> > > > MITgcm-support mailing list<br>
> > > > <a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
> > > > <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> > ><br>
> > > _______________________________________________<br>
> > > MITgcm-support mailing list<br>
> > > <a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
> > > <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> > _______________________________________________<br>
> > MITgcm-support mailing list<br>
> > <a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
> > <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> ><br>
> <br>
> -- <br>
> <br>
> This message and its contents, including attachments are intended solely <br>
> for the original recipient. If you are not the intended recipient or have <br>
> received this message in error, please notify me immediately and delete <br>
> this message from your computer system. Any unauthorized use or <br>
> distribution is prohibited. Please consider the environment before printing <br>
> this email.<br>
<br>
> _______________________________________________<br>
> MITgcm-support mailing list<br>
> <a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
> <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
<br>
_______________________________________________<br>
MITgcm-support mailing list<br>
<a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
<a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
</blockquote></div>
<br>
<div><hr></div><font face="Arial" size="1">This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.</font>