<div dir="ltr"><div>Thanks very much Martin and Jean-Michel for your suggestions,</div><div> <br></div><div>To update, by simply changing it from I4.4 to I5.5 for the writing of files STDERR*, STDOUT* and scratch* in few files related (I listed below), the model managed to run. I have not tried the #define SINGLE_DISK_IO (in CPP_EEOPTIONS.h), but it looks like a smarter solution, I will give it a try. <br></div><div><br></div><div>eeboot_minimal.F: WRITE(myProcessStr,'(I5.5)') myProcId<br>eeboot_minimal.F: WRITE(fNam,'(A,A)') 'STDERR.', myProcessStr(1:5)<br>eeboot_minimal.F: WRITE(fNam,'(A,A)') 'STDOUT.', myProcessStr(1:5)</div><div><br></div><div>eeset_parms.F: WRITE(scratchFile1,'(A,I5.5)') 'scratch1.', myProcId</div><div>eeset_parms.F: WRITE(scratchFile2,'(A,I5.5)') 'scratch2.', myProcId<br></div><div><br></div><div>open_copy_data_file.F: WRITE(scratchFile1,'(A,I5.5)') 'scratch1.', myProcId</div><div>open_copy_data_file.F: WRITE(scratchFile2,'(A,I5.5)') 'scratch2.', myProcId</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><font color="#888888"><div dir="ltr"><font face="trebuchet ms,sans-serif"><div><br>_____________________________</div><div>Daquan Guo<br></div><div dir="ltr">Post-doctoral Fellow</div><div>Physical Sciences and Engineering<br></div><div dir="ltr">King Abdullah University of Science and Technology (KAUST)<br>Bldg 1, Lv 4, 4700 KAUST, Thuwal 23955-6900, Jeddah, Saudi Arabia</div>
</font><div dir="ltr"><font face="trebuchet ms,sans-serif">Mobile: <a value="+966544701621">+966 541048507</a><b><br></b></font></div></div></font></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 10, 2020 at 10:26 PM Jean-Michel Campin <<a href="mailto:jmc@mit.edu">jmc@mit.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Daquan,<br>
<br>
Regarding STDOUT & STDERR files, you are right, this need to be fixed.<br>
Until now, the only time MITgcm has been run using more than 10000 procs<br>
was with #define SINGLE_DISK_IO (in CPP_EEOPTIONS.h).<br>
You might want to give it a try ?<br>
<br>
But regarding "scratch" files, the ones that are used to copy any parameter<br>
file (eedata, data and all data.* ) should have nine digits (FMT_PROC_ID = 'I9.9')<br>
for the proc number so it should be OK.<br>
May be our scratch file problem is coming from a different place ?<br>
<br>
Cheers,<br>
Jean-Michel<br>
<br>
On Tue, Mar 10, 2020 at 04:59:52PM +0100, Martin Losch wrote:<br>
> Hi Daquan,<br>
> <br>
> I have no experience with so many processors. I assume that there will be plenty of problems with order 1e4 files open (depending on your file system), but you can fix the ???*****??? problem by changing the definitions in eesupp/src/eeboot_minimal.F:<br>
> Look for ???USE_PDAF??? to see how the names of STDERR and STDOUT are changed to have longer numbers and do something similar for the default case.<br>
> <br>
> Alternatively you can define SINGLE_DISK_IO, but then only process 0 (0000) will write a STDOUT/STDERR pair.<br>
> <br>
> Martin<br>
> <br>
> > On 10. Mar 2020, at 12:52, Daquan Guo <<a href="mailto:Daquan.Guo@kaust.edu.sa" target="_blank">Daquan.Guo@kaust.edu.sa</a>> wrote:<br>
> > <br>
> > Dear mitgcm community and developers,<br>
> > <br>
> > I am running a case with 16650 cpus and facing a problem. <br>
> > It seems the files scratch.*, STDERR.* and STDOUT.* can not be written well if the number exceeds 9999, instead it generates one file named 'scratch.****', which can not be read and processed then the model crashed. <br>
> > I am wondering if anyone has experience on this and knows how to fix it?<br>
> > Thanks in advance.<br>
> > <br>
> > Best,<br>
> > Daquan<br>
> > <br>
> > <br>
> > <br>
> > <br>
> > <br>
> > <br>
> > This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email._______________________________________________<br>
> > MITgcm-support mailing list<br>
> > <a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
> > <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
> <br>
> _______________________________________________<br>
> MITgcm-support mailing list<br>
> <a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
> <a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
_______________________________________________<br>
MITgcm-support mailing list<br>
<a href="mailto:MITgcm-support@mitgcm.org" target="_blank">MITgcm-support@mitgcm.org</a><br>
<a href="http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support" rel="noreferrer" target="_blank">http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support</a><br>
</blockquote></div>
<br>
<div><hr></div><font face="Arial" size="1">This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.</font>