[MITgcm-devel] Fwd: job crashing with message forrtl: Text file busy

[An T Nguyen]

hi Matt, 

the first crash was in the middle of the run, error reading the runoff climatology.  Then over the following 16 hours, I resubmitted 8 times and they all crash immediately during model initialization with problems reading the grid files from links.  Next, I manually copied those grid files into run-time dir, the model read grid info ok, then crashed again due to error reading in a linked 3-d diffusivity file.   So the last test I did was copying all binary input files into runtime dir.  Then the model ran without a problem... (still running, fingers crossed it's not crashing...)

But yes, let's chat later today.  I still think my problem is not quite the same as yours.  I was surprised that the NAS people also didn't know why things were crashing, so they kept suggesting to me to just resubmit the job even though it seems to be very systematic.
An


On Jul 30, 2013, at 12:19 PM, Matthew Mazloff wrote:

> Hi An,
> 
> I've had this problem many times. It is an issue with overwhelming the NFS. Its not the link but the fact that you are opening and closing so many files at once. There are many possible fixes so it depends on your specific problem. Is it only during model initialization when this is happening?
> 
> Perhaps we can chat today about it?
> 
> Matt
> 
> 
> 
> On Jul 28, 2013, at 8:34 PM, An T Nguyen <antnguyen13 at gmail.com> wrote:
> 
>> Hello mitgcm-devel gurus,
>> 
>> Just thought i put this here.  Since yesterday, i kept having the mitgcm crashed on pleiades system with a series of error message:
>> 
>> forrtl: Text file busy
>> forrtl: severe: open failure, unit 9, file filename <-- where filename can be anything from grid, bathymetry, runoff, obcs, etc. which are LINKED from another directory.
>> 
>> I've ran ok for the last few years linking files instead of copying into run-time dir without issue until yesterday.  After 2 days talking to NAS, it seems they suggest that we fix on our end instead of them taking any action...
>> 
>> Just checking if anyone else has issue or just me.  I had a job running the last 2 days, then suddenly it crashed yesterday and wouldn't run again until i manually copy all input binary files into my runtime dir.  Atmospheric forcing seems to be the exception as I'm linking them and it's running without crashing.
>> 
>> An
>> 
>> Begin forwarded message:
>> 
>>> From: Johnny Chang <Johnny.Chang at nasa.gov>
>>> Date: July 28, 2013 10:39:03 PM EDT
>>> To: An T Nguyen <antnguyen13 at gmail.com>
>>> Subject: Re: INC000000053332 : APP : Low : job crashing with message forrtl: Text file busy
>>> 
>>> On 7/28/13 7:22 PM, An T Nguyen wrote:
>>>> i think it's related to files that are linked.
>>> 
>>> This makes more sense now.  When your program calls OPEN, it will
>>> open it for both read and write.  It's possible that you can't
>>> really write to those linked files, and hence the fortran runtime
>>> library (forrtl) complains "text file busy".  If you are only
>>> reading those linked files and not writing to them, then you
>>> can make the file read-only:
>>> 
>>> chmod u-wx some_file
>>> 
>>> will delete the (u)ser permission to (w)rite and e(x)ecute that some_file.
>>> 
>>> Alternately, you would need to modify the OPEN statement and put in
>>> action='read' to tell the fortran runtime library to OPEN the file with
>>> 'read' only.
>>> 
>>> Johnny
>>> 
>>> -- 
>>> Johnny Chang
>>> 650-604-4356
>> 
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> 
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel

--------
An T Nguyen, Ph.D. <atnguyen at mit.edu>
MIT Bldg 54-1410
77 Massachusetts Ave Cambridge MA 02139 USA
Phone: 617 253 2443; Fax: 617 253 4464




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-devel/attachments/20130730/85d6fceb/attachment.htm>