[MITgcm-support] Multiple outputfiles

[Stefano Querin]

Hi Ada,

just a wild guess...
Some months ago we had similar (VERY weird) problems running MITgcm on our cluster (with binary output). Every now and then the model was dumping output files at timesteps not prescribed by our namelists (i.e., dumptime-deltaT, dumptime, dumptime+deltaT). Those files (dumptime+/-deltaT) contained instant snapshots rather than the prescribed time averages. Maybe that's the reason for the instabilities you notice on your output: model velocity fields are characterized by instabilities, which you simply don't see in the averaged output...
In our case, the model was also running rather slowly.
Our system administrator found out that the problem was hardware/software (on the cluster) and it was not related to the MITgcm code.
In a few words, there were problems (instabilities) with the power supply of some nodes, which heavily interacted with the kernel module kipmi0.
As far as I understood, kipmi0 absorbed all the resources of some cores per node, slowing down the execution and creating that strange output...
You could check this issue with your system administrator or just try to run the code on other platforms to exclude this problem.

Hope this helps somehow...

Cheers!

SQ


On 16 Nov 2015, at 10:58:16, Ada Gjermundsen wrote:

> Hi all,
> I have a question regarding the output files from MITgcm. I'm using the mnc package to get netcdf output. Everything works out fine, but every now and then I get multiple output files (or 3 to be exact) for every output timestep (see example for the pickupfiles at the end of this email). Two of the files have numerical instabilties in the w field, while every third output file seems to be fine. Can someone explain why this is happening and if I can do something to prevent this?
> 
> Thanks in advance!
> 
> Ada 
> 
> ls -lrt build/pickup.*.001.001.data
> 
> pickup.0252287993.001.001.data
> pickup.0255441593.001.001.data
> pickup.0258595193.001.001.data
> pickup.0261748793.001.001.data
> pickup.0264902392.001.001.data
> pickup.0264902394.001.001.data
> pickup.0264902393.001.001.data
> pickup.0268055994.001.001.data
> pickup.0268055993.001.001.data
> pickup.0268055992.001.001.data
> pickup.0271209594.001.001.data
> pickup.0271209593.001.001.data
> pickup.0271209592.001.001.data
> pickup.0274363192.001.001.data
> pickup.0274363194.001.001.data
> pickup.0274363193.001.001.data
> 
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support