[MITgcm-support] Mysterious initialisation problem - debugging options?

[Liam Brannigan]

Dear MITgcmers

I was adding new diagnostics to the kpp package on my version of the model yesterday.  After making one small change and re-compiling, I have not been able to get the model to run.  This is causing me some heartache, as I'm getting very little information about why it won't run.  I'd appreciate on thoughts you had on further debugging options.

The scripts I modified were kpp_diagnostics_init.F and kpp_routines.F (which I've modified in a similar way a number of times before).  I have tried restoring these to how they were before the re-compile, but to no avail.  I've also tried replacing them from the website (taking care to get the right version number) and running with useKPP=.FALSE., in data.pkg to the same effect.

When I submit the job to the Hector supercomputer the mnc output folders are created and are populated by the grid, monitor, monitor_grid and state files at the initial time step.  This suggests that the model is initializing at least as far as WRITE_STATE and MONITOR (just about the last calls before the forward step begins) on the call tree http://mitgcm.org/sealion/code_reference/callTree.html.
I also tried to get the model to output a diagnostic (drhodr) which is calculated early in the time step, but nothing happened.

I ran the model with nIter=0, which ran without producing the exit code 136 in output.txt, suggesting that initialization is indeed working (the .e file stated 'NORMAL END' in this case).

I'm not sure how to proceed further, as the STDERR and STDOUT files are empty after the model runs and the job files simply give a message like: "_pmiu_daemon(SIGCHLD): PE RANK 8 exit signal Floating point exception".

Jean-Michel mentioned in a previous developer post (http://mitgcm.org/pipermail/mitgcm-devel/2011-May/004763.html) about accessing further debug info, but I'm not sure how this is switched on (and where the output can be accessed).  I tried adding debugLevel=4 to the data file in PARAM01 as suggested in the online manual, but nothing happened.

Apologies for the flood of info, thanks for any advice you can offer!

Liam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20130820/fc55f329/attachment.htm>