[MITgcm-devel] restart check

[Jean-Michel Campin]

Hi Matt,

I am answering to the list since this could be useful for others.

This is about the way to check the restart.

1) You could take a look at how it's done automatically (part of the daily
 tests) in the few testing scripts there: MITgcm/tools/example_scripts/
 such as this one: MITgcm/tools/example_scripts/csail/test_baudelaire
 The main script that is used to test the restart is "MITgcm/tools/do_tst_2+2"
 which is run just after testreport.
 However:
 a) In those script, it might not be very obvious what are the pieces
   relevant to the restart.
 b) and if restart test fails, the main script "do_tst_2+2" does not
   offer much help in finding where the problem is.

2) You could run the restart test for a single experiment, using
 MITgcm/tools/tst_2+2  (note that this script is used within "do_tst_2+2");
 The steps are:
  a) run testreport from "verification" dir., e.g.,
    > ./testreport -of ../tools/build_options/linux_amd64_gfortran -devel -t global_oce_biogeo_bling
     (assuming this additional experiment was downloaded in verification)
     this provides the first set of output ("primary" output) from which
     the restart test works (e.g., list of pickup files from end of the
     run, iter=endIter)
  b) from the individual experiment run dir:
    >  cd global_oce_biogeo_bling/run
    >  ../../../tools/tst_2+2
     A short "usage summary" is printed when "tst_2+2" is used without
     any argument.
  c) one step after the other:
    >  ../../../tools/tst_2+2 0
     this moves output from previous run and prepare input files
    >  ../../../tools/tst_2+2 1
     this run the 3 short tests, starting with the double length one:
       "2it" : endIter     --> endIter + 4
     and followed by the 2 consecutive short ones:
       "1iA" : endIter     --> endIter + 2
       "1iB" : endIter + 2 --> endIter + 4
    >  ../../../tools/tst_2+2 3
      this compares (diff) final pickup-files from "2it" run and "1iB" run.
    Since all output files from the 3 runs are saved (dir: res_2it, res_1iA
    and res_1iB) it's possible to check (matlab/python) where and when
    differences show-up.
  d) clean-up all "tst_2+2" output and move back results from the primary run.
    >  ../../../tools/tst_2+2 4
    this allows to repeat the "restart" test without the need to re-run
    testreport.

Let me know if something is not clear.

Cheers,
Jean-Michel