[MITgcm-support] optim_m1qn3: maximum iterations reached?

Martin Losch Martin.Losch at awi.de
Thu Sep 24 05:07:50 EDT 2020


the code hasn’t changed much since the initial check-in, a few small bug fixes that shouldn’t affect you very much.
omode = 6 means that m1qn3 can non longer find an improved solution. This is usually the case, when the gradient (ad-model) is not accurate enough for the required precision (espg). For example, in the testbed, you can get omode 6 by requiring espg = 1e-16 instead of 1e-8 in driver.F or driver_offline.F

If optim_m1qn3 turns out to be correct, I am afraid, the most likely candidate for error remains your calling shell script, if you don’t have a proper termination condition. Let’s see …

Don’t forget the output of optim_m1qn3 as well.

Martin

> On 24. Sep 2020, at 10:59, Daniel Goldberg <dan.goldberg at ed.ac.uk> wrote:
> 
> Hi Martin
> 
> Thank you very much for your helpful response. Thank you as well for directing me to m1qn3_output.txt -- now i recall your directing me to this before, which makes the "trace" of the optimisation much easier to follow.
> 
> I am skeptical as to whether this could be reproduced with a simpler run. I am using optim_m1qn3 at the moment to carry out an inversion of surface properties to yield basal properties with the package STREAMICE. I have done two separate optimisations, both very similar but using slightly different resolutions (1 km vs 1500m); and the slightly coarser simulation seemed to terminate in a slightly better state (omode=6; not the omode=1 i would hope for, but with a cost function progression i would expect for this type of problem).
> 
> It is very possible I am continuing the optimisation past a point where i should stop it, as out of pure laziness and slight ignorance i don't have appropriate termination conditions in the calling shell script. 
> 
> It is also somewhat possible I have not pulled from git recently enough, or even that i have unwittingly changed a source file; and it has been so long since i cloned from git (copying the source for each experiment) i cannot even recall where the cloned repo is, if i still have it. So this is something I can check. I can also re-run the optimisation with the debug message as you suggest, but this will take some time as the experiment is run on a supercomputer with long queues during the day. 
> 
> Perhaps the best thing is for me to send you m1qn3_output.txt for both experiments (via direct email as they are large) as well as my calling script. After this i will try cloning from the source and rerunning the optimisation (which will be queued for some time) and see if it does the same.
> 
> Best
> Dan
> 
> On Thu, Sep 24, 2020 at 8:49 AM Martin Losch <Martin.Losch at awi.de> wrote:
> Hi Dan,
> 
> thanks for using this routine. Do you think that we can reproduce this somewhat odd behavior with a simple optimization (i.e. with a cheap costfunction like the “testbed” in optim_m1qn3)?
> 
> nsim = numter*nfunc should not change during the optimization, only at the (successful) end, it is overwritten somewhere in m1qn3a or so to store the actual number of simulations. So my only guess is that you (accidentally) restart the optimization (with modified parameters) after m1qn3 thinks it’s over?
> 
> Maybe it would be helpful to have look at the output of optim_m1qn3, but also of m1qn3 itself (if you didn’t change it: fname_m1qn3='output_m1qn3.txt’), but also at the calling seqence (script). Depending on the size, you can send it directly to me.
> 
> Martin
> 
> > On 24. Sep 2020, at 09:26, Daniel Goldberg <dan.goldberg at ed.ac.uk> wrote:
> > 
> > Hi Martin
> > 
> > I am using optim_m1qn3 (installed from your github repo).
> > 
> > I have been using it in optimisations with data.optim parameters as follows:
> > 
> >  &OPTIM
> >  optimcycle=0,
> >  numiter=1000,
> >  nfunc=10,
> >  dfminfrac=0.001,
> >  iprint=10,
> >  nupdate=5,
> >  /
> > 
> > and am seeing the optimisation terminate with omode=5 after about 100 iterations. The manual for m1qn3 suggests the maximum number of simulations has been reached, which is a lot less than numiter*nfunc, which (according to the git readme) is the max number of simulations. (Though i do note that "nsim" in the text output from the optim_m1qn3 executable changes at some point in the optimisation from its initial value of 10000.) 
> > 
> > I then ran the optimisation again with nfunc=20 to see what would happen -- this time there is a termination with omode=4, the maximum number of iterations being reached -- this happened at optimcycle=116. Here, i notice that each subsequent optimcycle gives omode=1 in the optim_m1qn3 output -- im not sure if this is significant. 
> > 
> > I was wondering if this behaviour makes sense to you, and if you would be able to explain it? Happy to provide more output from optim_m1qn3 (from the more recent optimisation with nfunc=20; the previous nfunc=10 output is deleted).
> > 
> > Many thanks
> > Dan
> > 
> > -- 
> > --- PLEASE NOTE THAT I AM CURRENTLY WORKING FROM HOME AS A MEASURE OF SOCIAL DISTANCING DURING THE COVID-19 PANDEMIC ---
> > 
> > Daniel Goldberg, PhD
> > Reader in Glaciology
> > School of Geosciences, University of Edinburgh
> > Geography Building, Drummond Street, Edinburgh EH8 9XP
> > 
> > 
> > em: dan.goldberg at ed.ac.uk
> > web: https://www.geos.ed.ac.uk/homes/dgoldber
> > The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
> > _______________________________________________
> > MITgcm-support mailing list
> > MITgcm-support at mitgcm.org
> > http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
> 
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
> 
> 
> -- 
> --- PLEASE NOTE THAT I AM CURRENTLY WORKING FROM HOME AS A MEASURE OF SOCIAL DISTANCING DURING THE COVID-19 PANDEMIC ---
> 
> Daniel Goldberg, PhD
> Reader in Glaciology
> School of Geosciences, University of Edinburgh
> Geography Building, Drummond Street, Edinburgh EH8 9XP
> 
> 
> em: dan.goldberg at ed.ac.uk
> web: https://www.geos.ed.ac.uk/homes/dgoldber
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support



More information about the MITgcm-support mailing list