[MITgcm-support] Model optimization when using many ptracers

[Martin Losch]

Hi Stefano,

this is a wild guess:

There are exchange of momentum-related fields in both dynamics (only when obcs is active) and solve for pressure (always). I am guessing that the increased MPI traffic with 51 ptracers affects these exchanges mysterious ways.
One way to test this is to turn of the obcs package (hoping that the model will not explode, maybe you’ll have to close the open boundaries for that or use a shorter integration for this test) and see if the dynamics routine will still use much more time with the ptracers turned on.

If the MPI traffic with ptracers turn out to be the problem, one way to reduce the load is to modify ptracers_fields_blocking_exch.F. This routine sends each tracer field individually to MPI and calls exch_3D_RL PTRACER_numInUse times (so 51 times). This once slowed down my runs dramatically and I copied ptracers to a field ptrLoc(:,:,51*Nr,bi,bj) and passed this field once to exch_3D_RL and copied the result back to ptracers. I think you then will also have to modify something in exch_3d_rl. I am attaching the code that I used (it’s very old so please use it as a guideline in your much later version of the MITgcm, and it only refers to pkg/exch2).

Martin

On Sep 5, 2014, at 12:06 PM, Stefano Querin <squerin at ogs.trieste.it> wrote:

> Dear MITgcm-ers,
> 
> we are developing a coupled physical/biogeochemical model using MITgcm (c64v) and BFM (ERSEM-like biogeochemical model: http://bfm-community.eu/).
> We would like to make this model available to the community as soon as we finalize the coupling (the coupler is called by the GCHEM package, almost in the same way as DIC or DARWIN do). In the present configuration, the biogeochemical reactor handles 51 variables (ptracers).
> 
> We are experiencing a (HPC) problem with ptracers: in a few words, we sum up the main results of our performance tests below.
> 
> - we are running on a 336 x 512 x 72 points grid, using (among the others) the EXF, OBCS and KPP packages (namelists attached: please note that KPP is active only for physics);
> - if we suppose that the ocean physics alone takes 100 minutes to run, when we activate the biogeochemistry package (with 51 ptracers), the simulation lasts for 1300 minutes (adopting the same timestep). We expected a sensible increase in the computational burden, but not so large!
> - then we also used the LONGSTEP package (with deltaT*6): we reduced the wall-clock time to 815 minutes;
> - as you can see in the table below, LONGSTEP performs efficiently on two among the most time-consuming routines (BLOCKING_EXCHANGES and THERMODYNAMICS). As we expected, the wall clock time for DO_OCEANIC_PHYS does not change. Our concern is about DYNAMICS and SOLVE_FOR_PRESSURE, which increase of one order of magnitude from "physics" to "coupled", and are not affected by LONGSTEP. As far as we know, these routines are not involved in any ptracers computation, aren't they? Why does their computational cost increase so much? Are we missing something?
> 
> Thank you for any guess/suggestions!
> 
> Best regards,
> 
> Stefano and Gianpiero
> 
> 
> ****************************************************************
> 
> Synthesis of the performance tests [minutes] (we included ONLY the most "time consuming" routines):
> 
> routine  physics coupled
> coupled (with longstep)
> 
> FORWARD_STEP 100
> 1300 815
> DO_STATEVARS_DIAGS  1 19
> 4
> DO_OCEANIC_PHYS  32 33
> 34
> DYNAMICS 25
> 503 508
> SOLVE_FOR_PRESSURE  6
> 99 99
> BLOCKING_EXCHANGES  3
> 112 9
> THERMODYNAMICS  24 472
> 24
> LONGSTEP_THERMODYNAMICS
> 0
> 0 71
> 
> 
> ****************************************************************
> 
> 
> 
> <data.obcs><data.exf><data><data.kpp><data.longstep><data.ptracers><data.diagnostics><data.pkg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ptr.tgz
Type: application/octet-stream
Size: 3172 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140912/6892b225/attachment.obj>