[MITgcm-support] Model optimization when using many ptracers

[Stefano Querin]

Dear MITgcm-ers,

we are developing a coupled physical/biogeochemical model using MITgcm  
(c64v) and BFM (ERSEM-like biogeochemical model: http://bfm-community.eu/) 
.
We would like to make this model available to the community as soon as  
we finalize the coupling (the coupler is called by the GCHEM package,  
almost in the same way as DIC or DARWIN do). In the present  
configuration, the biogeochemical reactor handles 51 variables  
(ptracers).

We are experiencing a (HPC) problem with ptracers: in a few words, we  
sum up the main results of our performance tests below.

- we are running on a 336 x 512 x 72 points grid, using (among the  
others) the EXF, OBCS and KPP packages (namelists attached: please  
note that KPP is active only for physics);
- if we suppose that the ocean physics alone takes 100 minutes to run,  
when we activate the biogeochemistry package (with 51 ptracers), the  
simulation lasts for 1300 minutes (adopting the same timestep). We  
expected a sensible increase in the computational burden, but not so  
large!
- then we also used the LONGSTEP package (with deltaT*6): we reduced  
the wall-clock time to 815 minutes;
- as you can see in the table below, LONGSTEP performs efficiently on  
two among the most time-consuming routines (BLOCKING_EXCHANGES and  
THERMODYNAMICS). As we expected, the wall clock time for  
DO_OCEANIC_PHYS does not change. Our concern is about DYNAMICS and  
SOLVE_FOR_PRESSURE, which increase of one order of magnitude from  
"physics" to "coupled", and are not affected by LONGSTEP. As far as we  
know, these routines are not involved in any ptracers computation,  
aren't they? Why does their computational cost increase so much? Are  
we missing something?

Thank you for any guess/suggestions!

Best regards,

Stefano and Gianpiero


****************************************************************

Synthesis of the performance tests [minutes] (we included ONLY the  
most "time consuming" routines):

routine									physics		coupled		coupled (with longstep)

FORWARD_STEP						100			1300		815
DO_STATEVARS_DIAGS					1			19			4
DO_OCEANIC_PHYS					32			33			34
DYNAMICS								25			503			508
SOLVE_FOR_PRESSURE				6			99			99
BLOCKING_EXCHANGES				3			112			9
THERMODYNAMICS						24			472			24
LONGSTEP_THERMODYNAMICS			0			0			71


****************************************************************




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0009.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: data.obcs
Type: application/octet-stream
Size: 9505 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0008.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0010.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: data.exf
Type: application/octet-stream
Size: 3685 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0009.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0011.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: data
Type: application/octet-stream
Size: 2877 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0010.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0012.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: data.kpp
Type: application/octet-stream
Size: 163 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0011.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0013.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: data.longstep
Type: application/octet-stream
Size: 1214 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0012.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0014.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: data.ptracers
Type: application/octet-stream
Size: 20668 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0013.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0015.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: data.diagnostics
Type: application/octet-stream
Size: 10293 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0014.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0016.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: data.pkg
Type: application/octet-stream
Size: 196 bytes
Desc: not available
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0015.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20140905/c813c5f9/attachment-0017.htm>