[MITgcm-devel] vectorizing seaice and exf

Patrick Heimbach heimbach at MIT.EDU
Thu Sep 27 16:47:41 EDT 2007 

Martin,

great news.
I reckon you are confident that the
     IF (atemp ... = 0) THEN
doesn't cause additional problems...

Re. routine, I had a quick look, and it seems ok.
One thing is that the store directives for
rdn, ustar, qstar, tstar, tau
are now broken.
Could you add "(i,j)" to these?

I guess you'll run testreport before checkin(?)

Cheers
-Patrick



On Sep 27, 2007, at 2:32 PM, <Martin.Losch at awi.de>  
<Martin.Losch at awi.de> wrote:

> Just for the record: with my recent changes including the not yet  
> checked in vectorized exf_bulkformulae the seaice
> model (without any thermodynamics and a passive ocean) speeds up by  
> a factor of 2 on our NEC SX8!
>
> Martin
>
> Martin Losch
> Alfred Wegener Institute
> Postfach 120161, 27515 Bremerhaven, Germany;
> Tel./Fax: ++49(0471)4831-1872/1797
>
>
>
> ----- Original Message -----
> From: Martin Losch <Martin.Losch at awi.de>
> Date: Thursday, September 27, 2007 4:11 pm
> Subject: Re: [MITgcm-devel] vectorizing seaice and exf
>
>> Hi Patrick, Jean-Michel,
>>
>> I am all for retiring one of the two bulkformulae.
>> Currently the exf_bulkformulae.F seems to have more options than
>> the
>> other one, right? So I am not sure, how similar they are, your call.
>>
>> In the future one may want to replace exf_bulkformulae with a bulk
>> formulations of her/his own choise, eg. the COARE formulae etc ...
>>
>> Martin
>>
>> On 27 Sep 2007, at 16:03, Patrick Heimbach wrote:
>>
>>>
>>> Hi Martin,
>>>
>>> On Sep 27, 2007, at 9:46 AM, Martin Losch wrote:
>>>
>>>> Hi Patrick,
>>>>
>>>> in
>>>> /u/u0/mlosch/scratch/MITgcm/pkg/exf/exf_bulkformulae.F
>>>> you can find a version of exf_bulkformulae that does vectorize
>>>> (haven't checked the vector operation ratio yet, but the
>> compiler
>>>> no vectorizes the i-loop(s)). Tell me what you think (I get
>>>> identical results, but had to make 8 fields two dimensional:
>>>> tstar, ustar, qstar, tau, rdn, t0, delq, deltap).
>>>
>>> I'll have a look.
>>> 2-dim fields is not too much a problem for me, as I am a big
>> memory
>>> consumer anyways.
>>> Not sure what other people think.
>>>
>>>> now your questions (i'll refer to the NEC sxf90, which is
>> relevant
>>>> for anyone running at DKRZ as well):
>>>> you can make the sxf90 inline functions automatically, but only
>>>> those that are < 50 lines, and whose code are in the same file,
>> so
>>>> for exf_bulkformulae you'd have to put these functions into the
>>>> same *.F file (for automatic inlining). This works for, eg.
>>>> exf_interp.F and function lagran (which is part of exf_interp.F).
>>>> you can also make sxf90 inline functions > 50 lines and from
>> other
>>>> files by explicitly specifying at compile time, e.g. '-pi
>>>> exp=timestep_tracer expin=timestep_tracer.F',
>>>> BUT that's complicated and does not work with genmake2 (because
>>>> the compiler is looking for the specified files and crashes if
>> it
>>>> cannot find them, see my earlier email to Jean-Michel. My
>> solution
>>>> to this, an extra flag, is not general either). You'd have to
>>>> modify your makefile each time you run make makefile for this to
>>
>>>> work. So until this is not solved I would like to rely on
>>>> automatic inlining only.
>>>
>>> OK. That seems a good solution.
>>> The Cd functions are one-to-three liners anyways,
>>> and I'll just append them in the exf_bulkformulae.F file.
>>>
>>>> statements like
>>>> if ( atemp(i,j,bi,bj) .ne. 0. _d 0 ) then
>>>> don't seem to be a problem for the compiler
>>>
>>> Hmm, strange. Armin claims it to be a problem.
>>>
>>> Another thing:
>>> Jean-Michel and I were thinking of
>>> retiring one routine, either
>>> exf_bulkformulae.F or exf_bulk_largeyeager04.F
>>>
>>> They are extemely similar, and main difference is in the choice
>>> of the drag coeff. which we make handle differently.
>>> What do you (and others) think?
>>>
>>> -p.
>>>
>>>
>>>
>>>> Martin
>>>>
>>>>
>>>> On 27 Sep 2007, at 14:46, Patrick Heimbach wrote:
>>>>
>>>>>
>>>>> Hi Martin,
>>>>>
>>>>> On Sep 27, 2007, at 8:14 AM, Martin Losch wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>> ...
>>>>>
>>>>>> The bottleneck (from a vectorization point of view) is in
>>>>>> exf_bulkformulae (and exf_bulk_largeyeager04, I expect),
>> because
>>>>>> of the iteration as the innermost loop. Jens-Olaf's solution
>> is
>>>>>> unsatifactory: he noticed that niter_bulk = 2 always (per
>>>>>> parameter definition) and just copied the loop body twice,
>> thus
>>>>>> removing the iteratioin loop. I am still looking for a
>> directive
>>>>>> that can do that automatically, since the loop range is know
>> at
>>>>>> compile time. But I am afraid that the only real solution
>> would
>>>>>> be "loop pushing": move the iteration out of the ij-loops.
>> This
>>>>>> means redefining a few fields (the ones that are updated in
>> the
>>>>>> iteration, which are they?) as 2D.
>>>>>
>>>>> First, quick question:
>>>>> Do you know whether your compiler can inline function calls
>>>>> to improve vectorization?
>>>>> I've removed the three exf_bulk functions about a month ago
>>>>> to help with vecorization, but would be inclined to put some
>> back in
>>>>> (to be able to switch between different drag coeff. schemes).
>>>>> Good compilers would just inline them.
>>>>>
>>>>> A second thing to check is the
>>>>>           if ( atemp(i,j,bi,bj) .ne. 0. _d 0 ) then
>>>>> statement which likely hinders good vectorization.
>>>>>
>>>>>> I would like to do this, but what's the best way: have two
>>>>>> versions of exf_bulkformlulae, one as it is and one for vector
>>
>>>>>> machines? Or can we live with one version that is vectorizable
>>
>>>>>> at the cost of having a few extra 2d fields?
>>>>>
>>>>> A few that are currently non-fields:
>>>>> tau
>>>>> tstar
>>>>> ustar
>>>>> qstar
>>>>>
>>>>> -p.
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> MITgcm-devel mailing list
>>>>>> MITgcm-devel at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>
>>>>> ---
>>>>> Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
>>>>> MIT | EAPS 54-1518 | 77 Massachusetts Ave | Cambridge MA 02139 USA
>>>>> FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> MITgcm-devel mailing list
>>>>> MITgcm-devel at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>
>>>> _______________________________________________
>>>> MITgcm-devel mailing list
>>>> MITgcm-devel at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>
>>> ---
>>> Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
>>> MIT | EAPS 54-1518 | 77 Massachusetts Ave | Cambridge MA 02139 USA
>>> FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach
>>>
>>>
>>> _______________________________________________
>>> MITgcm-devel mailing list
>>> MITgcm-devel at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org
>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel

---
Patrick Heimbach | heimbach at mit.edu | http://www.mit.edu/~heimbach
MIT | EAPS 54-1518 | 77 Massachusetts Ave | Cambridge MA 02139 USA
FON +1-617-253-5259 | FAX +1-617-253-4464 | SKYPE patrick.heimbach

More information about the MITgcm-devel mailing list