[MITgcm-support] Wrapper bindings to OpenCL C library - The possibilities of GPGPU?

Tue Aug 4 14:15:14 EDT 2020

Hi Jody,

Thank you for your interest! Perhaps opening a GitHub issue on the
repository to discuss further (and make the discussion public) would be
better but to answer your questions:

1) Yes so right now it's constant grid spacing only (Δx, Δy, Δz can be
different of course). We have a branch that adds a vertically stretched
Cartesian grid, i.e. variable vertical grid spacing, but it still needs to
be fully tested. We currently have no plans to add grids other than
constant and vertically stretched as we can still use a fast/efficient
pressure solver based on FFTs with one stretched dimension. But with 2-3
stretched dimensions then we would have to implement an iterative solver,
e.g. conjugate gradient like MITgcm, which would be much slower. And for
now we're not tackling problems that require variable spacings in all three
dimensions so we haven't thought about supporting it yet.

2) Not officially in that there's no API to specify topography but we've
been using the forcing functions API to implement a continuous forcing
immersed boundary method to add topography. Basically you strongly relax
the velocity to zero inside the topography. We haven't been tackling
problems with topography yet so we haven't fleshed out how we handle
topography but we're hoping to add some tests to see if this immersed
boundary method can work well in oceanographic settings.

For an example, this PR adds an example/test for viscous flow around a
cylinder: https://github.com/CliMA/Oceananigans.jl/pull/693

These are the pertinent lines (PR needs updating a bit as API has changed,
improved we think):
https://github.com/CliMA/Oceananigans.jl/pull/693/files#diff-7331d4b1c4f866ed7dfe0952578bf1cfR18-R27

We're hoping to slowly expand the scope of what Oceananigans can do as we
tackle more complicated problems but for now trying to stick to
configurations where we can leverage the GPU using solvers we've already
implemented.

That said, we're more than happy to discuss new features. Sometimes they're
not difficult to add, and sometimes it can be implemented using an existing
API.

Cheers,
Ali

On Tue, Aug 4, 2020 at 1:44 PM Jody Klymak <jklymak at uvic.ca> wrote:

> Hi Ali,
>
> Thanks for the link - that looks pretty fun.
>
> Happy to discuss somewhere more appropriate if you send me there, but a
> couple of questions:
>
> 1) Are grids regular in that you can’t specify variable grid spacing (even
> if it is still Cartesian)?
>
> 2) It seems maybe no topography?
>
> I think both 1 and 2 make it really hard to use GPUs or spectral methods
> in general, but I could very well be ignorant of recent advances.
>
> Thanks,   Jody
>
> On 4 Aug 2020, at 10:27, Ali Ramadhan <alir at mit.edu> wrote:
>
> Hi Robin,
>
> Not an answer to your question but if your simulations are non-hydrostatic
> on regular Cartesian grids then you might be able to run Oceananigans.jl on
> a GPU: https://github.com/CliMA/Oceananigans.jl
>
> It uses mostly the same finite volume algorithm as MITgcm but with a more
> efficient pressure solver since it focuses on regular Cartesian grids. It's
> more for ocean process studies, i.e. simulation patches of ocean or
> incompressible fluid. Unfortunately, Oceananigans.jl does not run
> simulations on the sphere so it may not be suitable for your work.
>
> But if you think it might benefit your work, I'm more than happy to help
> out with setting up your problem in Oceananigans.jl and we can try running
> it on your GPU.
>
> Cheers,
> Ali
>
> On Tue, Aug 4, 2020 at 12:50 PM Robin Despouys <robin.despouys at gmail.com>
> wrote:
>
>> Hello Dear Community,
>>
>> I would like to know if there is any current work on trying to use
>> Graphics Processing Units in order to accelerate the simulations on
>> personal desktop.
>>
>> I just discovered this incredible work, MITgcm, thanks to a friend who is
>> doing simulations for his PhD. We managed to run his simulations on my
>> Computer but due to the limitations of my CPU 4 cores, 4 Ghz I
>> theoretically have 64 GFLOPS on best case scenario, and according to the
>> specs of my GPU I could theoretically have 7 TFLOPS (again best case
>> scenario).
>>
>> So today we started a simulation using MPI and 4 cores, my friend says it
>> will take roughly 4 days.  So with my simple supposition and ignoring all
>> the drawbacks and limitations due to the synchronisation of highly
>> multithreaded computation we could run this simulation 100 times faster
>> which would reduce the computation time to ~ 1 hour.
>>
>> I know that we could simply use MPI with a cluster of machines or with a
>> super-calculator.
>> I can't help but think : that there might be a way to harness the power
>> of GPUs by using an implementation of the OpenCL standard in C and make it
>> "callable" through the wrapper without having to change the numerical
>> input models.
>>
>> I am very new to FORTRAN and I have a reasonable understanding of C but
>> at the moment I have a very approximate knowledge of the Software
>> Architecture of MITGcm. Then... maybe what I am asking for is impossible.
>>
>> But if it is possible! Perhaps some of you could point me at the parts I
>> should focus on what/where should I update/add some code. (Yup it is a very
>> broad question^_^).
>>
>> Best regards,
>>
>> Robin
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> MITgcm-support mailing list
>> MITgcm-support at mitgcm.org
>> http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
>>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
>
>
> --
> Jody Klymak
> http://ocean-physics.seos.uvic.ca/~jklymak/
>
>
>
>
>
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mailman.mitgcm.org/mailman/listinfo/mitgcm-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.mitgcm.org/pipermail/mitgcm-support/attachments/20200804/0cec30d6/attachment.html>