Major Update of the MFEM Finite Element Library Broadens GPU Support

By Scott Gibson

The Center for Efficient Exascale Discretizations (CEED) recently released version 4.1 of its MFEM finite element library, which introduces features important for the nation’s first exascale supercomputers.

MFEM (modular finite element methods) is open-source scientific software that is used in computer modeling and simulation of nuclear reactors; subsurface flow for oil exploration; the strength of buildings, bridges, and engineering parts; additive manufacturing; electromagnetic devices like antennas, tokamaks, and MRI machines; climate phenomena; and the human body (the heart, for example). MFEM can be placed in a variety of codes to make their simulations more accurate and more performant on large-scale high-performance computing (HPC) systems.

To simulate physical phenomena, researchers take something in the real world and discretize it, meaning they split it into smaller components, or finite elements, that computers can model more efficiently. CEED, one of six co-design centers within the US Department of Energy’s (DOE’s) Exascale Computing Project (ECP), is developing the next-generation discretization software and algorithms that will enable a wide range of finite element applications to run efficiently on future hardware. Efficiency is measured by the accuracy achieved per unit of computational time.

Tzanio Kolev of Lawrence Livermore National Laboratory

Tzanio Kolev, principal investigator of CEED and project leader of MFEM, describes MFEM’s new features in the latest episode of ECP’s Let’s Talk Exascale podcast.

The conversation covers the value that MFEM brings to a variety of codes, the broadening of support to next-generation GPUs and to libCEED: the CEED operator evaluation library, improvements to the meshing and discretization algorithms, the addition of 18 new examples of mini-apps, and integrations with many other ECP software packages.

Interview Transcript

Gibson: Please provide us with a little background about MFEM and then describe the new features that are in version 4.1.

Kolev: As a co-design center, CEED aims to help many applications, and one way we do that is by providing software in the form of libraries that are easy to use, easy to incorporate in applications, and provide really optimized algorithms for high-order methods on general unstructured grids. And one of CEED’s major software products is MFEM, which is this open-source finite element library that we’ve been developing for quite a few years now.

One way to think about MFEM is like a very powerful and very tuned car engine, but it’s for computations. You can take this tuned and optimized engine and put it in a variety of codes to make simulations more accurate and more performant on large-scale HPC systems. Periodically, we have releases of MFEM, and 4.1 came up in March of 2020. It was a major development for the MFEM community and introduced a number of important new features. I will describe them briefly.

The first set of features has to do with support for GPUs. GPUs will power the exascale machines that we will get in this country, and, also, they’re prevalent already on large-scale HPC systems. But porting existing codes to GPUs has proven challenging, and MFEM, with its previous release, 4.0, was one of the first general-purpose finite element libraries that made GPU acceleration easy to use and available to large-scale simulation codes. In 4.1 we extended the GPU support in MFEM to AMD GPUs; these are the GPUs, for example, the next generation, that will power Frontier at Oak Ridge National Laboratory. We provided additional tools for GPU debugging, faster multi-GPU communication, improved memory management, more-performant kernels in the finite element and linear algebra modules of the libraries. And in many of the MFEM examples, we added the option for GPU acceleration to illustrate how to use that in applications.

Another of the major developments in MFEM 4.1 was the support for libCEED. libCEED is a library that we’re developing in the CEED project that is focused on just the operator evaluation of finite element operators. But the goal there is to provide very performant and very efficient, highly optimized kernels for operator evaluation that can run from the same code base on different architectures and CPUs, different brands of GPUs, and for different orders and for different applications.

So in MFEM 4.1, we took all this work that has been done in the libCEED project and made it easy to use and really available to application users at a very high level of abstraction and very user-friendly way.

There were a lot of other improvements in the meshing and discretization algorithms of MFEM, partial assembly, matrix discretization, nonconforming adaptive mesh confinement, support for complex valence systems, and many more. We added 18 new examples in miniapps, integrations with a lot of ECP packages: libCEED, Umpire, Ginkgo, ParaView, GSLIB, HIOP. And as of 4.1, we switched our license to a more permissive BSD open-source license.

Gibson: Thank you for being on the podcast.

Kolev: Thank you, Scott.