MFIX-Exa - Exascale Computing Project

Carbon capture and storage (CCS) technologies such as oxy-fuel combustion, chemical looping combustion, and post-combustion capture systems offer the most promising approaches for reducing CO₂ emissions from fossil fuel power plants. Large-scale commercial deployment of CO₂ capture technologies requires an understanding of how to scale laboratory designs of multiphase flow reactors to industrial sizes. However, the direct scale-up of such reactors is known to be unreliable, and the current approach requires building and testing physical systems at increasingly larger intermediate scales. The cost in both dollars and development time of having to build and extensively test systems at multiple intermediate scales is prohibitive. High-fidelity computational tools that use exascale computing power can be used to model emerging CCS technologies to enable the design and optimization of these systems, which are critical to controlling costs and reducing the risk of designs failing to meet performance standards.

Project Details

This work specifically targets scale-up of gas-solids reactors like chemical looping reactors (CLRs) through the creation of MFIX-Exa, an exascale capable computational fluid dynamics–discrete element method (CFD-DEM) code, which represents the next generation of the highly successful NETL-based MFIX code. CFD-DEM is an approach that allows for tracking of individual particles (DEM portion) within a continuum fluid phase (CFD portion). Rather than starting the new code from a completely blank slate, MFIX-Exa is built within the AMReX framework, an ECP co-design center for adaptive mesh refinement and block structured numerical algorithms at exascale. The multi-lab team integrates expertise in HPC directly with expertise in multiphase flow modeling and will outperform the existing MFIX by orders of magnitude. MFIX-Exa will enable the development of gas-solids reactors for the removal of CO₂ from point sources, such as fossil-fuel based power plants and industrial processes, or directly from the atmosphere, with minimum cost and energy penalty.

Figure 1: The 50kW Chemical Looping Reactor at NETL, (left) and details of particle flow possible in the riser section of the reactor (right).

The challenge problem for MFIX-Exa is a full-scale simulation of NETL’s 50kW CLR as depicted in Figure 1 and discussed by PI Jordan Musser in a Let’s Talk Exascale podcast.

A cold-flow (non-reacting, isothermal) simulation of only the riser section has been carried out on NETL’s Joule supercomputer. A snapshot of the cold-flow demonstration problem is also shown in the image above and an animation can be found in the MFIX-Exa gallery. A successful simulation of the full challenge problem will require representing the complex geometry and the various gas- solids flow regimes occurring in the CLR (bubbling bed, riser, cyclone, standpipe, and L-valve), and modeling the chemical reactions and interphase heat and mass transfer. Without the capabilities of MFiX-Exa at exascale, it is not possible to resolve the distribution in particle-scale properties (size, density, chemical conversion) in simulations of gas-solids reactors as large as NETL’s CLR.

Principal Investigator(s):

Jordan Musser, National Energy Technology Laboratory

Collaborators:

National Energy Technology Laboratory, Lawrence Berkeley National Laboratory, University of Colorado – Boulder

Progress to date

Replaced the SIMPLE algorithm for fluid flow with a modern projection method, achieving a 4x speed-up.
Extended physical modeling capabilities to include heat and mass transfer, species composition and chemical reactions.
Added a lower-fidelity particle-in-cell (PIC) needed to enable initial condition bootstrapping and save computational resources that would be otherwise required for evolving the (guessed) initial condition to a fully-developed state.
Implemented dynamic load-balancing strategies that account for particle density and a dual- grid approach that simultaneously optimizes CFD and DEM calculations.
Performed weak scaling studies using a simple setup to assess performance at scale, up to 8,192 GPUs on OLCF’s Summit supercomputer, nearly 30% of the leadership-class machine.
Enabled AMReX embedded boundary capability for the creation complex geometries, such as that of a CLR.
Fully ported the code from CPU to GPU and abstracted for next-generation GPU architectures through AMReX. This was aided by a NERSC hackathon, a second round of which recently led to a 3x GPU performance improvement.

MFIX-Exa will deliver high-fidelity multiphase flow modeling capabilities that can aid in the design process of emerging carbon capture technologies to reduce development costs and ensure that scaled-up reactors meet performance targets.