Power plants based on fossil fuels must employ cost-effective carbon capture and storage (CCS) technologies to ensure that the United States will continue to have sustainable, reliable, and affordable low-carbon energy.
One of the many novel combustion technologies that could greatly reduce the costs associated with the capture of carbon dioxide entails the use of what are called chemical looping reactors (CLRs). But CLRs have been demonstrated only in the laboratory and at small pilot scales, and they must be built to larger pilot and then industrial scales.
Highly detailed, or high-fidelity, computer simulations could reduce the cost and technical risk as CLRs transition from research and development to first demonstrations in the 2025–2030 timeframe, to meet a CCS technology goal of the US Department of Energy (DOE).
A DOE Exascale Computing Project (ECP) effort, led by Madhava Syamlal of the National Energy Technology Laboratory (NETL), is building a new tool, called MFiX-Exa, that will enable the needed high-fidelity simulations.
MFiX-Exa is a computational fluid dynamics–discrete element model (CFD-DEM) code designed to run efficiently on current and next-generation massively parallel supercomputing architectures. It is the latest creation based on the original MFiX code developed at NETL and is used widely in academia and industry.
By combining new algorithmic approaches and a new software infrastructure, MFiX-Exa will leverage future exascale machines to optimize CLRs. Exascale will provide 50 times more computational science and data analytic application power than is possible with DOE high-performance computing systems such as Titan at the Oak Ridge Leadership Computing Facility (OLCF) and Sequoia at Lawrence Livermore National Laboratory.
Tests have shown that the new MFiX-Exa algorithm reduces the computational time for the computational fluid dynamics calculations by 4x. The new algorithm is expected to perform even better in the ECP challenge problem simulation, which will use progressively more cores on an exascale machine.
The Challenge Problem
The MFiX-Exa efforts are directed at an ECP challenge problem that consists of a CFD-DEM simulation of NETL’s laboratory-scale CLR, which consists of a fuel reactor and an air reactor.
Rather than air, the fuel reactor uses oxygen from solid oxygen carriers, such as metal oxides, to combust fossil fuels. The spent oxygen carrier is sent to the air reactor where it is regenerated with oxygen from air. The air reactor produces a hot air stream that is used to raise steam to drive a turbine for power generation; the fuel reactor produces gases from which CO2 can be easily captured. The regenerated oxygen carrier is returned to the fuel reactor, completing the chemical looping cycle.
The approximately 5 billion oxygen-carrier particles in the NETL CLR is a quantity 40 times greater than the number of particles simulated in the largest CFD-DEM studies reported in the research literature in which open-source or commercial codes are used. The challenge problem simulation is a stepping stone for large pilot- and industrial-scale simulations, which, however, are not in the scope of the current project.
Another aspect of the robustness of the MFiX-Exa challenge problem is that it requires the simulation of a longer operational time and the handling of a complex reactor with multiple flow regimes and chemical reactions. The team expects the challenge problem simulation to be 5x longer or more than studies reported in the research literature.
Project Advances and Successes
The fundamental approach used to solve the ECP challenge problem is CFD-DEM. This methodology tracks individual particles using DEM while the gas flow is calculated with CFD. This method provides greater fidelity than the two-fluid model (TFM) and multiphase particle-in-cell (MP-PIC) methods currently popular in industry. By resolving the particles individually, the model does not need to use the approximations that reduce the fidelity of the TFM and MP-PIC methods.
Although MFiX-Exa builds on the multiphase modeling expertise embodied in NETL’s MFiX code, the core methodology has been both re-designed and re-implemented. The foundation for MFiX-Exa is the AMReX software framework supported by the ECP Block-Structured Adaptive Mesh Refinement (AMR) Co-Design Center.
In CFD-DEM, the entire volume of a simulated reactor is broken into a vast number of small contiguous volumes, over which the equations are solved. The collection of small volumes is called a mesh. The size of the mesh determines the fidelity of the simulation as well as the computational effort. AMR adjusts the computational effort locally to maintain a uniform level of accuracy throughout the reactor.
MFiX-Exa uses more efficient algorithms than MFiX for reducing the computational time. A new CFD algorithm has been implemented in MFiX-Exa that leverages discretizations (finite elements of geometry) and linear solvers (pieces of mathematical software) already available through the AMReX framework.
Tests have shown that the new MFiX-Exa algorithm reduces the computational time for the CFD calculations by 4x. The new algorithm is expected to perform even better in the challenge problem simulation, which will use progressively more cores on an exascale machine.
In the DEM, tracking the collisions between the particles and the reactor walls requires much computational time. A new algorithm that calculates the distance to the nearest wall once, stores that value, and reuses it for millions of repeated calculations, was implemented in MFiX-Exa. This improvement accelerated the DEM calculations for simple CLR geometries, and the team expects a greater speedup for the more complex CLR geometry.
The finer the mesh, the greater the accuracy with which geometry and flow features can be simulated—but also greater is the computational time required.
MFiX-Exa recently added the capability for local mesh refinement, which enables the use of a fine mesh near the walls that accurately resolves the reactor shape while not over-refining the interior of the reactor. Local mesh refinement will reduce the mesh size and, hence, the computational time required for the challenge problem.
The project also implemented the ability to eliminate unneeded mesh in regions outside the CLR itself—that is, the empty space between the fuel and air reactors. For the challenge problem geometry, this will reduce the mesh size by 10x.
The Collaborative Team
MFiX-Exa has brought together researchers from NETL, Lawrence Berkeley National Laboratory (LBNL), and the University of Colorado (CU). NETL and CU represent more than six decades of experience in multiphase modeling and the MFiX code, while LBNL brings the same level of expertise in large-scale, multiscale multiphysics applications.
In total, the MFiX-Exa team is characterized by more than 90 years of relevant experience and close collaborative ties: members interact daily, monthly, and yearly through Slack team message app channels, teleconferences, and all-hands meetings, respectively.
The most important next activity for the MFiX-Exa team is to ensure that MFiX-Exa code can run effectively on hybrid CPU/GPU architectures. The first stage of development has focused on running MFiX-Exa on multicore architectures such as the Cori supercomputer at the National Energy Research Scientific Computing Center.
The next stage will focus on running effectively on machines like the OLCF’s Summit system. Currently, the particle-particle collisions can be off-loaded to the GPUs, and work is in progress to migrate more of the algorithm to the GPUs to reap the benefit of their compute power.