The development of capable exascale systems was made possible by an interdisciplinary and collaborative co-design process. The Exascale Computing Project (ECP) created co-design centers such as AMReX to develop algorithms and software components optimized for cutting-edge hardware. The objective is to advance computational motifs common to multiple application development projects, thereby fostering participatory development processes to meet the complex and often conflicting needs of current and future exascale applications. The co-design teams worked closely with application developers to deliver efficient and reliable software products that are integral to the unprecedented results generated on exascale supercomputers such as Frontier.
Adaptive mesh refinement (AMR) is a simulation method that functions much like a computational microscope; it allows researchers across disciplines to zoom in on regions of a simulation that are more complex or scientifically relevant than others. For example, cosmologists can zoom in on cosmic filaments to explore them in greater detail, astrophysicists might focus on regions of nucleosynthesis, and combustion scientists may investigate the complex chemistry near a flame front. These capabilities significantly improve scientists’ ability to study computationally intensive problems in extreme detail by focusing computational power mostly on the areas of a simulation that require high fidelity. By combining AMR with next-generation supercomputers such as the Frontier exascale machine, researchers can generate simulations with unprecedented accuracy while integrating more realistic physical parameters and larger simulation sizes.
The ECP’s AMReX co-design center provides a unified software infrastructure with the functionality needed to effectively use AMR-based applications on machines ranging from laptops to exascale supercomputers. Many ECP application teams use AMReX designs, including those focused on particle accelerator design, additive manufacturing, astrophysics, combustion, cosmology, multiphase flow, and wind plant modeling.
The AMReX design provides a range of functions that afford application developers significant flexibility in how they interact with the software suite. These functions include core capabilities such as support for field data on a hierarchical mesh as well as tools to represent particles and various particle-mesh interactions to accommodate the diverse physics objectives of the ECP applications. Furthermore, AMReX supports complex boundary geometries within simulations and provides interfaces to several common frameworks for smooth in-situ data analysis and visualization. These capabilities represent some of the many tools that enable ECP developers to efficiently use AMR in creating new applications for exascale machines.
AMR is a critical tool for advancing simulation capabilities across the sciences. By incorporating this software approach into ECP applications on the world’s fastest supercomputers, researchers will create simulations of unprecedented size and fidelity. These improvements will advance basic research, provide information for engineering new technologies, and uncover new methods to further improve computational efficiency and capabilities.
Adaptive mesh refinement (AMR) is like a computational microscope; it allows scientists to “zoom in” on particular regions of space that are more interesting than others. Cosmologists might want to zoom in on detailed cosmic filaments, astrophysicists might focus on regions of nucleosynthesis, and combustion scientists might investigate the details of the chemistry near a flame front.
The AMReX (AMR for the Exascale) project supports the development of block-structured AMR algorithms for solving systems of partial differential equations on exascale architectures. Block-structured AMR provides the basis for the temporal and spatial discretization strategy for several ECP applications in the areas of accelerator design, additive manufacturing, astrophysics, combustion, cosmology, multiphase flow, and wind plant modeling.
AMReX is a software framework that provides a unified infrastructure with the functionality needed for these and other AMR applications to be able to effectively and efficiently use machines, from laptops to exascale architectures. AMR reduces the computational cost and memory footprint compared with a uniform mesh while preserving accurate descriptions of different physical processes in complex multiphysics algorithms.
AMReX supports a wide range of algorithms that solve systems of partial differential equations. In addition to the core capability to support field data on a hierarchical mesh, AMReX also provides data structures to represent particles and support for different particle and particle/mesh algorithms. AMReX also provides tools for an embedded boundary representation of complex problem geometry and components for designing algorithms that use that representation. Additionally, it provides native geometric multigrid solvers needed to support implicit discretizations and native asynchronous I/O capabilities used to write data for data analysis and visualization and for checkpoint/restart.
The AMReX design allows application developers to interact with the software at several different levels of abstraction. It is possible to use only the AMReX data containers and iterators and none of the higher level functionality. A more popular approach is to use the data structures and iterators for single- and multilevel operations but retain complete control over the time evolution algorithm (i.e., the ordering of algorithmic components at each level and across levels). In an alternative approach, the developer exploits additional functionality in AMReX that is designed specifically to support traditional subcycling-in-time algorithms. In this approach, stubs are provided for the necessary operations, such as advancing the solution on a level, correcting coarse grid fluxes with time- and space-averaged fine grid fluxes, averaging data from fine to coarse, and interpolating in space and time from coarse to fine. This layered design provides users with the ability to have complete control over their algorithm or to use an application template that can provide higher level functionality.
AMReX is designed for performance portability across a range of accelerator-based architectures for a variety of different applications. AMReX isolates applications from any particular architecture and programming model without sacrificing performance. A lightweight abstraction layer effectively hides the details of the architecture from the application. This layer provides constructs that allow users to specify what operations they want to perform on a block of data without specifying how those operations are carried out. AMReX then maps those operations onto the hardware at compile time so that the hardware is used effectively. For example, on a many-core node, an operation would be mapped onto a tiled execution model by using OpenMP to guarantee good cache performance, whereas on a different architecture, the same operation might be mapped to a kernel launch appropriate to a particular GPU.
Additionally, AMReX provides native functionality to support efficient parallel communication, parallel reductions, and memory management.