LLNL is developing next-generation multi-physics simulation capabilities for national security applications and has adopted a modular approach to code development through the Multi-physics on Advanced Platforms Project (MAPP). The software being developed as part of MAPP addresses the modeling needs of the high-energy-density physics (HEDP) community for simulating high-explosive, magnetic or laser-driven experiments such as inertial confinement fusion (ICF), pulsed-power magneto-hydrodynamics (MHD), and equation of state (EOS) and material strength studies as part of the NNSA’s stockpile stewardship program (SSP).
Fundamental to MAPP is the Axom computer science (CS) toolkit, a library of shared software components that provides various services for the development of modular, multi-physics application codes. MARBL, a next-generation code focused on ICF and pulsed-power applications, is one code in the project that is built on top of Axom. MARBL exemplifies MAPP’s overall philosophy of extreme modularity in physics and CS capabilities and includes multiple options for every major physics capability.
The Axom library consists of a collection of software components that provide core computer science infrastructure capabilities that can be shared by diverse high-performance computing (HPC) applications. The current set of capabilities that Axom provides includes customizable support for error, warning and diagnostic message reporting; coordination among components of integrated applications (e.g., physics packages, libraries, etc.), and an in-memory datastore for hierarchical, mesh-aware simulation data. Axom’s datastore supports data description, allocation, deallocation, and parallel l/0, along with mesh data model abstractions, enabling the development of computational algorithms that work with many different mesh types.
The MARBL application code has been designed from inception to support multiple diverse algorithms, including arbitrary Lagrangian-Eulerian (ALE) and direct Eulerian methods to solve the conservation laws associated with its various physics packages. One distinguishing feature of MARBL is the use of advanced, high-order numerical discretizations such as high-order finite element ALE and high-order finite difference Eulerian methods. High-order numerical methods were chosen because they have higher resolution/accuracy per unknown compared to standard low-order finite volume schemes and because they have computational characteristics which play to the strengths of current and emerging HPC architectures. Specifically, they have higher FLOP/byte ratios, meaning that more floating-point operations are performed for each piece of data retrieved from memory compared to low-order methods. This leads to increased computational efficiency and improved throughput on GPU platforms. The advanced simulation capabilities provided by MARBL improves user throughput along two axes: faster turnaround for multi-physics simulations on advanced architectures and less manual user intervention.
A key goal for MARBL is enhanced end-user productivity, including improved workflow for problem setup and meshing, simulation robustness, support for UQ and optimization-driven ensembles, and in-situ data visualization and analysis. High-order ALE and Eulerian schemes have proven to be more robust and to significantly improve the overall analysis workflow for users. As such, MAPP represents a massive software development effort, incorporating multiple physics, mathematics, and computer science packages into the overall integrated code.
The team collaborates with multiple ECP Software Technology (ST) projects to integrate new production quality capabilities, including software developed both internally at LLNL and externally from the ECP and the broader open-source community.
The success of MAPP will ultimately be determined by the degree of adoption of its simulation tools by the LLNL user community and beyond. To this end, emphasis at this relatively early stage of development is being placed on adding physics and capabilities to meet the current state of the art that users demand from today’s petascale production simulation codes. In the case of MARBL, this includes coupled multi-material radiation-magneto-hydrodynamics, thermonuclear burn for inertial confinement fusion calculations, general equations of state, material opacities and electrical conductivities, simulation diagnostics and queries, in-situ analytics/rendering, and parallel computational and file I/O performance at a massive scale. In addition, performance of the new codes on advanced architectures like the GPU-based Sierra and El Capitan systems at LLNL is critical. Portability of the software stack and long-term maintainability are critical as well, placing stringent demands on the integration and interoperability of high-quality production-level software libraries and tools. Finally, MARBL is the first demonstration of the viability of advanced high-order numerical approaches for production multi-physics simulation at scale in the NNSA and has already produced first-of-a-kind simulation results using such methods.