Sandia National Laboratories’ (Sandia’s) Advanced Technology Development and Mitigation (ATDM) components vision has enabled apps such as EMPIRE and SPARC to build on foundational capabilities developed and deployed by other teams, providing great leverage and potential for reuse and increased impact.

Project Details

EMPIRE: Preparing Electromagnetic Plasma Physics Codes for Exascale

Electromagnetic pulse (EMP) environments comprise system-generated and source-region-generated EMP conditions. Many EMP environments must be extrapolated from what can be realized with test facilities; thus, validated computational simulation tools are critical for meeting mission requirements. These problems are numerically challenging to simulate and can span vast length and timescales. Therefore, EMPIRE must deliver advanced electromagnetic and plasma physics code capabilities that will be performant on next-generation hardware architectures. To broaden the range of plasma conditions that can be efficiently simulated, EMPIRE includes kinetic (particle) and fluid (continuum) plasma representations.

The team has demonstrated the capabilities in EMPIRE with simulations of increasingly challenging plasma experiments at higher fidelity than was previously possible. This work is advancing toward the formal validation of EMPIRE capability in the regimes of interest to Sandia. EMPIRE results have also been compared with legacy code results for equivalent simulations, which demonstrated the performance and portability advances that have been enabled by the ATDM and Exascale Computing Project (ECP) program.

SPARC: Sandia Parallel Aerodynamics and Reentry Code Virtual Flight Testing

Engineering and physics applications for hypersonic reentry have multiple national security implications and represent the complex modeling of physical phenomena and engineering responses that significantly drive exascale computing requirements. SPARC will provide a state-of-the-art hypersonic flight simulation capability on next-generation hardware and will include hybrid RANS-LES turbulence models.

The pacing science challenge problem for SPARC is to perform a virtual flight test of a reentry vehicle in its entirety and to predict the structural and thermal response of the vehicle’s components under simulated reentry environments. Performing this analysis includes the simulation of the flow field around the vehicle by using a turbulence model suited for hypersonic, unsteady turbulent fluid dynamics. The thermal loads generated from the computational fluid dynamics simulation will be used to predict the response of the vehicle’s thermal protection system and internal components. The structural loads generated from pressure and shear stress fluctuations predictions by the turbulence models will be used to analyze the vibrational response of the vehicle and its internal components. This predictive capability, which is being validated simultaneously with the code’s development, will provide the ability to assess reentry vehicle response to trajectories in which little flight test data exists.

Sandia’s ATDM components vision has enabled apps such as EMPIRE and SPARC to build on foundational capabilities developed and deployed by other well-coordinated teams, providing great leverage and potential for reuse and increased impact. For example, EMPIRE has used discretization and linear solver technology deployed in Trilinos to make the development process more efficient. EMPIRE and SPARC both incorporate innovative approaches on several fronts, including the effective use of heterogeneous compute nodes via Kokkos, uncertainty quantification through Sacado integration, embedded mesh refinement and geometry, the implementation of state-of-the-art reentry physics and multiscale models, the use of advanced verification and validation methods, and enabling of improved workflows for users.


Principal Investigator(s):

Curtis Ober, Sandia National Laboratories

Progress to date


  • The EMPIRE team has demonstrated the effectiveness of the portable performance abstraction provided by Kokkos. No specialized code was required to port EMPIRE to run on the early access hardware for El Capitan. Initial scaling results are promising and consistent with prior assessments on past systems.
  • Progress in productionization of Sandia’s embedded components in support of exascale simulation: the UMR (Uniform Mesh Refinement) team pursued a focused effort to not only generate very large meshes (order billion elements), but to accurately and reliably refine the complex geometries of problems of interest with the ability to snap to underlying curved surfaces.
  • EMPIRE capabilities have been supported by ECP component’s efforts: linear solvers (Trilinos – Kokkos Kernels, MueLu and Ifpack2), automatic differentiation (Sacado), performance-portable I/O and mesh decomposition (SEACAS/IOSS) and checkpointing and load balancing (Darma).
  • EMPIRE has expanded the physics capabilities to address a broader range of plasma problems. Specialized boundary conditions and particle emission models, detailed collision physics, and other usability features have been added to the code, along with expanded verification and validation testing that is providing credibility for users performing analysis with EMPIRE.


  • SPARC has leveraged Kokkos and Kokkos Kernels (ECP Software Technologies) to enable its performance portability.  Every kernel in SPARC is written using Kokkos and it is absolutely essential to SPARC’s performance portability.  Additionally, Kokkos Kernels has been utilized to provide performance-portable block Jacobi and block tri-diagonal linear solvers to achieve milestone and ECP performance goals.
  • SPARC capabilities have been supported by ECP component’s efforts: linear solvers (Trilinos – Kokkos Kernels, Ifpack2, Belos and Teko), sensitivity analysis (Sacado), performance-portable I/O and mesh decomposition (SEACAS/IOSS), and mesh transfers and code coupling (STK).
  • SPARC has demonstrated the ability to scale to multiple Advanced Technology Systems (ATSs) (e.g., 4096 nodes of Trinity, 2048 nodes of Sierra, and 2048 nodes of Astra) as part of FY20 L1 milestone.  Recently SPARC has been ported to the early access hardware for El Capitan with improved performance with respect to prior platforms.
  • A comprehensive verification and validation (V&V) study of hypersonic flow was conducted in SPARC, validated by several experiments, and reviewed by an external review committee. The V&V process was thorough, including applicable frameworks, professional standards, code and solution verification, calibration, sensitivity analysis, and parametric uncertainty, and it has provided a basis for using SPARC as a credible analysis tool for hypersonic reentry flows, time integration, and checkpoint-restart components. The assessment included code verification, performance, and portability across available HPC architectures.

National Nuclear Security Administration logo U.S. Department of Energy Office of Science logo