EMPIRE AND SPARC

Sandia National Laboratories’ (Sandia’s) Advanced Technology Development and Mitigation (ATDM) components vision has enabled apps such as EMPIRE and SPARC to build on foundational capabilities developed and deployed by other teams, providing great leverage and potential for reuse and increased impact.

Project Details

EMPIRE: Preparing Electromagnetic Plasma Physics Codes for Exascale

Electromagnetic pulse (EMP) environments comprise system-generated and source-region-generated EMP conditions. Many EMP environments must be extrapolated from what can be realized with test facilities; thus, validated computational simulation tools are critical for meeting mission requirements. These problems are numerically challenging to simulate and can span vast length and timescales. Therefore, EMPIRE must deliver advanced electromagnetic and plasma physics code capabilities that will be performant on next-generation hardware architectures. To broaden the range of plasma conditions that can be efficiently simulated, EMPIRE includes kinetic (particle) and fluid (continuum) plasma representations.

The team has demonstrated the capabilities in EMPIRE with simulations of increasingly challenging plasma experiments at higher fidelity than was previously possible. This work is advancing toward the formal validation of EMPIRE capability in the regimes of interest to Sandia. EMPIRE results have also been compared with legacy code results for equivalent simulations, which demonstrated the performance and portability advances that have been enabled by the ATDM and Exascale Computing Project (ECP) program.

 

SPARC: Sandia Parallel Aerodynamics and Reentry Code Virtual Flight Testing

Engineering and physics applications for hypersonic reentry have multiple national security implications and represent the complex modeling of physical phenomena and engineering responses that significantly drive exascale computing requirements. SPARC will provide a state-of-the-art hypersonic flight simulation capability on next-generation hardware and will include hybrid RANS-LES turbulence models.

The pacing science challenge problem for SPARC is to perform a virtual flight test of a reentry vehicle in its entirety and to predict the structural and thermal response of the vehicle’s components under simulated reentry environments. Performing this analysis includes the simulation of the flow field around the vehicle by using a turbulence model suited for hypersonic, unsteady turbulent fluid dynamics. The thermal loads generated from the computational fluid dynamics simulation will be used to predict the response of the vehicle’s thermal protection system and internal components. The structural loads generated from pressure and shear stress fluctuations predictions by the turbulence models will be used to analyze the vibrational response of the vehicle and its internal components. This predictive capability, which is being validated simultaneously with the code’s development, will provide the ability to assess reentry vehicle response to trajectories in which little flight test data exists.

Sandia’s ATDM components vision has enabled apps such as EMPIRE and SPARC to build on foundational capabilities developed and deployed by other well-coordinated teams, providing great leverage and potential for reuse and increased impact. For example, EMPIRE has used discretization and linear solver technology deployed in Trilinos to make the development process more efficient. EMPIRE and SPARC both incorporate innovative approaches on several fronts, including the effective use of heterogeneous compute nodes via Kokkos, uncertainty quantification through Sacado integration, embedded mesh refinement and geometry, the implementation of state-of-the-art reentry physics and multiscale models, the use of advanced verification and validation methods, and enabling of improved workflows for users.

Principal Investigator(s):

EMPIRE: Matt Bettencourt, Sandia National Laboratories, SPARC: Micah Howard, Sandia National Laboratories

Progress to date

EMPIRE:

  • The status of next-generation components and physics models was assessed in EMPIRE. The assessment focused on the electromagnetic particle-in-cell solutions for EMPIRE and its associated solver, time integration, and checkpoint-restart components. The assessment included code verification, performance, and portability across available high-performance computing (HPC) architectures.
  • Next-generation readiness was based on the incorporation of portable performance abstractions—such as Kokkos (MPI+X), high-performance I/O (FAODEL), and time integration libraries—that allow for embedded sensitivity analysis (Tempus). Performance was tested on roughly half of each Trinity partition (KNL and HSW). The core particle-in-cell (PIC) algorithm was shown to have nearly perfect weak and strong scaling on up to 256,000 cores on Trinity and problems with up to 1.3 billion elements and 66 billion particles.
  • Recent scaling and performance studies have advanced EMPIRE simulations on Sierra to over 8,000 GPUs. Good strong and weak scaling results were observed thanks to massive improvements in the linear solver performance. These were achieved by the careful optimization of solver settings and code improvements aimed at reducing unnecessary communication and memory allocations.

SPARC:

  • The code performance and portability was assessed across Trinity-Haswell, Trinity-KNL, and GPU test platforms. Algorithmic improvements led to significant strong-scaling speedups since the start of FY19 (e.g., ~4× for GPU performance).
  • A new structured mesh refinement capability that honors CAD geometry was integrated. By honoring the CAD geometry, each refinement of the mesh more accurately captures the shape of curved surfaces, which helps achieve more accurate simulation results. Embedding the refinement process in the application will allow SPARC to read in a coarse mesh and refine in parallel to produce dramatically larger meshes for detailed analyses on large problems.
  • A comprehensive verification and validation (V&V) study of hypersonic flow was conducted in SPARC, validated by several experiments, and reviewed by an external review committee. The V&V process was thorough, including applicable frameworks, professional standards, code and solution verification, calibration, sensitivity analysis, and parametric uncertainty, and it has provided a basis for using SPARC as a credible analysis tool for hypersonic reentry flows, time integration, and checkpoint-restart components. The assessment included code verification, performance, and portability across available HPC architectures.
  • Next-generation readiness was based on the incorporation of portable performance abstractions—such as Kokkos (MPI+X), high-performance I/O (FAODEL), and time integration libraries—that allow for embedded sensitivity analysis (Tempus). Performance was tested on roughly half of each Trinity partition (KNL and HSW). The core PIC algorithm was shown to have nearly perfect weak and strong scaling on up to 256,000 cores on Trinity and problems with up to 1.3 billion elements and 66 billion particles.
  • Recent scaling and performance studies have advanced EMPIRE simulations on Sierra to over 8,000 GPUs. Good strong and weak scaling results were observed thanks to massive improvements in the linear solver performance. These were achieved by the careful optimization of solver settings and code improvements aimed at reducing unnecessary communication and memory allocations.

National Nuclear Security Administration logo U.S. Department of Energy Office of Science logo