Large earthquakes present a significant risk globally and addressing earthquake risk—both from the standpoint of life safety and damage/economic impact—is a significant societal challenge for virtually every element of the built environment, including energy, transportation, health, data/commerce, and urban infrastructure. The U.S. Department of Energy, in support of the safety of DOE mission-critical facilities, has long been at the forefront of building advanced capabilities for seismic hazard and risk assessment, and these capabilities can extend to broad societal benefit and impact. Earthquake Simulation (EQSIM) is tapping the tremendous developments that are occurring in high-performance computing (HPC) to advance the understanding of the physics of earthquake processes, data collection, and data exploitation to help advance earthquake hazard and risk assessments. EQSIM application codes are removing the reliance on traditional simplifying idealizations, approximations, and sparse empirical data. Instead, the focus is on resolving the fundamental physics uncertainties in earthquake processes. Through EQSIM, regional-scale ground motion simulations are becoming computationally feasible, and simulation models that rigorously connect the domains of seismology and geotechnical and structural engineering are becoming accessible. This simulation-based approach to evaluating earthquake risk is opening an entirely new avenue for the deeper understanding of earthquake phenomenon; allowing for the very detailed, virtual exploration of significant earthquakes; and ultimately providing an important practical tool for earthquake risk quantification.
The EQSIM application development project is focused on creating an unprecedented computational tool set and workflow for earthquake hazard and risk assessment. Starting with a set of existing codes—SW4 (a fourth-order, 3D seismic wave propagation model), NEVADA (a nonlinear, finite displacement program for building earthquake response), and OPENSEES (a nonlinear finite-element program for coupled soil-structure interaction)—EQSIM is creating an end-to-end capability to simulate from the initiation of fault rupture to surface ground motions (i.e., earthquake hazard) and ultimately to infrastructure response (i.e., earthquake risk). EQSIM’s ultimate goal is to remove computational limitations as a barrier to scientific exploration, understanding earthquake phenomenology, and practical earthquake hazard and risk assessments.
Traditional earthquake hazard and risk assessments for critical facilities have relied on empirically based approaches that use historical earthquake ground motions from many different locations to estimate future earthquake ground motions at a specific site of interest. Because ground motions for a particular site are strongly influenced by the physics of the specific earthquake processes—including the fault rupture mechanics, seismic wave propagation through a heterogeneous medium, and site response at the location of a particular facility—earthquake ground motions are very complex with significant spatial variation in frequency content and amplitude. The homogenization of many disparate records in traditional empirically based ground motion estimates cannot fully capture the complex site specificity of ground motion. Over the past decade, interest in using advanced simulations to characterize earthquake ground motions (i.e., earthquake hazard) and infrastructure response (i.e., earthquake risk) has accelerated significantly. However, the extreme computational demands required to execute hazard and risk simulations at a regional scale have been prohibitive. One fundamental objective of the EQSIM application development project is to advance regional-scale ground motion simulation capabilities from the historical computationally limited frequency range of ~0–2 Hz to the frequency range of interest for a breadth of engineered infrastructure of ~0–10 Hz. Another fundamental objective of this project is to implement an HPC framework and workflow that directly couple earthquake hazard and risk assessments through an end-to-end simulation framework that extends from earthquake rupture to structural response, thereby capturing the complexities of interaction between incident seismic waves and infrastructure systems (Figure 1).
To achieve the overall goals, two fundamental challenges must be addressed. First, regional-scale forward ground motion simulations must be effectively executed at an unprecedented frequency resolution with much larger, much faster models. Achieving fast earthquake simulations is essential for allowing the parametric variations needed to span critical problem parameters (e.g., multiple fault rupture scenarios). Second, as the ability to compute at higher frequencies progresses, there will be a need for better characterization of subsurface geologic structures at finer and finer scales; thus, a companion schema for representing fine-scale geologic heterogeneities in massive computational models must be developed.
To evaluate regional-scale simulations and measure the computational progress of the application development and exascale performance goals of this project, a representative large regional-scale detailed model of the San Francisco Bay Area (SFBA) was created (Figure 2). This model includes all the necessary geophysics modeling features (e.g., 3D geology, earth surface topography, material attenuation, nonreflecting boundaries, fault rupture models). For a 10 Hz simulation, the computational domain includes up to 300 billion grid points in the finite difference domain for models that contain fine-scale representations of soft near-surface sedimentary soils. The SFBA model provides a comprehensive basis for testing and evaluating advanced physics algorithms and computational implementations.
Figure of Merit (FOM) is a quantitative metric of the scientific work rate of an application. As the code is optimized to run faster, the FOM increases. Increasing the FOM is actually an enabler of more science. Running bigger problems faster provides for more realistic simulations because of the higher fidelity resolution of earthquake ground motions important to infrastructure systems. An increased FOM means major science advancement.
Significant progress was achieved in EQSIM performance that corresponds to significant increases in the EQSIM FOM from SFBA performance runs, as shown in Figure 4. The application FOM has increased from a baseline of 1.0 at the inception of the project (i.e., the first SFBA regional runs on the Cori computer at Lawrence Berkley National Laboratory) to a value of 189 in the most recent performance assessments on the Summit platform at Oak Ridge National Laboratory. The performance advancements were derived from both algorithm advancements and from the transition to GPU-based platforms, as summarized in Figure 4 and Table 1.
Table 1. Progression of EQSIM SFBA simulations.
|Code attributes||Frequency resolution
|Number of compute nodes||Wall-clock time
|Initial run of SW4 ported to Cori||3.67||2,048||23.9||1.0|
|SW4 with optimized hybrid message passing interface/OpenMP loops||4.17||6,528||12.0||3.32|
|SW4 with Cartesian mesh refinement||4.17||4,000||6.0||6.63|
|SW4 using all of Cori||5.0||8,192
(all of Cori)
|Initial run of SW4 ported to Summit||10.0||1,200
(¼ of Summit)
|Fall 2020 run of SW4, including enhanced I/O, curvilinear, and Cartesian mesh refinement||10.0||1,024
(<¼ of Summit)