By Rob Farber, contributing writer
The development of a whole device model (WDM) for a fusion reactor is critical for the science of magnetically confined fusion plasmas. In the next decade, the ITER fusion reactor project will realize plasmas well beyond the physics regimes accessible in any previous experiments.[1] Computer simulation offers the ability to gain insight into these regimes as ITER is being built and tested, first with deuterium–deuterium fuel by 2025–2026 and then adapted to perform higher power deuterium–tritium experiments by 2035.[2] The US Department of Energy (DOE) has identified WDM as a high-priority need for “assessments of reactor performance in order to minimize risk and qualify operating scenarios for next-step burning plasma experiments.”[3] This need is reflected in the WDMApp documentation overview, which states, “The Whole Device Model Application (WDMApp) in the DOE Exascale Computing Project (ECP) is developing a high-fidelity model of magnetically confined fusion plasmas, urgently needed to plan experiments on ITER and optimize the design of next-step fusion facilities.”[4]
Technical Introduction
The creation of a WDM planning and optimization tool for controlled fusion in a tokamak fusion device is ambitious. The path chosen by the WDMApp team is to create a first-principles coupled kinetic code for the whole plasma (including core and boundary) and demonstrate an extensible software integration framework for plasma codes that can run well on exascale supercomputers. Challenge problems are chosen to ensure that the software is performant and accurately models the tokamak physics.
In addressing their first challenge problem to test their modeling capabilities, the team used their extendable framework, called EFFIS 2.0, to combine—in a physically accurate manner—simulated results from the XGC gyrokinetic code (used to simulate the edge region of a tokamak) with the GENE or GEM gyrokinetic code (used to simulate the core region). The team has successfully ported to GPU all three codes used in the simulation.
This resulted in the first core-edge coupled simulation of turbulence in the plasma of a tokamak device. The following illustration shows the complexity of the simulated turbulence, along with key parts of a tokamak.[5]
Figure 1. WDMApp core-edge coupled simulation shows seamless turbulence coupling.
Seamlessly merging the results of these core and edge simulation codes to accurately model the behavior of the plasma required, among other things, mathematically matching the turbulence in physical time and the spatially overlapping regions between the two codes and other essential activities. To address the performance-critical data transfer between the codes, the team used the ECP ADIOS framework.
The following images demonstrate the turbulence modeling capabilities of the separate GENE and XGC codes and the complexity of the simulated results. The animation illustrates how the core evolves more slowly than the edge.
Figure 2. Core turbulence from GENE is shown on the left. Edge turbulence from XGC is shown on the right.
A detailed discussion of the science and computational work performed to seamlessly merge these codes is presented in a paper entitled The Exascale Framework for High Fidelity coupled Simulations (EFFIS): Enabling whole device modeling in fusion science.
Scott Gibson, ECP Communications Specialist, hosts an informative high-level podcast, Let’s Talk Exascale Code Development: WDMApp — XGC, GENE, GEM, with subject-matter experts Tim Williams of Argonne National Laboratory (ANL), Aaron Scheinberg of Jubilee Development, Kai Germaschewski of the University of New Hampshire, and Bryce Allen of ANL and the University of Chicago.
Figure 3. Clockwise from top left: Tim Williams of Argonne National Laboratory, Aaron Scheinberg of Jubilee Development, Kai Germaschewski of the University of New Hampshire, and Bryce Allen of Argonne National Laboratory and the University of Chicago.
Coupled Physics and Fidelity
WDMApp distinguishes itself by emphasizing a first principles–based approach where strong coupling between the core and edge plasma physics can play a significant role in the behavior of a fusion reactor.
- First-principles modeling holds the promise of very accurately modeling the plasma.[6]
- Strong coupling is necessary to ensure high-fidelity representation of real plasmas. [7]
The challenge is that first-principles models can incur a high computational cost. An alternative, complementary approach that utilizes simplified models and approximations for WDM is the Advanced Tokamak Modeling Environment (AToM) project, which is presently funded by the Scientific Discovery through Advanced Computing (SciDAC) program. The arena of WDM requires a hierarchy of models at different levels of fidelity.
Pedestal Height is Required to Determine Yield in the Core
Key to any WDM approach is the ability to accurately replicate experimental observations, including plasma confinement modes, edge physics, and pedestal height—which, as discussed below, is a key metric in determining the power output of the reactor.
Confinement Modes of Different Flavors
In studying magnetically contained fusion plasmas during the 1980s, scientists observed that under certain conditions, confinement of the plasma abruptly improved upon heating above a threshold. This accidental discovery of enhanced confinement was called H-mode (for “High-confinement mode”) to contrast it with the confinement mode obtained below the power threshold, called L-mode (for “Low‑confinement mode”). H mode enables improvement of confinement times by a factor of nearly two in comparison with L mode.[8] Since then, scientists have derived relationships (e.g., equations) from empirical data that link the H mode with confinement time and ultimately the ability of the fusion reactor to generate power. [9]
The transition to the high (H) mode is marked by the formation of an edge transport barrier at the plasma edge (shown in the image below on the left), which leads to a steep pressure gradient at the plasma edge, forming what is referred to as the pedestal (shown in both images below).
Figure 4. High-fidelity simulation of a whole-device burning plasma (specifically, ITER with full plasma current) operating in “high-mode” (H mode), and prediction of the plasma pressure “pedestal” shape (height and width).
The overall plasma performance in H mode—and thus the performance of the fusion reactor—strongly depends on the pedestal pressure.[10] Predictions indicate that the fusion power of ITER will approximately scale with [the] square of the pedestal pressure.[11] The H mode is currently designated as the operational scenario for the ITER fusion reactor project.
Edge Physics
It is very important to accurately model the edge physics around the edge transport barrier (shown on the right of Figure 4 and in Figure 5 below) because instabilities, called edge-localized modes (ELMs), can develop in this region due to steep pressure gradient.
These instabilities pose a major challenge in magnetic fusion research as they can overheat and damage the material wall. The WDMApp workflow uses the XGC code to model X-point–capable edge physics.[12] [13] Near the edge of tokamak plasmas (shown below), strong particle and energy sources and sinks (e.g., radiation, atomic interaction with neutral particles, contact with a material wall, etcetera) drive the plasma far away from thermal equilibrium, thereby invalidating the idealized assumptions underpinning simple fluid theories.[14] Simulation of the edge physics is expected to dominate the project’s computing budget (for the milestones specified) while a less demanding model (namely GENE or GEM) is used to perform the core simulations.[15]
Figure 5. Cross section of an ITER plasmas showing the X-point (source http://www.sci.utah.edu/publications/Suc2021a/10943420211019119.pdf).
The Importance of Pedestal Height
To understand how all this ties into the efficiency of a fusion reactor—a key metric for power generation—we need to understand the importance of the height of the pedestal illustrated in Figure 4 and the complicating factor of edge effects.
The pressure at the top of the edge transport barrier (labeled “pedestal” in Figure 4) and the exhaust heat from fusion strongly impact fusion performance and constrain material wall lifetime in the stationary operation phase; likewise, large ELMs, driven by the free energy in the pedestal region, can damage the wall material instantaneously.[16] More specifically, the predicted fusion gain, Q, is strongly affected by the temperature at the top of the pedestal.[17] At the same time, the stationary heat exhaust and the transient ELM-driven heat load on the divertor plates must be maintained at a tolerable level.[18] The paper Edge pedestal physics and its implications for ITER discusses the complexity of these relationships.
Pedestal stability studies have developed into a useful and standard tool for understanding both pedestal and ELM characteristics in tokamaks.[19] This makes accurate predictions of the pedestal height and ELM behavior a priority for the ITER project and an essential element in predicting and optimizing the performance of fusion reactors for WDM simulations. It is even unknown if ITER will obey the same edge physics and suffer from the same ELM instabilities as in the present tokamaks. [20]
In demonstrating the efficacy of the WDMApp approach and the EFFIS 2.0 computational framework, the WDMApp team notes in their paper that “Whole device modeling entails many-physics simulation and coupling. Different models and approximations are appropriate for the edge and core of the plasma. Core to edge energy transport occurs on time scales orders of magnitude longer than those needed to simulate micro-turbulence. Boundary and material interactions with the plasma must also be taken into account. A complete WDM must couple all components needed to reliably study ITER or other next generation devices.”[21]
The Importance of the WDMApp Result to Fusion Research
Publication of the WDMApp coupled edge-core simulation result addresses one of the key integration issues highlighted in the recommendation by the recent 2021 National Academies of Sciences, Engineering, and Medicine (NASEM) report, Bringing Fusion to the U.S. Grid: “As fusion concepts approach performance metrics required for a pilot plant, the Department of Energy should support innovative facilities capable of solving key integration physics issues including core-edge integration of a high fusion performance core with a boundary consistent with long-term survival of plasma-facing materials and exhaust of heat and helium ash. Support should incorporate extensive model and diagnostic development, and model validation.”
In terms of policy for the United States, the report notes that “Technology and research results from U.S. investments in the major fusion burning plasma experiment known as ITER, coupled with a strong foundation of research funded by the Department of Energy (DOE), position the United States to begin planning for its first fusion pilot plant.” [22]
Technology and research results from U.S. investments in the major fusion burning plasma experiment known as ITER, coupled with a strong foundation of research funded by the Department of Energy (DOE), position the United States to begin planning for its first fusion pilot plant. – NASEM report, Bringing Fusion to the U.S. Grid
Such an effort is necessary, the report states, “For the United States to be a leader in fusion and to make an impact on the transition to a low-carbon emission electrical system by 2050, the Department of Energy and the private sector should produce net electricity in a fusion pilot plant in the United States in the 2035–2040 timeframe.” [23]
For the United States to be a leader in fusion and to make an impact on the transition to a low-carbon emission electrical system by 2050, the Department of Energy and the private sector should produce net electricity in a fusion pilot plant in the United States in the 2035–2040 timeframe. – NASEM report, Bringing Fusion to the U.S. Grid
Laying out their vision for the future, the WDMApp team references the color coding in Figure 5, “The initial-phase objective (in black/yellow text) is first-principles strong coupling of core and edge physics.” Looking ahead, the authors note that “The 10 year goal then includes (in navy text) the coupling of plasma-material interaction (PMI), radio frequency (RF), neutral beam heating, extended magneto hydrodynamics (MHD), and energetic particles, as well as telescoping to transport time scales. Successful achievement of the project’s 10 year target will enable the development of scenarios for successful and stable operation of ITER and other future fusion reactors.”
Successful achievement of the project’s 10 year target will enable the development of scenarios for successful and stable operation of ITER and other future fusion reactors. – Eric Suchyta et al.[24]
Performance and Scaling to Meet Whole Device Modeling Needs
In completing the initial phase of the work, the team improved the performance of the plasma simulation codes by a factor of 50× over previous generation supercomputers.[25] [26]
Amitava Bhattacharjee (principal investigator of the ECP WDMApp project, professor of Astrophysical Sciences, and former head of the Theory Department of the Princeton Plasma Physics Laboratory) provides a more detailed breakdown: “This performance increase is the result of algorithm improvements and the speedup of the exascale hardware. Acceleration by GPUs is the essential hardware speedup, which is roughly 8×. Our research by a collaboration between physicists, applied mathematicians, and computer scientists have delivered the remaining performance on GPU accelerated hardware ranging between 8× to 50× when scaled from the older GPU-accelerated ORNL Titan supercomputer to the pre-exascale Summit supercomputer.”
Figure 6. Amitava Bhattacharjee, principal investigator of the ECP WDMApp project, professor of Astrophysical Sciences, and former head of the Theory Department of the Princeton Plasma Physics Laboratory.
In terms of scaling behavior, the WDMApp presentation to the International Atomic Energy Agency (IAEA) Accelerating Fusion through Advancements in Edge Turbulence Modeling and its Integration in a Whole Device Model highlighted the near-perfect scaling behavior exhibited by the XGC code on the pre-exascale Summit supercomputer, as shown below.
Figure 7. Near-perfect XGC scaling on the pre-exascale Summit supercomputer. Source: https://nucleus.iaea.org/sites/fusionportal/Shared%20Documents/FEC%202020/fec2020-material/slides1357.pdf
The team also achieved high performance and scalability with their GPU port and optimization of the GEM and GENE codes, as shown below. They note this is a critical requirement towards running WDMApp on the Frontier and Aurora exascale supercomputers.
Figure 8. Gem and GENE Performance and Scaling. Source: https://nucleus.iaea.org/sites/fusionportal/Shared%20Documents/FEC%202020/fec2020-material/slides1357.pdf.
More information plus access to the XGC, GENE or GEM, and EFFIS software can be found on the WDMApp website.
A Golden Opportunity for Physicists and Students
Simulating plasmas that are well beyond the physics regimes accessible in any previous experiments presents a golden opportunity for the physics community, researchers, and graduate students seeking a rewarding career.
To reproduce WDMApp core-edge coupled simulation, researchers need to request access to the XGC, GENE or GEM, and EFFIS software via the WDMApp website. Scientists with access to DOE systems can also view the WDMApp documentation.
Graduate students who wish to consider a career and/or and research opportunities in advancing the state-of-the-art by simulating these unexplored regimes of plasma physics should use the contact information on the WDMApp website. Alternatively, they can contact any of the project’s scientists.
Summary
The creation of computational tools such as WDMApp that leverage the power and versatility of the latest leadership class of supercomputers is a leading priority for the fusion community. Advancement in simulation capability requires exascale HPC hardware plus the creativity and expertise of a plethora of physicists, computer scientists, as well as AI and technology experts. The science is potentially transformational, the technology is leading-edge, and software plus algorithmic contributions are necessary to the development of fusion power systems in the 21st century.
To date, the team has focused on two primary goals: (1) Coupling of core gyrokinetic codes (GENE and GEM) and the edge gyrokinetic code (XGC), which achieved a figure of merit (FOM) greater than 50 compared to the previous generation, and (2) the development of a user-friendly extensible framework, EFFIS 2.0, for code coupling in WDMApp. Recent successes have advanced the field and laid the technology foundation to achieve the team’s goals for the next ten years of work.
This research was supported by the Exascale Computing Project (17-SC-20-SC), a joint project of the U.S. Department of Energy’s Office of Science and National Nuclear Security Administration, responsible for delivering a capable exascale ecosystem, including software, applications, and hardware technology, to support the nation’s exascale computing imperative.
Rob Farber is a global technology consultant and author with an extensive background in HPC and in developing machine learning technology that he applies at national laboratories and commercial organizations.
[1] Aymar R, Barabaschi P and Shimomura Y (2002) The ITER design. Plasma Physics and Controlled Fusion 44(5): 519–565.
[2] https://en.wikipedia.org/wiki/ITER
[3] http://www.sci.utah.edu/publications/Suc2021a/10943420211019119.pdf
[4] https://wdmapp.readthedocs.io/en/latest/
[5] J. Dominski et al., Physics of Plasmas 25, 072308 (2018) Visualization : Dave Pugmire (ORNL)
[6] http://www.sci.utah.edu/publications/Suc2021a/10943420211019119.pdf
[7] https://iopscience.iop.org/article/10.1088/0741-3335/49/7/B02/meta
[8] http://www-fusion-magnetique.cea.fr/gb/fusion/physique/modesconfinement.htm
[9] http://www-fusion-magnetique.cea.fr/gb/fusion/phy sique/lawson.htm#Le%20temps%20de%20confinement%20de%20l’%C3%A9nergie%20tE
[10] https://www.sciencedirect.com/science/article/pii/S2352179118301911
[11] https://fusion.gat.com/pubs-ext/APS11/Snydervgs.pdf
[12] http://www.sci.utah.edu/publications/Suc2021a/10943420211019119.pdf
[13] https://theory.pppl.gov/research/research.php?rid=10
[14] https://theory.pppl.gov/research/research.php?rid=10
[15] http://www.sci.utah.edu/publications/Suc2021a/10943420211019119.pdf
[16] https://www.researchgate.net/publication/231024625_Pedestal_stability_comparison_and_ITER_pedestal_prediction
[17] http://www-naweb.iaea.org/napc/physics/FEC/FEC2006/papers/it_1-3.pdf
[18] http://www-naweb.iaea.org/napc/physics/FEC/FEC2006/papers/it_1-3.pdf
[19] https://www.researchgate.net/publication/231024625_Pedestal_stability_comparison_and_ITER_pedestal_prediction
[20] C.S. Chang, S. Ku, R. Hager, R.M. Churchill, J. Hughes, F. Köchl, A. Loarte, V. Parail, R. Pitts, “Constructing a new predictive scaling formula for ITER’s divertor heat-load width informed by a simulation-anchored machine learning,” Physics of Plasmas 28, 022501 (2021); https://doi.org/10.1063/5.0027637
[21] http://www.sci.utah.edu/publications/Suc2021a/10943420211019119.pdf
[22] https://www.science.org/content/article/road-map-us-fusion-power-plant-comes-clearer-focus-sort
[23] https://www.science.org/content/article/road-map-us-fusion-power-plant-comes-clearer-focus-sort
[24] http://www.sci.utah.edu/publications/Suc2021a/10943420211019119.pdf
[25] https://sites.google.com/pppl.gov/ecpwdmapp
[26] https://nucleus.iaea.org/sites/fusionportal/Shared%20Documents/FEC%202020/fec2020-material/slides1357.pdf
