The impact of climate change on global and regional water cycles is one of the highest priorities and most difficult challenges in climate change prediction. Current earth system models possess limited ability to model the complex interactions between the large-scale, mostly 2D baroclinic atmospheric motions and the smaller-scale 3D convective motions found in clouds and individual storms. Exascale computers will enable full-resolution cloud and storm resolving models, but these simulations will remain much too expensive for long climate simulations. The Energy Exascale Earth System Model (E3SM)-multiscale modeling framework (MMF) is using an MMF for cloud resolving modeling that is often referred to as superparameterization. This next-generation model can accurately incorporate cloud physics while also obtaining the throughput necessary for multidecade, coupled high-resolution climate simulations. It will improve the scientific community’s ability to assess regional impacts of climate change on the water cycle that directly affect multiple sectors of US and global economies.
Summary
Climate change is influencing the global water cycle, resulting in droughts, changes to cloud formation and rainfall, as well as more frequent and powerful storms. These changes may negatively impact regional energy and water supplies, agriculture, and human and environmental health if not adequately addressed. Predicting changes in the water cycle gives critical information to regional policy makers in multiple sectors of the US and global economies, and advanced climate models are a key resource for providing this information. Researchers must consider complexities such as the microscale chemistry and physics of cloud formation and the impacts of anthropogenic climate change on cloud formation, and must incorporate these factors into simulations which routinely span decades while maintaining computational and economic feasibility. High performance computers are an essential part of this process, as they allow for greatly enhanced fidelity across longer timescales, leading to much more accurate and cost-effective predictions.
The Exascale Computing Project’s Energy Exascale Earth System Model (E3SM) has developed a computational model capable of running on the world’s fastest supercomputers, simulating climate-induced changes to cloud formation and weather events with unprecedented speed and accuracy. The E3SM application runs multidecade climate simulations at resolutions that accurately capture cloud formation through multiphysics integration and eddy capturing ocean models.
Accurately simulating the key processes in cloud formation requires minimum resolutions of 1 km in the atmosphere. Previous petascale computing systems are capable of such resolution, but only at great expense and for very short durations—measured in days rather than years. Running conventional climate models at the resolutions and durations needed for accurate and applicable cloud resolution simulations requires a 5,000× increase in computing resources. While modern exascale computers boast a more than 1000x improvement in processing speed compared with previous state-of-the-art machines, innovative software approaches were still needed to meet this benchmark.
The E3SM application team met and exceeded this computational challenge, running the first ever earth system model with a fully weather-resolving atmosphere and embedded cloud resolving model. The application uses superparameterization to resolve the convective processes responsible for cloud formation within 1 km grid boxes, with the rest of the model processes being resolved in 25 km boxes. These approaches, combined with efficient use of GPUs on exascale computers, improved E3SM’s simulation rate from initial speeds of .011 years per day to 5 simulated years per day.
The greatly improved simulation speed and fidelity generated by E3SM on exascale machines will give the scientific community the much-needed ability to predict, assess, and respond to the challenges imposed by local variations in the water cycle caused by global climate change in the years to come. These simulations will provide key information to policy makers, enabling the mitigation of potentially disastrous changes in weather patterns and supporting long-term stability in energy generation, agricultural production, and water quality and supply.
Technical Discussion
The goal of the E3SM-MMF project is to develop a cloud resolving earth system model with the throughput necessary for multidecade, coupled high-resolution climate simulations. This next-generation model could substantially reduce significant systematic precipitation errors found in current models due to its more realistic and explicit treatment of convective storms. These motions and their interactions, to first order, determine the spatial distributions and characteristics of regional precipitation. Complexities include the microscale chemistry and physics of cloud formation and the impacts of anthropogenic climate change on cloud formation. Properly resolving the key processes involved in cloud formation requires resolution (i.e., grid spacing) on the order of 1 km in the atmosphere. Today’s petascale computing systems are capable of such resolution but only at great expense and for very short times (i.e., several simulated days). Running conventional climate models at this resolution for 100 year simulations requires a 5,000× increase in computing resources.
This project is pursuing an alternative approach to cloud resolving modeling based on an MMF approach. This approach offers significant opportunities for unprecedented throughput and GPU acceleration but has yet to be fully explored due to limited computing resources. This project has integrated a cloud resolving convective parameterization (i.e., superparameterization) into the US Department of Energy’s E3SM by using MMF, referred to as E3SM-MMF. The team is now working to explore the model’s full potential to scientifically and computationally advance climate simulation and prediction. E3SM-MMF is ideal for GPU acceleration; compared with conventional approaches, it has ~100 times more work per node with no increase in the amount of internode communication. Although the focus is on the MMF approach, other components and utilities of the coupled system must also achieve the same throughput, so the project includes efforts to accelerate the ocean component and I/O capabilities.
The E3SM-MMF project’s challenge problem has several aspects: (1) a cloud resolving convective parameterization, defined as 1 km or finer grid spacing in horizontal and vertical directions; (2) achieving weather resolving resolution in the global atmosphere model, defined as 50–25 km average grid spacing in the horizontal directions with ~1 km grid spacing in the vertical direction (i.e., the resolution of today’s global operational forecast models); (3) achieving an eddy resolving ocean/ice model, defined as a minimum 18 km resolution in equatorial regions, decreasing to 6 km in polar regions; and (4) achieving the model throughput needed to perform the simulation campaign for the challenge problem in the course of one calendar year on the Frontier supercomputer.
In the MMF, every grid cell in the global atmosphere model (left) is coupled to a fine-scale model (inset) that can explicitly resolve convective circulations responsible for cloud formation.
The MMF approach makes it possible to couple the vastly different scales between the global model and cloud resolving fine-scale mode.
