E3SM-MMF

The impact of climate change on global and regional water cycles is one of the highest priorities and most difficult challenges in climate change prediction. Current earth system models possess limited ability to model the complex interactions between the large-scale, mostly 2D baroclinic atmospheric motions and the smaller-scale 3D convective motions found in clouds and individual storms. Exascale computers will enable full-resolution cloud and storm resolving models, but these simulations will remain much too expensive for long climate simulations. The Energy Exascale Earth System Model (E3SM)-multiscale modeling framework (MMF) is using an MMF for cloud resolving modeling that is often referred to as superparameterization. This next-generation model can accurately incorporate cloud physics while also obtaining the throughput necessary for multidecade, coupled high-resolution climate simulations. It will improve the scientific community’s ability to assess regional impacts of climate change on the water cycle that directly affect multiple sectors of US and global economies.

Project Details

The goal of the E3SM-MMF project is to develop a cloud resolving earth system model with the throughput necessary for multidecade, coupled high-resolution climate simulations. This next-generation model could substantially reduce significant systematic precipitation errors found in current models due to its more realistic and explicit treatment of convective storms. These motions and their interactions, to first order, determine the spatial distributions and characteristics of regional precipitation. Complexities include the microscale chemistry and physics of cloud formation and the impacts of anthropogenic climate change on cloud formation. Properly resolving the key processes involved in cloud formation requires resolution (i.e., grid spacing) on the order of 1 km in the atmosphere. Today’s petascale computing systems are capable of such resolution but only at great expense and for very short times (i.e., several simulated days). Running conventional climate models at this resolution for 100 year simulations requires a 5,000× increase in computing resources.

This project is pursuing an alternative approach to cloud resolving modeling based on an MMF approach. This approach offers significant opportunities for unprecedented throughput and GPU acceleration but has yet to be fully explored due to limited computing resources. This project has integrated a cloud resolving convective parameterization (i.e., superparameterization) into the US Department of Energy’s E3SM by using MMF, referred to as E3SM-MMF. The team is now working to explore the model’s full potential to scientifically and computationally advance climate simulation and prediction. E3SM-MMF is ideal for GPU acceleration; compared with conventional approaches, it has ~100 times more work per node with no increase in the amount of internode communication. Although the focus is on the MMF approach, other components and utilities of the coupled system must also achieve the same throughput, so the project includes efforts to accelerate the ocean component and I/O capabilities.

The E3SM-MMF project’s challenge problem has several aspects: (1) a cloud resolving convective parameterization, defined as 1 km or finer grid spacing in horizontal and vertical directions; (2) achieving weather resolving resolution in the global atmosphere model, defined as 50–25 km average grid spacing in the horizontal directions with ~1 km grid spacing in the vertical direction (i.e., the resolution of today’s global operational forecast models); (3) achieving an eddy resolving ocean/ice model, defined as a minimum 18 km resolution in equatorial regions, decreasing to 6 km in polar regions; and (4) achieving the model throughput needed to perform the simulation campaign for the challenge problem in the course of one calendar year on the Frontier supercomputer.

Weather modeling using E3SM-MMF

In the MMF, every grid cell in the global atmosphere model (left) is coupled to a fine-scale model (inset) that can explicitly resolve convective circulations responsible for cloud formation.

Weather modeling with E3SM-MMF

The MMF approach makes it possible to couple the vastly different scales between the global model and cloud resolving fine-scale mode.

Principal Investigator(s):

Mark Taylor, Sandia National Laboratories

Progress to date

  • The team developed Yet Another Kernel Launcher (YAKL), a performance portable C++ library designed to simplify the process of accelerating Fortran code.
  • Superparameterization subcomponents were ported to GPUs by using YAKL.
  • Other key subcomponents were partially ported, including radiation and many of the ocean kernels, by using OpenACC.
  • Currently, time-slice experiments (i.e., atmosphere/land/ice simulations with prescribed sea surface temperature and sea ice extent) are being run on Summit, obtaining a 15× GPU acceleration per node.

The E3SM enhancements enabled through the Exascale Computing Project will improve the scientific community’s ability to predict, assess, and respond to the challenges imposed by local variations in the water cycle caused by global climate change.

National Nuclear Security Administration logo U.S. Department of Energy Office of Science logo