E3SM-MMF

The impact of climate change on global and regional water cycles is one of the highest priorities and most difficult challenges in climate change prediction. Current earth system models possess limited ability to model the complex interactions between the large-scale, mostly 2D baroclinic atmospheric motions and the smaller-scale 3D convective motions found in clouds and individual storms. Exascale computers will enable full-resolution cloud and storm resolving models, but these simulations will remain much too expensive for long climate simulations. The Energy Exascale Earth System Model (E3SM)-multiscale modeling framework (MMF) is using an MMF for cloud resolving modeling that is often referred to as superparameterization. This next-generation model can accurately incorporate cloud physics while also obtaining the throughput necessary for multidecade, coupled high-resolution climate simulations. It will improve the scientific community’s ability to assess regional impacts of climate change on the water cycle that directly affect multiple sectors of US and global economies.

Project Details

The goal of the E3SM-MMF project is to develop a cloud resolving earth system model with the throughput necessary for multidecade, coupled high-resolution climate simulations. This next-generation model could substantially reduce significant systematic precipitation errors found in current models due to its more realistic and explicit treatment of convective storms. These motions and their interactions, to first order, determine the spatial distributions and characteristics of regional precipitation. Complexities include the microscale chemistry and physics of cloud formation and the impacts of anthropogenic climate change on cloud formation. Properly resolving the key processes involved in cloud formation requires resolution (i.e., grid spacing) on the order of 1 km in the atmosphere. Today’s petascale computing systems are capable of such resolution but only at great expense and for very short times (i.e., several simulated days). Running conventional climate models at this resolution for 100 year simulations requires a 5,000× increase in computing resources.

This project is pursuing an alternative approach to cloud resolving modeling based on an MMF approach. This approach offers significant opportunities for unprecedented throughput and GPU acceleration but has yet to be fully explored due to limited computing resources. This project has integrated a cloud resolving convective parameterization (i.e., superparameterization) into the US Department of Energy’s E3SM by using MMF, referred to as E3SM-MMF. The team is now working to explore the model’s full potential to scientifically and computationally advance climate simulation and prediction. E3SM-MMF is ideal for GPU acceleration; compared with conventional approaches, it has ~100 times more work per node with no increase in the amount of internode communication. Although the focus is on the MMF approach, other components and utilities of the coupled system must also achieve the same throughput, so the project includes efforts to accelerate the ocean component and I/O capabilities.

The E3SM-MMF project’s challenge problem has several aspects: (1) a cloud resolving convective parameterization, defined as 1 km or finer grid spacing in horizontal and vertical directions; (2) achieving weather resolving resolution in the global atmosphere model, defined as 50–25 km average grid spacing in the horizontal directions with ~1 km grid spacing in the vertical direction (i.e., the resolution of today’s global operational forecast models); (3) achieving an eddy resolving ocean/ice model, defined as a minimum 18 km resolution in equatorial regions, decreasing to 6 km in polar regions; and (4) achieving the model throughput needed to perform the simulation campaign for the challenge problem in the course of one calendar year on the Frontier supercomputer.

Weather modeling using E3SM-MMF

In the MMF, every grid cell in the global atmosphere model (left) is coupled to a fine-scale model (inset) that can explicitly resolve convective circulations responsible for cloud formation.

Weather modeling with E3SM-MMF

The MMF approach makes it possible to couple the vastly different scales between the global model and cloud resolving fine-scale mode.

Principal Investigator(s):

Mark Taylor, Sandia National Laboratories

Progress to date

  • The team developed Yet Another Kernel Launcher (YAKL), a performance portable C++ library designed to simplify the process of accelerating Fortran code.
  • Superparameterization subcomponents were ported to GPUs by using YAKL.
  • Other key subcomponents were partially ported, including radiation and many of the ocean kernels, by using OpenACC.
  • E3SM-MMF Time-slice experiments (i.e., atmosphere/land/ice simulations with prescribed sea surface temperature and sea ice extent) were completed on Summit, obtaining a 15× GPU acceleration per node.
  • E3SM-MMF multi-decadal fully coupled (atmosphere/land/ocean/ice) simulations were completed on Summit and are currently being analyzed
  • E3SM-MMF’s KPP-1 challenge problem, running a high-resolution version of E3SM-MMF has been run on both Summit and most recently on Frontier.  Frontier is the first machine to be able to meet the KPP-1 objective of an Earth system model with a fully weather- resolving atmosphere and cloud-resolving superparameterization, an eddy resolving ocean and ice components, all while obtaining the necessary throughput to run 10-100 member ensembles of 100-year simulations.
  • The project’s stretch goal was to run the E3SM’s global cloud resolving atmosphere model with a 3.25km global resolution on Frontier.  This model serves as the baseline for E3SM-MMF.  This stretch goal was completed in February 2023, running on 8K Frontier nodes, with the atmosphere component model running at over 1 simulated-year-per-day.

The E3SM enhancements enabled through the Exascale Computing Project will improve the scientific community’s ability to predict, assess, and respond to the challenges imposed by local variations in the water cycle caused by global climate change.

National Nuclear Security Administration logo Exascale Computing Project logo small U.S. Department of Energy Office of Science logo