Exascale Day Article: Contributing capabilities to E3SM-MMF

A subproject within the U.S. Department of Energy’s Exascale Computing Project (ECP), the Energy Exascale Earth Systems Modeling-Multiscale Modeling Framework (E3SM-MMF) has sought to improve simulation capabilities around the water cycle and precipitation. The project’s aim is to use the additional exascale computing power not only to accelerate the global E3SM model, but to add a new capability to simulate clouds and other fine-scale features using many cloud-resolving sub-models (MMF) within the global modeling system. Building on longstanding capabilities, Los Alamos Laboratory researchers within the project, led by E3SM performance group lead Phil Jones, have contributed improved ocean and sea ice components to complement the new atmosphere model improvements as E3SM-MMF brings the modeling and simulation onto exascale.

“In our work on E3SM-MMF, we’ve been able to modify codes to run on exascale GPU-accelerated systems and obtain useful speedups,” said Jones. “We can likely achieve another 2-times speed up soon. That’s significant because climate codes are not necessarily well-suited for these architectures, and it takes a lot of work to get a little improvement.”

The ocean and ice components for E3SM-MMF include the Model for Prediction Across Scales (MPAS), Ocean (MPAS-O), MPAS Sea Ice (MPAS-SI), and MPAS Land Ice (MPAS-LI). These utilize the same MPAS software framework that provides a capability to focus resolution and computational power on regions of the globe where ocean and ice interactions are important.

A central challenge to advancing modeling and simulation of earth systems is coupling the different scales between the global atmospheric model and the cloud-resolving fine scale model. Similarly, the smaller scales of the ocean and ice systems need to be incorporated into the overall modeling and simulation system. Ocean heat storage and transport are important pieces of the climate system and small-scale ocean eddies are critical to the dynamics of ocean circulation. The impacts of Arctic sea-ice retreat and projected sea-level rise associated with large ice sheets also require representing fine-scale processes within the ice and at the ice edges. Exascale improvements will enable resolving some of these processes and achieve a better representation of the climate.

The E3SM-MMF team looks to continue to maximize the throughput possible. In a project where simulation capability aims to be multi-decadal, the metric for measuring throughput is years able to be simulated in a CPU day. Currently the throughput is as high as one simulated year per day at 3-km resolution, and Jones is optimistic for improvement in that with more process improvements in the pipeline.

“A difficulty has been in finding enough parallelism and enough work for these new processors to operate efficiently on,” said Jones. “We have to find new algorithms and new ways of using these architectures efficiently, including breaking up horizontal meshes to distribute the work around. I think, project-wide, and including the E3SM-MMF piece, we’re all on track.”


E3SM simulation from 2018 representing sea surface temperature changes evolving as a hurricane (seen here approaching the U.S. East Coast) moves across the Atlantic and how the resultant cold wake affects subsequent intensification of the next hurricane.