E4S and SDK Efforts

The many software technologies being delivered to the Exascale Computing Project (ECP) application developers pose challenges, especially if the application uses several software technology products. The software development kit (SDK) efforts identify meaningful aggregations of products within the programming models and runtimes, development tools, math libraries, and data and visualization technical areas. They also leverage composable scientific workflow technologies with the goal of increasing the interoperability, availability, quality, and sustainability of the software technologies being developed in the ECP while improving developer productivity for the software and application development teams. The SDKs are packaged and delivered through the Extreme-Scale Scientific Software Stack (E4S).

Project Details

The forthcoming exascale systems require a sustainable, high-quality software ecosystem, and the ECP is chartered with delivering such an ecosystem that will continuously be improved by a robust research and development effort, deployed on advanced computing platforms, and broadly adopted by application teams and software developers to accelerate their science. The E4S and SDK efforts support a variety of activities, including establishing community policies aimed at increasing the interoperability between and sustainability of software technologies developed by the ECP and coordinating the delivery of those products through the E4S.

The Programming Models and Runtimes SDK effort identifies meaningful aggregations of products in this technical area. It provides the software infrastructure needed to enable and accelerate the development of exascale applications that perform well and are correct and robust while reducing the cost of initial development, ongoing porting, and maintenance.

The Development Tools SDK is a collection of independent projects specifically targeted to address performance analysis at scale. The team actively works to leverage techniques for common and identified problem patterns and creates new techniques for software quality assurance related to performance analysis tools while also supporting advanced techniques, such as autotuning and compiler integration, for upcoming heterogeneous architectures.

The Math Libraries SDK, or Extreme-Scale Scientific Software Development Kit (xSDK), predates ECP and is the largest ECP SDK effort. See more under Math Libraries.

The Data and Visualization SDK focuses on the delivery of efficient data management and storage libraries; services such as checkpoint/restart, monitoring, code coupling, and compression; and an efficient in situ visualization and analysis pipeline. The goal is to improve the deployment and usage of I/O and analysis capabilities.

The Scientific Workflows SDK is a new effort to create a rigorously tested community-curated collection of composable workflow technologies. The goal of this SDK is to ensure the exascale readiness of many scientific and other relevant ECP simulation workflows and improve their long-term sustainability. Toward this goal, the team is applying a community approach to design the SDK, integrate libraries and components, and define component interfaces.

The Software Ecosystem SDK effort coordinates activities between the various SDK projects, manages the release and testing of the E4S, and ensures that the software technologies within E4S can be built from source via the Spack package manager or used via prebuilt container images. Application developers can build the subset of the software technologies needed for their specific application. This effort fosters collaboration between software technologies and interacts heavily with the Hardware and Integration (HI) focus area to facilitate software product installation at the facilities.

Principal Investigator(s):

Sameer Shende, University of Oregon; Bart Miller, University of Wisconsin—Madison; Patrick O'Leary, Kitware Inc.; Dan Laney, Lawrence Livermore National Laboratory; Jim Willenbring, Sandia National Laboratories; Ulrike Yang, Lawrence Livermore National Laboratory; Todd Gamblin, Lawrence Livermore National Laboratory

Collaborators:

Argonne National Laboratory, Brookhaven National Laboratory, the HDF Group, Oak Ridge National Laboratory

Progress to date

  • E4S 23.02 was released, which contains 106 full products from across the software technologies and can be built from source via the Spack package manager or used via prebuilt container images and is available on AWS.
  • E4S has added extensive continuous integration (CI) testing. This testing is used to maintain interoperability and compatibility across the entire suite of E4S software and is a compliment to the CI testing done by individual software products.
  • The Data and Visualization SDK has established CI testing that is triggered by changes to the develop branch of Spack. This testing comprises about 160 separate package builds. The testing has already led to the quick resolution of many issues. This testing layer moves some of the burden of maintaining interoperability from the E4S team to staff who are more deeply familiar with data and visualization software.
  • The Tools SDK was verified to build on the pre-exascale systems at Argonne Leadership Computing Facility (ALCF) and National Energy Research Scientific Computing Center (NERSC), as well as the early-access system at ALCF. The SDK was integrated into the ALCF facility GitLab CI workflow and the Office of Scientific and Technical Information CI workflow hosted at NERSC. The test suites for two of the six SDK products were verified to successfully execute on the pre-exascale systems at NERSC, Lawrence Livermore National Laboratory, and ALCF.
  • Version 1 of the E4S community policies was released. These policies will serve as membership criteria for E4S and are modeled after the xSDK community policies.
  • This year, the winner and two of three finalists of the International Conference for High Performance Computing, Networking, Storage, and Analysis Gordon Bell Special Award for COVID-19 competition leveraged the scalable workflow technologies brought together by the ECP ExaWorks project.

National Nuclear Security Administration logo U.S. Department of Energy Office of Science logo