Helping Users Deploy High-Performance Computing Tools and Libraries in a Consistent Interoperable Software Stack

By Scott Gibson

Sameer Shende of the University of Oregon

Hi and welcome. This is where we explore the efforts of the Department of Energy’s (DOE’s) Exascale Computing Project (ECP)—from the development challenges and achievements to the ultimate expected impact of exascale computing on society.

In this episode, we get an update report on the Extreme-scale Scientific Software Stack (E4S). ECP supports the E4S project. E4S is a community effort to provide open-source software packages for developing, deploying, and running scientific applications on high-performance computing platforms. The packages distributed with E4S contain contributions from hundreds of open-source community developers.

Joining us is Sameer Shende. Sameer is a research associate professor and director, Performance Research Laboratory, University of Oregon. And for ECP he is principal investigator for Programming Models and Runtime for Software Development Kits (SDKs) project. That project is helping create E4S—which includes a subset of ECP Software Technology (ST) products and demonstrates the target approach for future delivery of the full ECP ST software stack.

Our topics: A review of what E4S is all about, the very important support for GPU architectures, and more.

Interview Transcript

Gibson: Sameer, will you get this discussion started with an overview of E4S?

Shende: Sure. The Extreme-scale Scientific software stack is a community effort to provide open source software packages for developers and it’s a platform a platform for delivering well-tested, interoperable HPC products.

It’s a Spack-based distribution of software that is tested for interoperability and portability to multiple architectures. Spack is the basis of E4S and a mature package manager that is used within E4S to manage different versions of products and interoperability and access to compilers and runtimes. So, starting with our initial release in October 2018, where we had 24 full E4S products, we now have a distribution of E4S that was just released at Supercomputing, the E4S 21.11, which has 91 full-release products across the x86_64 and ppc64le.

There are a wide range of products here covering areas such as numerical libraries, development tools, performance evaluation tools, runtime systems, data, and visualization tools. And it provides a platform where you can get a release of E4S and deploy it very easily on your systems.

Gibson: Who can use E4S?

Shende: The E4S users are the developers of HPC [high-performance computing] and AI/ML software. They are software developers. They are application developers. The users include systems integrators. It includes staff that manage HPC, systems, and a broad range of users across different communities.

The E4S 21.11 has support for three different GPU architectures. As you know, the Exascale Computing Project is relying on GPUs as a primary mechanism for getting large-scale computing power, and we have support in this release for Nvidia GPUs, including the A100s, and support for AMD GPUs, including the MI100 GPUs, and also Intel GPUs. There are runtime layers corresponding to each of these GPU architectures within the E4S 21.11 release. For instance, we have support for CUDA 11.4.2, and we have Envy HPC compilers. We have support for 21.9 version of Envy HPC. We have support for Intel compilers and Intel toolkits from the 2021.4 latest release, and also for ROCm 4.3 from AMD. So, we have support for all three architectures, and we have big products from E4S for different GPU architectures. And, thanks to Spack, they can coexist in a single distribution.

So, we have containers for Docker and Singularity, which include all three runtimes with all three products, which are ported to those different GPU architectures, as well as tools, tools such as Tau, which support all three GPU architectures in this release. And we can use individual containers which have support for only one GPU architecture or a fully featured container which has support for all three of them. So, you can customize your container and pick and choose the different packages from E4S for specific GPU architectures or you can create a customer container based on the base images that you provide which are well tested on these three different GPU architectures. Or you can just download a full image which has not only the three GPU runtimes but also the 91 different E4S products along with other AI/ML software such as TensorFlow and PyTorch which is ported to the different GPU architectures.

What is important to note here is that as our software becomes more complex, it’s getting harder and harder for individual users to deploy our HPC tools and libraries in a consistent interoperable software stack, and E4S helps that process. Not only can you use containers, [but] you can also deploy these packages on bare metal systems. You can use Spack as the package manager with the recipes that come from E4S to easily deploy this full software stack on bare metal systems or just download a container and image from the e4s.io web page and deploy a container based on that image.

We also have support for the AWS and ECP cloud systems that you can run a full image from E4S and AMI on an AWS system and get a remote desktop.

Gibson: Sameer, is there anything else you’d like to discuss about the latest release of E4S?

Shende: To help with the composition of applications, E4S provides a binary build cache for Spack, and we currently have over 75,000 binaries in this build cache of tools and E4S products. So, you can simplify the installation of your own custom software stack or build applications by pointing to the E4S Spack build cache and get a significant improvement in the time it takes to install these components, because these are just binaries that you can get through Spack and create your own custom software stack based on the packages. And if you are building some components in source form, like your application, and all of the dependencies are there in Spack in the binary build cache, then you can download those. And they’re well tested. They are ported to different GPU architectures. Depending on your configuration, you can use them and create custom images.

Gibson: Wow, that sounds very flexible and versatile. Will you remind users where they can get E4S? So they would go to e4s.io?

Shende: They would go to the E4S web page, which is e4s.io, and then download the containers over there and access the docker hub, the other build cache, and other resources that are available. We also have tutorials and videos that you can watch and learn more about E4S.

Gibson: Hey, Sameer, thanks so much for taking the time to share this excellent information with us.

Shende: Thank you, Scott.