Accelerating the Adoption of Container Technologies for Exascale Computing
By Scott Gibson
As the US Department of Energy’s (DOE) Exascale Computing Project (ECP) has evolved since its inception in 2016, what’s known as containers technology and how it fits into the wider scheme of exascale computing and high-performance computing (HPC) has been an area of ongoing interest in its own right within the HPC community.
Container technology has revolutionized software development and deployment for many industries and enterprises because it provides greater software flexibility, reliability, ease of deployment, and portability for users. But several challenges must be addressed to get containers ready for exascale computing.
The Supercontainers project, one of ECP’s newest efforts, aims to deliver containers and virtualization technologies for productivity, portability, and performance on the first exascale computing machines, which are planned for 2021.
ECP’s Let’s Talk Exascale podcast features as a guest Supercontainers project team member Andrew Younge of Sandia National Laboratories. The interview was recorded this past November in Denver at SC19: The International Conference for High Performance Computing, Networking, Storage, and Analysis.
Younge provided this perspective on containers technology:
“Essentially, it allows you to encompass your entire environment in a simple and reproducible way. So not only do I have my container image that has my application and my entire software stack with it, I also have a manifest for how I got there. That’s a really important notion for many people.”
The Supercontainers project is uniquely positioned by ECP’s collaborative framework to deliver first on the container runtimes research and development while helping colleagues across ECP leverage the technology, Younge said. And, he noted, the benefits will ripple across DOE labs.
“I have collaborators, and I’m able to leverage not only some of the great work that’s being done, say, at Berkeley Lab, but I’m also able to provide that in a centralized way to application developers across the entire DOE, from Brookhaven to Argonne, to any of the major labs,” he said.
One of the major objectives of the Supercontainers project is to provide training and outreach services to various ECP teams and to the DOE facilities. “It is getting our users familiar with and being able to leverage some of the modern tools with containers, working in sort of a DevOps model, which is a little bit different than how we’ve built HPC applications historically,” Younge said. The DevOps approach combines software development and information technology operations for efficiency in delivering the best quality software.
Younge shared some of the accomplishments of the Supercontainers team to date: It has developed a set of container images that can be deployed on a number of systems. And at least one container runtime is available on the vast majority of the pre-exascale machines. Performance and scalability numbers for the container runtimes have been very impressive. Also, some first images have been created with the Extreme-scale Software Stack (E4S).
“The good news is we already have this technology available today in some capacity,” he said. “We’re improving the scalability, and I think that’s one of the things that we’re actually really proud of. I’m surprised by how quickly we were able to demonstrate that containers don’t add fundamental overhead or performance impact.”
The team is optimizing the images and ensuring that the improvements reach the consumers of the container technologies—the ECP Application Development teams—immediately, and that the developers don’t have to fine-tune the solutions to meet their requirements, Younge said.
“They can take this cookie-cutter recipe or set of recipes and be able to quickly extend it to their needs without having to sacrifice performance,” he said. “There’s this continual balance between portability and performance.”
Open standards, interoperability, and the creation of tools that will, by design, outlast ECP, hold promise to be the central aspects of the enduring legacy of the Supercontainers project, Younge said.
ECP podcast episode 60: Simplifying the Deployment of High-Performance Computing Tools and Libraries, featuring Sameer Shende of the University of Oregon
The Department of Energy and the US DOE labs are working together to combat COVID-19. Learn more.