Pagoda: UPC++/GASNet

A computation being performed on one part of a large system often needs to access or provide data to another part of the system in order to complete a scientific simulation. The Partitioned Global Address Space (PGAS) model provides the appearance of shared memory accessible to all the compute nodes while implementing this shared memory behind the scenes using physical memory local to the nodes and primitives such as remote direct memory access (RDMA). The Pagoda project, as part of the US Department of Energy (DOE) Exascale Computing Project (ECP) is developing a productive and performant PGAS programming system to be deployed on exascale systems.

Project Details

The Pagoda project is developing a programming system to support exascale application development using the PGAS model, with a focus on supporting applications with irregular data structures and communication. There are two components to Pagoda: (1) a portable, high- performance, global-address-space communication library (GASNet-EX) and (2) a C++ template library (UPC++) that provides convenient methods for access and using the global address space. Together, these components enable the agile, lightweight communications that occur in applications, libraries, and frameworks running on exascale systems.

Pagoda enables effective scaling by avoiding the overhead of long, branchy serial code paths, leveraging hardware-offload support for RDMA, and supporting efficient fine-grained communication for both single- and multi-threaded environments. The importance of these properties is exacerbated by application trends; many applications in the ECP require the use of adaptive meshes, sparse matrices, dynamic load balancing, or similar techniques. Pagoda’s low-overhead communication mechanisms can maximize the injection rate and network utilization, tolerate latency through overlap, streamline unpredictable communication events, minimize synchronization, and efficiently support small- to medium-sized messages arising in such applications. Pagoda complements other programming models, enabling developers to focus their efforts on optimizing performance-critical communications.

The Pagoda team is focusing on developing new features that will support application and library requirements unique to the ECP and performance improvements that will enable the ECP software stack to exploit the best-available communication mechanisms, including novel features being developed by vendors, such as remote direct memory access mechanisms offered by network hardware and on-chip communication between distinct address spaces.

Principal Investigator(s):

Paul Hargrove, Lawrence Berkeley National Laboratory

Progress to date

  • The Pagoda team provides regular releases of their PGAS communication and C++ template libraries that typically include new features and performance improvements.
  • The C++ template library developed by the team supports remote direct memory access, remote procedure calls, future- based asynchrony, and remote atomics with offload to network hardware.
  • The team delivered new features in the C++ template library, including support for subset teams and collectives, and non-trivial serialization of user-defined types.
  • Developments in both libraries have enabled direct data movement between host and accelerator memories (local and remote), without host CPU involvement, on supported hardware.

National Nuclear Security Administration logo U.S. Department of Energy Office of Science logo