Software Technology

Data and Visualization

ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems

Principal Investigators: Suren Byna, lead, Lawrence Berkeley National Laboratory; Quincey Koziol, National Energy Research Scientific Computing Center; Scot Breitenfeld, The HDF Group; Venkat Vishwanath, Argonne National Laboratory (ANL); Preeti Malakar, ANL

This project is endeavoring to develop optimally performing parallel I/O strategies on upcoming exascale architectures, to maintain and optimize existing HDF5 features for ECP applications, and to release new features in HDF5 for broad deployment on HPC systems.

With the goal of addressing efficiency, fault-tolerance, and other challenges posed by data management and parallel I/O on exascale architectures, this will develop new capabilities in HDF5, the most popular parallel I/O library for scientific applications. This effort will productize features and techniques prototyped in the ExaHDF5 research and Fast Forward I/O projects, explore optimization strategies on upcoming architectures, maintain and optimize existing HDF5 features for ECP applications, and release these new features in HDF5 for broad deployment on HPC systems. Focusing on the challenges of exascale I/O, it will develop technologies based on the massively parallel storage hierarchies that are being built into pre-exascale systems. It will enhance HDF5 software to achieve efficient parallel I/O on exascale systems in ways that will impact a large number of DOE science applications. The HDF Group (THG) will release new HDF5 capabilities after rigorous software testing procedures.