Subfiling and Multiple dataset APIs: An introduction to two new features in HDF5 version 1.14
For parallel I/O, the principle behind Subfiling is to find the middle ground between a single shared file and one file per process, thereby avoiding the complexity of one file per process and minimizing the locking issues of a single shared file on a parallel file system. The first part of the talk will cover Subfiling’s implementation, its usage, and the performance benefits observed compared to a single shared file. The second part of the talk will introduce new HDF5 multiple dataset APIs and highlight the performance benefits when using them. The HDF5 library allows a data access operation to access one dataset at a time. However, accessing multiple datasets requires the user to issue an I/O call for each dataset. Hence, the new multiple dataset APIs allow users to access multiple datasets with a single I/O call. In addition, the new routines can improve performance, especially when data is accessed across several datasets from all processes.
Presenters: Neil Fortner and Jordan Henderson
The webinar will be held on September 30, 2022.
The IDEAS Productivity project, in partnership with the DOE Computing Facilities of the ALCF, OLCF, and NERSC, and the DOE Exascale Computing Project (ECP), organizes the webinar series on Best Practices for HPC Software Developers.
As part of this series, we offer one-hour webinars on topics in scientific software development and high-performance computing, approximately once a month. The October webinar is titled Investing in Code Reviews for Better Research Software; and will be presented by Thibault Lestang (Imperial College London), Dominik Krzemiński (University of Cambridge), and Valerio Maggio (Software Sustainability Institute). The webinar will take place on Wednesday, October 12, 2022 at 1:00 pm ET.
Abstract:
Code review is a development practice that improves readability and maintainability of software projects, in addition to making collaboration easier and teamwork more effective. Typically, code review is a conversation between reviewer(s) and the author(s) of the code under review. The code is dissected and analyzed in order to find areas of improvement according to the focus of the review. Examples include, but are not limited to, readability, security or performance improvements. Despite code review being an effective tool for improving software quality, it is still not a standard practice within the scientific software development process. The webinar will detail the benefits that code review can bring to scientific software developers, particularly improvements in software quality, improved teamwork and knowledge transfer. The presenters will highlight common difficulties faced by researchers to set up, perform and maintain frequent code reviews, and they will discuss several approaches and good practices to mitigate these difficulties. The presenters will also describe common tools that make code reviews easier and give examples of how to use them effectively, while explaining a typical code development cycle with continuous integration and automatic code checks.
The IDEAS Productivity project, in partnership with the DOE Computing Facilities of the ALCF, OLCF, and NERSC, and the DOE Exascale Computing Project (ECP), organizes the webinar series on Best Practices for HPC Software Developers.
As part of this series, we offer one-hour webinars on topics in scientific software development and high-performance computing, approximately once a month. The November webinar is titled Managing Academic Software Development; and will be presented by Sam Mangham (University of Southampton). The webinar will take place on Wednesday, November 9, 2022 at 1:00 pm ET.
Abstract:
Developing academic software can be an unusual exercise, especially compared to traditional software development. The goals and inputs can be undefined and fluctuating, whilst the code itself has traditionally been a stepping stone – a byproduct on the way to papers, ending up ad-hoc, unplanned and undocumented. Fortunately, things are changing. There are tools and techniques that make it easier to design, use, distribute and cite scientific software. This webinar discusses approaches to managing the development and release of academic software, ranging from coding best practices and project boards, to development environments and automated documentation that can help you write sustainable code that is easy to use, cite and collaborate with and on.
The IDEAS Productivity project, in partnership with the DOE Computing Facilities of the ALCF, OLCF, and NERSC, and the DOE Exascale Computing Project (ECP), organizes the webinar series on Best Practices for HPC Software Developers.
As part of this series, we offer one-hour webinars on topics in scientific software development and high-performance computing, approximately once a month. The December webinar is titled Lab Notebooks for Computational Mathematics, Sciences & Engineering; and will be presented by Jared O’Neal (Argonne National Laboratory). The webinar will take place on Wednesday, December 14, 2022 at 1:00 pm ET.
Abstract:
As computational mathematics, science, and engineering problems become larger, more ambitious, and more complex, it is increasingly important to develop and use tools and techniques that ensure that computational research is based on a strong foundation of general, low-level scientific best practices. In this webinar, the speaker will relate his experience of transitioning from working in the worlds of experimental and observational sciences to the world of computational sciences as well as his experience adapting experimental tools and techniques to computational research. In particular, the speaker will focus on the role of lab notebooks in experimental sciences and present concrete examples to address the challenges associated with adapting lab notebooks to computational research.
The IDEAS Productivity project, in partnership with the DOE Computing Facilities of the ALCF, OLCF, and NERSC, and the DOE Exascale Computing Project (ECP), organizes the webinar series on Best Practices for HPC Software Developers.
As part of this series, we offer one-hour webinars on topics in scientific software development and high-performance computing, approximately once a month. The January webinar is titled Openscapes: supporting better science for future us; and will be presented by Julia Stewart Lowndes (Openscapes). The webinar will take place on Wednesday, January 11, 2023 at 1:00 pm ET.
Abstract:
Openscapes champions open practices in environmental science to help uncover data-driven solutions faster. In this webinar the speaker will share how she transitioned from doing her own marine ecology research to founding Openscapes to support other researchers and grow the global Open Science movement. The speaker will share lessons learned from her work mentoring government, non-profit, and academic environmental and Earth teams, with specific stories from projects with NASA and NOAA Fisheries. The webinar will reuse parts of a recent keynote at RStudio::conf that was the global launch of Quarto, a new, open-source, scientific and technical publishing system. The webinar will include a demo on some features of Quarto for R and Python users and highlight how more reusing and less reinventing is critical for science. The speaker will also discuss how open source/science is a daily practice, and an important avenue to increase inclusion in science and contribute to the climate movement.
The IDEAS Productivity project, in partnership with the DOE Computing Facilities of the ALCF, OLCF, and NERSC, and the DOE Exascale Computing Project (ECP), organizes the webinar series on Best Practices for HPC Software Developers.
As part of this series, we offer one-hour webinars on topics in scientific software development and high-performance computing, approximately once a month. The March webinar is titled Our Road to Exascale: Particle Accelerator & Laser-Plasma Modeling; and will be presented by Axel Huebl (Lawrence Berkeley National Laboratory). The webinar will take place on Wednesday, March 15, 2023, at 1:00 pm ET.
Abstract:
Particle accelerators, among the largest, most complex devices, demand increasingly sophisticated computational tools for the design and optimization of the next generation of accelerators that will meet the challenges of increasing energy, intensity, accuracy, compactness, complexity and efficiency. It is key that contemporary software take advantage of the latest advances in computer hardware and scientific software engineering practices, delivering speed, reproducibility and feature composability for the aforementioned challenges.
The webinar will discuss the experience of the developers of WarpX in the US DOE Exascale Computing Project (ECP), which led to the 2022 ACM Gordon Bell Prize. Including the first Exascale supercomputer Frontier, WarpX uses GPUs and CPUs at massive scale; research efforts have advanced particle-in-cell algorithms such as dynamic load balancing, block-structured mesh-refinement, and modern relativistic Maxwell solvers. The webinar will present strategies and results in performance portability. In particular, the webinar will discuss the team-of-teams approach for software co-design in AMReX, software architecture, quality assurance, developer & user productivity, and ecosystem interplay that has lifted up accelerator modeling activities to be fast, open, modular and sustainable over the long term.
The IDEAS Productivity project, in partnership with the DOE Computing Facilities of the ALCF, OLCF, and NERSC, and the DOE Exascale Computing Project (ECP), organizes the webinar series on Best Practices for HPC Software Developers.
As part of this series, we offer one-hour webinars on topics in scientific software development and high-performance computing, approximately once a month. The April webinar is titled Facilitating Electronic Structure Calculations on GPU-based Exascale Platforms; and will be presented by Jean-Luc Fattebert (Oak Ridge National Laboratory). The webinar will take place on Wednesday, April 12, 2023, at 1:00 pm ET.
Abstract:
GPUs accelerators offer the prospect of speeding up ab initio molecular dynamics and other large-scale first-principles atomistic simulations. Taking advantage of these devices is, however, not a trivial task given their specificities. Some algorithms struggle, while others thrive with the high level of thread concurrency available on modern GPUs. The PROGRESS and BML libraries, developed within ECP’s Co-design Center for Particle Applications (CoPA) project, allow electronic structure codes to offload their most expensive kernels, with a unified interface for various matrix formats and computer architectures. The webinar will focus on implementations and algorithmic choices made in those libraries, and lessons learned while trying to achieve performance portability on exascale platforms. Specifically, the webinar will discuss eigensolvers and their alternatives, as well as strong scaling in fast time-to-solution in molecular dynamics.
The IDEAS Productivity project, in partnership with the DOE Computing Facilities of the ALCF, OLCF, and NERSC, and the DOE Exascale Computing Project (ECP), organizes the webinar series on Best Practices for HPC Software Developers.
As part of this series, we offer one-hour webinars on topics in scientific software development and high-performance computing, approximately once a month. The May webinar is titled Lessons Learned Developing Performance Portable QMCPACK; and will be presented by Paul Kent (Oak Ridge National Laboratory). The webinar will take place on Wednesday, May 10, 2023, at 1:00 pm ET.
Abstract:
During DOE’s Exascale Computing Project the open source QMCPACK code has been redesigned and reimplemented to run portably and performantly on multiple vendors GPUs as well as CPUs. The QMCPACK code implements Quantum Monte Carlo algorithms to predict the properties of materials with benchmark accuracy. The new implementation has now fully replaced the prior non-portable GPU solution. This webinar will outline some of the design considerations and new algorithms implemented both to run efficiently and to reduce burdens on the developers and maintainers. A key factor has been the adoption of modern development practices, including an extensive test suite. This has accelerated development, improved code quality, and also enabled isolation of problems in the wider HPC software stack, including in compilers and numerical libraries. The webinar will summarize these strategies and other recommendations for HPC application developers and facilities.
The IDEAS Productivity project, in partnership with the DOE Computing Facilities of the ALCF, OLCF, and NERSC, and the DOE Exascale Computing Project (ECP), organizes the webinar series on Best Practices for HPC Software Developers.
As part of this series, we offer one-hour webinars on topics in scientific software development and high-performance computing, approximately once a month. The June webinar is titled The OpenSSF Best Practices Badge Program; and will be presented by Roscoe A. Bartlett (Sandia National Laboratories). The webinar will take place on Wednesday, June 14, 2023, at 1:00 pm ET.
Abstract:
The Linux Foundation’s OpenSSF Best Practices Badge Program represents an impressive collection of the open source community’s knowledge base for creating, maintaining, and sustaining robust, high quality, and (most importantly) secure open source software. At its foundation is a featureful “Badge App” website, which provides a database of projects that document what best practices they have adopted and supporting evidence. This set of best practices (along with the detailed documentation and supporting justifications for each item) also serves as an incremental learning tool and as a foundation for incremental software process and quality improvements efforts. The webinar will provide an overview of this effort and describe some of its surprising benefits. The webinar will also describe how the OpenSSF Best Practices Badge Program can be used to help continue the recent advances in software quality and sustainability efforts in the computational science and engineering community going forward.
The IDEAS Productivity project, in partnership with the DOE Computing Facilities of the ALCF, OLCF, and NERSC, and the DOE Exascale Computing Project (ECP), organizes the webinar series on Best Practices for HPC Software Developers.
As part of this series, we offer one-hour webinars on topics in scientific software development and high-performance computing, approximately once a month. The July webinar is titled Writing Clean Scientific Software; and will be presented by Nick Murphy (Center for Astrophysics, Harvard & Smithsonian). The webinar will take place on Wednesday, July 12, 2023, at 1:00 pm ET.
Abstract:
Most scientists are largely self-taught as programmers. Even many of us who spend most of our time coding have never had formal training in writing software. This webinar is intended for students and scientists who have some experience writing code but who have had to learn mostly on their own. The webinar will describe tips and strategies on how to write readable, reusable, and maintainable code. These tips include writing short functions that do exactly one thing with no effects, and measuring the length of a variable name by the time needed to understand its meaning rather than by number of characters. The webinar will describe strategies for restructuring a complicated function into smaller and more manageable chunks, and provide tips on how to make the best use of comments and error messages. Overall, the webinar will embolden the CS&E community to think of code as communication.
