Researchers supported by the Exascale Computing Project (ECP) conducted the first comprehensive review of research examining the usefulness of mixed-precision algorithms to power exascale computations. Their paper was published in the March 2021 edition of the International Journal of High Performance Computing Applications.
There is a long history of efforts aiming to improve the performance of numerical algorithms by combining precision formats. The team scanned the literature for the best methods of combining low-precision and high-precision algorithms to yield mixed-precision strategies that deliver high computation speed with sufficient levels of precision.
The team noted that growing demand in the machine learning (ML) community for high compute power in low-precision formats is impacting both hardware and software design. Hardware vendors such as NVIDIA have designed low-precision special function units to power ML applications that provide an order of magnitude higher compute power than is possible with IEEE 754 double-precision converters but offer less precise outputs due to information loss.
Depending on the problem characteristics, mixed-precision algorithms can solve linear systems of equations up to four times faster on the Summit supercomputer than traditional fixed-precision linear solvers. Because speedups typically transfer to new processor architectures, new hardware functionality such as tensor cores can be leveraged to achieve higher performance on current supercomputers, and open-source algorithms that provide attractive speedups on all hardware, including future supercomputers, are being developed.
This research addresses the ECP’s interest in mixed-precision methods for designing and engineering novel algorithms.
Abdelfattah, A., H. Anzt, E.G. Boman, E. Carson, T. Cojean, J. Dongarra, A. Fox, M. Gates, N.J. Higham, X.S. Li, J. Loe, P. Luszczek, S. Pranesh, S. Rajamanickam, T. Ribizel, B.F. Smith, K. Swirydowicz, S. Thomas, S. Tomov, Y.M. Tsai, and U. Meier Yang. “A survey of numerical linear algebra methods utilizing mixed-precision arithmetic.” International Journal of High Performance Computing Applications (March 2021).
