With Shared Microexponents, A Little Shifting Goes a Long Way
Authors:
Bita Rouhani,
Ritchie Zhao,
Venmugil Elango,
Rasoul Shafipour,
Mathew Hall,
Maral Mesmakhosroshahi,
Ankit More,
Levi Melnick,
Maximilian Golub,
Girish Varatkar,
Lei Shao,
Gaurav Kolhe,
Dimitry Melts,
Jasmine Klar,
Renee L'Heureux,
Matt Perry,
Doug Burger,
Eric Chung,
Zhaoxia Deng,
Sam Naghshineh,
Jongsoo Park,
Maxim Naumov
Abstract:
This paper introduces Block Data Representations (BDR), a framework for exploring and evaluating a wide spectrum of narrow-precision formats for deep learning. It enables comparison of popular quantization standards, and through BDR, new formats based on shared microexponents (MX) are identified, which outperform other state-of-the-art quantization approaches, including narrow-precision floating-p…
▽ More
This paper introduces Block Data Representations (BDR), a framework for exploring and evaluating a wide spectrum of narrow-precision formats for deep learning. It enables comparison of popular quantization standards, and through BDR, new formats based on shared microexponents (MX) are identified, which outperform other state-of-the-art quantization approaches, including narrow-precision floating-point and block floating-point. MX utilizes multiple levels of quantization scaling with ultra-fine scaling factors based on shared microexponents in the hardware. The effectiveness of MX is demonstrated on real-world models including large-scale generative pretraining and inferencing, and production-scale recommendation systems.
△ Less
Submitted 12 April, 2023; v1 submitted 15 February, 2023;
originally announced February 2023.
The openCARP CDE -- Concept for and implementation of a sustainable collaborative development environment for research software
Authors:
Felix Bach,
Jochen Klar,
Axel Loewe,
Jorge Sánchez,
Gunnar Seemann,
Yung-Lin Huang,
Robert Ulrich
Abstract:
This work describes the setup of an advanced technical infrastructure for collaborative software development (CDE) in large, distributed projects based on GitLab. We present its customization and extension, additional features and processes like code review, continuous automated testing, DevOps practices, and sustainable life-cycle management including long-term preservation and citable publishing…
▽ More
This work describes the setup of an advanced technical infrastructure for collaborative software development (CDE) in large, distributed projects based on GitLab. We present its customization and extension, additional features and processes like code review, continuous automated testing, DevOps practices, and sustainable life-cycle management including long-term preservation and citable publishing of software releases along with relevant metadata. The environment is currently used for develo** the open cardiac simulation software openCARP and an evaluation showcases its capability and utility for collaboration and coordination of sizeable heterogeneous teams. As such, it could be a suitable and sustainable infrastructure solution for a wide range of research software projects.
△ Less
Submitted 12 January, 2022;
originally announced January 2022.
