-
DropCompute: simple and more robust distributed synchronous training via compute variance reduction
Authors:
Niv Giladi,
Shahar Gottlieb,
Moran Shkolnik,
Asaf Karnieli,
Ron Banner,
Elad Hoffer,
Kfir Yehuda Levy,
Daniel Soudry
Abstract:
Background: Distributed training is essential for large scale training of deep neural networks (DNNs). The dominant methods for large scale DNN training are synchronous (e.g. All-Reduce), but these require waiting for all workers in each step. Thus, these methods are limited by the delays caused by straggling workers. Results: We study a typical scenario in which workers are straggling due to vari…
▽ More
Background: Distributed training is essential for large scale training of deep neural networks (DNNs). The dominant methods for large scale DNN training are synchronous (e.g. All-Reduce), but these require waiting for all workers in each step. Thus, these methods are limited by the delays caused by straggling workers. Results: We study a typical scenario in which workers are straggling due to variability in compute time. We find an analytical relation between compute time properties and scalability limitations, caused by such straggling workers. With these findings, we propose a simple yet effective decentralized method to reduce the variation among workers and thus improve the robustness of synchronous training. This method can be integrated with the widely used All-Reduce. Our findings are validated on large-scale training tasks using 200 Gaudi Accelerators.
△ Less
Submitted 24 September, 2023; v1 submitted 18 June, 2023;
originally announced June 2023.
-
Automated Search for Conjectures on Mathematical Constants using Analysis of Integer Sequences
Authors:
Ofir Razon,
Yoav Harris,
Shahar Gottlieb,
Dan Carmon,
Ofir David,
Ido Kaminer
Abstract:
Formulas involving fundamental mathematical constants had a great impact on various fields of science and mathematics, for example aiding in proofs of irrationality of constants. However, the discovery of such formulas has historically remained scarce, often perceived as an act of mathematical genius by great mathematicians such as Ramanujan, Euler, and Gauss. Recent efforts to automate the discov…
▽ More
Formulas involving fundamental mathematical constants had a great impact on various fields of science and mathematics, for example aiding in proofs of irrationality of constants. However, the discovery of such formulas has historically remained scarce, often perceived as an act of mathematical genius by great mathematicians such as Ramanujan, Euler, and Gauss. Recent efforts to automate the discovery of formulas for mathematical constants, such as the Ramanujan Machine project, relied on exhaustive search. Despite several successful discoveries, exhaustive search remains limited by the space of options that can be covered and by the need for vast amounts of computational resources. Here we propose a fundamentally different method to search for conjectures on mathematical constants: through analysis of integer sequences. We introduce the Enumerated Signed-continued-fraction Massey Approve (ESMA) algorithm, which builds on the Berlekamp-Massey algorithm to identify patterns in integer sequences that represent mathematical constants. The ESMA algorithm found various known formulas for $e, e^2, tan(1)$, and ratios of values of Bessel functions. The algorithm further discovered a large number of new conjectures for these constants, some providing simpler representations and some providing faster numerical convergence than the corresponding simple continued fractions. Along with the algorithm, we present mathematical tools for manipulating continued fractions. These connections enable us to characterize what space of constants can be found by ESMA and quantify its algorithmic advantage in certain scenarios. Altogether, this work continues in the development of augmenting mathematical intuition by computer algorithms, to help reveal mathematical structures and accelerate mathematical research.
△ Less
Submitted 11 June, 2023; v1 submitted 13 December, 2022;
originally announced December 2022.
-
Performance Evaluation of Mixed-Precision Runge-Kutta Methods
Authors:
Ben Burnett,
Sigal Gottlieb,
Zachary J. Grant,
Alfa Heryudono
Abstract:
Additive Runge-Kutta methods designed for preserving highly accurate solutions in mixed-precision computation were proposed and analyzed in [8]. These specially designed methods use reduced precision or the implicit computations and full precision for the explicit computations. We develop a FORTRAN code to solve a nonlinear system of ordinary differential equations using the mixed precision additi…
▽ More
Additive Runge-Kutta methods designed for preserving highly accurate solutions in mixed-precision computation were proposed and analyzed in [8]. These specially designed methods use reduced precision or the implicit computations and full precision for the explicit computations. We develop a FORTRAN code to solve a nonlinear system of ordinary differential equations using the mixed precision additive Runge-Kutta (MP-ARK) methods on IBM POWER9 and Intel x86\_64 chips. The convergence, accuracy, runtime, and energy consumption of these methods is explored. We show that these MP-ARK methods efficiently produce accurate solutions with significant reductions in runtime (and by extension energy consumption).
△ Less
Submitted 7 July, 2021;
originally announced July 2021.
-
The Ramanujan Machine: Automatically Generated Conjectures on Fundamental Constants
Authors:
Gal Raayoni,
Shahar Gottlieb,
George Pisha,
Yoav Harris,
Yahel Manor,
Uri Mendlovic,
Doron Haviv,
Yaron Hadad,
Ido Kaminer
Abstract:
Fundamental mathematical constants like $e$ and $π$ are ubiquitous in diverse fields of science, from abstract mathematics to physics, biology and chemistry. For centuries, new formulas relating fundamental constants have been scarce and usually discovered sporadically. Here we propose a novel and systematic approach that leverages algorithms for deriving mathematical formulas for fundamental cons…
▽ More
Fundamental mathematical constants like $e$ and $π$ are ubiquitous in diverse fields of science, from abstract mathematics to physics, biology and chemistry. For centuries, new formulas relating fundamental constants have been scarce and usually discovered sporadically. Here we propose a novel and systematic approach that leverages algorithms for deriving mathematical formulas for fundamental constants and help reveal their underlying structure. Our algorithms find dozens of well-known as well as previously unknown continued fraction representations of $π$, $e$, Catalan's constant, and values of the Riemann zeta function. Two example conjectures found by our algorithm and so far unproven are: \begin{equation*} \frac{24}{π^2} = 2 + 7\cdot 0\cdot 1+ \frac{8\cdot1^4}{2 + 7\cdot 1\cdot 2 + \frac{8\cdot2^4}{2 + 7\cdot 2\cdot 3 + \frac{8\cdot3^4}{2 + 7\cdot 3\cdot 4 + \frac{8\cdot4^4}{..}}}} \quad\quad,\quad\quad \frac{8}{7 ζ(3)} = 1\cdot 1 - \frac{1^6}{3\cdot 7 - \frac{2^6}{5\cdot 19 - \frac{3^6}{7\cdot 37 - \frac{4^6}{..}}}} \end{equation*} We present two algorithms that proved useful in finding conjectures: a Meet-In-The-Middle (MITM) algorithm and a Gradient Descent (GD) tailored to the recurrent structure of continued fractions. Both algorithms are based on matching numerical values and thus they conjecture formulas without providing proofs and without requiring prior knowledge on any underlying mathematical structure. This approach is especially attractive for constants for which no mathematical structure is known, as it reverses the conventional approach of sequential logic in formal proofs. Instead, our work supports a different approach for research: algorithms utilizing numerical data to unveil mathematical structures, thus trying to play the role of intuition of great mathematicians of the past, providing leads to new mathematical research.
△ Less
Submitted 30 April, 2020; v1 submitted 29 June, 2019;
originally announced July 2019.
-
High Energy Physics Forum for Computational Excellence: Working Group Reports (I. Applications Software II. Software Libraries and Tools III. Systems)
Authors:
Salman Habib,
Robert Roser,
Tom LeCompte,
Zach Marshall,
Anders Borgland,
Brett Viren,
Peter Nugent,
Makoto Asai,
Lothar Bauerdick,
Hal Finkel,
Steve Gottlieb,
Stefan Hoeche,
Paul Sheldon,
Jean-Luc Vay,
Peter Elmer,
Michael Kirby,
Simon Patton,
Maxim Potekhin,
Brian Yanny,
Paolo Calafiura,
Eli Dart,
Oliver Gutsche,
Taku Izubuchi,
Adam Lyon,
Don Petravick
Abstract:
Computing plays an essential role in all aspects of high energy physics. As computational technology evolves rapidly in new directions, and data throughput and volume continue to follow a steep trend-line, it is important for the HEP community to develop an effective response to a series of expected challenges. In order to help shape the desired response, the HEP Forum for Computational Excellence…
▽ More
Computing plays an essential role in all aspects of high energy physics. As computational technology evolves rapidly in new directions, and data throughput and volume continue to follow a steep trend-line, it is important for the HEP community to develop an effective response to a series of expected challenges. In order to help shape the desired response, the HEP Forum for Computational Excellence (HEP-FCE) initiated a roadmap planning activity with two key overlap** drivers -- 1) software effectiveness, and 2) infrastructure and expertise advancement. The HEP-FCE formed three working groups, 1) Applications Software, 2) Software Libraries and Tools, and 3) Systems (including systems software), to provide an overview of the current status of HEP computing and to present findings and opportunities for the desired HEP computational roadmap. The final versions of the reports are combined in this document, and are presented along with introductory material.
△ Less
Submitted 28 October, 2015;
originally announced October 2015.
