TACOS: Topology-Aware Collective Algorithm Synthesizer for Distributed Machine Learning
Authors:
William Won,
Midhilesh Elavazhagan,
Sudarshan Srinivasan,
Ajaya Durg,
Samvit Kaul,
Swati Gupta,
Tushar Krishna
Abstract:
The surge of artificial intelligence, specifically large language models, has led to a rapid advent towards the development of large-scale machine learning training clusters. Collective communications within these clusters tend to be heavily bandwidth-bound, necessitating techniques to optimally utilize the available network bandwidth. This puts the routing algorithm for the collective at the fore…
▽ More
The surge of artificial intelligence, specifically large language models, has led to a rapid advent towards the development of large-scale machine learning training clusters. Collective communications within these clusters tend to be heavily bandwidth-bound, necessitating techniques to optimally utilize the available network bandwidth. This puts the routing algorithm for the collective at the forefront of determining the performance. Unfortunately, communication libraries used in distributed machine learning today are limited by a fixed set of routing algorithms. This constraints collective performance within the domain of next-generation training clusters that employ intricate, heterogeneous, and asymmetric, large-scale topologies. Further, the emergence of irregular topologies attributed to runtime phenomena such as device failures serves to compound the complexity of the challenge. To this end, this paper introduces TACOS, an automated synthesizer that generates topology-aware collective algorithms for common distributed machine learning collectives across arbitrary input network topologies. TACOS was able to synthesize All-Reduce algorithm for a heterogeneous 512-NPU system in just 6.09 minutes while achieving performance improvement up to 4.27x over state-of-the-art prior work. TACOS exhibits high scalability, with synthesis time scaling quadratically with the number of NPUs. In contrast to prior works' NP-hard approaches, TACOS with 40K NPUs completes in 2.52 hours.
△ Less
Submitted 29 March, 2024; v1 submitted 11 April, 2023;
originally announced April 2023.
Charged Embedded Horizons and their Area Evolution
Authors:
Anushka Durg,
Aryan Bethmangalkar
Abstract:
The Embedded Horizon is defined to be a horizon that is in equilibrium with the exterior of the black hole, that is, isolated on the outside, but dynamically evolving on the inside, analogous to the inner and outer event horizons of the Reissner-Nordstrom black hole. This is shown as the result of charge variance on the horizon, which is expressed in electrical angular coordinates. The mass and en…
▽ More
The Embedded Horizon is defined to be a horizon that is in equilibrium with the exterior of the black hole, that is, isolated on the outside, but dynamically evolving on the inside, analogous to the inner and outer event horizons of the Reissner-Nordstrom black hole. This is shown as the result of charge variance on the horizon, which is expressed in electrical angular coordinates. The mass and energy of the black hole are discussed. The intrinsic metric is calculated by taking the electric potential into consideration. An area law evolution law is formulated, which suggests that the inner dynamical horizon evolves until it reaches the radius of the isolated horizon. The dynamical horizon, then, reaches equilibrium with the exterior, turning into an isolated horizon, thus turning extremal.
△ Less
Submitted 14 November, 2019;
originally announced November 2019.
