Democratizing Production-Scale Distributed Deep Learning
Authors:
Minghuang Ma,
Hadi Pouransari,
Daniel Chao,
Saurabh Adya,
Santiago Akle Serrano,
Yi Qin,
Dan Gimnicher,
Dominic Walsh
Abstract:
The interest and demand for training deep neural networks have been experiencing rapid growth, spanning a wide range of applications in both academia and industry. However, training them distributed and at scale remains difficult due to the complex ecosystem of tools and hardware involved. One consequence is that the responsibility of orchestrating these complex components is often left to one-off…
▽ More
The interest and demand for training deep neural networks have been experiencing rapid growth, spanning a wide range of applications in both academia and industry. However, training them distributed and at scale remains difficult due to the complex ecosystem of tools and hardware involved. One consequence is that the responsibility of orchestrating these complex components is often left to one-off scripts and glue code customized for specific problems. To address these restrictions, we introduce \emph{Alchemist} - an internal service built at Apple from the ground up for \emph{easy}, \emph{fast}, and \emph{scalable} distributed training. We discuss its design, implementation, and examples of running different flavors of distributed training. We also present case studies of its internal adoption in the development of autonomous systems, where training times have been reduced by 10x to keep up with the ever-growing data collection.
△ Less
Submitted 3 November, 2018; v1 submitted 31 October, 2018;
originally announced November 2018.
