Parallel Training of Deep Networks with Local Updates
Authors:
Michael Laskin,
Luke Metz,
Seth Nabarro,
Mark Saroufim,
Badreddine Noune,
Carlo Luschi,
Jascha Sohl-Dickstein,
Pieter Abbeel
Abstract:
Deep learning models trained on large data sets have been widely successful in both vision and language domains. As state-of-the-art deep learning architectures have continued to grow in parameter count so have the compute budgets and times required to train them, increasing the need for compute-efficient methods that parallelize training. Two common approaches to parallelize the training of deep…
▽ More
Deep learning models trained on large data sets have been widely successful in both vision and language domains. As state-of-the-art deep learning architectures have continued to grow in parameter count so have the compute budgets and times required to train them, increasing the need for compute-efficient methods that parallelize training. Two common approaches to parallelize the training of deep networks have been data and model parallelism. While useful, data and model parallelism suffer from diminishing returns in terms of compute efficiency for large batch sizes. In this paper, we investigate how to continue scaling compute efficiently beyond the point of diminishing returns for large batches through local parallelism, a framework which parallelizes training of individual layers in deep networks by replacing global backpropagation with truncated layer-wise backpropagation. Local parallelism enables fully asynchronous layer-wise parallelism with a low memory footprint, and requires little communication overhead compared with model parallelism. We show results in both vision and language domains across a diverse set of architectures, and find that local parallelism is particularly effective in the high-compute regime.
△ Less
Submitted 15 June, 2021; v1 submitted 7 December, 2020;
originally announced December 2020.
Aren't we all nearest neighbors: Spatial trees, high dimensional reductions and batch nearest neighbor search
Authors:
Mark Saroufim
Abstract:
We start with a review of the pervasiveness of the nearest neighbor search problem and techniques used to solve it along with some experimental results. In the second chapter, we show reductions between two different classes of geo- metric proximity problems: the nearest neighbor problems to solve the Euclidean minimum spanning tree problem and the farthest neighbor problems to solve the k-centers…
▽ More
We start with a review of the pervasiveness of the nearest neighbor search problem and techniques used to solve it along with some experimental results. In the second chapter, we show reductions between two different classes of geo- metric proximity problems: the nearest neighbor problems to solve the Euclidean minimum spanning tree problem and the farthest neighbor problems to solve the k-centers problem. In the third chapter, we unify spatial partitioning trees un- der one framework the meta-tree. Finally, we propose a dual tree algorithm for Bichromatic Closest Pair and measure the complexity of batch nearest neighbor search.
△ Less
Submitted 13 July, 2015;
originally announced July 2015.
