-
Parallelized Training of Restricted Boltzmann Machines using Markov-Chain Monte Carlo Methods
Authors:
Pei Yang,
Srinivas Varadharajan,
Lucas A. Wilson,
Don D. Smith II,
John A Lockman III,
Vineet Gundecha,
Quy Ta
Abstract:
Restricted Boltzmann Machine (RBM) is a generative stochastic neural network that can be applied to collaborative filtering technique used by recommendation systems. Prediction accuracy of the RBM model is usually better than that of other models for recommendation systems. However, training the RBM model involves Markov-Chain Monte Carlo (MCMC) method, which is computationally expensive. In this…
▽ More
Restricted Boltzmann Machine (RBM) is a generative stochastic neural network that can be applied to collaborative filtering technique used by recommendation systems. Prediction accuracy of the RBM model is usually better than that of other models for recommendation systems. However, training the RBM model involves Markov-Chain Monte Carlo (MCMC) method, which is computationally expensive. In this paper, we have successfully applied distributed parallel training using Horovod framework to improve the training time of the RBM model. Our tests show that the distributed training approach of the RBM model has a good scaling efficiency. We also show that this approach effectively reduces the training time to little over 12 minutes on 64 CPU nodes compared to 5 hours on a single CPU node. This will make RBM models more practically applicable in recommendation systems.
△ Less
Submitted 13 October, 2019;
originally announced October 2019.
-
Densifying Assumed-sparse Tensors: Improving Memory Efficiency and MPI Collective Performance during Tensor Accumulation for Parallelized Training of Neural Machine Translation Models
Authors:
Derya Cavdar,
Valeriu Codreanu,
Can Karakus,
John A. Lockman III,
Damian Podareanu,
Vikram Saletore,
Alexander Sergeev,
Don D. Smith II,
Victor Suthichai,
Quy Ta,
Srinivas Varadharajan,
Lucas A. Wilson,
Rengan Xu,
Pei Yang
Abstract:
Neural machine translation - using neural networks to translate human language - is an area of active research exploring new neuron types and network topologies with the goal of dramatically improving machine translation performance. Current state-of-the-art approaches, such as the multi-head attention-based transformer, require very large translation corpuses and many epochs to produce models of…
▽ More
Neural machine translation - using neural networks to translate human language - is an area of active research exploring new neuron types and network topologies with the goal of dramatically improving machine translation performance. Current state-of-the-art approaches, such as the multi-head attention-based transformer, require very large translation corpuses and many epochs to produce models of reasonable quality. Recent attempts to parallelize the official TensorFlow "Transformer" model across multiple nodes have hit roadblocks due to excessive memory use and resulting out of memory errors when performing MPI collectives. This paper describes modifications made to the Horovod MPI-based distributed training framework to reduce memory usage for transformer models by converting assumed-sparse tensors to dense tensors, and subsequently replacing sparse gradient gather with dense gradient reduction. The result is a dramatic increase in scale-out capability, with CPU-only scaling tests achieving 91% weak scaling efficiency up to 1200 MPI processes (300 nodes), and up to 65% strong scaling efficiency up to 400 MPI processes (200 nodes) using the Stampede2 supercomputer.
△ Less
Submitted 10 May, 2019;
originally announced May 2019.
-
G28.17+0.05: An unusual giant HI cloud in the inner Galaxy
Authors:
Anthony H. Minter,
Felix J. Lockman,
Glen I. Langston,
Jennifer A. Lockman
Abstract:
New 21 cm HI observations have revealed a giant HI cloud in the Galactic plane that has unusual properties. It is quite well defined, about 150 pc in diameter at a distance of 5 kpc, and contains as much as 100,000 Solar Masses of atomic hydrogen. The outer parts of the cloud appear in HI emission above the HI background, while the central regions show HI self-absorption. Models which reproduce…
▽ More
New 21 cm HI observations have revealed a giant HI cloud in the Galactic plane that has unusual properties. It is quite well defined, about 150 pc in diameter at a distance of 5 kpc, and contains as much as 100,000 Solar Masses of atomic hydrogen. The outer parts of the cloud appear in HI emission above the HI background, while the central regions show HI self-absorption. Models which reproduce the observations have a core with a temperature <40 K and an outer envelope as much as an order of magnitude hotter. The cold core is elongated along the Galactic plane, whereas the overall outline of the cloud is approximately spherical. The warm and cold parts of the HI cloud have a similar, and relatively large, line width of approximately 7 km/s. The cloud core is a source of weak, anomalously-excited 1720 MHz OH emission, also with a relatively large line width, which delineates the region of HI self-absorption but is slightly blue-shifted in velocity. The intensity of the 1720 MHz OH emission is correlated with N(H) derived from models of the cold core. There is 12CO emission associated with the cloud core. Most of the cloud mass is in molecules, and the total mass is > 200,000 Solar Masses. In the cold core the HI mass fraction may be 10 percent. The cloud has only a few sites of current star formation. There may be about 100 more objects like this in the inner Galaxy; every line of sight through the Galactic plane within 50 degrees of the Galactic center probably intersects at least one. We suggest that G28.17+0.05 is a cloud being observed as it enters a spiral arm and that it is in the transition from the atomic to the molecular state.
△ Less
Submitted 8 January, 2001;
originally announced January 2001.
