-
The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage
Authors:
Daniel Galvez,
Greg Diamos,
Juan Ciro,
Juan Felipe Cerón,
Keith Achorn,
Anjali Gopi,
David Kanter,
Maximilian Lam,
Mark Mazumder,
Vijay Janapa Reddi
Abstract:
The People's Speech is a free-to-download 30,000-hour and growing supervised conversational English speech recognition dataset licensed for academic and commercial usage under CC-BY-SA (with a CC-BY subset). The data is collected via searching the Internet for appropriately licensed audio data with existing transcriptions. We describe our data collection methodology and release our data collection…
▽ More
The People's Speech is a free-to-download 30,000-hour and growing supervised conversational English speech recognition dataset licensed for academic and commercial usage under CC-BY-SA (with a CC-BY subset). The data is collected via searching the Internet for appropriately licensed audio data with existing transcriptions. We describe our data collection methodology and release our data collection system under the Apache 2.0 license. We show that a model trained on this dataset achieves a 9.98% word error rate on Librispeech's test-clean test set.Finally, we discuss the legal and ethical issues surrounding the creation of a sizable machine learning corpora and plans for continued maintenance of the project under MLCommons's sponsorship.
△ Less
Submitted 17 November, 2021;
originally announced November 2021.
-
MLPerf Inference Benchmark
Authors:
Vijay Janapa Reddi,
Christine Cheng,
David Kanter,
Peter Mattson,
Guenther Schmuelling,
Carole-Jean Wu,
Brian Anderson,
Maximilien Breughe,
Mark Charlebois,
William Chou,
Ramesh Chukka,
Cody Coleman,
Sam Davis,
Pan Deng,
Greg Diamos,
Jared Duke,
Dave Fick,
J. Scott Gardner,
Itay Hubara,
Sachin Idgunji,
Thomas B. Jablin,
Jeff Jiao,
Tom St. John,
Pankaj Kanwar,
David Lee
, et al. (22 additional authors not shown)
Abstract:
Machine-learning (ML) hardware and software system demand is burgeoning. Driven by ML applications, the number of different ML inference systems has exploded. Over 100 organizations are building ML inference chips, and the systems that incorporate existing models span at least three orders of magnitude in power consumption and five orders of magnitude in performance; they range from embedded devic…
▽ More
Machine-learning (ML) hardware and software system demand is burgeoning. Driven by ML applications, the number of different ML inference systems has exploded. Over 100 organizations are building ML inference chips, and the systems that incorporate existing models span at least three orders of magnitude in power consumption and five orders of magnitude in performance; they range from embedded devices to data-center solutions. Fueling the hardware are a dozen or more software frameworks and libraries. The myriad combinations of ML hardware and ML software make assessing ML-system performance in an architecture-neutral, representative, and reproducible manner challenging. There is a clear need for industry-wide standard ML benchmarking and evaluation criteria. MLPerf Inference answers that call. In this paper, we present our benchmarking method for evaluating ML inference systems. Driven by more than 30 organizations as well as more than 200 ML engineers and practitioners, MLPerf prescribes a set of rules and best practices to ensure comparability across systems with wildly differing architectures. The first call for submissions garnered more than 600 reproducible inference-performance measurements from 14 organizations, representing over 30 systems that showcase a wide range of capabilities. The submissions attest to the benchmark's flexibility and adaptability.
△ Less
Submitted 9 May, 2020; v1 submitted 6 November, 2019;
originally announced November 2019.
-
MLPerf Training Benchmark
Authors:
Peter Mattson,
Christine Cheng,
Cody Coleman,
Greg Diamos,
Paulius Micikevicius,
David Patterson,
Hanlin Tang,
Gu-Yeon Wei,
Peter Bailis,
Victor Bittorf,
David Brooks,
Dehao Chen,
Debojyoti Dutta,
Udit Gupta,
Kim Hazelwood,
Andrew Hock,
Xinyuan Huang,
Atsushi Ike,
Bill Jia,
Daniel Kang,
David Kanter,
Naveen Kumar,
Jeffery Liao,
Guokai Ma,
Deepak Narayanan
, et al. (12 additional authors not shown)
Abstract:
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time to solution exhibits h…
▽ More
Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time to solution exhibits high variance, and software and hardware systems are so diverse that fair benchmarking with the same binary, code, and even hyperparameters is difficult. We therefore present MLPerf, an ML benchmark that overcomes these challenges. Our analysis quantitatively evaluates MLPerf's efficacy at driving performance and scalability improvements across two rounds of results from multiple vendors.
△ Less
Submitted 2 March, 2020; v1 submitted 2 October, 2019;
originally announced October 2019.
-
Deep Learning Scaling is Predictable, Empirically
Authors:
Joel Hestness,
Sharan Narang,
Newsha Ardalani,
Gregory Diamos,
Heewoo Jun,
Hassan Kianinejad,
Md. Mostofa Ali Patwary,
Yang Yang,
Yanqi Zhou
Abstract:
Deep learning (DL) creates impactful advances following a virtuous recipe: model architecture search, creating large training data sets, and scaling computation. It is widely believed that growing training sets and models should improve accuracy and result in better products. As DL application domains grow, we would like a deeper understanding of the relationships between training set size, comput…
▽ More
Deep learning (DL) creates impactful advances following a virtuous recipe: model architecture search, creating large training data sets, and scaling computation. It is widely believed that growing training sets and models should improve accuracy and result in better products. As DL application domains grow, we would like a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements to advance the state-of-the-art.
This paper presents a large scale empirical characterization of generalization error and model size growth as training sets grow. We introduce a methodology for this measurement and test four machine learning domains: machine translation, language modeling, image processing, and speech recognition. Our empirical results show power-law generalization error scaling across a breadth of factors, resulting in power-law exponents---the "steepness" of the learning curve---yet to be explained by theoretical work. Further, model improvements only shift the error but do not appear to affect the power-law exponent. We also show that model size scales sublinearly with data size. These scaling relationships have significant implications on deep learning research, practice, and systems. They can assist model debugging, setting accuracy targets, and decisions about data set growth. They can also guide computing system design and underscore the importance of continued computational scaling.
△ Less
Submitted 1 December, 2017;
originally announced December 2017.
-
Block-Sparse Recurrent Neural Networks
Authors:
Sharan Narang,
Eric Undersander,
Gregory Diamos
Abstract:
Recurrent Neural Networks (RNNs) are used in state-of-the-art models in domains such as speech recognition, machine translation, and language modelling. Sparsity is a technique to reduce compute and memory requirements of deep learning models. Sparse RNNs are easier to deploy on devices and high-end server processors. Even though sparse operations need less compute and memory relative to their den…
▽ More
Recurrent Neural Networks (RNNs) are used in state-of-the-art models in domains such as speech recognition, machine translation, and language modelling. Sparsity is a technique to reduce compute and memory requirements of deep learning models. Sparse RNNs are easier to deploy on devices and high-end server processors. Even though sparse operations need less compute and memory relative to their dense counterparts, the speed-up observed by using sparse operations is less than expected on different hardware platforms. In order to address this issue, we investigate two different approaches to induce block sparsity in RNNs: pruning blocks of weights in a layer and using group lasso regularization to create blocks of weights with zeros. Using these techniques, we demonstrate that we can create block-sparse RNNs with sparsity ranging from 80% to 90% with small loss in accuracy. This allows us to reduce the model size by roughly 10x. Additionally, we can prune a larger dense network to recover this loss in accuracy while maintaining high block sparsity and reducing the overall parameter count. Our technique works with a variety of block sizes up to 32x32. Block-sparse RNNs eliminate overheads related to data storage and irregular memory accesses while increasing hardware efficiency compared to unstructured sparsity.
△ Less
Submitted 7 November, 2017;
originally announced November 2017.
-
Mixed Precision Training
Authors:
Paulius Micikevicius,
Sharan Narang,
Jonah Alben,
Gregory Diamos,
Erich Elsen,
David Garcia,
Boris Ginsburg,
Michael Houston,
Oleksii Kuchaiev,
Ganesh Venkatesh,
Hao Wu
Abstract:
Deep neural networks have enabled progress in a wide variety of applications. Growing the size of the neural network typically results in improved accuracy. As model sizes grow, the memory and compute requirements for training these models also increases. We introduce a technique to train deep neural networks using half precision floating point numbers. In our technique, weights, activations and g…
▽ More
Deep neural networks have enabled progress in a wide variety of applications. Growing the size of the neural network typically results in improved accuracy. As model sizes grow, the memory and compute requirements for training these models also increases. We introduce a technique to train deep neural networks using half precision floating point numbers. In our technique, weights, activations and gradients are stored in IEEE half-precision format. Half-precision floating numbers have limited numerical range compared to single-precision numbers. We propose two techniques to handle this loss of information. Firstly, we recommend maintaining a single-precision copy of the weights that accumulates the gradients after each optimizer step. This single-precision copy is rounded to half-precision format during training. Secondly, we propose scaling the loss appropriately to handle the loss of information with half-precision gradients. We demonstrate that this approach works for a wide variety of models including convolution neural networks, recurrent neural networks and generative adversarial networks. This technique works for large scale models with more than 100 million parameters trained on large datasets. Using this approach, we can reduce the memory consumption of deep learning models by nearly 2x. In future processors, we can also expect a significant computation speedup using half-precision hardware units.
△ Less
Submitted 15 February, 2018; v1 submitted 10 October, 2017;
originally announced October 2017.
