-
Flashlight: Enabling Innovation in Tools for Machine Learning
Authors:
Jacob Kahn,
Vineel Pratap,
Tatiana Likhomanenko,
Qiantong Xu,
Awni Hannun,
Jeff Cai,
Paden Tomasello,
Ann Lee,
Edouard Grave,
Gilad Avidov,
Benoit Steiner,
Vitaliy Liptchinsky,
Gabriel Synnaeve,
Ronan Collobert
Abstract:
As the computational requirements for machine learning systems and the size and complexity of machine learning frameworks increases, essential framework innovation has become challenging. While computational needs have driven recent compiler, networking, and hardware advancements, utilization of those advancements by machine learning tools is occurring at a slower pace. This is in part due to the…
▽ More
As the computational requirements for machine learning systems and the size and complexity of machine learning frameworks increases, essential framework innovation has become challenging. While computational needs have driven recent compiler, networking, and hardware advancements, utilization of those advancements by machine learning tools is occurring at a slower pace. This is in part due to the difficulties involved in prototy** new computational paradigms with existing frameworks. Large frameworks prioritize machine learning researchers and practitioners as end users and pay comparatively little attention to systems researchers who can push frameworks forward -- we argue that both are equally important stakeholders. We introduce Flashlight, an open-source library built to spur innovation in machine learning tools and systems by prioritizing open, modular, customizable internals and state-of-the-art, research-ready models and training setups across a variety of domains. Flashlight allows systems researchers to rapidly prototype and experiment with novel ideas in machine learning computation and has low overhead, competing with and often outperforming other popular machine learning frameworks. We see Flashlight as a tool enabling research that can benefit widely used libraries downstream and bring machine learning and systems researchers closer together. Flashlight is available at https://github.com/flashlight/flashlight .
△ Less
Submitted 22 June, 2022; v1 submitted 28 January, 2022;
originally announced January 2022.
-
Rethinking Evaluation in ASR: Are Our Models Robust Enough?
Authors:
Tatiana Likhomanenko,
Qiantong Xu,
Vineel Pratap,
Paden Tomasello,
Jacob Kahn,
Gilad Avidov,
Ronan Collobert,
Gabriel Synnaeve
Abstract:
Is pushing numbers on a single benchmark valuable in automatic speech recognition? Research results in acoustic modeling are typically evaluated based on performance on a single dataset. While the research community has coalesced around various benchmarks, we set out to understand generalization performance in acoustic modeling across datasets - in particular, if models trained on a single dataset…
▽ More
Is pushing numbers on a single benchmark valuable in automatic speech recognition? Research results in acoustic modeling are typically evaluated based on performance on a single dataset. While the research community has coalesced around various benchmarks, we set out to understand generalization performance in acoustic modeling across datasets - in particular, if models trained on a single dataset transfer to other (possibly out-of-domain) datasets. We show that, in general, reverberative and additive noise augmentation improves generalization performance across domains. Further, we demonstrate that when a large enough set of benchmarks is used, average word error rate (WER) performance over them provides a good proxy for performance on real-world noisy data. Finally, we show that training a single acoustic model on the most widely-used datasets - combined - reaches competitive performance on both research and real-world benchmarks.
△ Less
Submitted 2 May, 2021; v1 submitted 22 October, 2020;
originally announced October 2020.
-
Scaling Up Online Speech Recognition Using ConvNets
Authors:
Vineel Pratap,
Qiantong Xu,
Jacob Kahn,
Gilad Avidov,
Tatiana Likhomanenko,
Awni Hannun,
Vitaliy Liptchinsky,
Gabriel Synnaeve,
Ronan Collobert
Abstract:
We design an online end-to-end speech recognition system based on Time-Depth Separable (TDS) convolutions and Connectionist Temporal Classification (CTC). We improve the core TDS architecture in order to limit the future context and hence reduce latency while maintaining accuracy. The system has almost three times the throughput of a well tuned hybrid ASR baseline while also having lower latency a…
▽ More
We design an online end-to-end speech recognition system based on Time-Depth Separable (TDS) convolutions and Connectionist Temporal Classification (CTC). We improve the core TDS architecture in order to limit the future context and hence reduce latency while maintaining accuracy. The system has almost three times the throughput of a well tuned hybrid ASR baseline while also having lower latency and a better word error rate. Also important to the efficiency of the recognizer is our highly optimized beam search decoder. To show the impact of our design choices, we analyze throughput, latency, accuracy, and discuss how these metrics can be tuned based on the user requirements.
△ Less
Submitted 27 January, 2020;
originally announced January 2020.
