-
Lookahead When It Matters: Adaptive Non-causal Transformers for Streaming Neural Transducers
Authors:
Grant P. Strimel,
Yi Xie,
Brian King,
Martin Radfar,
Ariya Rastrow,
Athanasios Mouchtaris
Abstract:
Streaming speech recognition architectures are employed for low-latency, real-time applications. Such architectures are often characterized by their causality. Causal architectures emit tokens at each frame, relying only on current and past signal, while non-causal models are exposed to a window of future frames at each step to increase predictive accuracy. This dichotomy amounts to a trade-off fo…
▽ More
Streaming speech recognition architectures are employed for low-latency, real-time applications. Such architectures are often characterized by their causality. Causal architectures emit tokens at each frame, relying only on current and past signal, while non-causal models are exposed to a window of future frames at each step to increase predictive accuracy. This dichotomy amounts to a trade-off for real-time Automatic Speech Recognition (ASR) system design: profit from the low-latency benefit of strictly-causal architectures while accepting predictive performance limitations, or realize the modeling benefits of future-context models accompanied by their higher latency penalty. In this work, we relax the constraints of this choice and present the Adaptive Non-Causal Attention Transducer (ANCAT). Our architecture is non-causal in the traditional sense, but executes in a low-latency, streaming manner by dynamically choosing when to rely on future context and to what degree within the audio stream. The resulting mechanism, when coupled with our novel regularization algorithms, delivers comparable accuracy to non-causal configurations while improving significantly upon latency, closing the gap with their causal counterparts. We showcase our design experimentally by reporting comparative ASR task results with measures of accuracy and latency on both publicly accessible and production-scale, voice-assistant datasets.
△ Less
Submitted 9 May, 2023; v1 submitted 6 May, 2023;
originally announced May 2023.
-
Leveraging Redundancy in Multiple Audio Signals for Far-Field Speech Recognition
Authors:
Feng-Ju Chang,
Anastasios Alexandridis,
Rupak Vignesh Swaminathan,
Martin Radfar,
Harish Mallidi,
Maurizio Omologo,
Athanasios Mouchtaris,
Brian King,
Roland Maas
Abstract:
To achieve robust far-field automatic speech recognition (ASR), existing techniques typically employ an acoustic front end (AFE) cascaded with a neural transducer (NT) ASR model. The AFE output, however, could be unreliable, as the beamforming output in AFE is steered to a wrong direction. A promising way to address this issue is to exploit the microphone signals before the beamforming stage and a…
▽ More
To achieve robust far-field automatic speech recognition (ASR), existing techniques typically employ an acoustic front end (AFE) cascaded with a neural transducer (NT) ASR model. The AFE output, however, could be unreliable, as the beamforming output in AFE is steered to a wrong direction. A promising way to address this issue is to exploit the microphone signals before the beamforming stage and after the acoustic echo cancellation (post-AEC) in AFE. We argue that both, post-AEC and AFE outputs, are complementary and it is possible to leverage the redundancy between these signals to compensate for potential AFE processing errors. We present two fusion networks to explore this redundancy and aggregate these multi-channel (MC) signals: (1) Frequency-LSTM based, and (2) Convolutional Neural Network based fusion networks. We augment the MC fusion networks to a conformer transducer model and train it in an end-to-end fashion. Our experimental results on commercial virtual assistant tasks demonstrate that using the AFE output and two post-AEC signals with fusion networks offers up to 25.9% word error rate (WER) relative improvement over the model using the AFE output only, at the cost of <= 2% parameter increase.
△ Less
Submitted 1 March, 2023;
originally announced March 2023.
-
A Method for Crash Prediction and Avoidance Using Hidden Markov Models
Authors:
Avinash Prabu,
Lingxi Li,
Brian King,
Yaobin Chen
Abstract:
In recent years, automotive technology has made a steady progress. In particular, Advanced Driver Assistance System (ADAS) has enabled many safety features in commercial vehicles, for instance, pedestrian detection, lane kee** assist, emergency automatic braking, etc. Although these features provide drivers with a safer operational environment, crashes still happen occasionally due to the comple…
▽ More
In recent years, automotive technology has made a steady progress. In particular, Advanced Driver Assistance System (ADAS) has enabled many safety features in commercial vehicles, for instance, pedestrian detection, lane kee** assist, emergency automatic braking, etc. Although these features provide drivers with a safer operational environment, crashes still happen occasionally due to the complex road conditions and the unpredictable movement of road users including vehicles, pedestrians, bicyclists, and non-motorized vehicles. In this paper, we aim at predicting the possibilities of crashes between vehicles on highway and implementing an appropriate active safety system to prevent the same. In particular, hidden Markov models are developed for the traffic lanes and speed change of vehicles on highway. Algorithms are developed for the prediction of crash probabilities. Simulation experiments are conducted using Matlab, the results illustrate the effectiveness of the proposed research.
△ Less
Submitted 22 December, 2022;
originally announced December 2022.
-
Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparities
Authors:
Pranav Dheram,
Murugesan Ramakrishnan,
Anirudh Raju,
I-Fan Chen,
Brian King,
Katherine Powell,
Melissa Saboowala,
Karan Shetty,
Andreas Stolcke
Abstract:
As for other forms of AI, speech recognition has recently been examined with respect to performance disparities across different user cohorts. One approach to achieve fairness in speech recognition is to (1) identify speaker cohorts that suffer from subpar performance and (2) apply fairness mitigation measures targeting the cohorts discovered. In this paper, we report on initial findings with both…
▽ More
As for other forms of AI, speech recognition has recently been examined with respect to performance disparities across different user cohorts. One approach to achieve fairness in speech recognition is to (1) identify speaker cohorts that suffer from subpar performance and (2) apply fairness mitigation measures targeting the cohorts discovered. In this paper, we report on initial findings with both discovery and mitigation of performance disparities using data from a product-scale AI assistant speech recognition system. We compare cohort discovery based on geographic and demographic information to a more scalable method that groups speakers without human labels, using speaker embedding technology. For fairness mitigation, we find that oversampling of underrepresented cohorts, as well as modeling speaker cohort membership by additional input variables, reduces the gap between top- and bottom-performing cohorts, without deteriorating overall recognition accuracy.
△ Less
Submitted 22 July, 2022;
originally announced July 2022.
-
Reducing Geographic Disparities in Automatic Speech Recognition via Elastic Weight Consolidation
Authors:
Viet Anh Trinh,
Pegah Ghahremani,
Brian King,
Jasha Droppo,
Andreas Stolcke,
Roland Maas
Abstract:
We present an approach to reduce the performance disparity between geographic regions without degrading performance on the overall user population for ASR. A popular approach is to fine-tune the model with data from regions where the ASR model has a higher word error rate (WER). However, when the ASR model is adapted to get better performance on these high-WER regions, its parameters wander from t…
▽ More
We present an approach to reduce the performance disparity between geographic regions without degrading performance on the overall user population for ASR. A popular approach is to fine-tune the model with data from regions where the ASR model has a higher word error rate (WER). However, when the ASR model is adapted to get better performance on these high-WER regions, its parameters wander from the previous optimal values, which can lead to worse performance in other regions. In our proposed method, we utilize the elastic weight consolidation (EWC) regularization loss to identify directions in parameters space along which the ASR weights can vary to improve for high-error regions, while still maintaining performance on the speaker population overall. Our results demonstrate that EWC can reduce the word error rate (WER) in the region with highest WER by 3.2% relative while reducing the overall WER by 1.3% relative. We also evaluate the role of language and acoustic models in ASR fairness and propose a clustering algorithm to identify WER disparities based on geographic region.
△ Less
Submitted 16 July, 2022;
originally announced July 2022.
-
Compute Cost Amortized Transformer for Streaming ASR
Authors:
Yi Xie,
Jonathan Macoskey,
Martin Radfar,
Feng-Ju Chang,
Brian King,
Ariya Rastrow,
Athanasios Mouchtaris,
Grant P. Strimel
Abstract:
We present a streaming, Transformer-based end-to-end automatic speech recognition (ASR) architecture which achieves efficient neural inference through compute cost amortization. Our architecture creates sparse computation pathways dynamically at inference time, resulting in selective use of compute resources throughout decoding, enabling significant reductions in compute with minimal impact on acc…
▽ More
We present a streaming, Transformer-based end-to-end automatic speech recognition (ASR) architecture which achieves efficient neural inference through compute cost amortization. Our architecture creates sparse computation pathways dynamically at inference time, resulting in selective use of compute resources throughout decoding, enabling significant reductions in compute with minimal impact on accuracy. The fully differentiable architecture is trained end-to-end with an accompanying lightweight arbitrator mechanism operating at the frame-level to make dynamic decisions on each input while a tunable loss function is used to regularize the overall level of compute against predictive performance. We report empirical results from experiments using the compute amortized Transformer-Transducer (T-T) model conducted on LibriSpeech data. Our best model can achieve a 60% compute cost reduction with only a 3% relative word error rate (WER) increase.
△ Less
Submitted 4 July, 2022;
originally announced July 2022.
-
Investigation of Training Label Error Impact on RNN-T
Authors:
I-Fan Chen,
Brian King,
Jasha Droppo
Abstract:
In this paper, we propose an approach to quantitatively analyze impacts of different training label errors to RNN-T based ASR models. The result shows deletion errors are more harmful than substitution and insertion label errors in RNN-T training data. We also examined label error impact mitigation approaches on RNN-T and found that, though all the methods mitigate the label-error-caused degradati…
▽ More
In this paper, we propose an approach to quantitatively analyze impacts of different training label errors to RNN-T based ASR models. The result shows deletion errors are more harmful than substitution and insertion label errors in RNN-T training data. We also examined label error impact mitigation approaches on RNN-T and found that, though all the methods mitigate the label-error-caused degradation to some extent, they could not remove the performance gap between the models trained with and without the presence of label errors. Based on the analysis results, we suggest to design data pipelines for RNN-T with higher priority on reducing deletion label errors. We also find that ensuring high-quality training labels remains important, despite of the existence of the label error mitigation approaches.
△ Less
Submitted 1 December, 2021;
originally announced December 2021.
-
CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition
Authors:
Rupak Vignesh Swaminathan,
Brian King,
Grant P. Strimel,
Jasha Droppo,
Athanasios Mouchtaris
Abstract:
We propose a simple yet effective method to compress an RNN-Transducer (RNN-T) through the well-known knowledge distillation paradigm. We show that the transducer's encoder outputs naturally have a high entropy and contain rich information about acoustically similar word-piece confusions. This rich information is suppressed when combined with the lower entropy decoder outputs to produce the joint…
▽ More
We propose a simple yet effective method to compress an RNN-Transducer (RNN-T) through the well-known knowledge distillation paradigm. We show that the transducer's encoder outputs naturally have a high entropy and contain rich information about acoustically similar word-piece confusions. This rich information is suppressed when combined with the lower entropy decoder outputs to produce the joint network logits. Consequently, we introduce an auxiliary loss to distill the encoder logits from a teacher transducer's encoder, and explore training strategies where this encoder distillation works effectively. We find that tandem training of teacher and student encoders with an inplace encoder distillation outperforms the use of a pre-trained and static teacher transducer. We also report an interesting phenomenon we refer to as implicit distillation, that occurs when the teacher and student encoders share the same decoder. Our experiments show 5.37-8.4% relative word error rate reductions (WERR) on in-house test sets, and 5.05-6.18% relative WERRs on LibriSpeech test sets.
△ Less
Submitted 14 June, 2021;
originally announced June 2021.
-
Do You Listen with One or Two Microphones? A Unified ASR Model for Single and Multi-Channel Audio
Authors:
Gokce Keskin,
Minhua Wu,
Brian King,
Harish Mallidi,
Yang Gao,
Jasha Droppo,
Ariya Rastrow,
Roland Maas
Abstract:
Automatic speech recognition (ASR) models are typically designed to operate on a single input data type, e.g. a single or multi-channel audio streamed from a device. This design decision assumes the primary input data source does not change and if an additional (auxiliary) data source is occasionally available, it cannot be used. An ASR model that operates on both primary and auxiliary data can ac…
▽ More
Automatic speech recognition (ASR) models are typically designed to operate on a single input data type, e.g. a single or multi-channel audio streamed from a device. This design decision assumes the primary input data source does not change and if an additional (auxiliary) data source is occasionally available, it cannot be used. An ASR model that operates on both primary and auxiliary data can achieve better accuracy compared to a primary-only solution; and a model that can serve both primary-only (PO) and primary-plus-auxiliary (PPA) modes is highly desirable. In this work, we propose a unified ASR model that can serve both modes. We demonstrate its efficacy in a realistic scenario where a set of devices typically stream a single primary audio channel, and two additional auxiliary channels only when upload bandwidth allows it. The architecture enables a unique methodology that uses both types of input audio during training time. Our proposed approach achieves up to 12.5% relative word-error-rate reduction (WERR) compared to a PO baseline, and up to 16.0% relative WERR in low-SNR conditions. The unique training methodology achieves up to 2.5% relative WERR compared to a PPA baseline.
△ Less
Submitted 28 June, 2021; v1 submitted 4 June, 2021;
originally announced June 2021.
-
Attention-based Neural Beamforming Layers for Multi-channel Speech Recognition
Authors:
Bhargav Pulugundla,
Yang Gao,
Brian King,
Gokce Keskin,
Harish Mallidi,
Minhua Wu,
Jasha Droppo,
Roland Maas
Abstract:
Attention-based beamformers have recently been shown to be effective for multi-channel speech recognition. However, they are less capable at capturing local information. In this work, we propose a 2D Conv-Attention module which combines convolution neural networks with attention for beamforming. We apply self- and cross-attention to explicitly model the correlations within and between the input ch…
▽ More
Attention-based beamformers have recently been shown to be effective for multi-channel speech recognition. However, they are less capable at capturing local information. In this work, we propose a 2D Conv-Attention module which combines convolution neural networks with attention for beamforming. We apply self- and cross-attention to explicitly model the correlations within and between the input channels. The end-to-end 2D Conv-Attention model is compared with a multi-head self-attention and superdirective-based neural beamformers. We train and evaluate on an in-house multi-channel dataset. The results show a relative improvement of 3.8% in WER by the proposed model over the baseline neural beamformer.
△ Less
Submitted 14 May, 2021; v1 submitted 12 May, 2021;
originally announced May 2021.
-
End-to-End Multi-Channel Transformer for Speech Recognition
Authors:
Feng-Ju Chang,
Martin Radfar,
Athanasios Mouchtaris,
Brian King,
Siegfried Kunzmann
Abstract:
Transformers are powerful neural architectures that allow integrating different modalities using attention mechanisms. In this paper, we leverage the neural transformer architectures for multi-channel speech recognition systems, where the spectral and spatial information collected from different microphones are integrated using attention layers. Our multi-channel transformer network mainly consist…
▽ More
Transformers are powerful neural architectures that allow integrating different modalities using attention mechanisms. In this paper, we leverage the neural transformer architectures for multi-channel speech recognition systems, where the spectral and spatial information collected from different microphones are integrated using attention layers. Our multi-channel transformer network mainly consists of three parts: channel-wise self attention layers (CSA), cross-channel attention layers (CCA), and multi-channel encoder-decoder attention layers (EDA). The CSA and CCA layers encode the contextual relationship within and between channels and across time, respectively. The channel-attended outputs from CSA and CCA are then fed into the EDA layers to help decode the next token given the preceding ones. The experiments show that in a far-field in-house dataset, our method outperforms the baseline single-channel transformer, as well as the super-directive and neural beamformers cascaded with the transformers.
△ Less
Submitted 7 February, 2021;
originally announced February 2021.
-
Time-domain Representation of Passband Scattering Parameters
Authors:
Justin B. King
Abstract:
This paper presents a simple and accurate method for the inclusion of linear, time-invariant (LTI) networks, described by RF frequency-domain data, within equivalent baseband time-domain simulations. The time-domain representation is formulated as an equivalent baseband discrete-time impulse response, which may be convolved with the equivalent baseband form of the input signal, to obtain the corre…
▽ More
This paper presents a simple and accurate method for the inclusion of linear, time-invariant (LTI) networks, described by RF frequency-domain data, within equivalent baseband time-domain simulations. The time-domain representation is formulated as an equivalent baseband discrete-time impulse response, which may be convolved with the equivalent baseband form of the input signal, to obtain the corresponding equivalent baseband output. This allows networks which are most accurately described in the frequency domain, such as frequency-dispersive transmission lines, to be efficiently included as part of a transient time-domain simulation.
△ Less
Submitted 14 October, 2020;
originally announced October 2020.
-
Multi-view Frequency LSTM: An Efficient Frontend for Automatic Speech Recognition
Authors:
Maarten Van Segbroeck,
Harish Mallidih,
Brian King,
I-Fan Chen,
Gurpreet Chadha,
Roland Maas
Abstract:
Acoustic models in real-time speech recognition systems typically stack multiple unidirectional LSTM layers to process the acoustic frames over time. Performance improvements over vanilla LSTM architectures have been reported by prepending a stack of frequency-LSTM (FLSTM) layers to the time LSTM. These FLSTM layers can learn a more robust input feature to the time LSTM layers by modeling time-fre…
▽ More
Acoustic models in real-time speech recognition systems typically stack multiple unidirectional LSTM layers to process the acoustic frames over time. Performance improvements over vanilla LSTM architectures have been reported by prepending a stack of frequency-LSTM (FLSTM) layers to the time LSTM. These FLSTM layers can learn a more robust input feature to the time LSTM layers by modeling time-frequency correlations in the acoustic input signals. A drawback of FLSTM based architectures however is that they operate at a predefined, and tuned, window size and stride, referred to as 'view' in this paper. We present a simple and efficient modification by combining the outputs of multiple FLSTM stacks with different views, into a dimensionality reduced feature representation. The proposed multi-view FLSTM architecture allows to model a wider range of time-frequency correlations compared to an FLSTM model with single view. When trained on 50K hours of English far-field speech data with CTC loss followed by sMBR sequence training, we show that the multi-view FLSTM acoustic model provides relative Word Error Rate (WER) improvements of 3-7% for different speaker and acoustic environment scenarios over an optimized single FLSTM model, while retaining a similar computational footprint.
△ Less
Submitted 30 June, 2020;
originally announced July 2020.
