Extreme Encoder Output Frame Rate Reduction: Improving Computational Latencies of Large End-to-End Models
Authors:
Rohit Prabhavalkar,
Zhong Meng,
Weiran Wang,
Adam Stooke,
Xingyu Cai,
Yanzhang He,
Arun Narayanan,
Dongseong Hwang,
Tara N. Sainath,
Pedro J. Moreno
Abstract:
The accuracy of end-to-end (E2E) automatic speech recognition (ASR) models continues to improve as they are scaled to larger sizes, with some now reaching billions of parameters. Widespread deployment and adoption of these models, however, requires computationally efficient strategies for decoding. In the present work, we study one such strategy: applying multiple frame reduction layers in the enc…
▽ More
The accuracy of end-to-end (E2E) automatic speech recognition (ASR) models continues to improve as they are scaled to larger sizes, with some now reaching billions of parameters. Widespread deployment and adoption of these models, however, requires computationally efficient strategies for decoding. In the present work, we study one such strategy: applying multiple frame reduction layers in the encoder to compress encoder outputs into a small number of output frames. While similar techniques have been investigated in previous work, we achieve dramatically more reduction than has previously been demonstrated through the use of multiple funnel reduction layers. Through ablations, we study the impact of various architectural choices in the encoder to identify the most effective strategies. We demonstrate that we can generate one encoder output frame for every 2.56 sec of input speech, without significantly affecting word error rate on a large-scale voice search task, while improving encoder and decoder latencies by 48% and 92% respectively, relative to a strong but computationally expensive baseline.
△ Less
Submitted 26 February, 2024;
originally announced February 2024.
Massive End-to-end Models for Short Search Queries
Authors:
Weiran Wang,
Rohit Prabhavalkar,
Dongseong Hwang,
Qiujia Li,
Khe Chai Sim,
Bo Li,
James Qin,
Xingyu Cai,
Adam Stooke,
Zhong Meng,
CJ Zheng,
Yanzhang He,
Tara Sainath,
Pedro Moreno Mengibar
Abstract:
In this work, we investigate two popular end-to-end automatic speech recognition (ASR) models, namely Connectionist Temporal Classification (CTC) and RNN-Transducer (RNN-T), for offline recognition of voice search queries, with up to 2B model parameters. The encoders of our models use the neural architecture of Google's universal speech model (USM), with additional funnel pooling layers to signifi…
▽ More
In this work, we investigate two popular end-to-end automatic speech recognition (ASR) models, namely Connectionist Temporal Classification (CTC) and RNN-Transducer (RNN-T), for offline recognition of voice search queries, with up to 2B model parameters. The encoders of our models use the neural architecture of Google's universal speech model (USM), with additional funnel pooling layers to significantly reduce the frame rate and speed up training and inference. We perform extensive studies on vocabulary size, time reduction strategy, and its generalization performance on long-form test sets. Despite the speculation that, as the model size increases, CTC can be as good as RNN-T which builds label dependency into the prediction, we observe that a 900M RNN-T clearly outperforms a 1.8B CTC and is more tolerant to severe time reduction, although the WER gap can be largely removed by LM shallow fusion.
△ Less
Submitted 22 September, 2023;
originally announced September 2023.
