-
AudioPaLM: A Large Language Model That Can Speak and Listen
Authors:
Paul K. Rubenstein,
Chulayuth Asawaroengchai,
Duc Dung Nguyen,
Ankur Bapna,
Zalán Borsos,
Félix de Chaumont Quitry,
Peter Chen,
Dalia El Badawy,
Wei Han,
Eugene Kharitonov,
Hannah Muckenhirn,
Dirk Padfield,
James Qin,
Danny Rozenberg,
Tara Sainath,
Johan Schalkwyk,
Matt Sharifi,
Michelle Tadmor Ramanovich,
Marco Tagliasacchi,
Alexandru Tudor,
Mihajlo Velimirović,
Damien Vincent,
Jiahui Yu,
Yongqiang Wang,
Vicky Zayats
, et al. (5 additional authors not shown)
Abstract:
We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the…
▽ More
We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt. We release examples of our method at https://google-research.github.io/seanet/audiopalm/examples
△ Less
Submitted 22 June, 2023;
originally announced June 2023.
-
CycleGAN-Based Unpaired Speech Dereverberation
Authors:
Hannah Muckenhirn,
Aleksandr Safin,
Hakan Erdogan,
Felix de Chaumont Quitry,
Marco Tagliasacchi,
Scott Wisdom,
John R. Hershey
Abstract:
Typically, neural network-based speech dereverberation models are trained on paired data, composed of a dry utterance and its corresponding reverberant utterance. The main limitation of this approach is that such models can only be trained on large amounts of data and a variety of room impulse responses when the data is synthetically reverberated, since acquiring real paired data is costly. In thi…
▽ More
Typically, neural network-based speech dereverberation models are trained on paired data, composed of a dry utterance and its corresponding reverberant utterance. The main limitation of this approach is that such models can only be trained on large amounts of data and a variety of room impulse responses when the data is synthetically reverberated, since acquiring real paired data is costly. In this paper we propose a CycleGAN-based approach that enables dereverberation models to be trained on unpaired data. We quantify the impact of using unpaired data by comparing the proposed unpaired model to a paired model with the same architecture and trained on the paired version of the same dataset. We show that the performance of the unpaired model is comparable to the performance of the paired model on two different datasets, according to objective evaluation metrics. Furthermore, we run two subjective evaluations and show that both models achieve comparable subjective quality on the AMI dataset, which was not seen during training.
△ Less
Submitted 29 March, 2022;
originally announced March 2022.
-
VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Masking
Authors:
Quan Wang,
Hannah Muckenhirn,
Kevin Wilson,
Prashant Sridhar,
Zelin Wu,
John Hershey,
Rif A. Saurous,
Ron J. Weiss,
Ye Jia,
Ignacio Lopez Moreno
Abstract:
In this paper, we present a novel system that separates the voice of a target speaker from multi-speaker signals, by making use of a reference signal from the target speaker. We achieve this by training two separate neural networks: (1) A speaker recognition network that produces speaker-discriminative embeddings; (2) A spectrogram masking network that takes both noisy spectrogram and speaker embe…
▽ More
In this paper, we present a novel system that separates the voice of a target speaker from multi-speaker signals, by making use of a reference signal from the target speaker. We achieve this by training two separate neural networks: (1) A speaker recognition network that produces speaker-discriminative embeddings; (2) A spectrogram masking network that takes both noisy spectrogram and speaker embedding as input, and produces a mask. Our system significantly reduces the speech recognition WER on multi-speaker signals, with minimal WER degradation on single-speaker signals.
△ Less
Submitted 19 June, 2019; v1 submitted 10 October, 2018;
originally announced October 2018.
