-
Meta Learning Text-to-Speech Synthesis in over 7000 Languages
Authors:
Florian Lux,
Sarina Meyer,
Lyonel Behringer,
Frank Zalkow,
Phat Do,
Matt Coler,
Emanuël A. P. Habets,
Ngoc Thang Vu
Abstract:
In this work, we take on the challenging task of building a single text-to-speech synthesis system that is capable of generating speech in over 7000 languages, many of which lack sufficient data for traditional TTS development. By leveraging a novel integration of massively multilingual pretraining and meta learning to approximate language representations, our approach enables zero-shot speech syn…
▽ More
In this work, we take on the challenging task of building a single text-to-speech synthesis system that is capable of generating speech in over 7000 languages, many of which lack sufficient data for traditional TTS development. By leveraging a novel integration of massively multilingual pretraining and meta learning to approximate language representations, our approach enables zero-shot speech synthesis in languages without any available data. We validate our system's performance through objective measures and human evaluation across a diverse linguistic landscape. By releasing our code and models publicly, we aim to empower communities with limited linguistic resources and foster further innovation in the field of speech technology.
△ Less
Submitted 10 June, 2024;
originally announced June 2024.
-
The VoicePrivacy 2024 Challenge Evaluation Plan
Authors:
Natalia Tomashenko,
Xiaoxiao Miao,
Pierre Champion,
Sarina Meyer,
Xin Wang,
Emmanuel Vincent,
Michele Panariello,
Nicholas Evans,
Junichi Yamagishi,
Massimiliano Todisco
Abstract:
The task of the challenge is to develop a voice anonymization system for speech data which conceals the speaker's voice identity while protecting linguistic content and emotional states. The organizers provide development and evaluation datasets and evaluation scripts, as well as baseline anonymization systems and a list of training resources formed on the basis of the participants' requests. Part…
▽ More
The task of the challenge is to develop a voice anonymization system for speech data which conceals the speaker's voice identity while protecting linguistic content and emotional states. The organizers provide development and evaluation datasets and evaluation scripts, as well as baseline anonymization systems and a list of training resources formed on the basis of the participants' requests. Participants apply their developed anonymization systems, run evaluation scripts and submit evaluation results and anonymized speech data to the organizers. Results will be presented at a workshop held in conjunction with Interspeech 2024 to which all participants are invited to present their challenge systems and to submit additional workshop papers.
△ Less
Submitted 12 June, 2024; v1 submitted 3 April, 2024;
originally announced April 2024.
-
Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directions
Authors:
Florian Lux,
Pascal Tilli,
Sarina Meyer,
Ngoc Thang Vu
Abstract:
Customizing voice and speaking style in a speech synthesis system with intuitive and fine-grained controls is challenging, given that little data with appropriate labels is available. Furthermore, editing an existing human's voice also comes with ethical concerns. In this paper, we propose a method to generate artificial speaker embeddings that cannot be linked to a real human while offering intui…
▽ More
Customizing voice and speaking style in a speech synthesis system with intuitive and fine-grained controls is challenging, given that little data with appropriate labels is available. Furthermore, editing an existing human's voice also comes with ethical concerns. In this paper, we propose a method to generate artificial speaker embeddings that cannot be linked to a real human while offering intuitive and fine-grained control over the voice and speaking style of the embeddings, without requiring any labels for speaker or style. The artificial and controllable embeddings can be fed to a speech synthesis system, conditioned on embeddings of real humans during training, without sacrificing privacy during inference.
△ Less
Submitted 26 October, 2023;
originally announced October 2023.
-
The IMS Toucan System for the Blizzard Challenge 2023
Authors:
Florian Lux,
Julia Koch,
Sarina Meyer,
Thomas Bott,
Nadja Schauffler,
Pavel Denisov,
Antje Schweitzer,
Ngoc Thang Vu
Abstract:
For our contribution to the Blizzard Challenge 2023, we improved on the system we submitted to the Blizzard Challenge 2021. Our approach entails a rule-based text-to-phoneme processing system that includes rule-based disambiguation of homographs in the French language. It then transforms the phonemes to spectrograms as intermediate representations using a fast and efficient non-autoregressive synt…
▽ More
For our contribution to the Blizzard Challenge 2023, we improved on the system we submitted to the Blizzard Challenge 2021. Our approach entails a rule-based text-to-phoneme processing system that includes rule-based disambiguation of homographs in the French language. It then transforms the phonemes to spectrograms as intermediate representations using a fast and efficient non-autoregressive synthesis architecture based on Conformer and Glow. A GAN based neural vocoder that combines recent state-of-the-art approaches converts the spectrogram to the final wave. We carefully designed the data processing, training, and inference procedures for the challenge data. Our system identifier is G. Open source code and demo are available.
△ Less
Submitted 26 October, 2023;
originally announced October 2023.
-
VoicePAT: An Efficient Open-source Evaluation Toolkit for Voice Privacy Research
Authors:
Sarina Meyer,
Xiaoxiao Miao,
Ngoc Thang Vu
Abstract:
Speaker anonymization is the task of modifying a speech recording such that the original speaker cannot be identified anymore. Since the first Voice Privacy Challenge in 2020, along with the release of a framework, the popularity of this research topic is continually increasing. However, the comparison and combination of different anonymization approaches remains challenging due to the complexity…
▽ More
Speaker anonymization is the task of modifying a speech recording such that the original speaker cannot be identified anymore. Since the first Voice Privacy Challenge in 2020, along with the release of a framework, the popularity of this research topic is continually increasing. However, the comparison and combination of different anonymization approaches remains challenging due to the complexity of evaluation and the absence of user-friendly research frameworks. We therefore propose an efficient speaker anonymization and evaluation framework based on a modular and easily extendable structure, almost fully in Python. The framework facilitates the orchestration of several anonymization approaches in parallel and allows for interfacing between different techniques. Furthermore, we propose modifications to common evaluation methods which improves the quality of the evaluation and reduces their computation time by 65 to 95%, depending on the metric. Our code is fully open source.
△ Less
Submitted 21 December, 2023; v1 submitted 14 September, 2023;
originally announced September 2023.
-
Convolutional neural network-based single-shot speckle tracking for x-ray phase-contrast imaging
Authors:
Serena Qinyun Z. Shi,
Nadav Shapira,
Peter B. Noël,
Sebastian Meyer
Abstract:
X-ray phase-contrast imaging offers enhanced sensitivity for weakly-attenuating materials, such as breast and brain tissue, but has yet to be widely implemented clinically due to high coherence requirements and expensive x-ray optics. Speckle-based phase contrast imaging has been proposed as an affordable and simple alternative; however, obtaining high-quality phase-contrast images requires accura…
▽ More
X-ray phase-contrast imaging offers enhanced sensitivity for weakly-attenuating materials, such as breast and brain tissue, but has yet to be widely implemented clinically due to high coherence requirements and expensive x-ray optics. Speckle-based phase contrast imaging has been proposed as an affordable and simple alternative; however, obtaining high-quality phase-contrast images requires accurate tracking of sample-induced speckle pattern modulations. This study introduced a convolutional neural network to accurately retrieve sub-pixel displacement fields from pairs of reference (i.e., without sample) and sample images for speckle tracking. Speckle patterns were generated utilizing an in-house wave-optical simulation tool. These images were then randomly deformed and attenuated to generate training and testing datasets. The performance of the model was evaluated and compared against conventional speckle tracking algorithms: zero-normalized cross-correlation and unified modulated pattern analysis. We demonstrate improved accuracy (1.7 times better than conventional speckle tracking), bias (2.6 times), and spatial resolution (2.3 times), as well as noise robustness, window size independence, and computational efficiency. In addition, the model was validated with a simulated geometric phantom. Thus, in this study, we propose a novel convolutional-neural-network-based speckle-tracking method with enhanced performance and robustness that offers improved alternative tracking while further expanding the potential applications of speckle-based phase contrast imaging.
△ Less
Submitted 2 May, 2023;
originally announced May 2023.
-
Anonymizing Speech with Generative Adversarial Networks to Preserve Speaker Privacy
Authors:
Sarina Meyer,
Pascal Tilli,
Pavel Denisov,
Florian Lux,
Julia Koch,
Ngoc Thang Vu
Abstract:
In order to protect the privacy of speech data, speaker anonymization aims for hiding the identity of a speaker by changing the voice in speech recordings. This typically comes with a privacy-utility trade-off between protection of individuals and usability of the data for downstream applications. One of the challenges in this context is to create non-existent voices that sound as natural as possi…
▽ More
In order to protect the privacy of speech data, speaker anonymization aims for hiding the identity of a speaker by changing the voice in speech recordings. This typically comes with a privacy-utility trade-off between protection of individuals and usability of the data for downstream applications. One of the challenges in this context is to create non-existent voices that sound as natural as possible.
In this work, we propose to tackle this issue by generating speaker embeddings using a generative adversarial network with Wasserstein distance as cost function. By incorporating these artificial embeddings into a speech-to-text-to-speech pipeline, we outperform previous approaches in terms of privacy and utility. According to standard objective metrics and human evaluation, our approach generates intelligible and content-preserving yet privacy-protecting versions of the original recordings.
△ Less
Submitted 20 October, 2022; v1 submitted 13 October, 2022;
originally announced October 2022.
-
Speaker Anonymization with Phonetic Intermediate Representations
Authors:
Sarina Meyer,
Florian Lux,
Pavel Denisov,
Julia Koch,
Pascal Tilli,
Ngoc Thang Vu
Abstract:
In this work, we propose a speaker anonymization pipeline that leverages high quality automatic speech recognition and synthesis systems to generate speech conditioned on phonetic transcriptions and anonymized speaker embeddings. Using phones as the intermediate representation ensures near complete elimination of speaker identity information from the input while preserving the original phonetic co…
▽ More
In this work, we propose a speaker anonymization pipeline that leverages high quality automatic speech recognition and synthesis systems to generate speech conditioned on phonetic transcriptions and anonymized speaker embeddings. Using phones as the intermediate representation ensures near complete elimination of speaker identity information from the input while preserving the original phonetic content as much as possible. Our experimental results on LibriSpeech and VCTK corpora reveal two key findings: 1) although automatic speech recognition produces imperfect transcriptions, our neural speech synthesis system can handle such errors, making our system feasible and robust, and 2) combining speaker embeddings from different resources is beneficial and their appropriate normalization is crucial. Overall, our final best system outperforms significantly the baselines provided in the Voice Privacy Challenge 2020 in terms of privacy robustness against a lazy-informed attacker while maintaining high intelligibility and naturalness of the anonymized speech.
△ Less
Submitted 11 July, 2022;
originally announced July 2022.
-
Novel Receiver for Superparamagnetic Iron Oxide Nanoparticles in a Molecular Communication Setting
Authors:
Max Bartunik,
Maximilian Lübke,
Harald Unterweger,
Christoph Alexiou,
Sebastian Meyer,
Doaa Ahmed,
Georg Fischer,
Wayan Wicke,
Vahid Jamali Kooshkghazi,
Robert Schober,
Jens Kirchner
Abstract:
Superparamagnetic iron oxide nanoparticles (SPIONs) have recently been introduced as information carriers in a testbed for molecular communication (MC) in duct flow. Here, a new receiver for this testbed is presented, based on the concept of a bridge circuit. The capability for a reliable transmission using the testbed and detection of the proposed receiver was evaluated by sending a text message…
▽ More
Superparamagnetic iron oxide nanoparticles (SPIONs) have recently been introduced as information carriers in a testbed for molecular communication (MC) in duct flow. Here, a new receiver for this testbed is presented, based on the concept of a bridge circuit. The capability for a reliable transmission using the testbed and detection of the proposed receiver was evaluated by sending a text message and a 80 bit random sequence at a bit rate of 1/s, which resulted in a bit error rate of 0 %. Furthermore, the sensitivity of the device was assessed by a dilution series, which gave a limit for the detectability of peaks between 0.1 to 0.5 mg/mL. Compared to the commercial susceptometer that was previously used as receiver, the new detector provides an increased sampling rate of 100 samples/s and flexibility in the dimensions of the propagation channel. Furthermore, it allows to implement both single-ended and differential signaling in SPION-bases MC testbeds.
△ Less
Submitted 17 July, 2019;
originally announced July 2019.
