Leveraging Parameter-Efficient Transfer Learning for Multi-Lingual Text-to-Speech Adaptation

Abstract

Different languages have distinct phonetic systems and vary in their prosodic features making it challenging to develop a Text-to-Speech (TTS) model that can effectively synthesise speech in multilingual settings. Furthermore, TTS architecture needs to be both efficient enough to capture nuances in multiple languages and efficient enough to be practical for deployment. The standard approach is to build transformer based model such as SpeechT5 and train it on large multilingual dataset. As the size of these models grow the conventional fine-tuning for adapting these model becomes impractical due to heavy computational cost. In this paper, we proposes to integrate parameter-efficient transfer learning (PETL) methods such as adapters and hypernetwork with TTS architecture for multilingual speech synthesis. Notably, in our experiments PETL methods able to achieve comparable or even better performance compared to full fine-tuning with only $\sim$2.5% tunable parameters.

Leveraging Parameter-Efficient Transfer Learning for Multi-Lingual Text-to-Speech Adaptation

Yingting Li ¹, Ambuj Mehrish ², Bryan Chew ², Bo Cheng ¹, Soujanya Poria ², ¹Bei**g University of Posts and Telecommunications, ²Singapore University of Technology and Design, {cindyyting, chengbo}@bupt.edu.cn {bryan_chew}@mymail.sutd.edu.sg {ambuj_mehrish, sporia}@sutd.edu.sg}

1 Introduction

Multilingual speech synthesis, generating speech in multiple languages from text input, represents a major advancement in speech processing with wide-reaching implications for global communication Tan et al. (2021); Mehrish et al. (2023b). Unlike single-language systems, multilingual architectures break linguistic barriers, transforming education, entertainment, healthcare, and customer service by facilitating seamless communication across languages Marais et al. (2020); Seong et al. (2021); Le et al. (2024); Panda et al. (2020).

Current multilingual TTS architectures face challenges Nuthakki et al. (2023); Kaur and Singh (2023), including the complexity of modeling diverse linguistic structures, phonetic variations, and prosodic features across languages. Resource constraints, such as the availability of multilingual corpora and linguistic expertise, can impede model development, particularly for low-resource languages or underrepresented dialects Tan et al. (2021); Mehrish et al. (2023b). Addressing these challenges requires concerted efforts in data collection, model development, and evaluation.

Figure 1: (a) SpeechT5 TTS architecture. (b) Encoder/decoder architecture with adapter. (c) HyperGenrator architecture.

The advancement of architectural designs, coupled with pre-training models such as SpeechT5 Ao et al. (2021), reflects challenges similar to those encountered in NLP. Achieving optimal performance through fine-tuning these models for diverse downstream tasks or domain adaptations requires substantial task-specific datasets. Moreover, fine-tuning all model parameters necessitates significant memory resources allocated to each task. With limited data available for various underrepresented languages, full fine-tuning can further leads to poor generalization. Researchers have sought solutions to these challenges through the exploration of PETL methods Li et al. (2023); Oh et al. (2023); Chen et al. (2023); Sathyendra et al. (2022); Vanderreydt et al. (2023); Le et al. (2021). However, their investigation remains limited for TTS adaptation.

In this paper, we extends PETL approaches to the multilingual TTS, focusing on adapter Houlsby et al. (2019) and Hyper-Network Üstün et al. (2022). We pioneer the hyper-networks for multilingual TTS adaptation and introduces the Multi-Conditioned HyperGenerator for multilingual TTS. Our major contributions includes: (1) Regular & Dynamic Adapters: We embed language-specific parameters into SpeechT5 using regular adapters and explore a hyper-network to generate these parameters, referred to as HyperGenerator. (2) Parameter Efficiency: We achieve comparable or superior performance to full fine-tuning using only about $2.44\%$ of the parameters. (3) Improved Zero-shot Performance: HyperGenerator outperforms full finetuning and regular adapters on an unseen language with the same parameter count.

2 Related work

Research on PETL methods for large multilingual pretrained models like XLSR Vanderreydt et al. (2023) has gained significant attention, notably in Automatic Speech Recognition (ASR) Fu et al. (2023); Zhang et al. (2023); Yu et al. (2023); Yang et al. (2023); Shi and Kawahara (2024). Li et al. Li et al. (2023) propose a benchmark utilizing XLSR-53 Conneau et al. (2020), employing PETL such as regular adapters Pfeiffer et al. (2020), prefix tuning Li and Liang (2021), and LoRA Hu et al. (2021). Le et al. Le et al. (2021) and Zhao et al. Zhao et al. (2022) explore multilingual neural machine translation, focusing on lightweight adapter tuning. Morioka et al. Morioka et al. (2022) advocate for integrating regular adapters with TTS models for few-shot speaker adaptation, while Mehrish et al. Mehrish et al. (2023a) introduce a mixture of experts for low-resource speaker adaptation.

3 Methodology

3.1 Base Model Architecture: SpeechT5

SpeechT5 Ao et al. (2021) merges NLP and speech synthesis techniques, extending the transformer-based T5 architecture Raffel et al. (2020). It integrates self-attention mechanisms and CNNs to capture both temporal dependencies and spectral features in speech. By pre-training on large-scale speech corpora and fine-tuning on specific datasets, SpeechT5 excels in tasks such as speech recognition, TTS, and speech translation.

3.2 Adapter

In this work, we integrate language-specific parameters using adapter modules, commonly employed in the NLP for multilingual or multi-task scenarios Ansell et al. (2021). Following the formulation of Houlsby et al. (2019), we insert one adapter block after each convolutional block of every transformer module in the SpeechT5 model as shown in Figure 1. Each adapter module, with fewer parameters compared to the main network (SpeechT5), down-projects the input to a lower-dimensional space, applies a non-linearity, and then up-projects back to the original dimensions. A residual connection is added to produce the final output. During language adaptation, only the adapter parameters are updated while kee** the main network frozen.

Model	de		fr		fi		hu		nl		avg		Params
	MCD	CER	MCD	CER	MCD	CER	MCD	CER	MCD	CER	MCD	CER
Finetune Multilingual w/o Pretrain	$4.88$	$7.19$	$4.91$	$14.40$	$4.76$	$15.16$	$4.93$	$16.29$	$5.46$	$10.56$	$4.99$	$12.72$	144M(100%)
Finetune Multilingual w/ Pretrain	$4.87$	$6.66$	$4.95$	$11.82$	$4.82$	$12.64$	$5.00$	$15.52$	$5.51$	$10.28$	$5.03$	$11.38$	144M(100%)
Adapter Multilingual w/ Pretrain	$4.75$	6.30	$4.93$	$11.81$	$4.75$	$9.58$	$4.92$	$16.11$	$5.40$	$10.37$	$4.95$	$10.83$	3.56M(2.47%)
HyperAdapter Multilingual w/ Pretrain	$4.78$	$6.52$	$5.04$	$15.50$	$4.67$	7.05	4.87	13.13	5.36	$10.96$	$4.94$	10.63	3.52M(2.44%)
Finetune Monolingual w/ Pretrain	$4.86$	$6.44$	$4.85$	10.80	$4.67$	$15.09$	$4.90$	$15.54$	$5.53$	8.47	$4.96$	$11.27$	144M(100%)
Adapter Monolingual w/ Pretrain	$4.77$	$6.82$	$4.82$	$14.32$	4.63	$9.41$	$4.94$	$14.53$	$5.36$	$14.22$	4.90	$11.86$	3.56M(2.47%)
HyperAdapter Monolingual w/ Pretrain	4.71	$7.94$	4.82	$14.66$	$4.77$	$13.47$	$4.93$	$14.00$	$5.40$	$13.21$	$4.93$	$12.66$	3.52M(2.44%)

Table 1: Evaluation results for seen languages along with the percentage of parameters updated during training.

3.3 HyperGenerator

HyperGenerator consists of a hyper network Üstün et al. (2022) that generates the weights of all adapter modules. As depicted in Figure 1, a single hyper-network is employed to create adapters for multiple languages and layers, with conditioning on $(s,l,p)$, where $s$ denotes speaker embeddings, $l$ represents the target language, and $p$ indicates the encoder or decoder layer ID. This method, unlike traditional adapters, promotes cross-language and cross-layer information sharing, enabling the hyper-network to efficiently distribute its capacity among them. By adapting parameters based on speaker characteristics and language specifics, the hyper-network augments the effectiveness of adapters. Furthermore, to ensure network efficiency, we utilize a shared hyper-network to generate adapter parameters across all layers within the TTS backbone, further conditioning it with the layer ID $p$.

4 Experimental Setup

4.1 Baseline and Dataset

We developed a baselines with the following $3$ configurations for comparing the performance of the adapter and HyperGenerator with full fine-tuning.

Monolingual: We finetune the SpeechT5 individually for each language, uniquely optimizing its parameters to enhance speech synthesis performance.

Multilingual: We finetune the SpeechT5 with diverse speech data from multiple languages. This improves the model’s ability to understand and generate speech across various linguistic contexts, capturing cross-lingual patterns, phonetic variations, and language-specific features.

Figure 2: Multilingual Masked Text Pretraining. Where $\mathcal{L}_{mlm}$ and $\mathcal{L}_{tts}$ is mask language modeling and reconstruction loss respectively.

Multilingual Masked Text Pretraining: Multilingual models like multilingual BERT Devlin et al. (2018) have demonstrated strong cross-lingual transfer capabilities in NLP tasks. Leveraging multilingual pre-training improves generalization to other languages without specific target data. In this settings, we extend MLM pre-training to SpeechT5 to enhance pronunciation and prosody transfer. The left side of Figure 2 illustrates the unsupervised pre-training of SpeechT5’s text encoder and decoder using text-only data $\mathcal{D}{{}_{text}}$ with MLM. The pre-trained text encoder is then integrated into the TTS pipeline, as shown on the right side of Figure 2, and trained on paired speech-text data $\mathcal{D}{{}_{paired}}$.

(a)

(b)

Figure 3: Subjective evaluation on naturalness: MOS score

For fine-tuning using monolingual and multilingual configuration, as discussed in Section 4.1, we leverage German (de), French (fr), Finnish (fi), Hungarian (hu), and Dutch (nl)—as the five seen European languages from the CSS10 dataset Park and Mulc (2019). To evaluate zero-shot performance, we use Spanish (es) as an unseen language. For Multilingual Masked Text Pretraining, we utilize transcripts from VoxPopuli Wang et al. (2021), M-AILABS Bakhturina et al. (2021), and CSS10 Park and Mulc (2019) to pre-train the SpeechT5 text encoder-decoder for a character-based masked language modeling task.

4.2 Training and Evaluation

We follow the data partition outlined in Saeki et al. (2023). We use pretrained chekcpoint of SpeechT5 for all experiments. Speaker embeddings are set at a dimension of $256$, while language embeddings are initialized using pretrained weights from lang2vec Littell et al. (2017). The layer embedding dimension is set at $64$. The bottleneck dimension for adapters is $128$, whereas for HyperGenerator, is $32$ for ensuring the same number of parameters across both architectures. We employed MCD Kominek et al. (2008) and assess intelligibility using Character Error Rates (CERs) computed with the multilingual ASR Radford et al. (2023) as objective metrics. Furthermore, to evaluate naturalness, we conducted listening tests to calculate the MOS of synthesized speech. We recruited five native speaker via Amazon Mechanical Turk (AMT) for each of the languages.

Model	es
	MCD	CER
Finetune Multilingual w/o Pretrain	5.75	39.32
Finetune Multilingual w/ Pretrain	5.87	34.80
Adapter Multilingual w/ Pretrain	5.39	45.94
HyperAdapter Multilingual w/ Pretrain	5.28	18.79

Table 2: Evaluation results for unseen language (es).

5 Results

5.1 Objective and Subjective Evaluation

Table 1 shows that Finetune Multilingual with Pretraining outperforms Finetune Multilingual without Pretraining due to multilingual masked text pretraining. Both Adapter and HyperGenerator achieve similar or better performance than full fine-tuning with significantly fewer parameters. Full fine-tuning updates 144M parameters, while Adapter and HyperGenerator use only 3.56M and 3.52M parameters, respectively, with HyperGenerator showing superior performance in multilingual settings with text pretraining.

While the performance gain for HyperGenerator with seen languages is modest, it shows promise for zero-shot multilingual speech synthesis. Table 2 shows that HyperGenerator achieved a CER of 18.79% for Spanish, significantly lower than the over 30% CER for Multilingual Fine-tuning and adapter-based approaches. This highlights HyperGenerator’s dynamic adaptability and potential for efficient and accurate zero-shot synthesis. Figure 4 further demonstrates this, as speech from the same language clusters together, indicating HyperGenerator’s ability to adjust parameters based on language, unlike static adapters.

Figure 4: t-SNE plot of HyperGenerator parameters for 6 languages from the CSS10 test set, with same colors denoting speech samples from the same language.

MOS results (Figure 3(a) and Figure 3(b)) indicate that Adapters and HyperGenerator perform as well as or better than full fine-tuning in multilingual contexts. HyperGenerator consistently achieved the highest scores, with values of 3.09 for de and 3.81 for hu, demonstrating superior naturalness. Similar trends in monolingual scenarios highlight HyperGenerator’s effectiveness in generating high-quality speech across different languages.

6 Conclusion

In this paper, we advances multilingual speech synthesis using PETL methods like adapter fine-tuning, achieving SOTA performance with fewer parameters. Introducing regular and dynamic adapters with a hyper-network enhances efficiency and zero-shot performance. Future work could optimize adapters for specific languages, improve cross-lingual transfer learning, and reduce model complexity while maintaining high performance

Limitations

While our proposed approach shows promise in advancing multilingual TTS synthesis, there are several limitations that must be acknowledged. Addressing these challenges will be crucial for enhancing the robustness and applicability of our methods across a wider range of languages and use cases. The key limitations are as follows:

• The performance of hypernetworks and adapters can vary greatly depending on the hyperparameters used. Adjusting these settings for each language and task is often a complex and time-consuming process that requires significant computational resources.

• Languages such as Russian and Greek use scripts that differ from the Latin alphabet, like Cyrillic and Greek scripts, respectively. These scripts have unique rules for how letters and sounds are represented. The current PETL methods might not fully address these differences, resulting in lower quality speech synthesis for these languages.

• Symbolic languages, such as Chinese and Japanese, have unique linguistic elements like Chinese logograms and Japanese kana, as well as complex grammatical structures in languages like Russian and Greek. The proposed architecture in its current form can struggle to handle these diverse features effectively, which means modification to these adaptation techniques are needed to improve performance.

Potential Risk

While our research aims to advance multilingual TTS technology, it is crucial to acknowledge the potential risks associated with such systems. We will discuss some associated risks as follows :

• Malicious Use and Disinformation: The ability of TTS systems to generate highly realistic speech could be used to create disinformation. This could lead to the spread of false information, manipulation of opinion, and erosion of trust in digital content.

• Our research utilizes the publicly available CSS10 dataset, however utilization of personalized data to adapt these models can have the risk of privacy violations. Therefore it is important to follow best data management practices that do not inadvertently compromise privacy.

• TTS systems are vulnerable to adversarial attacks where small perturbations to the input can lead to significant changes in the output. Although the proposed framework is robust to noise, the necessary security measures and continuous testing of the system against potential attacks can enhance resilience.

Ethical Considerations

The TTS system could be used to produce misleading or harmful content. For instance, synthesized speech could be exploited to create fake audio recordings that mimic real individuals, potentially leading to misinformation or fraud. Additionally, the accessibility of TTS technology might raise concerns about the unauthorized use of voices, infringing on personal privacy and intellectual property rights.