Towards Building an End-to-End Multilingual Automatic Lyrics Transcription Model

Abstract

Multilingual automatic lyrics transcription (ALT) is a challenging task due to the limited availability of labelled data and the challenges introduced by singing, compared to multilingual automatic speech recognition. Although some multilingual singing datasets have been released recently, English continues to dominate these collections. Multilingual ALT remains underexplored due to the scale of data and annotation quality. In this paper, we aim to create a multilingual ALT system with available datasets. Inspired by architectures that have been proven effective for English ALT, we adapt these techniques to the multilingual scenario by expanding the target vocabulary set. We then evaluate the performance of the multilingual model in comparison to its monolingual counterparts. Additionally, we explore various conditioning methods to incorporate language information into the model. We apply analysis by language and combine it with the language classification performance. Our findings reveal that the multilingual model performs consistently better than the monolingual models trained on the language subsets. Furthermore, we demonstrate that incorporating language information significantly enhances performance.

I Introduction

Automatic lyrics transcription (ALT) is the task of recognising lyrics from singing voice. Access to lyrics enriches the listening experience by arousing sympathy and building a deeper connection between the music and the listeners. Moreover, lyrics transcription can benefit other music analysis tasks as well, including lyrics alignment [4], singing pronunciation analysis [3], and cover song identification [19]. While ALT shares the same input/output format and similar objectives with automatic speech recognition (ASR), it is more challenging due to larger variations in rhythm, pitch, and pronunciation [6, 11]. In recent years, significant progress has been achieved in ALT for English songs using end-to-end models [13, 7, 16]. Stoller et al. [16] developed the first end-to-end lyrics alignment model using the Wave-U-Net architecture and connectionist temporal classification (CTC) loss [8]. Gao et al. enhanced performance by fine-tuning the ALT model in conjunction with a source separation frontend [7]. Ou et al. [13] leveraged wav2vec2 features and applied transfer learning techniques, resulting in a significant performance boost.

Multilingual lyrics transcription, however, remains under-explored due to the limited publicly available training and evaluation data. Although DALI v2 [12] is a singing dataset of moderate size with lyrics annotations, it is dominated by English songs, comprising over 80% of the dataset. Whisper was introduced by OpenAI [14]. It is a robust ASR model trained on numerous audio-transcript pairs collected from the Internet, and the training data remains unreleased. This model has demonstrated its effectiveness in multilingual ALT [24]. Building on this work, Wang et al. [21] investigated the potential of adapting it to Mandarin Chinese ALT. A multilingual ALT dataset called MulJam was created by post-processing Whisper’s output [24]. With access to these datasets, we develop multilingual ALT models using publicly available data by combining DALI and MulJam.

Compared to English ALT, multilingual ALT faces several additional challenges: Firstly, unless explicitly specified, models must implicitly identify the underlying language of the singing to ensure that the predicted lyrics match the correct character set. Secondly, there is a language imbalance in the datasets. Languages like English typically dominate the majority of songs, while other languages are considered low-resource in the context of ALT development. Thirdly, specific characters appear in different languages’ alphabets but adhere to different pronunciation rules, adding complexity to the problem.

The multilingual ASR task shares many of the challenges mentioned above. Previous research has extensively studied and compared mono-, cross-, and multilingual models using a single model [9, 18]. These works demonstrate that multilingual models tend to perform better than their mono-/cross-lingual counterparts, particularly in low-resource settings. Furthermore, some studies observe performance gains by incorporating language information through conditioning [18] or predicting language-specific tokens [23]. More recent research takes advantage of unlabeled data through self-supervised learning, such as wav2vec2 [1, 2].

(a) The multilingual and monolingual models.

(b) The language-informed model.

(c) The language self-conditioned model.

Figure 1: Proposed model architectures at training. Our models consists of a convolutional block $\mathit{CNNBlock}$, a transformer encoder $\mathit{TfmEnc}$, a transformer decoder $\mathit{TfmDec}$, and several fully connected layers $\mathit{FC_{ctc}}$, $\mathit{FC_{s2s}}$, and $\mathit{FC_{lang}}$. The dotted lines indicate feature concatenation (with map** for 1(c)). In 1(b), $emb$ denotes the language embedding.

In this work, we aim to investigate the development of multilingual ALT models using publicly accessible data, building on existing work specifically designed for English ALT. We tackle the low-resource aspects of ALT model development by training jointly on data from across a wide range of languages. Additionally, we study the impact of language information via conditioning and multi-task learning. Our contributions can be summarized as follows: Firstly, as one of the first attempts towards multilingual ALT, we propose a small-scale multilingual ALT model trained on publicly available datasets. Secondly, we compare the performance of multilingual models with their monolingual counterparts, revealing that training data in additional languages benefits ALT in low-resource scenarios. Thirdly, we show that language conditioning has a positive impact on performance, while the amount of improvement varies across different languages.

We acknowledge that previous research in multilingual ASR has explored similar approaches, as mentioned above. However, ALT diverges significantly due to different acoustic characteristics of the singing voice, the severe shortage of resources, and language imbalances. Therefore, fundamental assumptions need to be verified carefully. Our results indicate that our work will provide a solid baseline for future research, as well as an initial step in addressing the new challenges.

II Method

II-A Model

Our models are built upon a similar architecture to the state-of-the-art transformer models [7], utilising the hybrid CTC/Attention architecture [22]. Fig. 1(a) illustrates this architecture at the training stage. The input is an 80-dim Mel-spectrogram computed at a sampling rate of 16kHz, with an FFT size of 400 and a hop size of 10 ms. The model consists of a convolutional block, a transformer encoder, a transformer decoder and two fully connected layers. The target dimension of the two fully connected layers is equal to the size of the target character set $N$. Let $y$ represent the target lyrics token list, $y^{bos}$ denote $y$ with a <bos> token added at the beginning, and $y^{eos}$ indicate $y$ with a <eos> token appended at the end. During training, the Mel-spectrogram is processed through the convolutional block, before being passed to the transformer encoder and decoder. Teacher forcing is adopted for faster convergence (The model is provided with the ground truth tokens $y^{bos}$ and predicts the next ones $y^{eos}$.):

$$\mathit{feat} =\mathit{CNNBlock}(mel)$$ (1)

$$h =\mathit{TfmEnc}(\mathit{feat})$$ (2)

$$o =\mathit{TfmDec}(h,y^{bos})$$ (3)

where $h$ and $o$ are the output from the encoder and the decoder. Then $h$ and $o$ are passed to the two fully connected layers $FC_{ctc}$ and $FC_{s2s}$. These layers are responsible for generating the posteriorgrams for both the CTC branch and the sequence-to-sequence (seq2seq) branch.

$$p_{ctc} =\mathit{FC_{ctc}}(h)$$ (4)

$$p_{s2s} =\mathit{FC_{s2s}}(o)$$ (5)

The loss function is a weighted sum of two components: the CTC loss for alignment-free training (computed from the CTC branch), and the Kullback-Leibler (KL) divergence loss for smooth predictions (computed from the seq2seq branch):

$$Loss=\alpha L_{ctc}(p_{ctc},y)+(1-\alpha)L_{s2s}(p_{s2s},y^{eos})$$ (6)

II-B Multilingual model and monolingual models

Let there be $M$ languages in the training set $\{\mathcal{L}_{1},...,\mathcal{L}_{M}\}$, where $\mathcal{C}_{i}$ represents the character set for language $\mathcal{L}_{i}$. In the multilingual setting, the target character set $\mathcal{C}$ is formed by taking the union of all independent character sets $\cup^{M}_{i=1}\mathcal{C}_{i}$.

In addition, we train individual monolingual models for each language to provide comparative analysis with multilingual models. The corresponding training and validation sets are the language-specific subsets derived from the multilingual data.

II-C Language-informed models

We condition our multilingual model with language information to study its influence. By providing the model with knowledge of the target language, it has the potential to learn language-specific features through the encoder and predict characters belonging to the target language alphabet through the decoder.

To be more specific, an embedding is assigned to each language (Fig. 1(b)). During training, we explore three approaches: appending the language embedding to the input of the encoder ($\mathit{feat}$), to the input of the decoder ($h$), and to both. The three conditioned models are respectively denoted as Enc-Cond, Dec-Cond, and EncDec-Cond.

II-D Language self-conditioned model

To gain deeper insights into the model’s capability to identify the correct language, we make the language identification ability measurable by taking a multi-task learning approach (Fig. 1(c)). The output of the encoder is averaged over time and passed to a fully connected layer $\mathit{FC_{lang}}$ to predict language ID. The predicted language probability $p_{l}$ is used as a self-conditioning vector, mapped to the embedding dimension, and appended to the input of the decoder. In this configuration, a cross-entropy loss term for language identification is added to the overall loss function, where $l$ is the language label:

$$\begin{split}Loss=&\alpha L_{ctc}(p_{ctc},y)+(1-\alpha)L_{s2s}(p_{s2s},y^{eos})\\ &+\beta L_{CE}(p_{l},l)\end{split}$$ (7)

	Train		Valid	Test
	DALI	MulJam	MulJam	Jamendo
English	295444	85773	542	868
French	11959	33322	760	809
Spanish	10317	15146	566	881
German	26343	3208	710	871
Italian	9164	7807	616	0
Russian	0	1805	317	0

TABLE I: The numbers of utterances in the training, validation, and test sets in each language.

III Experiments

III-A Datasets

The models are trained on the DALI v2 [12] and the MulJam [24] datasets. DALI v2 contains 7756 songs in total in more than 30 languages, among which we take the 5 languages that have more than 200 songs each: English, French, German, Spanish, and Italian. We segment the songs to line level, with paired lyrics annotations. MulJam contains 6031 songs with line-level lyrics annotations in 6 languages: English, French, German, Spanish, Italian, and Russian. For each language, 20 songs are randomly selected for validation.

The training set is a combination of DALI-train and MulJam-train. It is important to note that the lyrics annotations in DALI do not include accented or special characters. Instead, they are converted to the Latin alphabet. Therefore, the validation and test sets for DALI v2 may not represent the real multilingual problem. We exclusively use the MulJam validation set for validation purposes. For training and validation sets, we apply additional filtering to exclude utterances with incorrectly annotated durations, excessively long durations ($>$30s), and abnormally high character rates ($>$37.5 Hz). All utterances are source-separated by Open-Unmix [17].

All models are evaluated on the MultiLang Jamendo dataset [5] at line level. It consists of 80 songs in 4 languages: English, French, Spanish, and German. Line-level segments are prepared according to the line-level timestamps provided by the dataset. Tab. I shows the statistics of the data for multilingual and monolingual experiments .

III-B Model Configuration

The convolutional block contains 3 CNN blocks with 64 channels. The first two layers have a kernel size of 5 and a stride of 2, while the last layer has both the kernel and stride set to 1. Positional encoding [20] is added to the transformer input before passing through the encoder. The transformer encoder has 12 layers and the transformer decoder has 6 layers. Each encoder layer consists of a multi-head attention and a position-wise feed-forward layer. Each decoder layer contains the same except that the attention layer is causal. The attention dimension is set to 512, the number of heads is 4 and the position-wise feed-forward layer dimension is 2048.

The loss weighting parameter $\alpha$ is set to 0.3, and $\beta$ is set to 0.1. The language embedding for language-informed and language self-conditioned models has a fixed size of 5 for all 6 languages. The union character set size, encompassing all 6 languages, is 91. This includes the Latin alphabet, accented and special characters, and the Cyrillic alphabet for Russian.

Figure 2: The multilingual vocabulary. <bos> and <eos> denote the beginning and the end of a line. <unk> is the unknown token. Epsilon $\varepsilon$ is included for the CTC computation.

III-C Training and Inference

Our models are built upon the speechbrain [15] transformer recipe for ASR . The number of languages $M$ is 6. They are trained using the Adam optimizer [10] and Noam learning rate scheduler [20]. The initial learning rate is 0.001 and the number of warm-up steps is 25000. The number of epochs is 50 for all models, except for the non-English monolingual ones, which have 70 epochs. The checkpoint with the lowest word error rate on the validation set is selected. During validation and testing, beam search is employed on the transformer decoder to select the best prediction autoregressively. The beam size is 10 at validation and 66 at testing. We use Word Error Rate (WER) to assess the performance of ALT models.

III-D Models for comparison

For our core experiments, we intentionally avoid using ALT models or feature extractors pretrained with speech data, such as wav2vec2 and Whisper, although we are aware that incorporating these could benefit the performance. This is since using speech models would introduce impact from the data distribution of the pretrained models, making it difficult to estimate if any difference is due to training on more languages’ data or the knowledge acquired during pretraining. For similar reasons, we avoid using language models.

We still report the WER using a multilingual wav2vec2 (W2V2) variant (large-xlsr-53 [2]) as a reference to the WER range that an adapted state-of-the-art English ALT method [13] can achieve. The W2V2 ALT model uses a wav2vec2 feature extractor frontend, and a hybrid CTC/attention backend, similar to the proposed multilingual transformer. The CTC branch is a fully connected layer while the seq2seq branch is a one-layer recurrent neural network with attention. The attention dimension is set to 256 and the hidden size is also 256. The target character set is the multilingual vocabulary $\mathcal{C}$. All other configurations are the same as in [20].

IV Results

	Transformer	W2V2	Whisper
	Multilingual	XLSR-53	large-v3
English	51.45	42.67	36.80
French	68.40	54.74	49.33
Spanish	68.02	45.02	41.15
German	70.18	49.29	44.52
All	64.31	47.95	42.95

TABLE II: WER (%) of the multilingual transformer, the wav2vec2-based multilingual models, and Whisper.

IV-A Comparison with the State-of-The-Art

Tab. II lists the performance of our multilingual model, the wav2vec2-based model, and Whisper. As expected, the W2V2-based model outperforms our multilingual transformer, and Whisper performs the best due to its training with diverse data in different environments and setups. Even when utilizing pretrained models, the WERs for non-English singing remain higher than that reported for the English monolingual model in [13], indicating greater challenges of multilingual ALT.

	Monolingual	Multilingual	Self-condition
English	53.19	51.45	50.99
French	76.06	68.40	70.26
Spanish	79.35	68.02	68.17
German	82.32	70.18	67.48
All	72.37	64.31	64.07

TABLE III: WER (%) of monolingual, multilingual, and language self-conditioned models.

	Multilingual	Enc-Cond	Dec-Cond	EncDec-Cond
English	51.45	51.19	50.61	50.80
French	68.40	65.01	67.33	65.22
Spanish	68.02	62.27	65.44	61.38
German	70.18	63.23	63.95	62.07
All	64.31	60.32	61.71	59.79

TABLE IV: WER (%) of multilingual and language-informed models.

IV-B Monolingual, multilingual and language self-conditioning

Tab. III lists the performance of monolingual, multilingual, and the language self-conditioned models. Notably, for the monolingual models, despite having more data than Spanish, the German model yields worse WER. This aligns with the nature of the two languages: Spanish is more phonetic and consistent in its spelling and pronunciation than German. Additionally, German uses compounding more frequently than Spanish, leading to more variations in pronunciation and stress patterns. This suggests that the challenge and data requirements for training ALT models vary across languages.

The multilingual model outperforms monolingual models in every language, indicating that having more training data in various languages benefits low-resource language ALT. Specifically, while the improvement for English is small ($\sim$2%), it exceeds 7% for all other languages. This suggests that leveraging high-resource language data (English) can be beneficial for low-resource ALT, when the target languages exhibit similarities in pronunciation and spelling rules.

Compared to the results of the multilingual model, the self-conditioned model is able to perform the additional language classification task without compromising ALT performance. It is not surprising that having the auxiliary task does not bring significant improvements to ALT, as the language class can be rather easily inferred from the predicted lyrics.

IV-C Language-informed models and the Language Classification Accuracy

Tab. IV shows the performance of the multilingual and language-informed models. It can be observed that after providing the language class as input results in improved WER for all languages. Among the three conditioning methods, Enc-Cond performs better than Dec-Cond except for English. EncDec-Cond gives the best overall WER, but the trend varies for each language. After conditioning on both encoder and decoder, there is a clear improvement for non-English languages, while the English WER remains nearly the same as that for the multilingual model. The French monolingual model has a better WER than Spanish and German, but this reverses after language conditioning. This indicates that with sufficient data, French ALT might be more challenging than the other two due to its complexity and frequent silent letters.

To gain a deeper understanding of the distinctions among languages, we examine the language classification confusion matrix of the self-conditioned model in Fig. 3. As depicted, languages other than English are often misclassified as English. Additionally, Spanish singing is more commonly mistaken for Italian than English. It is understandable that for the dominant language in the training set, the classification accuracy for English is close to 100%, which explains why language conditioning has minimal impact on English ALT.

Figure 3: Confusion matrix for the self-conditioned model (%). Languages other than English frequently get confused with English.

V Conclusion and future work

In summary, our study addresses the challenges of multilingual ALT model development when training only from publicly available data, particularly in enhancing low-resource languages. We illustrate that multilingual ALT surpasses monolingual ALT for all languages, primarily due to the shared phonetic similarities among them. Our language-conditioned experiments indicate that incorporating language information enhances performance.

In this study, we take a straightforward approach by merging grapheme sets for multilingual vocabulary processing. In future research, we intend to explore more advanced strategies, including adapting target vocabularies using phoneme representations, subword units, and transliteration techniques. We believe that reducing target ambiguity is particularly critical for effective training in low-resource settings.