Technology Pipeline for Large Scale Cross-Lingual Dubbing of Lecture Videos into Multiple Indian Languages
Authors:
Anusha Prakash,
Arun Kumar,
Ashish Seth,
Bhagyashree Mukherjee,
Ishika Gupta,
Jom Kuriakose,
Jordan Fernandes,
K V Vikram,
Mano Ranjith Kumar M,
Metilda Sagaya Mary,
Mohammad Wajahat,
Mohana N,
Mudit Batra,
Navina K,
Nihal John George,
Nithya Ravi,
Pruthwik Mishra,
Sudhanshu Srivastava,
Vasista Sai Lodagala,
Vandan Mujadia,
Kada Sai Venkata Vineeth,
Vrunda Sukhadia,
Dipti Sharma,
Hema Murthy,
Pushpak Bhattacharya
, et al. (2 additional authors not shown)
Abstract:
Cross-lingual dubbing of lecture videos requires the transcription of the original audio, correction and removal of disfluencies, domain term discovery, text-to-text translation into the target language, chunking of text using target language rhythm, text-to-speech synthesis followed by isochronous lipsyncing to the original video. This task becomes challenging when the source and target languages…
▽ More
Cross-lingual dubbing of lecture videos requires the transcription of the original audio, correction and removal of disfluencies, domain term discovery, text-to-text translation into the target language, chunking of text using target language rhythm, text-to-speech synthesis followed by isochronous lipsyncing to the original video. This task becomes challenging when the source and target languages belong to different language families, resulting in differences in generated audio duration. This is further compounded by the original speaker's rhythm, especially for extempore speech. This paper describes the challenges in regenerating English lecture videos in Indian languages semi-automatically. A prototype is developed for dubbing lectures into 9 Indian languages. A mean-opinion-score (MOS) is obtained for two languages, Hindi and Tamil, on two different courses. The output video is compared with the original video in terms of MOS (1-5) and lip synchronisation with scores of 4.09 and 3.74, respectively. The human effort also reduces by 75%.
△ Less
Submitted 1 November, 2022;
originally announced November 2022.
S-vectors and TESA: Speaker Embeddings and a Speaker Authenticator Based on Transformer Encoder
Authors:
N J Metilda Sagaya Mary,
S Umesh,
Sandesh V Katta
Abstract:
One of the most popular speaker embeddings is x-vectors, which are obtained from an architecture that gradually builds a larger temporal context with layers. In this paper, we propose to derive speaker embeddings from Transformer's encoder trained for speaker classification. Self-attention, on which Transformer's encoder is built, attends to all the features over the entire utterance and might be…
▽ More
One of the most popular speaker embeddings is x-vectors, which are obtained from an architecture that gradually builds a larger temporal context with layers. In this paper, we propose to derive speaker embeddings from Transformer's encoder trained for speaker classification. Self-attention, on which Transformer's encoder is built, attends to all the features over the entire utterance and might be more suitable in capturing the speaker characteristics in an utterance. We refer to the speaker embeddings obtained from the proposed speaker classification model as s-vectors to emphasize that they are obtained from an architecture that heavily relies on self-attention. Through experiments, we demonstrate that s-vectors perform better than x-vectors. In addition to the s-vectors, we also propose a new architecture based on Transformer's encoder for speaker verification as a replacement for speaker verification based on conventional probabilistic linear discriminant analysis (PLDA). This architecture is inspired by the next sentence prediction task of bidirectional encoder representations from Transformers (BERT), and we feed the s-vectors of two utterances to verify whether they belong to the same speaker. We name this architecture the Transformer encoder speaker authenticator (TESA). Our experiments show that the performance of s-vectors with TESA is better than s-vectors with conventional PLDA-based speaker verification.
△ Less
Submitted 12 December, 2021; v1 submitted 11 August, 2020;
originally announced August 2020.
