Skip to main content

Showing 1–17 of 17 results for author: Khurana, S

Searching in archive eess. Search in all archives.
.
  1. arXiv:2404.02252  [pdf, other

    cs.SD eess.AS

    SMITIN: Self-Monitored Inference-Time INtervention for Generative Music Transformers

    Authors: Junghyun Koo, Gordon Wichern, Francois G. Germain, Sameer Khurana, Jonathan Le Roux

    Abstract: We introduce Self-Monitored Inference-Time INtervention (SMITIN), an approach for controlling an autoregressive generative music transformer using classifier probes. These simple logistic regression probes are trained on the output of each attention head in the transformer using a small dataset of audio examples both exhibiting and missing a specific musical trait (e.g., the presence/absence of dr… ▽ More

    Submitted 2 April, 2024; originally announced April 2024.

  2. arXiv:2402.17907  [pdf, other

    eess.AS cs.SD

    NIIRF: Neural IIR Filter Field for HRTF Upsampling and Personalization

    Authors: Yoshiki Masuyama, Gordon Wichern, François G. Germain, Zexu Pan, Sameer Khurana, Chiori Hori, Jonathan Le Roux

    Abstract: Head-related transfer functions (HRTFs) are important for immersive audio, and their spatial interpolation has been studied to upsample finite measurements. Recently, neural fields (NFs) which map from sound source direction to HRTF have gained attention. Existing NF-based methods focused on estimating the magnitude of the HRTF from a given sound source direction, and the magnitude is converted to… ▽ More

    Submitted 27 February, 2024; originally announced February 2024.

    Comments: Accepted to ICASSP 2024

  3. arXiv:2312.07513  [pdf, other

    eess.AS cs.SD

    NeuroHeed+: Improving Neuro-steered Speaker Extraction with Joint Auditory Attention Detection

    Authors: Zexu Pan, Gordon Wichern, Francois G. Germain, Sameer Khurana, Jonathan Le Roux

    Abstract: Neuro-steered speaker extraction aims to extract the listener's brain-attended speech signal from a multi-talker speech signal, in which the attention is derived from the cortical activity. This activity is usually recorded using electroencephalography (EEG) devices. Though promising, current methods often have a high speaker confusion error, where the interfering speaker is extracted instead of t… ▽ More

    Submitted 12 December, 2023; originally announced December 2023.

  4. arXiv:2310.19644  [pdf, other

    eess.AS cs.MM

    Scenario-Aware Audio-Visual TF-GridNet for Target Speech Extraction

    Authors: Zexu Pan, Gordon Wichern, Yoshiki Masuyama, Francois G. Germain, Sameer Khurana, Chiori Hori, Jonathan Le Roux

    Abstract: Target speech extraction aims to extract, based on a given conditioning cue, a target speech signal that is corrupted by interfering sources, such as noise or competing speakers. Building upon the achievements of the state-of-the-art (SOTA) time-frequency speaker separation model TF-GridNet, we propose AV-GridNet, a visual-grounded variant that incorporates the face recording of a target speaker a… ▽ More

    Submitted 30 October, 2023; originally announced October 2023.

    Comments: Accepted by ASRU 2023

  5. arXiv:2310.10604  [pdf, other

    eess.AS cs.SD

    Generation or Replication: Auscultating Audio Latent Diffusion Models

    Authors: Dimitrios Bralios, Gordon Wichern, François G. Germain, Zexu Pan, Sameer Khurana, Chiori Hori, Jonathan Le Roux

    Abstract: The introduction of audio latent diffusion models possessing the ability to generate realistic sound clips on demand from a text description has the potential to revolutionize how we work with audio. In this work, we make an initial attempt at understanding the inner workings of audio latent diffusion models by investigating how their audio outputs compare with the training data, similar to how a… ▽ More

    Submitted 16 October, 2023; originally announced October 2023.

    Comments: Submitted to ICASSP 2024

  6. arXiv:2309.07478  [pdf, other

    cs.CL cs.LG cs.SD eess.AS

    Direct Text to Speech Translation System using Acoustic Units

    Authors: Victoria Mingote, Pablo Gimeno, Luis Vicente, Sameer Khurana, Antoine Laurent, Jarod Duret

    Abstract: This paper proposes a direct text to speech translation system using discrete acoustic units. This framework employs text in different source languages as input to generate speech in the target language without the need for text transcriptions in this language. Motivated by the success of acoustic units in previous works for direct speech to speech translation systems, we use the same pipeline to… ▽ More

    Submitted 14 September, 2023; originally announced September 2023.

    Comments: 5 pages, 4 figures

  7. Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers

    Authors: Yuan Gong, Sameer Khurana, Leonid Karlinsky, James Glass

    Abstract: In this paper, we focus on Whisper, a recent automatic speech recognition model trained with a massive 680k hour labeled speech corpus recorded in diverse conditions. We first show an interesting finding that while Whisper is very robust against real-world background sounds (e.g., music), its audio representation is actually not noise-invariant, but is instead highly correlated to non-speech sound… ▽ More

    Submitted 6 July, 2023; originally announced July 2023.

    Comments: Accepted at Interspeech 2023. Code at https://github.com/yuangongnd/whisper-at

    Journal ref: Proceedings of Interspeech 2023

  8. arXiv:2306.00789  [pdf, other

    cs.CL cs.AI eess.AS eess.SP

    Improved Cross-Lingual Transfer Learning For Automatic Speech Translation

    Authors: Sameer Khurana, Nauman Dawalatabad, Antoine Laurent, Luis Vicente, Pablo Gimeno, Victoria Mingote, James Glass

    Abstract: Research in multilingual speech-to-text translation is topical. Having a single model that supports multiple translation tasks is desirable. The goal of this work it to improve cross-lingual transfer learning in multilingual speech-to-text translation via semantic knowledge distillation. We show that by initializing the encoder of the encoder-decoder sequence-to-sequence translation model with SAM… ▽ More

    Submitted 25 January, 2024; v1 submitted 1 June, 2023; originally announced June 2023.

  9. arXiv:2305.12606  [pdf, other

    cs.CL cs.SD eess.AS

    Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages

    Authors: Andrew Rouditchenko, Sameer Khurana, Samuel Thomas, Rogerio Feris, Leonid Karlinsky, Hilde Kuehne, David Harwath, Brian Kingsbury, James Glass

    Abstract: Recent models such as XLS-R and Whisper have made multilingual speech technologies more accessible by pre-training on audio from around 100 spoken languages each. However, there are thousands of spoken languages worldwide, and adapting to new languages is an important problem. In this work, we aim to understand which model adapts better to languages unseen during pre-training. We fine-tune both mo… ▽ More

    Submitted 30 May, 2023; v1 submitted 21 May, 2023; originally announced May 2023.

    Comments: Accepted at Interspeech 2023

  10. arXiv:2211.07795  [pdf, other

    eess.AS cs.AI cs.LG

    On Unsupervised Uncertainty-Driven Speech Pseudo-Label Filtering and Model Calibration

    Authors: Nauman Dawalatabad, Sameer Khurana, Antoine Laurent, James Glass

    Abstract: Pseudo-label (PL) filtering forms a crucial part of Self-Training (ST) methods for unsupervised domain adaptation. Dropout-based Uncertainty-driven Self-Training (DUST) proceeds by first training a teacher model on source domain labeled data. Then, the teacher model is used to provide PLs for the unlabeled target domain data. Finally, we train a student on augmented labeled and pseudo-labeled data… ▽ More

    Submitted 14 November, 2022; originally announced November 2022.

  11. arXiv:2205.08180  [pdf, other

    cs.CL cs.LG cs.SD eess.AS

    SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation

    Authors: Sameer Khurana, Antoine Laurent, James Glass

    Abstract: We propose the SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation learning framework. Unlike previous works on speech representation learning, which learns multilingual contextual speech embedding at the resolution of an acoustic frame (10-20ms), this work focuses on learning multimodal (speech-text) multilingual speech embedding at the resolution of a s… ▽ More

    Submitted 17 May, 2022; originally announced May 2022.

  12. arXiv:2203.06760  [pdf, other

    cs.SD cs.AI eess.AS

    CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification

    Authors: Yuan Gong, Sameer Khurana, Andrew Rouditchenko, James Glass

    Abstract: Audio classification is an active research area with a wide range of applications. Over the past decade, convolutional neural networks (CNNs) have been the de-facto standard building block for end-to-end audio classification models. Recently, neural networks based solely on self-attention mechanisms such as the Audio Spectrogram Transformer (AST) have been shown to outperform CNNs. In this paper,… ▽ More

    Submitted 13 March, 2022; originally announced March 2022.

  13. Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0

    Authors: Sameer Khurana, Antoine Laurent, James Glass

    Abstract: We propose a simple and effective cross-lingual transfer learning method to adapt monolingual wav2vec-2.0 models for Automatic Speech Recognition (ASR) in resource-scarce languages. We show that a monolingual wav2vec-2.0 is a good few-shot ASR learner in several languages. We improve its performance further via several iterations of Dropout Uncertainty-Driven Self-Training (DUST) by using a modera… ▽ More

    Submitted 7 October, 2021; originally announced October 2021.

  14. arXiv:2106.05933  [pdf, other

    cs.CL cs.LG cs.SD eess.AS

    PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition

    Authors: Cheng-I Jeff Lai, Yang Zhang, Alexander H. Liu, Shiyu Chang, Yi-Lun Liao, Yung-Sung Chuang, Kaizhi Qian, Sameer Khurana, David Cox, James Glass

    Abstract: Self-supervised speech representation learning (speech SSL) has demonstrated the benefit of scale in learning rich representations for Automatic Speech Recognition (ASR) with limited paired data, such as wav2vec 2.0. We investigate the existence of sparse subnetworks in pre-trained speech SSL models that achieve even better low-resource ASR results. However, directly applying widely adopted prunin… ▽ More

    Submitted 26 October, 2021; v1 submitted 10 June, 2021; originally announced June 2021.

  15. arXiv:2011.13439  [pdf, other

    cs.CL cs.LG cs.SD eess.AS

    Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training

    Authors: Sameer Khurana, Niko Moritz, Takaaki Hori, Jonathan Le Roux

    Abstract: The performance of automatic speech recognition (ASR) systems typically degrades significantly when the training and test data domains are mismatched. In this paper, we show that self-training (ST) combined with an uncertainty-based pseudo-label filtering approach can be effectively used for domain adaptation. We propose DUST, a dropout-based uncertainty-driven self-training technique which uses a… ▽ More

    Submitted 16 February, 2021; v1 submitted 26 November, 2020; originally announced November 2020.

    Comments: ICASSP 2021

  16. arXiv:2006.02814  [pdf, other

    eess.AS cs.CL cs.LG cs.SD

    CSTNet: Contrastive Speech Translation Network for Self-Supervised Speech Representation Learning

    Authors: Sameer Khurana, Antoine Laurent, James Glass

    Abstract: More than half of the 7,000 languages in the world are in imminent danger of going extinct. Traditional methods of documenting language proceed by collecting audio data followed by manual annotation by trained linguists at different levels of granularity. This time consuming and painstaking process could benefit from machine learning. Many endangered languages do not have any orthographic form but… ▽ More

    Submitted 5 August, 2020; v1 submitted 4 June, 2020; originally announced June 2020.

  17. arXiv:2006.02547  [pdf, other

    eess.AS cs.CL cs.LG cs.SD

    A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning

    Authors: Sameer Khurana, Antoine Laurent, Wei-Ning Hsu, Jan Chorowski, Adrian Lancucki, Ricard Marxer, James Glass

    Abstract: Probabilistic Latent Variable Models (LVMs) provide an alternative to self-supervised learning approaches for linguistic representation learning from speech. LVMs admit an intuitive probabilistic interpretation where the latent structure shapes the information extracted from the signal. Even though LVMs have recently seen a renewed interest due to the introduction of Variational Autoencoders (VAEs… ▽ More

    Submitted 8 September, 2020; v1 submitted 3 June, 2020; originally announced June 2020.

    Comments: Proceedings of Interspeech, 2020