-
Predictive Analysis for Optimizing Port Operations
Authors:
Aniruddha Rajendra Rao,
Haiyan Wang,
Chetan Gupta
Abstract:
Maritime transport is a pivotal logistics mode for the long-distance and bulk transportation of goods. However, the intricate planning involved in this mode is often hindered by uncertainties, including weather conditions, cargo diversity, and port dynamics, leading to increased costs. Consequently, accurately estimating vessel total (stay) time at port and potential delays becomes imperative for…
▽ More
Maritime transport is a pivotal logistics mode for the long-distance and bulk transportation of goods. However, the intricate planning involved in this mode is often hindered by uncertainties, including weather conditions, cargo diversity, and port dynamics, leading to increased costs. Consequently, accurately estimating vessel total (stay) time at port and potential delays becomes imperative for effective planning and scheduling in port operations. This study aims to develop a port operation solution with competitive prediction and classification capabilities for estimating vessel Total and Delay times. This research addresses a significant gap in port analysis models for vessel Stay and Delay times, offering a valuable contribution to the field of maritime logistics. The proposed solution is designed to assist decision-making in port environments and predict service delays. This is demonstrated through a case study on Brazil ports. Additionally, feature analysis is used to understand the key factors impacting maritime logistics, enhancing the overall understanding of the complexities involved in port operations.
△ Less
Submitted 25 January, 2024;
originally announced January 2024.
-
Optimal Load Shedding for Public Safety Power Shutoffs
Authors:
Aniruddha Rajendra Rao,
Chandrasekar Venkatraman,
Robert Ellis,
Chetan Gupta
Abstract:
Public utilities are faced with situations where high winds can bring trees and debris into contact with energized power lines and other equipments, which could ignite wildfires. As a result, they need to turn off power during severe weather to help prevent wildfires. This is called Public Safety Power Shutoff (PSPS). We present a method for load reduction using a multi-step genetic algorithm for…
▽ More
Public utilities are faced with situations where high winds can bring trees and debris into contact with energized power lines and other equipments, which could ignite wildfires. As a result, they need to turn off power during severe weather to help prevent wildfires. This is called Public Safety Power Shutoff (PSPS). We present a method for load reduction using a multi-step genetic algorithm for Public Safety Power Shutoff events. The proposed method optimizes load shedding using partial load shedding based on load importance (critical loads like hospitals, fire stations, etc). The multi-step genetic algorithm optimizes load shedding while minimizing the impact on important loads and preserving grid stability. The effectiveness of the method is demonstrated through network examples. The results show that the proposed method achieves minimal load shedding while maintaining the critical loads at acceptable levels. This approach will help utilities to effectively manage PSPS events and reduce the risk of wildfires caused by the power lines.
△ Less
Submitted 13 November, 2023;
originally announced November 2023.
-
Systematic Review of Experimental Paradigms and Deep Neural Networks for Electroencephalography-Based Cognitive Workload Detection
Authors:
Vishnu KN,
Cota Navin Gupta
Abstract:
This article summarizes a systematic review of the electroencephalography (EEG)-based cognitive workload (CWL) estimation. The focus of the article is twofold: identify the disparate experimental paradigms used for reliably eliciting discreet and quantifiable levels of cognitive load and the specific nature and representational structure of the commonly used input formulations in deep neural netwo…
▽ More
This article summarizes a systematic review of the electroencephalography (EEG)-based cognitive workload (CWL) estimation. The focus of the article is twofold: identify the disparate experimental paradigms used for reliably eliciting discreet and quantifiable levels of cognitive load and the specific nature and representational structure of the commonly used input formulations in deep neural networks (DNNs) used for signal classification. The analysis revealed a number of studies using EEG signals in its native representation of a two-dimensional matrix for offline classification of CWL. However, only a few studies adopted an online or pseudo-online classification strategy for real-time CWL estimation. Further, only a couple of interpretable DNNs and a single generative model were employed for cognitive load detection till date during this review. More often than not, researchers were using DNNs as black-box type models. In conclusion, DNNs prove to be valuable tools for classifying EEG signals, primarily due to the substantial modeling power provided by the depth of their network architecture. It is further suggested that interpretable and explainable DNN models must be employed for cognitive workload estimation since existing methods are limited in the face of the non-stationary nature of the signal.
△ Less
Submitted 11 September, 2023;
originally announced September 2023.
-
Example-Based Framework for Perceptually Guided Audio Texture Generation
Authors:
Purnima Kamath,
Chitralekha Gupta,
Lonce Wyse,
Suranga Nanayakkara
Abstract:
Controllable generation using StyleGANs is usually achieved by training the model using labeled data. For audio textures, however, there is currently a lack of large semantically labeled datasets. Therefore, to control generation, we develop a method for semantic control over an unconditionally trained StyleGAN in the absence of such labeled datasets. In this paper, we propose an example-based fra…
▽ More
Controllable generation using StyleGANs is usually achieved by training the model using labeled data. For audio textures, however, there is currently a lack of large semantically labeled datasets. Therefore, to control generation, we develop a method for semantic control over an unconditionally trained StyleGAN in the absence of such labeled datasets. In this paper, we propose an example-based framework to determine guidance vectors for audio texture generation based on user-defined semantic attributes. Our approach leverages the semantically disentangled latent space of an unconditionally trained StyleGAN. By using a few synthetic examples to indicate the presence or absence of a semantic attribute, we infer the guidance vectors in the latent space of the StyleGAN to control that attribute during generation. Our results show that our framework can find user-defined and perceptually relevant guidance vectors for controllable generation for audio textures. Furthermore, we demonstrate an application of our framework to other tasks, such as selective semantic attribute transfer.
△ Less
Submitted 14 April, 2024; v1 submitted 22 August, 2023;
originally announced August 2023.
-
OrcoDCS: An IoT-Edge Orchestrated Online Deep Compressed Sensing Framework
Authors:
Cheng-Wei Ching,
Chirag Gupta,
Zi Huang,
Liting Hu
Abstract:
Compressed data aggregation (CDA) over wireless sensor networks (WSNs) is task-specific and subject to environmental changes. However, the existing compressed data aggregation (CDA) frameworks (e.g., compressed sensing-based data aggregation, deep learning(DL)-based data aggregation) do not possess the flexibility and adaptivity required to handle distinct sensing tasks and environmental changes.…
▽ More
Compressed data aggregation (CDA) over wireless sensor networks (WSNs) is task-specific and subject to environmental changes. However, the existing compressed data aggregation (CDA) frameworks (e.g., compressed sensing-based data aggregation, deep learning(DL)-based data aggregation) do not possess the flexibility and adaptivity required to handle distinct sensing tasks and environmental changes. Additionally, they do not consider the performance of follow-up IoT data-driven deep learning (DL)-based applications. To address these shortcomings, we propose OrcoDCS, an IoT-Edge orchestrated online deep compressed sensing framework that offers high flexibility and adaptability to distinct IoT device groups and their sensing tasks, as well as high performance for follow-up applications. The novelty of our work is the design and deployment of IoT-Edge orchestrated online training framework over WSNs by leveraging an specially-designed asymmetric autoencoder, which can largely reduce the encoding overhead and improve the reconstruction performance and robustness. We show analytically and empirically that OrcoDCS outperforms the state-of-the-art DCDA on training time, significantly improves flexibility and adaptability when distinct reconstruction tasks are given, and achieves higher performance for follow-up applications.
△ Less
Submitted 5 August, 2023;
originally announced August 2023.
-
Unique Brain Network Identification Number for Parkinson's Individuals Using Structural MRI
Authors:
Tanmayee Samantaray,
Utsav Gupta,
Jitender Saini,
Cota Navin Gupta
Abstract:
We propose a novel algorithm called Unique Brain Network Identification Number, UBNIN for encoding the brain networks of individual subjects. To realize this objective, we employed structural MRI on 180 Parkinsons disease PD patients and 70 healthy controls HC from the National Institute of Mental Health and Neurosciences, India. We parcellated each subjects brain volume and constructed an individ…
▽ More
We propose a novel algorithm called Unique Brain Network Identification Number, UBNIN for encoding the brain networks of individual subjects. To realize this objective, we employed structural MRI on 180 Parkinsons disease PD patients and 70 healthy controls HC from the National Institute of Mental Health and Neurosciences, India. We parcellated each subjects brain volume and constructed an individual adjacency matrix using the correlation between the gray matter volumes of every pair of regions. The unique code is derived from values representing connections for every node i, weighted by a factor of 2^1-i. The numerical representation UBNIN was observed to be distinct for each individual brain network, which may also be applied to other neuroimaging modalities. This model may be implemented as a neural signature of a persons unique brain connectivity, thereby making it useful for brainprinting applications. Additionally, we segregated the above datasets into five age cohorts to study the variation in network topology over age. Sparsity was adopted as the threshold estimate to binarize each age-based correlation matrix. For each age cohort, a decreasing trend was observed in the mean clustering coefficient with increasing sparsity. Significantly different clustering coefficients were noted in PD between age cohort B and C, C and E, and in HC between E and B, E and C, E and D, and C and D. Our findings suggest network connectivity patterns change with age, indicating network disruption may be due to the underlying neuropathology. Varying clustering coefficients for different cohorts indicate that information transfer between neighboring nodes changes with age. This provides evidence of age related brain shrinkage and network degeneration. We also discuss limitations and provide an open-access link to software codes and a help file for the entire study.
△ Less
Submitted 19 September, 2023; v1 submitted 2 June, 2023;
originally announced June 2023.
-
An ensemble of convolution-based methods for fault detection using vibration signals
Authors:
Xian Yeow Lee,
Aman Kumar,
Lasitha Vidyaratne,
Aniruddha Rajendra Rao,
Ahmed Farahat,
Chetan Gupta
Abstract:
This paper focuses on solving a fault detection problem using multivariate time series of vibration signals collected from planetary gearboxes in a test rig. Various traditional machine learning and deep learning methods have been proposed for multivariate time-series classification, including distance-based, functional data-oriented, feature-driven, and convolution kernel-based methods. Recent st…
▽ More
This paper focuses on solving a fault detection problem using multivariate time series of vibration signals collected from planetary gearboxes in a test rig. Various traditional machine learning and deep learning methods have been proposed for multivariate time-series classification, including distance-based, functional data-oriented, feature-driven, and convolution kernel-based methods. Recent studies have shown using convolution kernel-based methods like ROCKET, and 1D convolutional neural networks with ResNet and FCN, have robust performance for multivariate time-series data classification. We propose an ensemble of three convolution kernel-based methods and show its efficacy on this fault detection problem by outperforming other approaches and achieving an accuracy of more than 98.8\%.
△ Less
Submitted 4 May, 2023;
originally announced May 2023.
-
Towards Controllable Audio Texture Morphing
Authors:
Chitralekha Gupta,
Purnima Kamath,
Yize Wei,
Zhuoyao Li,
Suranga Nanayakkara,
Lonce Wyse
Abstract:
In this paper, we propose a data-driven approach to train a Generative Adversarial Network (GAN) conditioned on "soft-labels" distilled from the penultimate layer of an audio classifier trained on a target set of audio texture classes. We demonstrate that interpolation between such conditions or control vectors provides smooth morphing between the generated audio textures, and shows similar or bet…
▽ More
In this paper, we propose a data-driven approach to train a Generative Adversarial Network (GAN) conditioned on "soft-labels" distilled from the penultimate layer of an audio classifier trained on a target set of audio texture classes. We demonstrate that interpolation between such conditions or control vectors provides smooth morphing between the generated audio textures, and shows similar or better audio texture morphing capability compared to the state-of-the-art methods. The proposed approach results in a well-organized latent space that generates novel audio outputs while remaining consistent with the semantics of the conditioning parameters. This is a step towards a general data-driven approach to designing generative audio models with customized controls capable of traversing out-of-distribution regions for novel sound synthesis.
△ Less
Submitted 23 April, 2023;
originally announced April 2023.
-
Graph-based Simultaneous Coverage and Exploration Planning for Fast Multi-robot Search
Authors:
Indraneel Patil,
Rachel Zheng,
Charvi Gupta,
Jaekyung Song,
Narendar Sriram,
Katia Sycara
Abstract:
In large unknown environments, search operations can be much more time-efficient with the use of multi-robot fleets by parallelizing efforts. This means robots must efficiently perform collaborative map** (exploration) while simultaneously searching an area for victims (coverage). Previous simultaneous map** and planning techniques treat these problems as separate and do not take advantage of…
▽ More
In large unknown environments, search operations can be much more time-efficient with the use of multi-robot fleets by parallelizing efforts. This means robots must efficiently perform collaborative map** (exploration) while simultaneously searching an area for victims (coverage). Previous simultaneous map** and planning techniques treat these problems as separate and do not take advantage of the possibility for a unified approach. We propose a novel exploration-coverage planner which bridges the map** and search domains by growing sets of random trees rooted upon a pose graph produced through map** to generate points of interest, or tasks. Furthermore, it is important for the robots to first prioritize high information tasks to locate the greatest number of victims in minimum time by balancing coverage and exploration, which current methods do not address. Towards this goal, we also present a new multi-robot task allocator that formulates a notion of a hierarchical information heuristic for time-critical collaborative search. Our results show that our algorithm produces 20% more coverage efficiency, defined as average covered area per second, compared to the existing state-of-the-art. Our algorithms and the rest of our multi-robot search stack is based in ROS and made open source
△ Less
Submitted 3 March, 2023;
originally announced March 2023.
-
Parameter Sensitivity of Deep-Feature based Evaluation Metrics for Audio Textures
Authors:
Chitralekha Gupta,
Yize Wei,
Zequn Gong,
Purnima Kamath,
Zhuoyao Li,
Lonce Wyse
Abstract:
Standard evaluation metrics such as the Inception score and Fréchet Audio Distance provide a general audio quality distance metric between the synthesized audio and reference clean audio. However, the sensitivity of these metrics to variations in the statistical parameters that define an audio texture is not well studied. In this work, we provide a systematic study of the sensitivity of some of th…
▽ More
Standard evaluation metrics such as the Inception score and Fréchet Audio Distance provide a general audio quality distance metric between the synthesized audio and reference clean audio. However, the sensitivity of these metrics to variations in the statistical parameters that define an audio texture is not well studied. In this work, we provide a systematic study of the sensitivity of some of the existing audio quality evaluation metrics to parameter variations in audio textures. Furthermore, we also study three more potentially parameter-sensitive metrics for audio texture synthesis, (a) a Gram matrix based distance, (b) an Accumulated Gram metric using a summarized version of the Gram matrices, and (c) a cochlear-model based statistical features metric. These metrics use deep features that summarize the statistics of any given audio texture, thus being inherently sensitive to variations in the statistical parameters that define an audio texture. We study and evaluate the sensitivity of existing standard metrics as well as Gram matrix and cochlear-model based metrics to control-parameter variations in audio textures across a wide range of texture and parameter types, and validate with subjective evaluation. We find that each of the metrics is sensitive to different sets of texture-parameter types. This is the first step towards investigating objective metrics for assessing parameter sensitivity in audio textures.
△ Less
Submitted 23 August, 2022;
originally announced August 2022.
-
Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
Authors:
Takuya Kanazawa,
Haiyan Wang,
Chetan Gupta
Abstract:
Uncertainty quantification is one of the central challenges for machine learning in real-world applications. In reinforcement learning, an agent confronts two kinds of uncertainty, called epistemic uncertainty and aleatoric uncertainty. Disentangling and evaluating these uncertainties simultaneously stands a chance of improving the agent's final performance, accelerating training, and facilitating…
▽ More
Uncertainty quantification is one of the central challenges for machine learning in real-world applications. In reinforcement learning, an agent confronts two kinds of uncertainty, called epistemic uncertainty and aleatoric uncertainty. Disentangling and evaluating these uncertainties simultaneously stands a chance of improving the agent's final performance, accelerating training, and facilitating quality assurance after deployment. In this work, we propose an uncertainty-aware reinforcement learning algorithm for continuous control tasks that extends the Deep Deterministic Policy Gradient algorithm (DDPG). It exploits epistemic uncertainty to accelerate exploration and aleatoric uncertainty to learn a risk-sensitive policy. We conduct numerical experiments showing that our variant of DDPG outperforms vanilla DDPG without uncertainty estimation in benchmark tasks on robotic control and power-grid optimization.
△ Less
Submitted 27 July, 2022;
originally announced July 2022.
-
PoLyScriber: Integrated Fine-tuning of Extractor and Lyrics Transcriber for Polyphonic Music
Authors:
Xiaoxue Gao,
Chitralekha Gupta,
Haizhou Li
Abstract:
Lyrics transcription of polyphonic music is challenging as the background music affects lyrics intelligibility. Typically, lyrics transcription can be performed by a two-step pipeline, i.e. a singing vocal extraction front end, followed by a lyrics transcriber back end, where the front end and back end are trained separately. Such a two-step pipeline suffers from both imperfect vocal extraction an…
▽ More
Lyrics transcription of polyphonic music is challenging as the background music affects lyrics intelligibility. Typically, lyrics transcription can be performed by a two-step pipeline, i.e. a singing vocal extraction front end, followed by a lyrics transcriber back end, where the front end and back end are trained separately. Such a two-step pipeline suffers from both imperfect vocal extraction and mismatch between front end and back end. In this work, we propose a novel end-to-end integrated fine-tuning framework, that we call PoLyScriber, to globally optimize the vocal extractor front end and lyrics transcriber back end for lyrics transcription in polyphonic music. The experimental results show that our proposed PoLyScriber achieves substantial improvements over the existing approaches on publicly available test datasets.
△ Less
Submitted 5 May, 2023; v1 submitted 15 July, 2022;
originally announced July 2022.
-
Sound Model Factory: An Integrated System Architecture for Generative Audio Modelling
Authors:
Lonce Wyse,
Purnima Kamath,
Chitralekha Gupta
Abstract:
We introduce a new system for data-driven audio sound model design built around two different neural network architectures, a Generative Adversarial Network(GAN) and a Recurrent Neural Network (RNN), that takes advantage of the unique characteristics of each to achieve the system objectives that neither is capable of addressing alone. The objective of the system is to generate interactively contro…
▽ More
We introduce a new system for data-driven audio sound model design built around two different neural network architectures, a Generative Adversarial Network(GAN) and a Recurrent Neural Network (RNN), that takes advantage of the unique characteristics of each to achieve the system objectives that neither is capable of addressing alone. The objective of the system is to generate interactively controllable sound models given (a) a range of sounds the model should be able to synthesize, and (b) a specification of the parametric controls for navigating that space of sounds. The range of sounds is defined by a dataset provided by the designer, while the means of navigation is defined by a combination of data labels and the selection of a sub-manifold from the latent space learned by the GAN. Our proposed system takes advantage of the rich latent space of a GAN that consists of sounds that fill out the spaces ''between" real data-like sounds. This augmented data from the GAN is then used to train an RNN for its ability to respond immediately and continuously to parameter changes and to generate audio over unlimited periods of time. Furthermore, we develop a self-organizing map technique for ``smoothing" the latent space of GAN that results in perceptually smooth interpolation between audio timbres. We validate this process through user studies. The system contributes advances to the state of the art for generative sound model design that include system configuration and components for improving interpolation and the expansion of audio modeling capabilities beyond musical pitch and percussive instrument sounds into the more complex space of audio textures.
△ Less
Submitted 27 June, 2022;
originally announced June 2022.
-
Genre-conditioned Acoustic Models for Automatic Lyrics Transcription of Polyphonic Music
Authors:
Xiaoxue Gao,
Chitralekha Gupta,
Haizhou Li
Abstract:
Lyrics transcription of polyphonic music is challenging not only because the singing vocals are corrupted by the background music, but also because the background music and the singing style vary across music genres, such as pop, metal, and hip hop, which affects lyrics intelligibility of the song in different ways. In this work, we propose to transcribe the lyrics of polyphonic music using a nove…
▽ More
Lyrics transcription of polyphonic music is challenging not only because the singing vocals are corrupted by the background music, but also because the background music and the singing style vary across music genres, such as pop, metal, and hip hop, which affects lyrics intelligibility of the song in different ways. In this work, we propose to transcribe the lyrics of polyphonic music using a novel genre-conditioned network. The proposed network adopts pre-trained model parameters, and incorporates the genre adapters between layers to capture different genre peculiarities for lyrics-genre pairs, thereby only requiring lightweight genre-specific parameters for training. Our experiments show that the proposed genre-conditioned network outperforms the existing lyrics transcription systems.
△ Less
Submitted 7 April, 2022;
originally announced April 2022.
-
Music-robust Automatic Lyrics Transcription of Polyphonic Music
Authors:
Xiaoxue Gao,
Chitralekha Gupta,
Haizhou Li
Abstract:
Lyrics transcription of polyphonic music is challenging because singing vocals are corrupted by the background music. To improve the robustness of lyrics transcription to the background music, we propose a strategy of combining the features that emphasize the singing vocals, i.e. music-removed features that represent singing vocal extracted features, and the features that capture the singing vocal…
▽ More
Lyrics transcription of polyphonic music is challenging because singing vocals are corrupted by the background music. To improve the robustness of lyrics transcription to the background music, we propose a strategy of combining the features that emphasize the singing vocals, i.e. music-removed features that represent singing vocal extracted features, and the features that capture the singing vocals as well as the background music, i.e. music-present features. We show that these two sets of features complement each other, and their combination performs better than when they are used alone, thus improving the robustness of the acoustic model to the background music. Furthermore, language model interpolation between a general-purpose language model and an in-domain lyrics-specific language model provides further improvement in transcription results. Our experiments show that our proposed strategy outperforms the existing lyrics transcription systems for polyphonic music. Moreover, we find that our proposed music-robust features specially improve the lyrics transcription performance in metal genre of songs, where the background music is loud and dominant.
△ Less
Submitted 22 April, 2022; v1 submitted 7 April, 2022;
originally announced April 2022.
-
Data-driven Residual Generation for Early Fault Detection with Limited Data
Authors:
Hamed Khorasgani,
Ahmed Farahat,
Chetan Gupta
Abstract:
Traditionally, fault detection and isolation community has used system dynamic equations to generate diagnosers and to analyze detectability and isolability of the dynamic systems. Model-based fault detection and isolation methods use system model to generate a set of residuals as the bases for fault detection and isolation. However, in many complex systems it is not feasible to develop highly acc…
▽ More
Traditionally, fault detection and isolation community has used system dynamic equations to generate diagnosers and to analyze detectability and isolability of the dynamic systems. Model-based fault detection and isolation methods use system model to generate a set of residuals as the bases for fault detection and isolation. However, in many complex systems it is not feasible to develop highly accurate models for the systems and to keep the models updated during the system lifetime. Recently, data-driven solutions have received an immense attention in the industries systems for several practical reasons. First, these methods do not require the initial investment and expertise for develo** accurate models. Moreover, it is possible to automatically update and retrain the diagnosers as the system or the environment change over time. Finally, unlike the model-based methods it is straight forward to combine time series measurements such as pressure and voltage with other sources of information such as system operating hours to achieve a higher accuracy. In this paper, we extend the traditional model-based fault detection and isolation concepts such as residuals, and detectable and isolable faults to the data-driven domain. We then propose an algorithm to automatically generate residuals from the normal operating data. We present the performance of our proposed approach through a comparative case study.
△ Less
Submitted 27 September, 2021;
originally announced October 2021.
-
Signal Representations for Synthesizing Audio Textures with Generative Adversarial Networks
Authors:
Chitralekha Gupta,
Purnima Kamath,
Lonce Wyse
Abstract:
Generative Adversarial Networks (GANs) currently achieve the state-of-the-art sound synthesis quality for pitched musical instruments using a 2-channel spectrogram representation consisting of log magnitude and instantaneous frequency (the "IFSpectrogram"). Many other synthesis systems use representations derived from the magnitude spectra, and then depend on a backend component to invert the outp…
▽ More
Generative Adversarial Networks (GANs) currently achieve the state-of-the-art sound synthesis quality for pitched musical instruments using a 2-channel spectrogram representation consisting of log magnitude and instantaneous frequency (the "IFSpectrogram"). Many other synthesis systems use representations derived from the magnitude spectra, and then depend on a backend component to invert the output magnitude spectrograms that generally result in audible artefacts associated with the inversion process. However, for signals that have closely-spaced frequency components such as non-pitched and other noisy sounds, training the GAN on the 2-channel IFSpectrogram representation offers no advantage over the magnitude spectra based representations. In this paper, we propose that training GANs on single-channel magnitude spectra, and using the Phase Gradient Heap Integration (PGHI) inversion algorithm is a better comprehensive approach for audio synthesis modeling of diverse signals that include pitched, non-pitched, and dynamically complex sounds. We show that this method produces higher-quality output for wideband and noisy sounds, such as pops and chirps, compared to using the IFSpectrogram. Furthermore, the sound quality for pitched sounds is comparable to using the IFSpectrogram, even while using a simpler representation with half the memory requirements.
△ Less
Submitted 12 March, 2021;
originally announced March 2021.
-
Automatic Lyrics Alignment and Transcription in Polyphonic Music: Does Background Music Help?
Authors:
Chitralekha Gupta,
Emre Yılmaz,
Haizhou Li
Abstract:
Background music affects lyrics intelligibility of singing vocals in a music piece. Automatic lyrics alignment and transcription in polyphonic music are challenging tasks because the singing vocals are corrupted by the background music. In this work, we propose to learn music genre-specific characteristics to train polyphonic acoustic models. We first compare several automatic speech recognition p…
▽ More
Background music affects lyrics intelligibility of singing vocals in a music piece. Automatic lyrics alignment and transcription in polyphonic music are challenging tasks because the singing vocals are corrupted by the background music. In this work, we propose to learn music genre-specific characteristics to train polyphonic acoustic models. We first compare several automatic speech recognition pipelines for the application of lyrics transcription. We then present the lyrics alignment and transcription performance of music-informed acoustic models for the best-performing pipeline, and systematically study the impact of music genre and language model on the performance. With such genre-based approach, we explicitly model the music without removing it during acoustic modeling. The proposed approach outperforms all competing systems in the lyrics alignment and transcription tasks on several well-known polyphonic test datasets.
△ Less
Submitted 22 October, 2019; v1 submitted 23 September, 2019;
originally announced September 2019.
-
Acoustic Modeling for Automatic Lyrics-to-Audio Alignment
Authors:
Chitralekha Gupta,
Emre Yılmaz,
Haizhou Li
Abstract:
Automatic lyrics to polyphonic audio alignment is a challenging task not only because the vocals are corrupted by background music, but also there is a lack of annotated polyphonic corpus for effective acoustic modeling. In this work, we propose (1) using additional speech and music-informed features and (2) adapting the acoustic models trained on a large amount of solo singing vocals towards poly…
▽ More
Automatic lyrics to polyphonic audio alignment is a challenging task not only because the vocals are corrupted by background music, but also there is a lack of annotated polyphonic corpus for effective acoustic modeling. In this work, we propose (1) using additional speech and music-informed features and (2) adapting the acoustic models trained on a large amount of solo singing vocals towards polyphonic music using a small amount of in-domain data. Incorporating additional information such as voicing and auditory features together with conventional acoustic features aims to bring robustness against the increased spectro-temporal variations in singing vocals. By adapting the acoustic model using a small amount of polyphonic audio data, we reduce the domain mismatch between training and testing data. We perform several alignment experiments and present an in-depth alignment error analysis on acoustic features, and model adaptation techniques. The results demonstrate that the proposed strategy provides a significant error reduction of word boundary alignment over comparable existing systems, especially on more challenging polyphonic data with long-duration musical interludes.
△ Less
Submitted 25 June, 2019;
originally announced June 2019.
