Showing 1–50 of 100 results for author: Du, J

Exploring Audio-Visual Information Fusion for Sound Event Localization and Detection In Low-Resource Realistic Scenarios

Authors: Ya Jiang, Qing Wang, Jun Du, Maocheng Hu, Pengfei Hu, Zeyan Liu, Shi Cheng, Zhaoxu Nian, Yuxuan Dong, Mingqi Cai, Xin Fang, Chin-Hui Lee

Abstract: This study presents an audio-visual information fusion approach to sound event localization and detection (SELD) in low-resource scenarios. We aim at utilizing audio and video modality information through cross-modal learning and multi-modal fusion. First, we propose a cross-modal teacher-student learning (TSL) framework to transfer information from an audio-only teacher model, trained on a rich c… ▽ More This study presents an audio-visual information fusion approach to sound event localization and detection (SELD) in low-resource scenarios. We aim at utilizing audio and video modality information through cross-modal learning and multi-modal fusion. First, we propose a cross-modal teacher-student learning (TSL) framework to transfer information from an audio-only teacher model, trained on a rich collection of audio data with multiple data augmentation techniques, to an audio-visual student model trained with only a limited set of multi-modal data. Next, we propose a two-stage audio-visual fusion strategy, consisting of an early feature fusion and a late video-guided decision fusion to exploit synergies between audio and video modalities. Finally, we introduce an innovative video pixel swap** (VPS) technique to extend an audio channel swap** (ACS) method to an audio-visual joint augmentation. Evaluation results on the Detection and Classification of Acoustic Scenes and Events (DCASE) 2023 Challenge data set demonstrate significant improvements in SELD performances. Furthermore, our submission to the SELD task of the DCASE 2023 Challenge ranks first place by effectively integrating the proposed techniques into a model ensemble. △ Less

Submitted 21 June, 2024; originally announced June 2024.

Comments: accepted by icme2024

AS-70: A Mandarin stuttered speech dataset for automatic speech recognition and stuttering event detection

Authors: Rong Gong, Hongfei Xue, Lezhi Wang, Xin Xu, Qisheng Li, Lei Xie, Hui Bu, Shaomei Wu, Jiaming Zhou, Yong Qin, Binbin Zhang, Jun Du, Jia Bin, Ming Li

Abstract: The rapid advancements in speech technologies over the past two decades have led to human-level performance in tasks like automatic speech recognition (ASR) for fluent speech. However, the efficacy of these models diminishes when applied to atypical speech, such as stuttering. This paper introduces AS-70, the first publicly available Mandarin stuttered speech dataset, which stands out as the large… ▽ More The rapid advancements in speech technologies over the past two decades have led to human-level performance in tasks like automatic speech recognition (ASR) for fluent speech. However, the efficacy of these models diminishes when applied to atypical speech, such as stuttering. This paper introduces AS-70, the first publicly available Mandarin stuttered speech dataset, which stands out as the largest dataset in its category. Encompassing conversational and voice command reading speech, AS-70 includes verbatim manual transcription, rendering it suitable for various speech-related tasks. Furthermore, baseline systems are established, and experimental results are presented for ASR and stuttering event detection (SED) tasks. By incorporating this dataset into the model fine-tuning, significant improvements in the state-of-the-art ASR models, e.g., Whisper and Hubert, are observed, enhancing their inclusivity in addressing stuttered speech. △ Less

Submitted 11 June, 2024; originally announced June 2024.

Comments: Accepted by Interspeech 2024

Neural Codec-based Adversarial Sample Detection for Speaker Verification

Authors: Xuanjun Chen, Jiawei Du, Haibin Wu, Jyh-Shing Roger Jang, Hung-yi Lee

Abstract: Automatic Speaker Verification (ASV), increasingly used in security-critical applications, faces vulnerabilities from rising adversarial attacks, with few effective defenses available. In this paper, we propose a neural codec-based adversarial sample detection method for ASV. The approach leverages the codec's ability to discard redundant perturbations and retain essential information. Specificall… ▽ More Automatic Speaker Verification (ASV), increasingly used in security-critical applications, faces vulnerabilities from rising adversarial attacks, with few effective defenses available. In this paper, we propose a neural codec-based adversarial sample detection method for ASV. The approach leverages the codec's ability to discard redundant perturbations and retain essential information. Specifically, we distinguish between genuine and adversarial samples by comparing ASV score differences between original and re-synthesized audio (by codec models). This comprehensive study explores all open-source neural codecs and their variant models for experiments. The Descript-audio-codec model stands out by delivering the highest detection rate among 15 neural codecs and surpassing seven prior state-of-the-art (SOTA) detection methods. Note that, our single-model method even outperforms a SOTA ensemble method by a large margin. △ Less

Submitted 6 June, 2024; originally announced June 2024.

Comments: Accepted by Interspeech 2024

A DAFT Based Unified Waveform Design Framework for High-Mobility Communications

Authors: Xingyao Zhang, Haoran Yin, Yanqun Tang, Yu Zhou, Yuqing Liu, **ming Du, Yipeng Ding

Abstract: With the increasing demand for multi-carrier communication in high-mobility scenarios, it is urgent to design new multi-carrier communication waveforms that can resist large delay-Doppler spreads. Various multi-carrier waveforms in the transform domain were proposed for the fast time-varying channels, including orthogonal time frequency space (OTFS), orthogonal chirp division multiplexing (OCDM),… ▽ More With the increasing demand for multi-carrier communication in high-mobility scenarios, it is urgent to design new multi-carrier communication waveforms that can resist large delay-Doppler spreads. Various multi-carrier waveforms in the transform domain were proposed for the fast time-varying channels, including orthogonal time frequency space (OTFS), orthogonal chirp division multiplexing (OCDM), and affine frequency division multiplexing (AFDM). Among these, the AFDM is a strong candidate for its low implementation complexity and ability to achieve optimal diversity. This paper unifies the waveforms based on the discrete affine Fourier transform (DAFT) by using the chirp slope factor "k" in the time-frequency representation to construct a unified design framework for high-mobility communications. The design framework is employed to verify that the bit error rate performance of the DAFT-based waveform can be enhanced when the signal-to-noise ratio (SNR) is sufficiently high by adjusting the chirp slope factor "k". △ Less

Submitted 4 June, 2024; originally announced June 2024.

A Variance-Preserving Interpolation Approach for Diffusion Models with Applications to Single Channel Speech Enhancement and Recognition

Authors: Zilu Guo, Qing Wang, Jun Du, Jia Pan, Qing-Feng Liu, Chin-Hui

Abstract: In this paper, we propose a variance-preserving interpolation framework to improve diffusion models for single-channel speech enhancement (SE) and automatic speech recognition (ASR). This new variance-preserving interpolation diffusion model (VPIDM) approach requires only 25 iterative steps and obviates the need for a corrector, an essential element in the existing variance-exploding interpolation… ▽ More In this paper, we propose a variance-preserving interpolation framework to improve diffusion models for single-channel speech enhancement (SE) and automatic speech recognition (ASR). This new variance-preserving interpolation diffusion model (VPIDM) approach requires only 25 iterative steps and obviates the need for a corrector, an essential element in the existing variance-exploding interpolation diffusion model (VEIDM). Two notable distinctions between VPIDM and VEIDM are the scaling function of the mean of state variables and the constraint imposed on the variance relative to the mean's scale. We conduct a systematic exploration of the theoretical mechanism underlying VPIDM and develop insights regarding VPIDM's applications in SE and ASR using VPIDM as a frontend. Our proposed approach, evaluated on two distinct data sets, demonstrates VPIDM's superior performances over conventional discriminative SE algorithms. Furthermore, we assess the performance of the proposed model under varying signal-to-noise ratio (SNR) levels. The investigation reveals VPIDM's improved robustness in target noise elimination when compared to VEIDM. Furthermore, utilizing the mid-outputs of both VPIDM and VEIDM results in enhanced ASR accuracies, thereby highlighting the practical efficacy of our proposed approach. △ Less

Submitted 27 May, 2024; originally announced May 2024.

Quality-aware Masked Diffusion Transformer for Enhanced Music Generation

Authors: Chang Li, Ruoyu Wang, Lijuan Liu, Jun Du, Yixuan Sun, Zilu Guo, Zhenrong Zhang, Yuan Jiang

Abstract: In recent years, diffusion-based text-to-music (TTM) generation has gained prominence, offering a novel approach to synthesizing musical content from textual descriptions. Achieving high accuracy and diversity in this generation process requires extensive, high-quality data, which often constitutes only a fraction of available datasets. Within open-source datasets, the prevalence of issues like mi… ▽ More In recent years, diffusion-based text-to-music (TTM) generation has gained prominence, offering a novel approach to synthesizing musical content from textual descriptions. Achieving high accuracy and diversity in this generation process requires extensive, high-quality data, which often constitutes only a fraction of available datasets. Within open-source datasets, the prevalence of issues like mislabeling, weak labeling, unlabeled data, and low-quality music waveform significantly hampers the development of music generation models. To overcome these challenges, we introduce a novel quality-aware masked diffusion transformer (QA-MDT) approach that enables generative models to discern the quality of input music waveform during training. Building on the unique properties of musical signals, we have adapted and implemented a MDT model for TTM task, while further unveiling its distinct capacity for quality control. Moreover, we address the issue of low-quality captions with a caption refinement data processing approach. Our demo page is shown in https://qa-mdt.github.io/. Code on https://github.com/ivcylc/qa-mdt △ Less

Submitted 24 May, 2024; originally announced May 2024.

Large coordinate kernel attention network for lightweight image super-resolution

Authors: Fangwei Hao, Jiesheng Wu, Haotian Lu, Ji Du, **g Xu

Abstract: The multi-scale receptive field and large kernel attention (LKA) module have been shown to significantly improve performance in the lightweight image super-resolution task. However, existing lightweight super-resolution (SR) methods seldom pay attention to designing efficient building block with multi-scale receptive field for local modeling, and their LKA modules face a quadratic increase in comp… ▽ More The multi-scale receptive field and large kernel attention (LKA) module have been shown to significantly improve performance in the lightweight image super-resolution task. However, existing lightweight super-resolution (SR) methods seldom pay attention to designing efficient building block with multi-scale receptive field for local modeling, and their LKA modules face a quadratic increase in computational and memory footprints as the convolutional kernel size increases. To address the first issue, we propose the multi-scale blueprint separable convolutions (MBSConv) as highly efficient building block with multi-scale receptive field, it can focus on the learning for the multi-scale information which is a vital component of discriminative representation. As for the second issue, we revisit the key properties of LKA in which we find that the adjacent direct interaction of local information and long-distance dependencies is crucial to provide remarkable performance. Thus, taking this into account and in order to mitigate the complexity of LKA, we propose a large coordinate kernel attention (LCKA) module which decomposes the 2D convolutional kernels of the depth-wise convolutional layers in LKA into horizontal and vertical 1-D kernels. LCKA enables the adjacent direct interaction of local information and long-distance dependencies not only in the horizontal direction but also in the vertical. Besides, LCKA allows for the direct use of extremely large kernels in the depth-wise convolutional layers to capture more contextual information, which helps to significantly improve the reconstruction performance, and it incurs lower computational complexity and memory footprints. Integrating MBSConv and LCKA, we propose a large coordinate kernel attention network (LCAN). △ Less

Submitted 15 May, 2024; originally announced May 2024.

NTIRE 2024 Challenge on Short-form UGC Video Quality Assessment: Methods and Results

Authors: Xin Li, Kun Yuan, Ya**g Pei, Yiting Lu, Ming Sun, Chao Zhou, Zhibo Chen, Radu Timofte, Wei Sun, Haoning Wu, Zicheng Zhang, Jun Jia, Zhichao Zhang, Linhan Cao, Qiubo Chen, Xiongkuo Min, Weisi Lin, Guangtao Zhai, Jianhui Sun, Tianyi Wang, Lei Li, Han Kong, Wenxuan Wang, Bing Li, Cheng Luo , et al. (43 additional authors not shown)

Abstract: This paper reviews the NTIRE 2024 Challenge on Shortform UGC Video Quality Assessment (S-UGC VQA), where various excellent solutions are submitted and evaluated on the collected dataset KVQ from popular short-form video platform, i.e., Kuaishou/Kwai Platform. The KVQ database is divided into three parts, including 2926 videos for training, 420 videos for validation, and 854 videos for testing. The… ▽ More This paper reviews the NTIRE 2024 Challenge on Shortform UGC Video Quality Assessment (S-UGC VQA), where various excellent solutions are submitted and evaluated on the collected dataset KVQ from popular short-form video platform, i.e., Kuaishou/Kwai Platform. The KVQ database is divided into three parts, including 2926 videos for training, 420 videos for validation, and 854 videos for testing. The purpose is to build new benchmarks and advance the development of S-UGC VQA. The competition had 200 participants and 13 teams submitted valid solutions for the final testing phase. The proposed solutions achieved state-of-the-art performances for S-UGC VQA. The project can be found at https://github.com/lixinustc/KVQChallenge-CVPR-NTIRE2024. △ Less

Submitted 17 April, 2024; originally announced April 2024.

Comments: Accepted by CVPR2024 Workshop. The challenge report for CVPR NTIRE2024 Short-form UGC Video Quality Assessment Challenge

DeepFunction: Deep Metric Learning-based Imbalanced Classification for Diagnosing Threaded Pipe Connection Defects using Functional Data

Authors: Yukun Xie, Juan Du, Chen Zhang

Abstract: In modern manufacturing, most of the product lines are conforming. Few products are nonconforming but with different defect types. The identification of defect types can help further root cause diagnosis of production lines. With the sensing development, signals of process variables can be collected in high resolution, which can be regarded as multichannel functional data. They have abundant infor… ▽ More In modern manufacturing, most of the product lines are conforming. Few products are nonconforming but with different defect types. The identification of defect types can help further root cause diagnosis of production lines. With the sensing development, signals of process variables can be collected in high resolution, which can be regarded as multichannel functional data. They have abundant information to characterize the process and help identify the defect types. Motivated by a real example from the pipe tightening process, we focus on defect classification where each sample is a multichannel functional data. However, the available samples for each defect type are limited and imbalanced. Moreover, the functions are incomplete since the pre-tightening process before the pipe tightening process is unobserved. To classify the defect samples based on imbalanced, multichannel, and incomplete functional data is very important but challenging. Thus, we propose an innovative classification framework based on deep metric learning using functional data (DeepFunction). The framework leverages the power of deep metric learning to train on imbalanced datasets. A neural network specially crafted for processing functional data is also proposed to handle multichannel and incomplete functional data. The results from a real-world case study demonstrate the superior accuracy of our framework when compared to existing benchmarks. △ Less

Submitted 24 April, 2024; v1 submitted 4 April, 2024; originally announced April 2024.

Comments: Revised version for submission to IISE Transactions

Budget Recycling Differential Privacy

Authors: Bo Jiang, Jian Du, Sagar Shamar, Qiang Yan

Abstract: Differential Privacy (DP) mechanisms usually {force} reduction in data utility by producing "out-of-bound" noisy results for a tight privacy budget. We introduce the Budget Recycling Differential Privacy (BR-DP) framework, designed to provide soft-bounded noisy outputs for a broad range of existing DP mechanisms. By "soft-bounded," we refer to the mechanism's ability to release most outputs within… ▽ More Differential Privacy (DP) mechanisms usually {force} reduction in data utility by producing "out-of-bound" noisy results for a tight privacy budget. We introduce the Budget Recycling Differential Privacy (BR-DP) framework, designed to provide soft-bounded noisy outputs for a broad range of existing DP mechanisms. By "soft-bounded," we refer to the mechanism's ability to release most outputs within a predefined error boundary, thereby improving utility and maintaining privacy simultaneously. The core of BR-DP consists of two components: a DP kernel responsible for generating a noisy answer per iteration, and a recycler that probabilistically recycles/regenerates or releases the noisy answer. We delve into the privacy accounting of BR-DP, culminating in the development of a budgeting principle that optimally sub-allocates the available budget between the DP kernel and the recycler. Furthermore, we introduce algorithms for tight BR-DP accounting in composition scenarios, and our findings indicate that BR-DP achieves reduced privacy leakage post-composition compared to DP. Additionally, we explore the concept of privacy amplification via subsampling within the BR-DP framework and propose optimal sampling rates for BR-DP across various queries. We experiment with real data, and the results demonstrate BR-DP's effectiveness in lifting the utility-privacy tradeoff provided by DP mechanisms. △ Less

Submitted 16 April, 2024; v1 submitted 17 March, 2024; originally announced March 2024.

Multitask frame-level learning for few-shot sound event detection

Authors: Liang Zou, Genwei Yan, Ruoyu Wang, Jun Du, Meng Lei, Tian Gao, Xin Fang

Abstract: This paper focuses on few-shot Sound Event Detection (SED), which aims to automatically recognize and classify sound events with limited samples. However, prevailing methods methods in few-shot SED predominantly rely on segment-level predictions, which often providing detailed, fine-grained predictions, particularly for events of brief duration. Although frame-level prediction strategies have been… ▽ More This paper focuses on few-shot Sound Event Detection (SED), which aims to automatically recognize and classify sound events with limited samples. However, prevailing methods methods in few-shot SED predominantly rely on segment-level predictions, which often providing detailed, fine-grained predictions, particularly for events of brief duration. Although frame-level prediction strategies have been proposed to overcome these limitations, these strategies commonly face difficulties with prediction truncation caused by background noise. To alleviate this issue, we introduces an innovative multitask frame-level SED framework. In addition, we introduce TimeFilterAug, a linear timing mask for data augmentation, to increase the model's robustness and adaptability to diverse acoustic environments. The proposed method achieves a F-score of 63.8%, securing the 1st rank in the few-shot bioacoustic event detection category of the Detection and Classification of Acoustic Scenes and Events Challenge 2023. △ Less

Submitted 17 March, 2024; originally announced March 2024.

Comments: 6 pages, 4 figures, conference

SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluation

Authors: Jiayu Du, **peng Li, Guoguo Chen, Wei-Qiang Zhang

Abstract: In the wake of the surging tide of deep learning over the past decade, Automatic Speech Recognition (ASR) has garnered substantial attention, leading to the emergence of numerous publicly accessible ASR systems that are actively being integrated into our daily lives. Nonetheless, the impartial and replicable evaluation of these ASR systems encounters challenges due to various crucial subtleties. I… ▽ More In the wake of the surging tide of deep learning over the past decade, Automatic Speech Recognition (ASR) has garnered substantial attention, leading to the emergence of numerous publicly accessible ASR systems that are actively being integrated into our daily lives. Nonetheless, the impartial and replicable evaluation of these ASR systems encounters challenges due to various crucial subtleties. In this paper we introduce the SpeechColab Leaderboard, a general-purpose, open-source platform designed for ASR evaluation. With this platform: (i) We report a comprehensive benchmark, unveiling the current state-of-the-art panorama for ASR systems, covering both open-source models and industrial commercial services. (ii) We quantize how distinct nuances in the scoring pipeline influence the final benchmark outcomes. These include nuances related to capitalization, punctuation, interjection, contraction, synonym usage, compound words, etc. These issues have gained prominence in the context of the transition towards an End-to-End future. (iii) We propose a practical modification to the conventional Token-Error-Rate (TER) evaluation metric, with inspirations from Kolmogorov complexity and Normalized Information Distance (NID). This adaptation, called modified-TER (mTER), achieves proper normalization and symmetrical treatment of reference and hypothesis. By leveraging this platform as a large-scale testing ground, this study demonstrates the robustness and backward compatibility of mTER when compared to TER. The SpeechColab Leaderboard is accessible at https://github.com/SpeechColab/Leaderboard △ Less

Submitted 12 March, 2024; originally announced March 2024.

A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition

Authors: Yusheng Dai, Hang Chen, Jun Du, Ruoyu Wang, Shihao Chen, Jiefeng Ma, Haotian Wang, Chin-Hui Lee

Abstract: Advanced Audio-Visual Speech Recognition (AVSR) systems have been observed to be sensitive to missing video frames, performing even worse than single-modality models. While applying the dropout technique to the video modality enhances robustness to missing frames, it simultaneously results in a performance loss when dealing with complete data input. In this paper, we investigate this contrasting p… ▽ More Advanced Audio-Visual Speech Recognition (AVSR) systems have been observed to be sensitive to missing video frames, performing even worse than single-modality models. While applying the dropout technique to the video modality enhances robustness to missing frames, it simultaneously results in a performance loss when dealing with complete data input. In this paper, we investigate this contrasting phenomenon from the perspective of modality bias and reveal that an excessive modality bias on the audio caused by dropout is the underlying reason. Moreover, we present the Modality Bias Hypothesis (MBH) to systematically describe the relationship between modality bias and robustness against missing modality in multimodal systems. Building on these findings, we propose a novel Multimodal Distribution Approximation with Knowledge Distillation (MDA-KD) framework to reduce over-reliance on the audio modality and to maintain performance and robustness simultaneously. Finally, to address an entirely missing modality, we adopt adapters to dynamically switch decision strategies. The effectiveness of our proposed approach is evaluated and validated through a series of comprehensive experiments using the MISP2021 and MISP2022 datasets. Our code is available at https://github.com/dalision/ModalBiasAVSR △ Less

Submitted 7 March, 2024; originally announced March 2024.

Comments: the paper is accepted by CVPR2024

EMO-SUPERB: An In-depth Look at Speech Emotion Recognition

Authors: Haibin Wu, Huang-Cheng Chou, Kai-Wei Chang, Lucas Goncalves, Jiawei Du, Jyh-Shing Roger Jang, Chi-Chun Lee, Hung-Yi Lee

Abstract: Speech emotion recognition (SER) is a pivotal technology for human-computer interaction systems. However, 80.77% of SER papers yield results that cannot be reproduced. We develop EMO-SUPERB, short for EMOtion Speech Universal PERformance Benchmark, which aims to enhance open-source initiatives for SER. EMO-SUPERB includes a user-friendly codebase to leverage 15 state-of-the-art speech self-supervi… ▽ More Speech emotion recognition (SER) is a pivotal technology for human-computer interaction systems. However, 80.77% of SER papers yield results that cannot be reproduced. We develop EMO-SUPERB, short for EMOtion Speech Universal PERformance Benchmark, which aims to enhance open-source initiatives for SER. EMO-SUPERB includes a user-friendly codebase to leverage 15 state-of-the-art speech self-supervised learning models (SSLMs) for exhaustive evaluation across six open-source SER datasets. EMO-SUPERB streamlines result sharing via an online leaderboard, fostering collaboration within a community-driven benchmark and thereby enhancing the development of SER. On average, 2.58% of annotations are annotated using natural language. SER relies on classification models and is unable to process natural languages, leading to the discarding of these valuable annotations. We prompt ChatGPT to mimic annotators, comprehend natural language annotations, and subsequently re-label the data. By utilizing labels generated by ChatGPT, we consistently achieve an average relative gain of 3.08% across all settings. △ Less

Submitted 12 March, 2024; v1 submitted 20 February, 2024; originally announced February 2024.

Comments: webpage: https://emosuperb.github.io/

Tensor-based process control and monitoring for semiconductor manufacturing with unstable disturbances

Authors: Yanrong Li, Juan Du, Fugee Tsung, Wei Jiang

Abstract: With the development and popularity of sensors installed in manufacturing systems, complex data are collected during manufacturing processes, which brings challenges for traditional process control methods. This paper proposes a novel process control and monitoring method for the complex structure of high-dimensional image-based overlay errors (modeled in tensor form), which are collected in semic… ▽ More With the development and popularity of sensors installed in manufacturing systems, complex data are collected during manufacturing processes, which brings challenges for traditional process control methods. This paper proposes a novel process control and monitoring method for the complex structure of high-dimensional image-based overlay errors (modeled in tensor form), which are collected in semiconductor manufacturing processes. The proposed method aims to reduce overlay errors using limited control recipes. We first build a high-dimensional process model and propose different tensor-on-vector regression algorithms to estimate parameters in the model to alleviate the curse of dimensionality. Then, based on the estimate of tensor parameters, the exponentially weighted moving average (EWMA) controller for tensor data is designed whose stability is theoretically guaranteed. Considering the fact that low-dimensional control recipes cannot compensate for all high-dimensional disturbances on the image, control residuals are monitored to prevent significant drifts of uncontrollable high-dimensional disturbances. Through extensive simulations and real case studies, the performances of parameter estimation algorithms and the EWMA controller in tensor space are evaluated. Compared with existing image-based feedback controllers, the superiority of our method is verified especially when disturbances are not stable. △ Less

Submitted 30 January, 2024; originally announced January 2024.

Comments: 30 pages, 5 figures

Backscatter Measurements and Models for RF Sensing Applications in Cluttered Environments

Authors: Dmitry Chizhik, **feng Du, Jakub Sapis, Reinaldo A. Valenzuela, Abhishek Adhikari, Gil Zussman, Manuel A. Almendra, Mauricio Rodriguez, Rodolfo Feick

Abstract: A statistical backscatter channel model for indoor clutter is developed for indoor RF sensing applications based on measurements. A narrowband 28 GHz sounder used a quazi-monostatic radar arrangement with an omnidirectional transmit antenna illuminating an indoor scene and a spinning horn receive antenna less than 1 m away collecting backscattered power as a function of azimuth. Median average bac… ▽ More A statistical backscatter channel model for indoor clutter is developed for indoor RF sensing applications based on measurements. A narrowband 28 GHz sounder used a quazi-monostatic radar arrangement with an omnidirectional transmit antenna illuminating an indoor scene and a spinning horn receive antenna less than 1 m away collecting backscattered power as a function of azimuth. Median average backscatter power was found to vary over a 12 dB range, with average power generally decreasing with increasing room size. A deterministic model of average backscattered power dependent on distance to nearest wall and wall reflection coefficient reproduces observations with 4.0 dB RMS error. Distribution of power variation in azimuth around this average is reproduced within 1 dB by a random azimuth spectrum with a lognormal amplitude distribution and uniformly random phase. The model is extended to provide power distribution over both azimuth and delay (conveying range to scatterer) by combining azimuthal distribution with published results on power delay profiles in reverberant environments. The statistical model does not require a detailed room layout description, aiming to reproduce backscatter clutter statistics, as opposed to a deterministic response. △ Less

Submitted 26 January, 2024; originally announced January 2024.

On Inhomogeneous Infinite Products of Stochastic Matrices and Applications

Authors: Zhaoyue Xia, Jun Du, Chunxiao Jiang, H. Vincent Poor, Zhu Han, Yong Ren

Abstract: With the growth of magnitude of multi-agent networks, distributed optimization holds considerable significance within complex systems. Convergence, a pivotal goal in this domain, is contingent upon the analysis of infinite products of stochastic matrices (IPSMs). In this work, convergence properties of inhomogeneous IPSMs are investigated. The convergence rate of inhomogeneous IPSMs towards an abs… ▽ More With the growth of magnitude of multi-agent networks, distributed optimization holds considerable significance within complex systems. Convergence, a pivotal goal in this domain, is contingent upon the analysis of infinite products of stochastic matrices (IPSMs). In this work, convergence properties of inhomogeneous IPSMs are investigated. The convergence rate of inhomogeneous IPSMs towards an absolute probability sequence $π$ is derived. We also show that the convergence rate is nearly exponential, which coincides with existing results on ergodic chains. The methodology employed relies on delineating the interrelations among Sarymsakov matrices, scrambling matrices, and positive-column matrices. Based on the theoretical results on inhomogeneous IPSMs, we propose a decentralized projected subgradient method for time-varying multi-agent systems with graph-related stretches in (sub)gradient descent directions. The convergence of the proposed method is established for convex objective functions, and extended to non-convex objectives that satisfy Polyak-Lojasiewicz conditions. To corroborate the theoretical findings, we conduct numerical simulations, aligning the outcomes with the established theoretical framework. △ Less

Submitted 25 January, 2024; originally announced January 2024.

Measured and Modeled Outdoor Indoor Coverage at 28 GHz into High Thermal Efficiency Buildings

Authors: Dmitry Chizhik, **feng Du, Reinaldo Valenzuela, Andrea Bedin, Martti Moisio, Rodolfo Feick

Abstract: 28 GHz outdoor-indoor coverage into modern office buildings with high thermal efficiency windows is found to be severely limited due to 46 dB median penetration loss at normal incidence and additional 15 dB median oblique incidence loss. The study is based on measurements of path gain over 280 outdoor-indoor links, at ranges up to 100 m. A simple theoretical path gain model is extended to include… ▽ More 28 GHz outdoor-indoor coverage into modern office buildings with high thermal efficiency windows is found to be severely limited due to 46 dB median penetration loss at normal incidence and additional 15 dB median oblique incidence loss. The study is based on measurements of path gain over 280 outdoor-indoor links, at ranges up to 100 m. A simple theoretical path gain model is extended to include building penetration through multiple sides of the building as well as a reflection from another building. The theoretical model accounts for the building orientation relative to the source, resulting in 4.9 dB RMSE relative to data, as compared to 5.7 dB RMSE from a linear fit and 14.7 dB RMSE for the 3GPP recommended model. Only coarse description of the buildings is required: building orientation and exterior wall composition, without any interior details. Coverage range for SNR>-8 dB from an outdoor base to a terminal just inside a high-efficiency building is under 35 m △ Less

Submitted 4 September, 2023; originally announced January 2024.

Comments: 2 pages, 3 figures. Presented at IEEE International Symposium on Antennas and Propagation and USNC-URSI Radio Science Meeting

A Low-Complexity Range Estimation with Adjusted Affine Frequency Division Multiplexing Waveform

Authors: Jiajun Zhu, Yanqun Tang, Xizhang Wei, Haoran Yin, **ming Du, Zhengpeng Wang, Yuqinng Liu

Abstract: Affine frequency division multiplexing (AFDM) is a recently proposed communication waveform for time-varying channel scenarios. As a chirp-based multicarrier modulation technique it can not only satisfy the needs of multiple scenarios in future mobile communication networks but also achieve good performance in radar sensing by adjusting the built-in parameters, making it a promising air interface… ▽ More Affine frequency division multiplexing (AFDM) is a recently proposed communication waveform for time-varying channel scenarios. As a chirp-based multicarrier modulation technique it can not only satisfy the needs of multiple scenarios in future mobile communication networks but also achieve good performance in radar sensing by adjusting the built-in parameters, making it a promising air interface waveform in integrated sensing and communication (ISAC) applications. In this paper, we investigate an AFDM-based radar system and analyze the radar ambiguity function of AFDM with different built-in parameters, based on which we find an AFDM waveform with the specific parameter c2 owns the near-optimal time-domain ambiguity function. Then a low-complexity algorithm based on matched filtering for high-resolution target range estimation is proposed for this specific AFDM waveform. Through simulation and analysis, the specific AFDM waveform has near-optimal range estimation performance with the proposed low-complexity algorithm while having the same bit error rate (BER) performance as orthogonal time frequency space (OTFS) using simple linear minimum mean square error (LMMSE) equalizer. △ Less

Submitted 29 December, 2023; v1 submitted 18 December, 2023; originally announced December 2023.

Comments: The paper has been submitted to IEEE WCNC 2024 WS-13: Mobile Sensing-Communication-Computation Synergy for 6G Internet of Things

CDSD: Chinese Dysarthria Speech Database

Authors: Mengyi Sun, Ming Gao, Xinchen Kang, Shiru Wang, Jun Du, Dengfeng Yao, Su-**g Wang

Abstract: We present the Chinese Dysarthria Speech Database (CDSD) as a valuable resource for dysarthria research. This database comprises speech data from 24 participants with dysarthria. Among these participants, one recorded an additional 10 hours of speech data, while each recorded one hour, resulting in 34 hours of speech material. To accommodate participants with varying cognitive levels, our text poo… ▽ More We present the Chinese Dysarthria Speech Database (CDSD) as a valuable resource for dysarthria research. This database comprises speech data from 24 participants with dysarthria. Among these participants, one recorded an additional 10 hours of speech data, while each recorded one hour, resulting in 34 hours of speech material. To accommodate participants with varying cognitive levels, our text pool primarily consists of content from the AISHELL-1 dataset and speeches by primary and secondary school students. When participants read these texts, they must use a mobile device or the ZOOM F8n multi-track field recorder to record their speeches. In this paper, we elucidate the data collection and annotation processes and present an approach for establishing a baseline for dysarthric speech recognition. Furthermore, we conducted a speaker-dependent dysarthric speech recognition experiment using an additional 10 hours of speech data from one of our participants. Our research findings indicate that, through extensive data-driven model training, fine-tuning limited quantities of specific individual data yields commendable results in speaker-dependent dysarthric speech recognition. However, we observe significant variations in recognition results among different dysarthric speakers. These insights provide valuable reference points for speaker-dependent dysarthric speech recognition. △ Less

Submitted 24 October, 2023; originally announced October 2023.

Comments: 9 pages, 3 figures

A generalization of the achievable rate of a MISO system using Bode-Fano wideband matching theory

Authors: Nitish Deshpande, Miguel R. Castellanos, Saeed R. Khosravirad, **feng Du, Harish Viswanathan, Robert W. Heath Jr

Abstract: Impedance-matching networks affect power transfer from the radio frequency (RF) chains to the antennas. Their design impacts the signal to noise ratio (SNR) and the achievable rate. In this paper, we maximize the information-theoretic achievable rate of a multiple-input-single-output (MISO) system with wideband matching constraints. Using a multiport circuit theory approach with frequency-selectiv… ▽ More Impedance-matching networks affect power transfer from the radio frequency (RF) chains to the antennas. Their design impacts the signal to noise ratio (SNR) and the achievable rate. In this paper, we maximize the information-theoretic achievable rate of a multiple-input-single-output (MISO) system with wideband matching constraints. Using a multiport circuit theory approach with frequency-selective scattering parameters, we propose a general framework for optimizing the MISO achievable rate that incorporates Bode-Fano wideband matching theory. We express the solution to the achievable rate optimization problem in terms of the optimized transmission coefficient and the Lagrangian parameters corresponding to the Bode-Fano inequality constraints. We apply this framework to a single electric Chu's antenna and an array of two electric Chu's antennas. We compare the optimized achievable rate obtained numerically with other benchmarks like the ideal achievable rate computed by disregarding matching constraints and the achievable rate obtained by using sub-optimal matching strategies like conjugate matching and frequency-flat transmission. We also propose a practical methodology to approximate the achievable rate bound by using the optimal transmission coefficient to derive a physically realizable matching network through the ADS software. △ Less

Submitted 14 October, 2023; originally announced October 2023.

Continuous Modeling of the Denoising Process for Speech Enhancement Based on Deep Learning

Authors: Zilu Guo, Jun Du, CHin-Hui Lee

Abstract: In this paper, we explore a continuous modeling approach for deep-learning-based speech enhancement, focusing on the denoising process. We use a state variable to indicate the denoising process. The starting state is noisy speech and the ending state is clean speech. The noise component in the state variable decreases with the change of the state index until the noise component is 0. During traini… ▽ More In this paper, we explore a continuous modeling approach for deep-learning-based speech enhancement, focusing on the denoising process. We use a state variable to indicate the denoising process. The starting state is noisy speech and the ending state is clean speech. The noise component in the state variable decreases with the change of the state index until the noise component is 0. During training, a UNet-like neural network learns to estimate every state variable sampled from the continuous denoising process. In testing, we introduce a controlling factor as an embedding, ranging from zero to one, to the neural network, allowing us to control the level of noise reduction. This approach enables controllable speech enhancement and is adaptable to various application scenarios. Experimental results indicate that preserving a small amount of noise in the clean target benefits speech enhancement, as evidenced by improvements in both objective speech measures and automatic speech recognition performance. △ Less

Submitted 7 January, 2024; v1 submitted 17 September, 2023; originally announced September 2023.

Comments: We found the results are got from some wrong experimental settings. We needs new experiments

MFRL-BI: Design of a Model-free Reinforcement Learning Process Control Scheme by Using Bayesian Inference

Authors: Yanrong Li, Juan Du, Wei Jiang

Abstract: Design of process control scheme is critical for quality assurance to reduce variations in manufacturing systems. Taking semiconductor manufacturing as an example, extensive literature focuses on control optimization based on certain process models (usually linear models), which are obtained by experiments before a manufacturing process starts. However, in real applications, pre-defined models may… ▽ More Design of process control scheme is critical for quality assurance to reduce variations in manufacturing systems. Taking semiconductor manufacturing as an example, extensive literature focuses on control optimization based on certain process models (usually linear models), which are obtained by experiments before a manufacturing process starts. However, in real applications, pre-defined models may not be accurate, especially for a complex manufacturing system. To tackle model inaccuracy, we propose a model-free reinforcement learning (MFRL) approach to conduct experiments and optimize control simultaneously according to real-time data. Specifically, we design a novel MFRL control scheme by updating the distribution of disturbances using Bayesian inference to reduce their large variations during manufacturing processes. As a result, the proposed MFRL controller is demonstrated to perform well in a nonlinear chemical mechanical planarization (CMP) process when the process model is unknown. Theoretical properties are also guaranteed when disturbances are additive. The numerical studies also demonstrate the effectiveness and efficiency of our methodology. △ Less

Submitted 17 September, 2023; originally announced September 2023.

Comments: 31 pages, 7 figures, and 3 tables

Neural Speaker Diarization Using Memory-Aware Multi-Speaker Embedding with Sequence-to-Sequence Architecture

Authors: Gaobin Yang, Maokui He, Shutong Niu, Ruoyu Wang, Yanyan Yue, Shuangqing Qian, Shilong Wu, Jun Du, Chin-Hui Lee

Abstract: We propose a novel neural speaker diarization system using memory-aware multi-speaker embedding with sequence-to-sequence architecture (NSD-MS2S), which integrates the strengths of memory-aware multi-speaker embedding (MA-MSE) and sequence-to-sequence (Seq2Seq) architecture, leading to improvement in both efficiency and performance. Next, we further decrease the memory occupation of decoding by in… ▽ More We propose a novel neural speaker diarization system using memory-aware multi-speaker embedding with sequence-to-sequence architecture (NSD-MS2S), which integrates the strengths of memory-aware multi-speaker embedding (MA-MSE) and sequence-to-sequence (Seq2Seq) architecture, leading to improvement in both efficiency and performance. Next, we further decrease the memory occupation of decoding by incorporating input features fusion and then employ a multi-head attention mechanism to capture features at different levels. NSD-MS2S achieved a macro diarization error rate (DER) of 15.9% on the CHiME-7 EVAL set, which signifies a relative improvement of 49% over the official baseline system, and is the key technique for us to achieve the best performance for the main track of CHiME-7 DASR Challenge. Additionally, we introduce a deep interactive module (DIM) in MA-MSE module to better retrieve a cleaner and more discriminative multi-speaker embedding, enabling the current model to outperform the system we used in the CHiME-7 DASR Challenge. Our code will be available at https://github.com/liyunlongaaa/NSD-MS2S. △ Less

Submitted 26 December, 2023; v1 submitted 17 September, 2023; originally announced September 2023.

Comments: Accepted by ICASSP 2024

The Multimodal Information Based Speech Processing (MISP) 2023 Challenge: Audio-Visual Target Speaker Extraction

Authors: Shilong Wu, Chenxi Wang, Hang Chen, Yusheng Dai, Chenyue Zhang, Ruoyu Wang, Hongbo Lan, Jun Du, Chin-Hui Lee, **gdong Chen, Shinji Watanabe, Sabato Marco Siniscalchi, Odette Scharenborg, Zhong-Qiu Wang, Jia Pan, Jianqing Gao

Abstract: Previous Multimodal Information based Speech Processing (MISP) challenges mainly focused on audio-visual speech recognition (AVSR) with commendable success. However, the most advanced back-end recognition systems often hit performance limits due to the complex acoustic environments. This has prompted a shift in focus towards the Audio-Visual Target Speaker Extraction (AVTSE) task for the MISP 2023… ▽ More Previous Multimodal Information based Speech Processing (MISP) challenges mainly focused on audio-visual speech recognition (AVSR) with commendable success. However, the most advanced back-end recognition systems often hit performance limits due to the complex acoustic environments. This has prompted a shift in focus towards the Audio-Visual Target Speaker Extraction (AVTSE) task for the MISP 2023 challenge in ICASSP 2024 Signal Processing Grand Challenges. Unlike existing audio-visual speech enhance-ment challenges primarily focused on simulation data, the MISP 2023 challenge uniquely explores how front-end speech processing, combined with visual clues, impacts back-end tasks in real-world scenarios. This pioneering effort aims to set the first benchmark for the AVTSE task, offering fresh insights into enhancing the ac-curacy of back-end speech recognition systems through AVTSE in challenging and real acoustic environments. This paper delivers a thorough overview of the task setting, dataset, and baseline system of the MISP 2023 challenge. It also includes an in-depth analysis of the challenges participants may encounter. The experimental results highlight the demanding nature of this task, and we look forward to the innovative solutions participants will bring forward. △ Less

Submitted 15 September, 2023; originally announced September 2023.

Comments: 5 pages, 4 figures

Hierarchical Audio-Visual Information Fusion with Multi-label Joint Decoding for MER 2023

Authors: Haotian Wang, Yuxuan Xi, Hang Chen, Jun Du, Yan Song, Qing Wang, Hengshun Zhou, Chenxi Wang, Jiefeng Ma, Pengfei Hu, Ya Jiang, Shi Cheng, Jie Zhang, Yuzhe Weng

Abstract: In this paper, we propose a novel framework for recognizing both discrete and dimensional emotions. In our framework, deep features extracted from foundation models are used as robust acoustic and visual representations of raw video. Three different structures based on attention-guided feature gathering (AFG) are designed for deep feature fusion. Then, we introduce a joint decoding structure for e… ▽ More In this paper, we propose a novel framework for recognizing both discrete and dimensional emotions. In our framework, deep features extracted from foundation models are used as robust acoustic and visual representations of raw video. Three different structures based on attention-guided feature gathering (AFG) are designed for deep feature fusion. Then, we introduce a joint decoding structure for emotion classification and valence regression in the decoding stage. A multi-task loss based on uncertainty is also designed to optimize the whole process. Finally, by combining three different structures on the posterior probability level, we obtain the final predictions of discrete and dimensional emotions. When tested on the dataset of multimodal emotion recognition challenge (MER 2023), the proposed framework yields consistent improvements in both emotion classification and valence regression. Our final system achieves state-of-the-art performance and ranks third on the leaderboard on MER-MULTI sub-challenge. △ Less

Submitted 10 September, 2023; originally announced September 2023.

Comments: 5 pages, 4 figures

Journal ref: The 31st ACM International Conference on Multimedia (MM'23), 2023

Trade-Off Between Beamforming and Macro-Diversity Gains in Distributed mMIMO

Authors: Eduardo Noboro Tominaga, Hsuan-Jung Su, **feng Du, Sivarama Venkatesan, Richard Demo Souza, Hirley Alves

Abstract: Industry and academia have been working towards the evolution from Centralized massive Multiple-Input Multiple-Output (CmMIMO) to Distributed mMIMO (DmMIMO) architectures. Instead of splitting a coverage area into many cells, each served by a single Base Station equipped with several antennas, the whole coverage area is jointly covered by several Access Points (AP) equipped with few or single ante… ▽ More Industry and academia have been working towards the evolution from Centralized massive Multiple-Input Multiple-Output (CmMIMO) to Distributed mMIMO (DmMIMO) architectures. Instead of splitting a coverage area into many cells, each served by a single Base Station equipped with several antennas, the whole coverage area is jointly covered by several Access Points (AP) equipped with few or single antennas. Nevertheless, when choosing between deploying more APs with few or single antennas or fewer APs equipped with many antennas, one observes an inherent trade-off between the beamforming and macro-diversity gains that has not been investigated in the literature. Given a total number of antenna elements and total downlink power, under a channel model that takes into account a probability of Line-of-Sight (LoS) as a function of the distance between the User Equipments (UEs) and APs, our numerical results show that there exists a ``sweet spot" on the optimal number of APs and of antenna elements per AP which is a function of the physical dimensions of the coverage area. △ Less

Submitted 10 September, 2023; originally announced September 2023.

Comments: 6 pages, 3 figures. Manuscript submitted to the IEEE Wireless Communications and Networking Conference (WCNC) 2024, Dubai, United Arab Emirates

The USTC-NERCSLIP Systems for the CHiME-7 DASR Challenge

Authors: Ruoyu Wang, Maokui He, Jun Du, Hengshun Zhou, Shutong Niu, Hang Chen, Yanyan Yue, Gaobin Yang, Shilong Wu, Lei Sun, Yanhui Tu, Haitao Tang, Shuangqing Qian, Tian Gao, Mengzhi Wang, Genshun Wan, Jia Pan, Jianqing Gao, Chin-Hui Lee

Abstract: This technical report details our submission system to the CHiME-7 DASR Challenge, which focuses on speaker diarization and speech recognition under complex multi-speaker scenarios. Additionally, it also evaluates the efficiency of systems in handling diverse array devices. To address these issues, we implemented an end-to-end speaker diarization system and introduced a rectification strategy base… ▽ More This technical report details our submission system to the CHiME-7 DASR Challenge, which focuses on speaker diarization and speech recognition under complex multi-speaker scenarios. Additionally, it also evaluates the efficiency of systems in handling diverse array devices. To address these issues, we implemented an end-to-end speaker diarization system and introduced a rectification strategy based on multi-channel spatial information. This approach significantly diminished the word error rates (WER). In terms of recognition, we utilized publicly available pre-trained models as the foundational models to train our end-to-end speech recognition models. Our system attained a Macro-averaged diarization-attributed WER (DA-WER) of 21.01% on the CHiME-7 evaluation set, which signifies a relative improvement of 62.04% over the official baseline system. △ Less

Submitted 10 October, 2023; v1 submitted 28 August, 2023; originally announced August 2023.

Comments: Accepted by 2023 CHiME Workshop, Oral

Research on the Influence of Underwater Environment on the Dynamic Performance of the Mechanical Leg of a Deep-sea Crawling and Swimming Robot

Authors: Lihui Liao, Baoren Li, Dijia Zhang, Lu** Gao, Mboulé Ngwa, **gmin Du

Abstract: The performance of underwater crawling and adjustment of the body posture for underwater manipulating of the deep-sea crawling and swimming robot (DCSR) is directly influenced by the dynamic performance of the underwater mechanical legs (UWML), as it serves as the executive mechanism of the DCSR. Compared with the mechanical legs of legged robots working on land, the UWML of the DCSR not only poss… ▽ More The performance of underwater crawling and adjustment of the body posture for underwater manipulating of the deep-sea crawling and swimming robot (DCSR) is directly influenced by the dynamic performance of the underwater mechanical legs (UWML), as it serves as the executive mechanism of the DCSR. Compared with the mechanical legs of legged robots working on land, the UWML of the DCSR not only possesses the characteristics of the land used mechanical legs, but is also affected by the influence of the deep-sea underwater working environment (i.e., the hydrodynamic force, viscous resistance and dynamic seal resistance). To reduce these influence, firstly, the hydrodynamic force of the UWML were researched based on theory and experiment, and the hydrodynamic model was established with the fitted hydrodynamic parameters. Secondly, the oil viscous resistance and the dynamic seal resistance were studied experimentally, and the change laws of both with respect to the joint speed and the ambient pressure (depth of operation) were obtained. The results provide a basis for the subsequent research on the structure optimization and high performance control of the UWML and DCSR. △ Less

Submitted 28 August, 2023; originally announced August 2023.

Comments: conference for 2023 IEEE 9th International Conference on Fluid Power and Mechatronics (FPM2023)

MSC Class: 93C40 ACM Class: C.5

Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder

Authors: Yusheng Dai, Hang Chen, Jun Du, Xiaofei Ding, Ning Ding, Feijun Jiang, Chin-Hui Lee

Abstract: In recent research, slight performance improvement is observed from automatic speech recognition systems to audio-visual speech recognition systems in the end-to-end framework with low-quality videos. Unmatching convergence rates and specialized input representations between audio and visual modalities are considered to cause the problem. In this paper, we propose two novel techniques to improve a… ▽ More In recent research, slight performance improvement is observed from automatic speech recognition systems to audio-visual speech recognition systems in the end-to-end framework with low-quality videos. Unmatching convergence rates and specialized input representations between audio and visual modalities are considered to cause the problem. In this paper, we propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework. First, we explore the correlation between lip shapes and syllable-level subword units in Mandarin to establish good frame-level syllable boundaries from lip shapes. This enables accurate alignment of video and audio streams during visual model pre-training and cross-modal fusion. Next, we propose an audio-guided cross-modal fusion encoder (CMFE) neural network to utilize main training parameters for multiple cross-modal attention layers to make full use of modality complementarity. Experiments on the MISP2021-AVSR data set show the effectiveness of the two proposed techniques. Together, using only a relatively small amount of training data, the final system achieves better performances than state-of-the-art systems with more complex front-ends and back-ends. △ Less

Submitted 8 March, 2024; v1 submitted 14 August, 2023; originally announced August 2023.

Comments: 6 pages, 2 figures, published in ICME2023

DiVa: An Iterative Framework to Harvest More Diverse and Valid Labels from User Comments for Music

Authors: Hongru Liang, **gyao Liu, Yuanxin Xiang, Jiachen Du, Lanjun Zhou, Shushen Pan, Wenqiang Lei

Abstract: Towards sufficient music searching, it is vital to form a complete set of labels for each song. However, current solutions fail to resolve it as they cannot produce diverse enough map**s to make up for the information missed by the gold labels. Based on the observation that such missing information may already be presented in user comments, we propose to study the automated music labeling in an… ▽ More Towards sufficient music searching, it is vital to form a complete set of labels for each song. However, current solutions fail to resolve it as they cannot produce diverse enough map**s to make up for the information missed by the gold labels. Based on the observation that such missing information may already be presented in user comments, we propose to study the automated music labeling in an essential but under-explored setting, where the model is required to harvest more diverse and valid labels from the users' comments given limited gold labels. To this end, we design an iterative framework (DiVa) to harvest more $\underline{\text{Di}}$verse and $\underline{\text{Va}}$lid labels from user comments for music. The framework makes a classifier able to form complete sets of labels for songs via pseudo-labels inferred from pre-trained classifiers and a novel joint score function. The experiment on a densely annotated testing set reveals the superiority of the Diva over state-of-the-art solutions in producing more diverse labels missed by the gold labels. We hope our work can inspire future research on automated music labeling. △ Less

Submitted 9 August, 2023; originally announced August 2023.

Comments: 11 pages, 5 figures, published to ACM MM 2023

Semi-supervised multi-channel speaker diarization with cross-channel attention

Authors: Shilong Wu, Jun Du, Maokui He, Shutong Niu, Hang Chen, Haitao Tang, Chin-Hui Lee

Abstract: Most neural speaker diarization systems rely on sufficient manual training data labels, which are hard to collect under real-world scenarios. This paper proposes a semi-supervised speaker diarization system to utilize large-scale multi-channel training data by generating pseudo-labels for unlabeled data. Furthermore, we introduce cross-channel attention into the Neural Speaker Diarization Using Me… ▽ More Most neural speaker diarization systems rely on sufficient manual training data labels, which are hard to collect under real-world scenarios. This paper proposes a semi-supervised speaker diarization system to utilize large-scale multi-channel training data by generating pseudo-labels for unlabeled data. Furthermore, we introduce cross-channel attention into the Neural Speaker Diarization Using Memory-Aware Multi-Speaker Embedding (NSD-MA-MSE) to learn channel contextual information of speaker embeddings better. Experimental results on the CHiME-7 Mixer6 dataset which only contains partial speakers' labels of the training set, show that our system achieved 57.01% relative DER reduction compared to the clustering-based model on the development set. We further conducted experiments on the CHiME-6 dataset to simulate the scenario of missing partial training set labels. When using 80% and 50% labeled training data, our system performs comparably to the results obtained using 100% labeled data for training. △ Less

Submitted 17 July, 2023; originally announced July 2023.

Comments: 8 pages,3 figures

Variance-Preserving-Based Interpolation Diffusion Models for Speech Enhancement

Authors: Zilu Guo, Jun Du, Chin-Hui Lee, Yu Gao, Wenbin Zhang

Abstract: The goal of this study is to implement diffusion models for speech enhancement (SE). The first step is to emphasize the theoretical foundation of variance-preserving (VP)-based interpolation diffusion under continuous conditions. Subsequently, we present a more concise framework that encapsulates both the VP- and variance-exploding (VE)-based interpolation diffusion methods. We demonstrate that th… ▽ More The goal of this study is to implement diffusion models for speech enhancement (SE). The first step is to emphasize the theoretical foundation of variance-preserving (VP)-based interpolation diffusion under continuous conditions. Subsequently, we present a more concise framework that encapsulates both the VP- and variance-exploding (VE)-based interpolation diffusion methods. We demonstrate that these two methods are special cases of the proposed framework. Additionally, we provide a practical example of VP-based interpolation diffusion for the SE task. To improve performance and ease model training, we analyze the common difficulties encountered in diffusion models and suggest amenable hyper-parameters. Finally, we evaluate our model against several methods using a public benchmark to showcase the effectiveness of our approach △ Less

Submitted 17 September, 2023; v1 submitted 14 June, 2023; originally announced June 2023.

An Overview of Resource Allocation in Integrated Sensing and Communication

Authors: **ming Du, Yanqun Tang, Xizhang Wei, Jiaojiao Xiong, Jiajun Zhu, Haoran Yin, Chi Zhang, Haibo Chen

Abstract: Integrated sensing and communication (ISAC) is considered as a promising solution for improving spectrum efficiency and relieving wireless spectrum congestion. This paper systematically introduces the evolutionary path of ISAC technologies, then sorts out and summarizes the current research status of ISAC resource allocation. From the perspective of different integrated levels of ISAC, we introduc… ▽ More Integrated sensing and communication (ISAC) is considered as a promising solution for improving spectrum efficiency and relieving wireless spectrum congestion. This paper systematically introduces the evolutionary path of ISAC technologies, then sorts out and summarizes the current research status of ISAC resource allocation. From the perspective of different integrated levels of ISAC, we introduce and elaborate the research progress of resource allocation in different stages, namely, resource separated, orthogonal, converged, and collaborative stages. In addition, we give in-depth consideration to propose a new resource allocation framework from a multi-granularity perspective. Finally, we demonstrate the feasibility of our proposed framework with a case of full-duplex ISAC system. △ Less

Submitted 15 May, 2023; originally announced May 2023.

Comments: 6 pages,4 figures,conference

Maximin Headway Control of Automated Vehicles for System Optimal Dynamic Traffic Assignment in General Networks

Authors: **xiao Du, Wei Ma

Abstract: This study develops the headway control framework in a fully automated road network, as we believe headway of Automated Vehicles (AVs) is another influencing factor to traffic dynamics in addition to conventional vehicle behaviors (e.g. route and departure time choices). Specifically, we aim to search for the optimal time headway between AVs on each link that achieves the network-wide system optim… ▽ More This study develops the headway control framework in a fully automated road network, as we believe headway of Automated Vehicles (AVs) is another influencing factor to traffic dynamics in addition to conventional vehicle behaviors (e.g. route and departure time choices). Specifically, we aim to search for the optimal time headway between AVs on each link that achieves the network-wide system optimal dynamic traffic assignment (SO-DTA). To this end, the headway-dependent fundamental diagram (HFD) and headway-dependent double queue model (HDQ) are developed to model the effect of dynamic headway on roads, and a dynamic network model is built. It is rigorously proved that the minimum headway could always achieve SO-DTA, yet the optimal headway is non-unique. Motivated by these two findings, this study defines a novel concept of maximin headway, which is the largest headway that still achieves SO-DTA in the network. Mathematical properties regarding maximin headway are analyzed and an efficient solution algorithm is developed. Numerical experiments on both a small and large network verify the effectiveness of the maximin headway control framework as well as the properties of maximin headway. This study sheds light on deriving the desired solution among the non-unique solutions in SO-DTA and provides implications regarding the safety margin of AVs under SO-DTA. △ Less

Submitted 11 June, 2024; v1 submitted 29 March, 2023; originally announced March 2023.

Overview and Performance Analysis of Various Waveforms in High Mobility Scenarios

Authors: Yu Zhou, Haoran Yin, Jiaojiao Xiong, Shiyu Song, Jiajun Zhu, **ming Du, Haibo Chen, Yanqun Tang

Abstract: In the high-mobility scenarios of next-generation wireless communication systems (beyond 5G/6G), the performance of orthogonal frequency division multiplexing (OFDM) deteriorates drastically due to the loss of orthogonality between the subcarriers caused by large Doppler frequency shifts. Various emerging waveforms have been proposed for fast time-varying channels with excellent results. In this p… ▽ More In the high-mobility scenarios of next-generation wireless communication systems (beyond 5G/6G), the performance of orthogonal frequency division multiplexing (OFDM) deteriorates drastically due to the loss of orthogonality between the subcarriers caused by large Doppler frequency shifts. Various emerging waveforms have been proposed for fast time-varying channels with excellent results. In this paper, we classify these waveforms from the perspective of their modulation domain and establish a unified framework to provide a comprehensive system structure comparison. Then we analyze bit error rate (BER) performance of each waveform in doubly selective channels. Through the discussions on their complexity and compatibility with OFDM systems, we finally give the candidate waveform suggestions. △ Less

Submitted 27 February, 2023; originally announced February 2023.

Energy Efficient Extreme MIMO: Design Goals and Directions

Authors: Stefan Wesemann, **feng Du, Harish Viswanathan

Abstract: Ever since the invention of Bell Laboratories Layer Space-Time (BLAST) in mid 1990s, the focus of MIMO research and development has been largely on pushing the limit of spectral efficiency. While massive MIMO technologies laid the foundation of high spectrum efficiency in 5G and beyond, the challenge remains in improving energy efficiency given the increasing complexity of the associated radio sys… ▽ More Ever since the invention of Bell Laboratories Layer Space-Time (BLAST) in mid 1990s, the focus of MIMO research and development has been largely on pushing the limit of spectral efficiency. While massive MIMO technologies laid the foundation of high spectrum efficiency in 5G and beyond, the challenge remains in improving energy efficiency given the increasing complexity of the associated radio systems. With the substantial negative implications of climate change looming ever closer, minimizing energy use is a key dimension of achieving sustainability and is of paramount importance for any future technology. Thus, every aspect of future extreme MIMO system design, implementation, and operation will be scrutinized to maximize energy efficiency. An analysis of the massive MIMO 5G radio energy consumption at different loads leads to qualitative energy efficiency design goals for emerging extreme MIMO systems. Following this, we focus on novel operational and component technology innovations to minimize energy consumption. △ Less

Submitted 22 June, 2023; v1 submitted 3 January, 2023; originally announced January 2023.

Comments: This work has been accepted for publication by IEEE Communications Magazine. Copyright may be transferred without notice

Journal ref: IEEE Communications Magazine, 2023

Gradient and Channel Aware Dynamic Scheduling for Over-the-Air Computation in Federated Edge Learning Systems

Authors: Jun Du, Bingqing Jiang, Chunxiao Jiang, Yuanming Shi, Zhu Han

Abstract: To satisfy the expected plethora of computation-heavy applications, federated edge learning (FEEL) is a new paradigm featuring distributed learning to carry the capacities of low-latency and privacy-preserving. To further improve the efficiency of wireless data aggregation and model learning, over-the-air computation (AirComp) is emerging as a promising solution by using the superposition characte… ▽ More To satisfy the expected plethora of computation-heavy applications, federated edge learning (FEEL) is a new paradigm featuring distributed learning to carry the capacities of low-latency and privacy-preserving. To further improve the efficiency of wireless data aggregation and model learning, over-the-air computation (AirComp) is emerging as a promising solution by using the superposition characteristics of wireless channels. However, the fading and noise of wireless channels can cause aggregate distortions in AirComp enabled federated learning. In addition, the quality of collected data and energy consumption of edge devices may also impact the accuracy and efficiency of model aggregation as well as convergence. To solve these problems, this work proposes a dynamic device scheduling mechanism, which can select qualified edge devices to transmit their local models with a proper power control policy so as to participate the model training at the server in federated learning via AirComp. In this mechanism, the data importance is measured by the gradient of local model parameter, channel condition and energy consumption of the device jointly. In particular, to fully use distributed datasets and accelerate the convergence rate of federated learning, the local updates of unselected devices are also retained and accumulated for future potential transmission, instead of being discarded directly. Furthermore, the Lyapunov drift-plus-penalty optimization problem is formulated for searching the optimal device selection strategy. Simulation results validate that the proposed scheduling mechanism can achieve higher test accuracy and faster convergence rate, and is robust against different channel conditions. △ Less

Submitted 1 December, 2022; originally announced December 2022.

Speech-to-Speech Translation For A Real-world Unwritten Language

Authors: Peng-Jen Chen, Kevin Tran, Yilin Yang, **gfei Du, Justine Kao, Yu-An Chung, Paden Tomasello, Paul-Ambroise Duquenne, Holger Schwenk, Hongyu Gong, Hirofumi Inaguma, Sravya Popuri, Changhan Wang, Juan Pino, Wei-Ning Hsu, Ann Lee

Abstract: We study speech-to-speech translation (S2ST) that translates speech from one language into another language and focuses on building systems to support languages without standard text writing systems. We use English-Taiwanese Hokkien as a case study, and present an end-to-end solution from training data collection, modeling choices to benchmark dataset release. First, we present efforts on creating… ▽ More We study speech-to-speech translation (S2ST) that translates speech from one language into another language and focuses on building systems to support languages without standard text writing systems. We use English-Taiwanese Hokkien as a case study, and present an end-to-end solution from training data collection, modeling choices to benchmark dataset release. First, we present efforts on creating human annotated data, automatically mining data from large unlabeled speech datasets, and adopting pseudo-labeling to produce weakly supervised data. On the modeling, we take advantage of recent advances in applying self-supervised discrete representations as target for prediction in S2ST and show the effectiveness of leveraging additional text supervision from Mandarin, a language similar to Hokkien, in model training. Finally, we release an S2ST benchmark set to facilitate future research in this field. The demo can be found at https://huggingface.co/spaces/facebook/Hokkien_Translation . △ Less

Submitted 11 November, 2022; originally announced November 2022.

SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations

Authors: Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, **gfei Du, Ann Lee, Vedanuj Goswani, Changhan Wang, Juan Pino, Benoît Sagot, Holger Schwenk

Abstract: We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive basel… ▽ More We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on EuroParl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pre-training and sparse scaling using Mixture-of-Experts bring large gains to translation performance. The mined data and models are freely available. △ Less

Submitted 8 November, 2022; originally announced November 2022.

Comments: 18 pages

Deep Learning Based Audio-Visual Multi-Speaker DOA Estimation Using Permutation-Free Loss Function

Authors: Qing Wang, Hang Chen, Ya Jiang, Zhe Wang, Yuyang Wang, Jun Du, Chin-Hui Lee

Abstract: In this paper, we propose a deep learning based multi-speaker direction of arrival (DOA) estimation with audio and visual signals by using permutation-free loss function. We first collect a data set for multi-modal sound source localization (SSL) where both audio and visual signals are recorded in real-life home TV scenarios. Then we propose a novel spatial annotation method to produce the ground… ▽ More In this paper, we propose a deep learning based multi-speaker direction of arrival (DOA) estimation with audio and visual signals by using permutation-free loss function. We first collect a data set for multi-modal sound source localization (SSL) where both audio and visual signals are recorded in real-life home TV scenarios. Then we propose a novel spatial annotation method to produce the ground truth of DOA for each speaker with the video data by transformation between camera coordinate and pixel coordinate according to the pin-hole camera model. With spatial location information served as another input along with acoustic feature, multi-speaker DOA estimation could be solved as a classification task of active speaker detection. Label permutation problem in multi-speaker related tasks will be addressed since the locations of each speaker are used as input. Experiments conducted on both simulated data and real data show that the proposed audio-visual DOA estimation model outperforms audio-only DOA estimation model by a large margin. △ Less

Submitted 26 October, 2022; originally announced October 2022.

Comments: 5 pages, 3 figures, accepted by ISCSLP 2022

MS-DCANet: A Novel Segmentation Network For Multi-Modality COVID-19 Medical Images

Authors: Xiaoyu Pan, Huazheng Zhu, **glong Du, Guangtao Hu, Baoru Han, Yuanyuan Jia

Abstract: The Coronavirus Disease 2019 (COVID-19) pandemic has increased the public health burden and brought profound disaster to humans. For the particularity of the COVID-19 medical images with blurred boundaries, low contrast and different infection sites, some researchers have improved the accuracy by adding more complexity. Also, they overlook the complexity of lesions, which hinder their ability to c… ▽ More The Coronavirus Disease 2019 (COVID-19) pandemic has increased the public health burden and brought profound disaster to humans. For the particularity of the COVID-19 medical images with blurred boundaries, low contrast and different infection sites, some researchers have improved the accuracy by adding more complexity. Also, they overlook the complexity of lesions, which hinder their ability to capture the relationship between segmentation sites and the background, as well as the edge contours and global context. However, increasing the computational complexity, parameters and inference speed is unfavorable for model transfer from laboratory to clinic. A perfect segmentation network needs to balance the above three factors completely. To solve the above issues, this paper propose a symmetric automatic segmentation framework named MS-DCANet. We introduce Tokenized MLP block, a novel attention scheme that use a shift-window mechanism to conditionally fuse local and global features to get more continuous boundaries and spatial positioning capabilities. It has greater understanding of irregular lesions contours. MS-DCANet also uses several Dual Channel blocks and a Res-ASPP block to improve the ability to recognize small targets. On multi-modality COVID-19 tasks, MS-DCANet achieved state-of-the-art performance compared with other baselines. It can well trade off the accuracy and complexity. To prove the strong generalization ability of our proposed model, we apply it to other tasks (ISIC 2018 and BAA) and achieve satisfactory results. △ Less

Submitted 19 July, 2023; v1 submitted 22 October, 2022; originally announced October 2022.

Comments: 21pages,13 figures,9 tables

Journal ref: J Multidiscip Healthc. 2023;16:2023-2043

Convergence Theory of Generalized Distributed Subgradient Method with Random Quantization

Authors: Zhaoyue Xia, Jun Du, Yong Ren

Abstract: The distributed subgradient method (DSG) is a widely discussed algorithm to cope with large-scale distributed optimization problems in the arising machine learning applications. Most exisiting works on DSG focus on ideal communication between the cooperative agents such that the shared information between agents is exact and perfect. This assumption, however, could lead to potential privacy concer… ▽ More The distributed subgradient method (DSG) is a widely discussed algorithm to cope with large-scale distributed optimization problems in the arising machine learning applications. Most exisiting works on DSG focus on ideal communication between the cooperative agents such that the shared information between agents is exact and perfect. This assumption, however, could lead to potential privacy concerns and is not feasible when the wireless transmission links are not of good quality. To overcome the challenge, a common approach is to quantize the data locally before transmission, which avoids exposure of raw data and significantly reduces the size of data. Compared with perfect data, quantization poses fundamental challenges on loss of data accuracy, which further impacts the convergence of the algorithms. To settle the problem, we propose a generalized distributed subgradient method with random quantization, which can be intepreted as a two time-scale stochastic approximation method. We provide comprehensive results on the convergence of the algorithm and derive upper bounds on the convergence rates in terms of the quantization bit, stepsizes and the number of network agents. Our results extend the existing results, where only special cases are considered and general conclusions for the convergence rates are missing. Finally, numerical simulations are conducted on linear regression problems to support our theoretical results. △ Less

Submitted 23 August, 2022; v1 submitted 22 July, 2022; originally announced July 2022.

A wideband generalization of the near-field region for extremely large phased-arrays

Authors: Nitish Deshpande, Miguel R. Castellanos, Saeed R. Khosravirad, **feng Du, Harish Viswanathan, Robert W. Heath Jr

Abstract: The narrowband and far-field assumption in conventional wireless system design leads to a mismatch with the optimal beamforming required for wideband and near-field systems. This discrepancy is exacerbated for larger apertures and bandwidths. To characterize the behavior of near-field and wideband systems, we derive the beamforming gain expression achieved by a frequency-flat phased array designed… ▽ More The narrowband and far-field assumption in conventional wireless system design leads to a mismatch with the optimal beamforming required for wideband and near-field systems. This discrepancy is exacerbated for larger apertures and bandwidths. To characterize the behavior of near-field and wideband systems, we derive the beamforming gain expression achieved by a frequency-flat phased array designed for plane-wave propagation. To determine the far-field to near-field boundary for a wideband system, we propose a frequency-selective distance metric. The proposed far-field threshold increases for frequencies away from the center frequency. The analysis results in a fundamental upper bound on the product of the array aperture and the system bandwidth. We present numerical results to illustrate how the gain threshold affects the maximum usable bandwidth for the n260 and n261 5G NR bands. △ Less

Submitted 29 June, 2022; v1 submitted 28 June, 2022; originally announced June 2022.

Outdoor-to-Indoor 28 GHz Wireless Measurements in Manhattan: Path Loss, Environmental Effects, and 90% Coverage

Authors: Manav Kohli, Abhishek Adhikari, Gulnur Avci, Sienna Brent, Aditya Dash, Jared Moser, Sabbir Hossain, Igor Kadota, Carson Garland, Shivan Mukherjee, Rodolfo Feick, Dmitry Chizhik, **feng Du, Reinaldo A. Valenzuela, Gil Zussman

Abstract: Outdoor-to-indoor (OtI) signal propagation further challenges the already tight link budgets at millimeter-wave (mmWave). To gain insight into OtI mmWave scenarios at 28 GHz, we conducted an extensive measurement campaign consisting of over 2,200 link measurements. In total, 43 OtI scenarios were measured in West Harlem, New York City, covering seven highly diverse buildings. The measured OtI path… ▽ More Outdoor-to-indoor (OtI) signal propagation further challenges the already tight link budgets at millimeter-wave (mmWave). To gain insight into OtI mmWave scenarios at 28 GHz, we conducted an extensive measurement campaign consisting of over 2,200 link measurements. In total, 43 OtI scenarios were measured in West Harlem, New York City, covering seven highly diverse buildings. The measured OtI path gain can vary by up to 40 dB for a given link distance, and the empirical path gain model for all data shows an average of 30 dB excess loss over free space at distances beyond 50 m, with an RMS fitting error of 11.7 dB. The type of glass is found to be the single dominant feature for OtI loss, with 20 dB observed difference between empirical path gain models for scenarios with low-loss and high-loss glass. The presence of scaffolding, tree foliage, or elevated subway tracks, as well as difference in floor height are each found to have an impact between 5-10 dB. We show that for urban buildings with high-loss glass, OtI coverage can support 500 Mbps for 90% of indoor user equipment (UEs) with a base station (BS) antenna placed up to 49 m away. For buildings with low-loss glass, such as our case study covering multiple classrooms of a public school, data rates over 2.5/1.2 Gbps are possible from a BS 68/175 m away from the school building, when a line-of-sight path is available. We expect these results to be useful for the deployment of mmWave networks in dense urban environments as well as the development of relevant scheduling and beam management algorithms. △ Less

Submitted 19 May, 2022; originally announced May 2022.

Comments: 13 pages, 13 figures

Unsupervised Denoising of Optical Coherence Tomography Images with Dual_Merged CycleWGAN

Authors: Jie Du, Xujian Yang, Kecheng **, Xuanzheng Qi, Hu Chen

Abstract: Nosie is an important cause of low quality Optical coherence tomography (OCT) image. The neural network model based on Convolutional neural networks(CNNs) has demonstrated its excellent performance in image denoising. However, OCT image denoising still faces great challenges because many previous neural network algorithms required a large number of labeled data, which might cost much time or is ex… ▽ More Nosie is an important cause of low quality Optical coherence tomography (OCT) image. The neural network model based on Convolutional neural networks(CNNs) has demonstrated its excellent performance in image denoising. However, OCT image denoising still faces great challenges because many previous neural network algorithms required a large number of labeled data, which might cost much time or is expensive. Besides, these CNN-based algorithms need numerous parameters and good tuning techniques, which is hardware resources consuming. To solved above problems, We proposed a new Cycle-Consistent Generative Adversarial Nets called Dual-Merged Cycle-WGAN for retinal OCT image denoiseing, which has remarkable performance with less unlabeled traning data. Our model consists of two Cycle-GAN networks with imporved generator, descriminator and wasserstein loss to achieve good training stability and better performance. Using image merge technique between two Cycle-GAN networks, our model could obtain more detailed information and hence better training effect. The effectiveness and generality of our proposed network has been proved via ablation experiments and comparative experiments. Compared with other state-of-the-art methods, our unsupervised method obtains best subjective visual effect and higher evaluation objective indicators. △ Less

Submitted 2 May, 2022; originally announced May 2022.

Comments: Mr. Hu Chen is our corresponding author

A study on joint modeling and data augmentation of multi-modalities for audio-visual scene classification

Authors: Qing Wang, Jun Du, Siyuan Zheng, Yunqing Li, Yajian Wang, Yuzhong Wu, Hu Hu, Chao-Han Huck Yang, Sabato Marco Siniscalchi, Yannan Wang, Chin-Hui Lee

Abstract: In this paper, we propose two techniques, namely joint modeling and data augmentation, to improve system performances for audio-visual scene classification (AVSC). We employ pre-trained networks trained only on image data sets to extract video embedding; whereas for audio embedding models, we decide to train them from scratch. We explore different neural network architectures for joint modeling to… ▽ More In this paper, we propose two techniques, namely joint modeling and data augmentation, to improve system performances for audio-visual scene classification (AVSC). We employ pre-trained networks trained only on image data sets to extract video embedding; whereas for audio embedding models, we decide to train them from scratch. We explore different neural network architectures for joint modeling to effectively combine the video and audio modalities. Moreover, data augmentation strategies are investigated to increase audio-visual training set size. For the video modality the effectiveness of several operations in RandAugment is verified. An audio-video joint mixup scheme is proposed to further improve AVSC performances. Evaluated on the development set of TAU Urban Audio Visual Scenes 2021, our final system can achieve the best accuracy of 94.2% among all single AVSC systems submitted to DCASE 2021 Task 1b. △ Less

Submitted 31 August, 2022; v1 submitted 7 March, 2022; originally announced March 2022.

Comments: 5 pages, 1 figure

Dense Urban Outdoor-Indoor Coverage from 3.5 to 28 GHz

Authors: Dipankar Shakya, Dmitry Chizhik, **feng Du, Reinaldo A. Valenzuela, Theodore S. Rappaport

Abstract: In the US, people spend 87% of their time indoors and have an average of four connected devices per person (in 2020). As such, providing indoor coverage has always been a challenge but becomes even more difficult as carrier frequencies increase to mmWave and beyond. This paper investigates the outdoor and outdoor-indoor coverage of an urban network comparing globally standardized building penetrat… ▽ More In the US, people spend 87% of their time indoors and have an average of four connected devices per person (in 2020). As such, providing indoor coverage has always been a challenge but becomes even more difficult as carrier frequencies increase to mmWave and beyond. This paper investigates the outdoor and outdoor-indoor coverage of an urban network comparing globally standardized building penetration models and implementing models to corresponding scenarios. The glass used in windows of buildings in the grid plays a pivotal role in determining the outdoor-to-indoor propagation loss. For 28 GHz with 1 W/polarization transmit power in the urban street grid, the downlink data rates for 90% of outdoor users are estimated at over 250 Mbps. In contrast, 15% of indoor users are estimated to be in outage, with SNR $<-$3 dB when base stations are 400 m apart with one-fifth of the buildings imposing high penetration loss ($\sim$ 35 dB). At 3.5 GHz, base stations may achieve over 250 Mbps for 90% indoor users if 400 MHz bandwidth with 100 W/polarization transmit power is available. The methods and models presented can be used to facilitate decisions regarding the density and transmit power required to provide high data rates to majority users in urban centers. △ Less

Submitted 7 March, 2022; originally announced March 2022.

Comments: 6 pages, 8 figures, ICC 2022 conference

Sampling and Reconstructing Angular Domains with Uniform Arrays

Authors: Silvio Mandelli, Marcus Henninger, **feng Du

Abstract: The surge of massive antenna arrays in wireless networks calls for the adoption of analog/hybrid array solutions, where multiple antenna elements are driven by a common radio front end to form a beam along a specific angle in order to maximize the beamforming gain. Many heuristics have been proposed to sample the angular domain by trading off between sampling step size and overhead, where arbitrar… ▽ More The surge of massive antenna arrays in wireless networks calls for the adoption of analog/hybrid array solutions, where multiple antenna elements are driven by a common radio front end to form a beam along a specific angle in order to maximize the beamforming gain. Many heuristics have been proposed to sample the angular domain by trading off between sampling step size and overhead, where arbitrarily small angular step size is only attainable with infinite sampling overhead. We show that, for uniform linear and rectangular arrays, lossless reconstruction of the array's angular responses at arbitrary angular precision is possible using a finite number of samples without resorting to assumptions of angular sparsity. The proposed method, sampling and reconstructing angular domain (SARA), defines how many and which angles to be sampled and the corresponding reconstruction. This general solution to scan the angular domain can therefore be applied not only to beam acquisition and channel estimation, but also to radio imaging techniques, making it a candidate for future integrated sensing and communications (ISAC). Extensive simulation results for target detection and radio imaging have demonstrated clear advantages of SARA versus other considered baselines, both in terms of angular reconstruction performance and computational complexity. △ Less

Submitted 3 July, 2023; v1 submitted 17 February, 2022; originally announced February 2022.

Comments: 15 pages, 10 figures. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. This version corrects a typo in Section IV-D of the published paper in IEEE Transactions of Wireless Communications

A Study of Designing Compact Audio-Visual Wake Word Spotting System Based on Iterative Fine-Tuning in Neural Network Pruning

Authors: Hengshun Zhou, Jun Du, Chao-Han Huck Yang, Shifu Xiong, Chin-Hui Lee

Abstract: Audio-only-based wake word spotting (WWS) is challenging under noisy conditions due to environmental interference in signal transmission. In this paper, we investigate on designing a compact audio-visual WWS system by utilizing visual information to alleviate the degradation. Specifically, in order to use visual information, we first encode the detected lips to fixed-size vectors with MobileNet an… ▽ More Audio-only-based wake word spotting (WWS) is challenging under noisy conditions due to environmental interference in signal transmission. In this paper, we investigate on designing a compact audio-visual WWS system by utilizing visual information to alleviate the degradation. Specifically, in order to use visual information, we first encode the detected lips to fixed-size vectors with MobileNet and concatenate them with acoustic features followed by the fusion network for WWS. However, the audio-visual model based on neural networks requires a large footprint and a high computational complexity. To meet the application requirements, we introduce a neural network pruning strategy via the lottery ticket hypothesis in an iterative fine-tuning manner (LTH-IF), to the single-modal and multi-modal models, respectively. Tested on our in-house corpus for audio-visual WWS in a home TV scene, the proposed audio-visual system achieves significant performance improvements over the single-modality (audio-only or video-only) system under different noisy conditions. Moreover, LTH-IF pruning can largely reduce the network parameters and computations with no degradation of WWS performance, leading to a potential product solution for the TV wake-up scenario. △ Less

Submitted 17 February, 2022; originally announced February 2022.

Comments: Accepted to ICASSP 2022. H. Zhou et al