-
Multi-modal Mood Reader: Pre-trained Model Empowers Cross-Subject Emotion Recognition
Authors:
Yihang Dong,
Xuhang Chen,
Yanyan Shen,
Michael Kwok-Po Ng,
Tao Qian,
Shuqiang Wang
Abstract:
Emotion recognition based on Electroencephalography (EEG) has gained significant attention and diversified development in fields such as neural signal processing and affective computing. However, the unique brain anatomy of individuals leads to non-negligible natural differences in EEG signals across subjects, posing challenges for cross-subject emotion recognition. While recent studies have attem…
▽ More
Emotion recognition based on Electroencephalography (EEG) has gained significant attention and diversified development in fields such as neural signal processing and affective computing. However, the unique brain anatomy of individuals leads to non-negligible natural differences in EEG signals across subjects, posing challenges for cross-subject emotion recognition. While recent studies have attempted to address these issues, they still face limitations in practical effectiveness and model framework unity. Current methods often struggle to capture the complex spatial-temporal dynamics of EEG signals and fail to effectively integrate multimodal information, resulting in suboptimal performance and limited generalizability across subjects. To overcome these limitations, we develop a Pre-trained model based Multimodal Mood Reader for cross-subject emotion recognition that utilizes masked brain signal modeling and interlinked spatial-temporal attention mechanism. The model learns universal latent representations of EEG signals through pre-training on large scale dataset, and employs Interlinked spatial-temporal attention mechanism to process Differential Entropy(DE) features extracted from EEG data. Subsequently, a multi-level fusion layer is proposed to integrate the discriminative features, maximizing the advantages of features across different dimensions and modalities. Extensive experiments on public datasets demonstrate Mood Reader's superior performance in cross-subject emotion recognition tasks, outperforming state-of-the-art methods. Additionally, the model is dissected from attention perspective, providing qualitative analysis of emotion-related brain areas, offering valuable insights for affective research in neural signal processing.
△ Less
Submitted 28 May, 2024;
originally announced May 2024.
-
Hyperspectral and multispectral image fusion with arbitrary resolution through self-supervised representations
Authors:
Ting Wang,
Zipei Yan,
Jizhou Li,
Xile Zhao,
Chao Wang,
Michael Ng
Abstract:
The fusion of a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) has emerged as an effective technique for achieving HSI super-resolution (SR). Previous studies have mainly concentrated on estimating the posterior distribution of the latent high-resolution hyperspectral image (HR-HSI), leveraging an appropriate image prior and likelihood computed from…
▽ More
The fusion of a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) has emerged as an effective technique for achieving HSI super-resolution (SR). Previous studies have mainly concentrated on estimating the posterior distribution of the latent high-resolution hyperspectral image (HR-HSI), leveraging an appropriate image prior and likelihood computed from the discrepancy between the latent HSI and observed images. Low rankness stands out for preserving latent HSI characteristics through matrix factorization among the various priors. However, this method only enhances resolution within the dimensions of the two modalities. To overcome this limitation, we propose a novel continuous low-rank factorization (CLoRF) by integrating two neural representations into the matrix factorization, capturing spatial and spectral information, respectively. This approach enables us to harness both the low rankness from the matrix factorization and the continuity from neural representation in a self-supervised manner. Theoretically, we prove the low-rank property and Lipschitz continuity in the proposed continuous low-rank factorization. Experimentally, our method significantly surpasses existing techniques and achieves user-desired resolutions without the need for neural network retraining.
△ Less
Submitted 28 May, 2024;
originally announced May 2024.
-
Automatic Assessment of Dysarthria Using Audio-visual Vowel Graph Attention Network
Authors:
Xiaokang Liu,
Xiaoxia Du,
Juan Liu,
Rongfeng Su,
Manwa Lawrence Ng,
Yumei Zhang,
Yudong Yang,
Shaofeng Zhao,
Lan Wang,
Nan Yan
Abstract:
Automatic assessment of dysarthria remains a highly challenging task due to high variability in acoustic signals and the limited data. Currently, research on the automatic assessment of dysarthria primarily focuses on two approaches: one that utilizes expert features combined with machine learning, and the other that employs data-driven deep learning methods to extract representations. Research ha…
▽ More
Automatic assessment of dysarthria remains a highly challenging task due to high variability in acoustic signals and the limited data. Currently, research on the automatic assessment of dysarthria primarily focuses on two approaches: one that utilizes expert features combined with machine learning, and the other that employs data-driven deep learning methods to extract representations. Research has demonstrated that expert features are effective in representing pathological characteristics, while deep learning methods excel at uncovering latent features. Therefore, integrating the advantages of expert features and deep learning to construct a neural network architecture based on expert knowledge may be beneficial for interpretability and assessment performance. In this context, the present paper proposes a vowel graph attention network based on audio-visual information, which effectively integrates the strengths of expert knowledges and deep learning. Firstly, various features were combined as inputs, including knowledge based acoustical features and deep learning based pre-trained representations. Secondly, the graph network structure based on vowel space theory was designed, allowing for a deep exploration of spatial correlations among vowels. Finally, visual information was incorporated into the model to further enhance its robustness and generalizability. The method exhibited superior performance in regression experiments targeting Frenchay scores compared to existing approaches.
△ Less
Submitted 6 May, 2024; v1 submitted 6 May, 2024;
originally announced May 2024.
-
Semi-on-Demand Hybrid Transit Route Design with Shared Autonomous Mobility Services
Authors:
Max T. M. Ng,
Florian Dandl,
Hani S. Mahmassani,
Klaus Bogenberger
Abstract:
This study examines the route design of a semi-on-demand hybrid route directional service in the public transit network, offering on-demand flexible route service in low-density areas and fixed route service in higher-density areas with Shared Autonomous Mobility Service (SAMS). The study develops analytically tractable cost expressions that capture access, waiting, and riding costs for users, and…
▽ More
This study examines the route design of a semi-on-demand hybrid route directional service in the public transit network, offering on-demand flexible route service in low-density areas and fixed route service in higher-density areas with Shared Autonomous Mobility Service (SAMS). The study develops analytically tractable cost expressions that capture access, waiting, and riding costs for users, and distance-based operating and time-based vehicle costs for operators. Two formulations are presented for strategic and tactical decisions in flexible route portion, fleet size, headway, and vehicle size optimization, enabling the determination of route types between fixed, hybrid, and flexible routes based on demand, cost, and operational parameters. The practical applications and benefits of semi-on-demand feeders are demonstrated with numerical examples and a large-scale case study in the Chicago metropolitan area. Findings reveal scenarios in which flexible route portions serving passengers located further away reduce total costs, particularly user costs. Lower operating costs in lower-demand areas favor more flexible routes, whereas higher demand densities favor more traditional line-based operations. On two studied lines, a current cost forecast favors smaller vehicles with flexible routes, but operating constraints and higher operating costs would favor bigger vehicles with hybrid routes. The study provides an analytical tool to design SAMS as directional services and transit feeders, and tractable continuous approximation formulations for future research in transit network design.
△ Less
Submitted 23 March, 2024;
originally announced March 2024.
-
Uniform Recovery Guarantees for Quantized Corrupted Sensing Using Structured or Generative Priors
Authors:
Junren Chen,
Zhaoqiang Liu,
Meng Ding,
Michael K. Ng
Abstract:
This paper studies quantized corrupted sensing where the measurements are contaminated by unknown corruption and then quantized by a dithered uniform quantizer. We establish uniform guarantees for Lasso that ensure the accurate recovery of all signals and corruptions using a single draw of the sub-Gaussian sensing matrix and uniform dither. For signal and corruption with structured priors (e.g., s…
▽ More
This paper studies quantized corrupted sensing where the measurements are contaminated by unknown corruption and then quantized by a dithered uniform quantizer. We establish uniform guarantees for Lasso that ensure the accurate recovery of all signals and corruptions using a single draw of the sub-Gaussian sensing matrix and uniform dither. For signal and corruption with structured priors (e.g., sparsity, low-rankness), our uniform error rate for constrained Lasso typically coincides with the non-uniform one [Sun, Cui and Liu, 2022] up to logarithmic factors. By contrast, our uniform error rate for unconstrained Lasso exhibits worse dependence on the structured parameters due to regularization parameters larger than the ones for non-uniform recovery. For signal and corruption living in the ranges of some Lipschitz continuous generative models (referred to as generative priors), we achieve uniform recovery via constrained Lasso with a measurement number proportional to the latent dimensions of the generative models. Our treatments to the two kinds of priors are (nearly) unified and share the common key ingredients of (global) quantized product embedding (QPE) property, which states that the dithered uniform quantization (universally) preserves inner product. As a by-product, our QPE result refines the one in [Xu and Jacques, 2020] under sub-Gaussian random matrix, and in this specific instance we are able to sharpen the uniform error decaying rate (for the projected-back projection estimator with signals in some convex symmetric set) presented therein from $O(m^{-1/16})$ to $O(m^{-1/8})$.
△ Less
Submitted 16 January, 2024;
originally announced January 2024.
-
BDHT: Generative AI Enables Causality Analysis for Mild Cognitive Impairment
Authors:
Qiankun Zuo,
Ling Chen,
Yanyan Shen,
Michael Kwok-Po Ng,
Baiying Lei,
Shuqiang Wang
Abstract:
Effective connectivity estimation plays a crucial role in understanding the interactions and information flow between different brain regions. However, the functional time series used for estimating effective connectivity is derived from certain software, which may lead to large computing errors because of different parameter settings and degrade the ability to model complex causal relationships b…
▽ More
Effective connectivity estimation plays a crucial role in understanding the interactions and information flow between different brain regions. However, the functional time series used for estimating effective connectivity is derived from certain software, which may lead to large computing errors because of different parameter settings and degrade the ability to model complex causal relationships between brain regions. In this paper, a brain diffuser with hierarchical transformer (BDHT) is proposed to estimate effective connectivity for mild cognitive impairment (MCI) analysis. To our best knowledge, the proposed brain diffuser is the first generative model to apply diffusion models to the application of generating and analyzing multimodal brain networks. Specifically, the BDHT leverages structural connectivity to guide the reverse processes in an efficient way. It makes the denoising process more reliable and guarantees effective connectivity estimation accuracy. To improve denoising quality, the hierarchical denoising transformer is designed to learn multi-scale features in topological space. By stacking the multi-head attention and graph convolutional network, the graph convolutional transformer (GraphConformer) module is devised to enhance structure-function complementarity and improve the ability in noise estimation. Experimental evaluations of the denoising diffusion model demonstrate its effectiveness in estimating effective connectivity. The proposed model achieves superior performance in terms of accuracy and robustness compared to existing approaches. Moreover, the proposed model can identify altered directional connections and provide a comprehensive understanding of parthenogenesis for MCI treatment.
△ Less
Submitted 28 May, 2024; v1 submitted 14 December, 2023;
originally announced December 2023.
-
A Unified Framework for Uniform Signal Recovery in Nonlinear Generative Compressed Sensing
Authors:
Junren Chen,
Jonathan Scarlett,
Michael K. Ng,
Zhaoqiang Liu
Abstract:
In generative compressed sensing (GCS), we want to recover a signal $\mathbf{x}^* \in \mathbb{R}^n$ from $m$ measurements ($m\ll n$) using a generative prior $\mathbf{x}^*\in G(\mathbb{B}_2^k(r))$, where $G$ is typically an $L$-Lipschitz continuous generative model and $\mathbb{B}_2^k(r)$ represents the radius-$r$ $\ell_2$-ball in $\mathbb{R}^k$. Under nonlinear measurements, most prior results ar…
▽ More
In generative compressed sensing (GCS), we want to recover a signal $\mathbf{x}^* \in \mathbb{R}^n$ from $m$ measurements ($m\ll n$) using a generative prior $\mathbf{x}^*\in G(\mathbb{B}_2^k(r))$, where $G$ is typically an $L$-Lipschitz continuous generative model and $\mathbb{B}_2^k(r)$ represents the radius-$r$ $\ell_2$-ball in $\mathbb{R}^k$. Under nonlinear measurements, most prior results are non-uniform, i.e., they hold with high probability for a fixed $\mathbf{x}^*$ rather than for all $\mathbf{x}^*$ simultaneously. In this paper, we build a unified framework to derive uniform recovery guarantees for nonlinear GCS where the observation model is nonlinear and possibly discontinuous or unknown. Our framework accommodates GCS with 1-bit/uniformly quantized observations and single index models as canonical examples. Specifically, using a single realization of the sensing ensemble and generalized Lasso, {\em all} $\mathbf{x}^*\in G(\mathbb{B}_2^k(r))$ can be recovered up to an $\ell_2$-error at most $ε$ using roughly $\tilde{O}({k}/{ε^2})$ samples, with omitted logarithmic factors typically being dominated by $\log L$. Notably, this almost coincides with existing non-uniform guarantees up to logarithmic factors, hence the uniformity costs very little. As part of our technical contributions, we introduce the Lipschitz approximation to handle discontinuous observation models. We also develop a concentration inequality that produces tighter bounds for product processes whose index sets have low metric entropy. Experimental results are presented to corroborate our theory.
△ Less
Submitted 9 October, 2023; v1 submitted 25 September, 2023;
originally announced October 2023.
-
Solving Quadratic Systems with Full-Rank Matrices Using Sparse or Generative Priors
Authors:
Junren Chen,
Shuai Huang,
Michael K. Ng,
Zhaoqiang Liu
Abstract:
The problem of recovering a signal $\boldsymbol{x} \in \mathbb{R}^n$ from a quadratic system $\{y_i=\boldsymbol{x}^\top\boldsymbol{A}_i\boldsymbol{x},\ i=1,\ldots,m\}$ with full-rank matrices $\boldsymbol{A}_i$ frequently arises in applications such as unassigned distance geometry and sub-wavelength imaging. With i.i.d. standard Gaussian matrices $\boldsymbol{A}_i$, this paper addresses the high-d…
▽ More
The problem of recovering a signal $\boldsymbol{x} \in \mathbb{R}^n$ from a quadratic system $\{y_i=\boldsymbol{x}^\top\boldsymbol{A}_i\boldsymbol{x},\ i=1,\ldots,m\}$ with full-rank matrices $\boldsymbol{A}_i$ frequently arises in applications such as unassigned distance geometry and sub-wavelength imaging. With i.i.d. standard Gaussian matrices $\boldsymbol{A}_i$, this paper addresses the high-dimensional case where $m\ll n$ by incorporating prior knowledge of $\boldsymbol{x}$. First, we consider a $k$-sparse $\boldsymbol{x}$ and introduce the thresholded Wirtinger flow (TWF) algorithm that does not require the sparsity level $k$. TWF comprises two steps: the spectral initialization that identifies a point sufficiently close to $\boldsymbol{x}$ (up to a sign flip) when $m=O(k^2\log n)$, and the thresholded gradient descent (with a good initialization) that produces a sequence linearly converging to $\boldsymbol{x}$ with $m=O(k\log n)$ measurements. Second, we explore the generative prior, assuming that $\boldsymbol{x}$ lies in the range of an $L$-Lipschitz continuous generative model with $k$-dimensional inputs in an $\ell_2$-ball of radius $r$. We develop the projected gradient descent (PGD) algorithm that also comprises two steps: the projected power method that provides an initial vector with $O\big(\sqrt{\frac{k \log L}{m}}\big)$ $\ell_2$-error given $m=O(k\log(Lnr))$ measurements, and the projected gradient descent that refines the $\ell_2$-error to $O(δ)$ at a geometric rate when $m=O(k\log\frac{Lrn}{δ^2})$. Experimental results corroborate our theoretical findings and show that: (i) our approach for the sparse case notably outperforms the existing provable algorithm sparse power factorization; (ii) leveraging the generative prior allows for precise image recovery in the MNIST dataset from a small number of quadratic measurements.
△ Less
Submitted 16 September, 2023;
originally announced September 2023.
-
Redesigning Large-Scale Multimodal Transit Networks with Shared Autonomous Mobility Services
Authors:
Max T. M. Ng,
Hani S. Mahmassani,
Ömer Verbas,
Taner Cokyasar,
Roman Engelhardt
Abstract:
This study addresses a large-scale multimodal transit network design problem, with Shared Autonomous Mobility Services (SAMS) as both transit feeders and an origin-to-destination mode. The framework captures spatial demand and modal characteristics, considers intermodal transfers and express services, determines transit infrastructure investment and path flows, and generates transit routes. A syst…
▽ More
This study addresses a large-scale multimodal transit network design problem, with Shared Autonomous Mobility Services (SAMS) as both transit feeders and an origin-to-destination mode. The framework captures spatial demand and modal characteristics, considers intermodal transfers and express services, determines transit infrastructure investment and path flows, and generates transit routes. A system-optimal multimodal transit network is designed with minimum total door-to-door generalized costs of users and operators, satisfying transit origin-destination demand within a pre-set infrastructure budget. Firstly, the geography, demand, and modes in each zone are characterized with continuous approximation. The decisions of network link investment and multimodal path flows in zonal connection optimization are formulated as a minimum-cost multi-commodity network flow (MCNF) problem and solved efficiently with a mixed-integer linear programming (MILP) solver. Subsequently, the route generation problem is solved by expanding the MCNF formulation to minimize intramodal transfers. The model is illustrated through a set of experiments with the Chicago network comprised of 50 zones and seven modes, under three scenarios. The computational results present savings in traveler journey time and operator cost demonstrating the potential benefits of collaboration between multimodal transit systems and SAMS.
△ Less
Submitted 27 March, 2024; v1 submitted 29 July, 2023;
originally announced July 2023.
-
Quantized Low-Rank Multivariate Regression with Random Dithering
Authors:
Junren Chen,
Yueqi Wang,
Michael K. Ng
Abstract:
Low-rank multivariate regression (LRMR) is an important statistical learning model that combines highly correlated tasks as a multiresponse regression problem with low-rank priori on the coefficient matrix. In this paper, we study quantized LRMR, a practical setting where the responses and/or the covariates are discretized to finite precision. We focus on the estimation of the underlying coefficie…
▽ More
Low-rank multivariate regression (LRMR) is an important statistical learning model that combines highly correlated tasks as a multiresponse regression problem with low-rank priori on the coefficient matrix. In this paper, we study quantized LRMR, a practical setting where the responses and/or the covariates are discretized to finite precision. We focus on the estimation of the underlying coefficient matrix. To make consistent estimator that could achieve arbitrarily small error possible, we employ uniform quantization with random dithering, i.e., we add appropriate random noise to the data before quantization. Specifically, uniform dither and triangular dither are used for responses and covariates, respectively. Based on the quantized data, we propose the constrained Lasso and regularized Lasso estimators, and derive the non-asymptotic error bounds. With the aid of dithering, the estimators achieve minimax optimal rate, while quantization only slightly worsens the multiplicative factor in the error rate. Moreover, we extend our results to a low-rank regression model with matrix responses. We corroborate and demonstrate our theoretical results via simulations on synthetic data or image restoration.
△ Less
Submitted 6 October, 2023; v1 submitted 22 February, 2023;
originally announced February 2023.
-
Low-Rank Tensor Function Representation for Multi-Dimensional Data Recovery
Authors:
Yisi Luo,
Xile Zhao,
Zhemin Li,
Michael K. Ng,
Deyu Meng
Abstract:
Since higher-order tensors are naturally suitable for representing multi-dimensional data in real-world, e.g., color images and videos, low-rank tensor representation has become one of the emerging areas in machine learning and computer vision. However, classical low-rank tensor representations can only represent data on finite meshgrid due to their intrinsical discrete nature, which hinders their…
▽ More
Since higher-order tensors are naturally suitable for representing multi-dimensional data in real-world, e.g., color images and videos, low-rank tensor representation has become one of the emerging areas in machine learning and computer vision. However, classical low-rank tensor representations can only represent data on finite meshgrid due to their intrinsical discrete nature, which hinders their potential applicability in many scenarios beyond meshgrid. To break this barrier, we propose a low-rank tensor function representation (LRTFR), which can continuously represent data beyond meshgrid with infinite resolution. Specifically, the suggested tensor function, which maps an arbitrary coordinate to the corresponding value, can continuously represent data in an infinite real space. Parallel to discrete tensors, we develop two fundamental concepts for tensor functions, i.e., the tensor function rank and low-rank tensor function factorization. We theoretically justify that both low-rank and smooth regularizations are harmoniously unified in the LRTFR, which leads to high effectiveness and efficiency for data continuous representation. Extensive multi-dimensional data recovery applications arising from image processing (image inpainting and denoising), machine learning (hyperparameter optimization), and computer graphics (point cloud upsampling) substantiate the superiority and versatility of our method as compared with state-of-the-art methods. Especially, the experiments beyond the original meshgrid resolution (hyperparameter optimization) or even beyond meshgrid (point cloud upsampling) validate the favorable performances of our method for continuous representation.
△ Less
Submitted 30 November, 2022;
originally announced December 2022.
-
Phase Retrieval of Quaternion Signal via Wirtinger Flow
Authors:
Junren Chen,
Michael K. Ng
Abstract:
The main aim of this paper is to study quaternion phase retrieval (QPR), i.e., the recovery of quaternion signal from the magnitude of quaternion linear measurements. We show that all $d$-dimensional quaternion signals can be reconstructed up to a global right quaternion phase factor from $O(d)$ phaseless measurements. We also develop the scalable algorithm quaternion Wirtinger flow (QWF) for solv…
▽ More
The main aim of this paper is to study quaternion phase retrieval (QPR), i.e., the recovery of quaternion signal from the magnitude of quaternion linear measurements. We show that all $d$-dimensional quaternion signals can be reconstructed up to a global right quaternion phase factor from $O(d)$ phaseless measurements. We also develop the scalable algorithm quaternion Wirtinger flow (QWF) for solving QPR, and establish its linear convergence guarantee. Compared with the analysis of complex Wirtinger flow, a series of different treatments are employed to overcome the difficulties of the non-commutativity of quaternion multiplication. Moreover, we develop a variant of QWF that can effectively utilize a pure quaternion priori (e.g., for color images) by incorporating a quaternion phase factor estimate into QWF iterations. The estimate can be computed efficiently as it amounts to finding a singular vector of a $4\times 4$ real matrix. Motivated by the variants of Wirtinger flow in prior work, we further propose quaternion truncated Wirtinger flow (QTWF), quaternion truncated amplitude flow (QTAF) and their pure quaternion versions. Experimental results on synthetic data and color images are presented to validate our theoretical results. In particular, for pure quaternion signal recovery, our quaternion method often succeeds with measurements notably fewer than real methods based on monochromatic model or concatenation model.
△ Less
Submitted 23 July, 2023; v1 submitted 25 October, 2022;
originally announced October 2022.
-
Spherical Image Inpainting with Frame Transformation and Data-driven Prior Deep Networks
Authors:
Jianfei Li,
Chaoyan Huang,
Raymond Chan,
Han Feng,
Micheal Ng,
Tieyong Zeng
Abstract:
Spherical image processing has been widely applied in many important fields, such as omnidirectional vision for autonomous cars, global climate modelling, and medical imaging. It is non-trivial to extend an algorithm developed for flat images to the spherical ones. In this work, we focus on the challenging task of spherical image inpainting with deep learning-based regularizer. Instead of a naive…
▽ More
Spherical image processing has been widely applied in many important fields, such as omnidirectional vision for autonomous cars, global climate modelling, and medical imaging. It is non-trivial to extend an algorithm developed for flat images to the spherical ones. In this work, we focus on the challenging task of spherical image inpainting with deep learning-based regularizer. Instead of a naive application of existing models for planar images, we employ a fast directional spherical Haar framelet transform and develop a novel optimization framework based on a sparsity assumption of the framelet transform. Furthermore, by employing progressive encoder-decoder architecture, a new and better-performed deep CNN denoiser is carefully designed and works as an implicit regularizer. Finally, we use a plug-and-play method to handle the proposed optimization model, which can be implemented efficiently by training the CNN denoiser prior. Numerical experiments are conducted and show that the proposed algorithms can greatly recover damaged spherical images and achieve the best performance over purely using deep learning denoiser and plug-and-play model.
△ Less
Submitted 29 September, 2022;
originally announced September 2022.
-
A cross-corpus study on speech emotion recognition
Authors:
Rosanna Milner,
Md Asif Jalal,
Raymond W. M. Ng,
Thomas Hain
Abstract:
For speech emotion datasets, it has been difficult to acquire large quantities of reliable data and acted emotions may be over the top compared to less expressive emotions displayed in everyday life. Lately, larger datasets with natural emotions have been created. Instead of ignoring smaller, acted datasets, this study investigates whether information learnt from acted emotions is useful for detec…
▽ More
For speech emotion datasets, it has been difficult to acquire large quantities of reliable data and acted emotions may be over the top compared to less expressive emotions displayed in everyday life. Lately, larger datasets with natural emotions have been created. Instead of ignoring smaller, acted datasets, this study investigates whether information learnt from acted emotions is useful for detecting natural emotions. Cross-corpus research has mostly considered cross-lingual and even cross-age datasets, and difficulties arise from different methods of annotating emotions causing a drop in performance. To be consistent, four adult English datasets covering acted, elicited and natural emotions are considered. A state-of-the-art model is proposed to accurately investigate the degradation of performance. The system involves a bi-directional LSTM with an attention mechanism to classify emotions across datasets. Experiments study the effects of training models in a cross-corpus and multi-domain fashion and results show the transfer of information is not successful. Out-of-domain models, followed by adapting to the missing dataset, and domain adversarial training (DAT) are shown to be more suitable to generalising to emotions across datasets. This shows positive information transfer from acted datasets to those with more natural emotions and the benefits from training on different corpora.
△ Less
Submitted 5 July, 2022;
originally announced July 2022.
-
High Dimensional Statistical Estimation under Uniformly Dithered One-bit Quantization
Authors:
Junren Chen,
Cheng-Long Wang,
Michael K. Ng,
Di Wang
Abstract:
In this paper, we propose a uniformly dithered 1-bit quantization scheme for high-dimensional statistical estimation. The scheme contains truncation, dithering, and quantization as typical steps. As canonical examples, the quantization scheme is applied to the estimation problems of sparse covariance matrix estimation, sparse linear regression (i.e., compressed sensing), and matrix completion. We…
▽ More
In this paper, we propose a uniformly dithered 1-bit quantization scheme for high-dimensional statistical estimation. The scheme contains truncation, dithering, and quantization as typical steps. As canonical examples, the quantization scheme is applied to the estimation problems of sparse covariance matrix estimation, sparse linear regression (i.e., compressed sensing), and matrix completion. We study both sub-Gaussian and heavy-tailed regimes, where the underlying distribution of heavy-tailed data is assumed to have bounded moments of some order. We propose new estimators based on 1-bit quantized data. In sub-Gaussian regime, our estimators achieve near minimax rates, indicating that our quantization scheme costs very little. In heavy-tailed regime, while the rates of our estimators become essentially slower, these results are either the first ones in an 1-bit quantized and heavy-tailed setting, or already improve on existing comparable results from some respect. Under the observations in our setting, the rates are almost tight in compressed sensing and matrix completion. Our 1-bit compressed sensing results feature general sensing vector that is sub-Gaussian or even heavy-tailed. We also first investigate a novel setting where both the covariate and response are quantized. In addition, our approach to 1-bit matrix completion does not rely on likelihood and represent the first method robust to pre-quantization noise with unknown distribution. Experimental results on synthetic data are presented to support our theoretical analysis.
△ Less
Submitted 20 January, 2023; v1 submitted 26 February, 2022;
originally announced February 2022.
-
Morphological feature visualization of Alzheimer's disease via Multidirectional Perception GAN
Authors:
Wen Yu,
Baiying Lei,
Yanyan Shen,
Shuqiang Wang,
Yong Liu,
Zhiguang Feng,
Yong Hu,
Michael K. Ng
Abstract:
The diagnosis of early stages of Alzheimer's disease (AD) is essential for timely treatment to slow further deterioration. Visualizing the morphological features for the early stages of AD is of great clinical value. In this work, a novel Multidirectional Perception Generative Adversarial Network (MP-GAN) is proposed to visualize the morphological features indicating the severity of AD for patient…
▽ More
The diagnosis of early stages of Alzheimer's disease (AD) is essential for timely treatment to slow further deterioration. Visualizing the morphological features for the early stages of AD is of great clinical value. In this work, a novel Multidirectional Perception Generative Adversarial Network (MP-GAN) is proposed to visualize the morphological features indicating the severity of AD for patients of different stages. Specifically, by introducing a novel multidirectional map** mechanism into the model, the proposed MP-GAN can capture the salient global features efficiently. Thus, by utilizing the class-discriminative map from the generator, the proposed model can clearly delineate the subtle lesions via MR image transformations between the source domain and the pre-defined target domain. Besides, by integrating the adversarial loss, classification loss, cycle consistency loss and \emph{L}1 penalty, a single generator in MP-GAN can learn the class-discriminative maps for multiple-classes. Extensive experimental results on Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset demonstrate that MP-GAN achieves superior performance compared with the existing methods. The lesions visualized by MP-GAN are also consistent with what clinicians observe.
△ Less
Submitted 24 November, 2021;
originally announced November 2021.
-
FastHyMix: Fast and Parameter-free Hyperspectral Image Mixed Noise Removal
Authors:
Lina Zhuang,
Michael K. Ng
Abstract:
Hyperspectral imaging with high spectral resolution plays an important role in finding objects, identifying materials, or detecting processes. The decrease of the widths of spectral bands leads to a decrease in the signal-to-noise ratio (SNR) of measurements. The decreased SNR reduces the reliability of measured features or information extracted from HSIs. Furthermore, the image degradations linke…
▽ More
Hyperspectral imaging with high spectral resolution plays an important role in finding objects, identifying materials, or detecting processes. The decrease of the widths of spectral bands leads to a decrease in the signal-to-noise ratio (SNR) of measurements. The decreased SNR reduces the reliability of measured features or information extracted from HSIs. Furthermore, the image degradations linked with various mechanisms also result in different types of noise, such as Gaussian noise, impulse noise, deadlines, and stripes. This paper introduces a fast and parameter-free hyperspectral image mixed noise removal method (termed FastHyMix), which characterizes the complex distribution of mixed noise by using a Gaussian mixture model and exploits two main characteristics of hyperspectral data, namely low-rankness in the spectral domain and high correlation in the spatial domain. The Gaussian mixture model enables us to make a good estimation of Gaussian noise intensity and the location of sparse noise. The proposed method takes advantage of the low-rankness using subspace representation and the spatial correlation of HSIs by adding a powerful deep image prior, which is extracted from a neural denoising network. An exhaustive array of experiments and comparisons with state-of-the-art denoisers were carried out. The experimental results show significant improvement in both synthetic and real datasets. A MATLAB demo of this work will be available at https://github.com/LinaZhuang for the sake of reproducibility.
△ Less
Submitted 18 September, 2021;
originally announced September 2021.
-
Signal Reconstruction from Phase-only Measurements: Uniqueness Condition, Minimal Measurement Number and Beyond
Authors:
Junren Chen,
Michael K. Ng
Abstract:
This paper studies the phase-only reconstruction problem of recovering a complex-valued signal $\textbf{x}$ in $\mathbb{C}^d$ from the phase of $\textbf{Ax}$ where $\textbf{A}$ is a given measurement matrix in $\mathbb{C}^{m\times d}$. The reconstruction, if possible, should be up to a positive scaling factor. By using the rank of discriminant matrices, uniqueness conditions are derived to charact…
▽ More
This paper studies the phase-only reconstruction problem of recovering a complex-valued signal $\textbf{x}$ in $\mathbb{C}^d$ from the phase of $\textbf{Ax}$ where $\textbf{A}$ is a given measurement matrix in $\mathbb{C}^{m\times d}$. The reconstruction, if possible, should be up to a positive scaling factor. By using the rank of discriminant matrices, uniqueness conditions are derived to characterize whether the underlying signal can be uniquely reconstructed. We are also interested in the problem of minimal measurement number. We show that at least $2d$ but no more than $4d-2$ measurements are needed for the reconstruction of all $\textbf{x}\in\mathbb{C}^d$, whereas the minimal measurement number is exactly $2d-1$ if we pursue the recovery of almost all signals. Moreover, when adapted to the phase-only reconstruction of $\textbf{x}\in\mathbb{R}^d$, our uniqueness conditions are more practical and general than existing ones. Our theoretical results can be straightforwardly extended to affine phase-only reconstruction where the phase of $\textbf{Ax}+\textbf{b}$ is observed for some $\textbf{b}\in\mathbb{C}^m$.
△ Less
Submitted 30 January, 2023; v1 submitted 8 September, 2021;
originally announced September 2021.
-
Self-Supervised Nonlinear Transform-Based Tensor Nuclear Norm for Multi-Dimensional Image Recovery
Authors:
Yi-Si Luo,
Xi-Le Zhao,
Tai-Xiang Jiang,
Yi Chang,
Michael K. Ng,
Chao Li
Abstract:
In this paper, we study multi-dimensional image recovery. Recently, transform-based tensor nuclear norm minimization methods are considered to capture low-rank tensor structures to recover third-order tensors in multi-dimensional image processing applications. The main characteristic of such methods is to perform the linear transform along the third mode of third-order tensors, and then compute te…
▽ More
In this paper, we study multi-dimensional image recovery. Recently, transform-based tensor nuclear norm minimization methods are considered to capture low-rank tensor structures to recover third-order tensors in multi-dimensional image processing applications. The main characteristic of such methods is to perform the linear transform along the third mode of third-order tensors, and then compute tensor nuclear norm minimization on the transformed tensor so that the underlying low-rank tensors can be recovered. The main aim of this paper is to propose a nonlinear multilayer neural network to learn a nonlinear transform via the observed tensor data under self-supervision. The proposed network makes use of low-rank representation of transformed tensors and data-fitting between the observed tensor and the reconstructed tensor to construct the nonlinear transformation. Extensive experimental results on tensor completion, background subtraction, robust tensor completion, and snapshot compressive imaging are presented to demonstrate that the performance of the proposed method is better than that of state-of-the-art methods.
△ Less
Submitted 29 May, 2021;
originally announced May 2021.
-
Estimating Uncertainty in Neural Networks for Cardiac MRI Segmentation: A Benchmark Study
Authors:
Matthew Ng,
Fumin Guo,
Labonny Biswas,
Steffen E. Petersen,
Stefan K. Piechnik,
Stefan Neubauer,
Graham Wright
Abstract:
Objective: Convolutional neural networks (CNNs) have demonstrated promise in automated cardiac magnetic resonance image segmentation. However, when using CNNs in a large real-world dataset, it is important to quantify segmentation uncertainty and identify segmentations which could be problematic. In this work, we performed a systematic study of Bayesian and non-Bayesian methods for estimating unce…
▽ More
Objective: Convolutional neural networks (CNNs) have demonstrated promise in automated cardiac magnetic resonance image segmentation. However, when using CNNs in a large real-world dataset, it is important to quantify segmentation uncertainty and identify segmentations which could be problematic. In this work, we performed a systematic study of Bayesian and non-Bayesian methods for estimating uncertainty in segmentation neural networks.
Methods: We evaluated Bayes by Backprop, Monte Carlo Dropout, Deep Ensembles, and Stochastic Segmentation Networks in terms of segmentation accuracy, probability calibration, uncertainty on out-of-distribution images, and segmentation quality control.
Results: We observed that Deep Ensembles outperformed the other methods except for images with heavy noise and blurring distortions. We showed that Bayes by Backprop is more robust to noise distortions while Stochastic Segmentation Networks are more resistant to blurring distortions. For segmentation quality control, we showed that segmentation uncertainty is correlated with segmentation accuracy for all the methods. With the incorporation of uncertainty estimates, we were able to reduce the percentage of poor segmentation to 5% by flagging 31--48% of the most uncertain segmentations for manual review, substantially lower than random review without using neural network uncertainty (reviewing 75--78% of all images).
Conclusion: This work provides a comprehensive evaluation of uncertainty estimation methods and showed that Deep Ensembles outperformed other methods in most cases.
Significance: Neural network uncertainty measures can help identify potentially inaccurate segmentations and alert users for manual review.
△ Less
Submitted 30 December, 2022; v1 submitted 31 December, 2020;
originally announced December 2020.
-
Tensorizing GAN with High-Order Pooling for Alzheimer's Disease Assessment
Authors:
Wen Yu,
Baiying Lei,
Michael K. Ng,
Albert C. Cheung,
Yanyan Shen,
Shuqiang Wang
Abstract:
It is of great significance to apply deep learning for the early diagnosis of Alzheimer's Disease (AD). In this work, a novel tensorizing GAN with high-order pooling is proposed to assess Mild Cognitive Impairment (MCI) and AD. By tensorizing a three-player cooperative game based framework, the proposed model can benefit from the structural information of the brain. By incorporating the high-order…
▽ More
It is of great significance to apply deep learning for the early diagnosis of Alzheimer's Disease (AD). In this work, a novel tensorizing GAN with high-order pooling is proposed to assess Mild Cognitive Impairment (MCI) and AD. By tensorizing a three-player cooperative game based framework, the proposed model can benefit from the structural information of the brain. By incorporating the high-order pooling scheme into the classifier, the proposed model can make full use of the second-order statistics of the holistic Magnetic Resonance Imaging (MRI) images. To the best of our knowledge, the proposed Tensor-train, High-pooling and Semi-supervised learning based GAN (THS-GAN) is the first work to deal with classification on MRI images for AD diagnosis. Extensive experimental results on Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset are reported to demonstrate that the proposed THS-GAN achieves superior performance compared with existing methods, and to show that both tensor-train and high-order pooling can enhance classification performance. The visualization of generated samples also shows that the proposed model can generate plausible samples for semi-supervised learning purpose.
△ Less
Submitted 3 August, 2020;
originally announced August 2020.
-
Nonnegative Low Rank Tensor Approximation and its Application to Multi-dimensional Images
Authors:
Tai-Xiang Jiang,
Michael K. Ng,
Junjun Pan,
Guang**g Song
Abstract:
The main aim of this paper is to develop a new algorithm for computing nonnegative low rank tensor approximation for nonnegative tensors that arise in many multi-dimensional imaging applications. Nonnegativity is one of the important property as each pixel value refers to nonzero light intensity in image data acquisition. Our approach is different from classical nonnegative tensor factorization (N…
▽ More
The main aim of this paper is to develop a new algorithm for computing nonnegative low rank tensor approximation for nonnegative tensors that arise in many multi-dimensional imaging applications. Nonnegativity is one of the important property as each pixel value refers to nonzero light intensity in image data acquisition. Our approach is different from classical nonnegative tensor factorization (NTF) which requires each factorized matrix and/or tensor to be nonnegative. In this paper, we determine a nonnegative low Tucker rank tensor to approximate a given nonnegative tensor. We propose an alternating projections algorithm for computing such nonnegative low rank tensor approximation, which is referred to as NLRT. The convergence of the proposed manifold projection method is established. Experimental results for synthetic data and multi-dimensional images are presented to demonstrate the performance of NLRT is better than state-of-the-art NTF methods.
△ Less
Submitted 26 September, 2021; v1 submitted 28 July, 2020;
originally announced July 2020.
-
Framelet Representation of Tensor Nuclear Norm for Third-Order Tensor Completion
Authors:
Tai-Xiang Jiang,
Michael K. Ng,
Xi-Le Zhao,
Ting-Zhu Huang
Abstract:
The main aim of this paper is to develop a framelet representation of the tensor nuclear norm for third-order tensor completion. In the literature, the tensor nuclear norm can be computed by using tensor singular value decomposition based on the discrete Fourier transform matrix, and tensor completion can be performed by the minimization of the tensor nuclear norm which is the relaxation of the su…
▽ More
The main aim of this paper is to develop a framelet representation of the tensor nuclear norm for third-order tensor completion. In the literature, the tensor nuclear norm can be computed by using tensor singular value decomposition based on the discrete Fourier transform matrix, and tensor completion can be performed by the minimization of the tensor nuclear norm which is the relaxation of the sum of matrix ranks from all Fourier transformed matrix frontal slices. These Fourier transformed matrix frontal slices are obtained by applying the discrete Fourier transform on the tubes of the original tensor. In this paper, we propose to employ the framelet representation of each tube so that a framelet transformed tensor can be constructed. Because of framelet basis redundancy, the representation of each tube is sparsely represented. When the matrix slices of the original tensor are highly correlated, we expect the corresponding sum of matrix ranks from all framelet transformed matrix frontal slices would be small, and the resulting tensor completion can be performed much better. The proposed minimization model is convex and global minimizers can be obtained. Numerical results on several types of multi-dimensional data (videos, multispectral images, and magnetic resonance imaging data) have tested and shown that the proposed method outperformed the other testing methods.
△ Less
Submitted 25 January, 2020; v1 submitted 16 September, 2019;
originally announced September 2019.
-
Deep Plug-and-play Prior for Low-rank Tensor Completion
Authors:
Xi-Le Zhao,
Wen-Hao Xu,
Tai-Xiang Jiang,
Yao Wang,
Michael Ng
Abstract:
Multi-dimensional images, such as color images and multi-spectral images, are highly correlated and contain abundant spatial and spectral information. However, real-world multi-dimensional images are usually corrupted by missing entries. By integrating deterministic low-rankness prior to the data-driven deep prior, we suggest a novel regularized tensor completion model for multi-dimensional image…
▽ More
Multi-dimensional images, such as color images and multi-spectral images, are highly correlated and contain abundant spatial and spectral information. However, real-world multi-dimensional images are usually corrupted by missing entries. By integrating deterministic low-rankness prior to the data-driven deep prior, we suggest a novel regularized tensor completion model for multi-dimensional image completion. In the objective function, we adopt the newly emerged tensor nuclear norm (TNN) to characterize the global low-rankness prior of the multi-dimensional images. We also formulate an implicit regularizer by plugging into a denoising neural network (termed as deep denoiser), which is convinced to express the deep image prior learned from a large number of natural images. The resulting model can be solved by the alternating directional method of multipliers algorithm under the plug-and-play (PnP) framework. Experimental results on color images, videos, and multi-spectral images demonstrate that the proposed method can recover both the global structure and fine details very well and achieve superior performance over competing methods in terms of quality metrics and visual effects.
△ Less
Submitted 4 May, 2020; v1 submitted 11 May, 2019;
originally announced May 2019.
-
Graph Spectral Image Processing
Authors:
Gene Cheung,
Enrico Magli,
Yuichi Tanaka,
Michael Ng
Abstract:
Recent advent of graph signal processing (GSP) has spurred intensive studies of signals that live naturally on irregular data kernels described by graphs (e.g., social networks, wireless sensor networks). Though a digital image contains pixels that reside on a regularly sampled 2D grid, if one can design an appropriate underlying graph connecting pixels with weights that reflect the image structur…
▽ More
Recent advent of graph signal processing (GSP) has spurred intensive studies of signals that live naturally on irregular data kernels described by graphs (e.g., social networks, wireless sensor networks). Though a digital image contains pixels that reside on a regularly sampled 2D grid, if one can design an appropriate underlying graph connecting pixels with weights that reflect the image structure, then one can interpret the image (or image patch) as a signal on a graph, and apply GSP tools for processing and analysis of the signal in graph spectral domain. In this article, we overview recent graph spectral techniques in GSP specifically for image / video processing. The topics covered include image compression, image restoration, image filtering and image segmentation.
△ Less
Submitted 16 January, 2018; v1 submitted 15 January, 2018;
originally announced January 2018.
