-
MIMO Capacity Analysis and Channel Estimation for Electromagnetic Information Theory
Authors:
Jieao Zhu,
Vincent Y. F. Tan,
Linglong Dai
Abstract:
Electromagnetic information theory (EIT) is an interdisciplinary subject that serves to integrate deterministic electromagnetic theory with stochastic Shannon's information theory. Existing EIT analysis operates in the continuous space domain, which is not aligned with the practical algorithms working in the discrete space domain. This mismatch leads to a significant difficulty in application of E…
▽ More
Electromagnetic information theory (EIT) is an interdisciplinary subject that serves to integrate deterministic electromagnetic theory with stochastic Shannon's information theory. Existing EIT analysis operates in the continuous space domain, which is not aligned with the practical algorithms working in the discrete space domain. This mismatch leads to a significant difficulty in application of EIT methodologies to practical discrete space systems, which is called as the discrete-continuous gap in this paper. To bridge this gap, we establish the discrete-continuous correspondence with a prolate spheroidal wave function (PSWF)-based ergodic capacity analysis framework. Specifically, we state and prove some discrete-continuous correspondence lemmas to establish a firm theoretical connection between discrete information-theoretic quantities to their continuous counterparts. With these lemmas, we apply the PSWF ergodic capacity bound to advanced MIMO architectures such as continuous-aperture MIMO (CAP-MIMO) and extremely large-scale MIMO (XL-MIMO). From this PSWF capacity bound, we discover the capacity saturation phenomenon both theoretically and empirically. Although the growth of MIMO performance is fundamentally limited in this EIT-based analysis framework, we reveal new opportunities in MIMO channel estimation by exploiting the EIT knowledge about the channel. Inspired by the PSWF capacity bound, we utilize continuous PSWFs to improve the pilot design of discrete MIMO channel estimators, which is called as the PSWF channel estimator (PSWF-CE). Simulation results demonstrate improved performances of the proposed PSWF-CE, compared to traditional minimum mean squared error (MMSE) and compressed sensing-based estimators.
△ Less
Submitted 7 June, 2024;
originally announced June 2024.
-
Learning Regularized Monotone Graphon Mean-Field Games
Authors:
Fengzhuo Zhang,
Vincent Y. F. Tan,
Zhaoran Wang,
Zhuoran Yang
Abstract:
This paper studies two fundamental problems in regularized Graphon Mean-Field Games (GMFGs). First, we establish the existence of a Nash Equilibrium (NE) of any $λ$-regularized GMFG (for $λ\geq 0$). This result relies on weaker conditions than those in previous works for analyzing both unregularized GMFGs ($λ=0$) and $λ$-regularized MFGs, which are special cases of GMFGs. Second, we propose provab…
▽ More
This paper studies two fundamental problems in regularized Graphon Mean-Field Games (GMFGs). First, we establish the existence of a Nash Equilibrium (NE) of any $λ$-regularized GMFG (for $λ\geq 0$). This result relies on weaker conditions than those in previous works for analyzing both unregularized GMFGs ($λ=0$) and $λ$-regularized MFGs, which are special cases of GMFGs. Second, we propose provably efficient algorithms to learn the NE in weakly monotone GMFGs, motivated by Lasry and Lions [2007]. Previous literature either only analyzed continuous-time algorithms or required extra conditions to analyze discrete-time algorithms. In contrast, we design a discrete-time algorithm and derive its convergence rate solely under weakly monotone conditions. Furthermore, we develop and analyze the action-value function estimation procedure during the online learning process, which is absent from algorithms for monotone GMFGs. This serves as a sub-module in our optimization algorithm. The efficiency of the designed algorithm is corroborated by empirical evaluations.
△ Less
Submitted 12 October, 2023;
originally announced October 2023.
-
Deep Unrolling for Nonconvex Robust Principal Component Analysis
Authors:
Elizabeth Z. C. Tan,
Caroline Chaux,
Emmanuel Soubies,
Vincent Y. F. Tan
Abstract:
We design algorithms for Robust Principal Component Analysis (RPCA) which consists in decomposing a matrix into the sum of a low rank matrix and a sparse matrix. We propose a deep unrolled algorithm based on an accelerated alternating projection algorithm which aims to solve RPCA in its nonconvex form. The proposed procedure combines benefits of deep neural networks and the interpretability of the…
▽ More
We design algorithms for Robust Principal Component Analysis (RPCA) which consists in decomposing a matrix into the sum of a low rank matrix and a sparse matrix. We propose a deep unrolled algorithm based on an accelerated alternating projection algorithm which aims to solve RPCA in its nonconvex form. The proposed procedure combines benefits of deep neural networks and the interpretability of the original algorithm and it automatically learns hyperparameters. We demonstrate the unrolled algorithm's effectiveness on synthetic datasets and also on a face modeling problem, where it leads to both better numerical and visual performances.
△ Less
Submitted 11 July, 2023;
originally announced July 2023.
-
Asymptotic Nash Equilibrium for the $M$-ary Sequential Adversarial Hypothesis Testing Game
Authors:
Jiachun Pan,
Yonglong Li,
Vincent Y. F. Tan
Abstract:
In this paper, we consider a novel $M$-ary sequential hypothesis testing problem in which an adversary is present and perturbs the distributions of the samples before the decision maker observes them. This problem is formulated as a sequential adversarial hypothesis testing game played between the decision maker and the adversary. This game is a zero-sum and strategic one. We assume the adversary…
▽ More
In this paper, we consider a novel $M$-ary sequential hypothesis testing problem in which an adversary is present and perturbs the distributions of the samples before the decision maker observes them. This problem is formulated as a sequential adversarial hypothesis testing game played between the decision maker and the adversary. This game is a zero-sum and strategic one. We assume the adversary is active under \emph{all} hypotheses and knows the underlying distribution of observed samples. We adopt this framework as it is the worst-case scenario from the perspective of the decision maker. The goal of the decision maker is to minimize the expectation of the stop** time to ensure that the test is as efficient as possible; the adversary's goal is, instead, to maximize the stop** time. We derive a pair of strategies under which the asymptotic Nash equilibrium of the game is attained. We also consider the case in which the adversary is not aware of the underlying hypothesis and hence is constrained to apply the same strategy regardless of which hypothesis is in effect. Numerical results corroborate our theoretical findings.
△ Less
Submitted 20 June, 2022;
originally announced June 2022.
-
Towards Adversarially Robust Deep Image Denoising
Authors:
Hanshu Yan,
**gfeng Zhang,
Jiashi Feng,
Masashi Sugiyama,
Vincent Y. F. Tan
Abstract:
This work systematically investigates the adversarial robustness of deep image denoisers (DIDs), i.e, how well DIDs can recover the ground truth from noisy observations degraded by adversarial perturbations. Firstly, to evaluate DIDs' robustness, we propose a novel adversarial attack, namely Observation-based Zero-mean Attack ({\sc ObsAtk}), to craft adversarial zero-mean perturbations on given no…
▽ More
This work systematically investigates the adversarial robustness of deep image denoisers (DIDs), i.e, how well DIDs can recover the ground truth from noisy observations degraded by adversarial perturbations. Firstly, to evaluate DIDs' robustness, we propose a novel adversarial attack, namely Observation-based Zero-mean Attack ({\sc ObsAtk}), to craft adversarial zero-mean perturbations on given noisy images. We find that existing DIDs are vulnerable to the adversarial noise generated by {\sc ObsAtk}. Secondly, to robustify DIDs, we propose an adversarial training strategy, hybrid adversarial training ({\sc HAT}), that jointly trains DIDs with adversarial and non-adversarial noisy data to ensure that the reconstruction quality is high and the denoisers around non-adversarial data are locally smooth. The resultant DIDs can effectively remove various types of synthetic and adversarial noise. We also uncover that the robustness of DIDs benefits their generalization capability on unseen real-world noise. Indeed, {\sc HAT}-trained DIDs can recover high-quality clean images from real-world noise even without training on real noisy data. Extensive experiments on benchmark datasets, including Set68, PolyU, and SIDD, corroborate the effectiveness of {\sc ObsAtk} and {\sc HAT}.
△ Less
Submitted 13 January, 2022; v1 submitted 12 January, 2022;
originally announced January 2022.
-
Exact Recovery in the General Hypergraph Stochastic Block Model
Authors:
Qiaosheng Zhang,
Vincent Y. F. Tan
Abstract:
This paper investigates fundamental limits of exact recovery in the general d-uniform hypergraph stochastic block model (d-HSBM), wherein n nodes are partitioned into k disjoint communities with relative sizes (p1,..., pk). Each subset of nodes with cardinality d is generated independently as an order-d hyperedge with a certain probability that depends on the ground-truth communities that the d no…
▽ More
This paper investigates fundamental limits of exact recovery in the general d-uniform hypergraph stochastic block model (d-HSBM), wherein n nodes are partitioned into k disjoint communities with relative sizes (p1,..., pk). Each subset of nodes with cardinality d is generated independently as an order-d hyperedge with a certain probability that depends on the ground-truth communities that the d nodes belong to. The goal is to exactly recover the k hidden communities based on the observed hypergraph. We show that there exists a sharp threshold such that exact recovery is achievable above the threshold and impossible below the threshold (apart from a small regime of parameters that will be specified precisely). This threshold is represented in terms of a quantity which we term as the generalized Chernoff-Hellinger divergence between communities. Our result for this general model recovers prior results for the standard SBM and d-HSBM with two symmetric communities as special cases. En route to proving our achievability results, we develop a polynomial-time two-stage algorithm that meets the threshold. The first stage adopts a certain hypergraph spectral clustering method to obtain a coarse estimate of communities, and the second stage refines each node individually via local refinement steps to ensure exact recovery.
△ Less
Submitted 9 September, 2022; v1 submitted 10 May, 2021;
originally announced May 2021.
-
Adversarially-Trained Nonnegative Matrix Factorization
Authors:
Ting Cai,
Vincent Y. F. Tan,
Cédric Févotte
Abstract:
We consider an adversarially-trained version of the nonnegative matrix factorization, a popular latent dimensionality reduction technique. In our formulation, an attacker adds an arbitrary matrix of bounded norm to the given data matrix. We design efficient algorithms inspired by adversarial training to optimize for dictionary and coefficient matrices with enhanced generalization abilities. Extens…
▽ More
We consider an adversarially-trained version of the nonnegative matrix factorization, a popular latent dimensionality reduction technique. In our formulation, an attacker adds an arbitrary matrix of bounded norm to the given data matrix. We design efficient algorithms inspired by adversarial training to optimize for dictionary and coefficient matrices with enhanced generalization abilities. Extensive simulations on synthetic and benchmark datasets demonstrate the superior predictive performance on matrix completion tasks of our proposed method compared to state-of-the-art competitors, including other variants of adversarial nonnegative matrix factorization.
△ Less
Submitted 22 June, 2021; v1 submitted 10 April, 2021;
originally announced April 2021.
-
On the Error Exponent of Approximate Sufficient Statistics for M-ary Hypothesis Testing
Authors:
Jiachun Pan,
Yonglong Li,
Vincent Y. F. Tan,
Yonina C. Eldar
Abstract:
Consider the problem of detecting one of M i.i.d. Gaussian signals corrupted in white Gaussian noise. Conventionally, matched filters are used for detection. We first show that the outputs of the matched filter form a set of asymptotically optimal sufficient statistics in the sense of maximizing the error exponent of detecting the true signal. In practice, however, M may be large which motivates t…
▽ More
Consider the problem of detecting one of M i.i.d. Gaussian signals corrupted in white Gaussian noise. Conventionally, matched filters are used for detection. We first show that the outputs of the matched filter form a set of asymptotically optimal sufficient statistics in the sense of maximizing the error exponent of detecting the true signal. In practice, however, M may be large which motivates the design and analysis of a reduced set of N statistics which we term approximate sufficient statistics. Our construction of these statistics is based on a small set of filters that project the outputs of the matched filters onto a lower-dimensional vector using a sensing matrix. We consider a sequence of sensing matrices that has the desiderata of row orthonormality and low coherence. We analyze the performance of the resulting maximum likelihood (ML) detector, which leads to an achievable bound on the error exponent based on the approximate sufficient statistics; this bound recovers the original error exponent when N = M. We compare this to a bound that we obtain by analyzing a modified form of the Reduced Dimensionality Detector (RDD) proposed by Xie, Eldar, and Goldsmith [IEEE Trans. on Inform. Th., 59(6):3858-3874, 2013]. We show that by setting the sensing matrices to be column-normalized group Hadamard matrices, the exponents derived are ensemble-tight, i.e., our analysis is tight on the exponential scale given the sensing matrices and the decoding rule. Finally, we derive some properties of the exponents, showing, in particular, that they increase linearly in the compression ratio N/M.
△ Less
Submitted 17 August, 2020;
originally announced August 2020.
-
Positive Semidefinite Matrix Factorization: A Connection with Phase Retrieval and Affine Rank Minimization
Authors:
Dana Lahat,
Yanbin Lang,
Vincent Y. F. Tan,
Cédric Févotte
Abstract:
Positive semidefinite matrix factorization (PSDMF) expresses each entry of a nonnegative matrix as the inner product of two positive semidefinite (psd) matrices. When all these psd matrices are constrained to be diagonal, this model is equivalent to nonnegative matrix factorization. Applications include combinatorial optimization, quantum-based statistical models, and recommender systems, among ot…
▽ More
Positive semidefinite matrix factorization (PSDMF) expresses each entry of a nonnegative matrix as the inner product of two positive semidefinite (psd) matrices. When all these psd matrices are constrained to be diagonal, this model is equivalent to nonnegative matrix factorization. Applications include combinatorial optimization, quantum-based statistical models, and recommender systems, among others. However, despite the increasing interest in PSDMF, only a few PSDMF algorithms were proposed in the literature. In this work, we provide a collection of tools for PSDMF, by showing that PSDMF algorithms can be designed based on phase retrieval (PR) and affine rank minimization (ARM) algorithms. This procedure allows a shortcut in designing new PSDMF algorithms, as it allows to leverage some of the useful numerical properties of existing PR and ARM methods to the PSDMF framework. Motivated by this idea, we introduce a new family of PSDMF algorithms based on iterative hard thresholding (IHT). This family subsumes previously-proposed projected gradient PSDMF methods. We show that there is high variability among PSDMF optimization problems that makes it beneficial to try a number of methods based on different principles to tackle difficult problems. In certain cases, our proposed methods are the only algorithms able to find a solution. In certain other cases, they converge faster. Our results support our claim that the PSDMF framework can inherit desired numerical properties from PR and ARM algorithms, leading to more efficient PSDMF algorithms, and motivate further study of the links between these models.
△ Less
Submitted 2 April, 2021; v1 submitted 24 July, 2020;
originally announced July 2020.
-
MC2G: An Efficient Algorithm for Matrix Completion with Social and Item Similarity Graphs
Authors:
Qiaosheng Zhang,
Geewon Suh,
Changho Suh,
Vincent Y. F. Tan
Abstract:
In this paper, we design and analyze MC2G (Matrix Completion with 2 Graphs), an algorithm that performs matrix completion in the presence of social and item similarity graphs. MC2G runs in quasilinear time and is parameter free. It is based on spectral clustering and local refinement steps. The expected number of sampled entries required for MC2G to succeed (i.e., recover the clusters in the graph…
▽ More
In this paper, we design and analyze MC2G (Matrix Completion with 2 Graphs), an algorithm that performs matrix completion in the presence of social and item similarity graphs. MC2G runs in quasilinear time and is parameter free. It is based on spectral clustering and local refinement steps. The expected number of sampled entries required for MC2G to succeed (i.e., recover the clusters in the graphs and complete the matrix) matches an information-theoretic lower bound up to a constant factor for a wide range of parameters. We show via extensive experiments on both synthetic and real datasets that MC2G outperforms other state-of-the-art matrix completion algorithms that leverage graph side information.
△ Less
Submitted 7 June, 2021; v1 submitted 8 June, 2020;
originally announced June 2020.
-
RAIN: A Simple Approach for Robust and Accurate Image Classification Networks
Authors:
Jiawei Du,
Hanshu Yan,
Vincent Y. F. Tan,
Joey Tianyi Zhou,
Rick Siow Mong Goh,
Jiashi Feng
Abstract:
It has been shown that the majority of existing adversarial defense methods achieve robustness at the cost of sacrificing prediction accuracy. The undesirable severe drop in accuracy adversely affects the reliability of machine learning algorithms and prohibits their deployment in realistic applications. This paper aims to address this dilemma by proposing a novel preprocessing framework, which we…
▽ More
It has been shown that the majority of existing adversarial defense methods achieve robustness at the cost of sacrificing prediction accuracy. The undesirable severe drop in accuracy adversely affects the reliability of machine learning algorithms and prohibits their deployment in realistic applications. This paper aims to address this dilemma by proposing a novel preprocessing framework, which we term Robust and Accurate Image classificatioN(RAIN), to improve the robustness of given CNN classifiers and, at the same time, preserve their high prediction accuracies. RAIN introduces a new randomization-enhancement scheme. It applies randomization over inputs to break the ties between the model forward prediction path and the backward gradient path, thus improving the model robustness. However, similar to existing preprocessing-based methods, the randomized process will degrade the prediction accuracy. To understand why this is the case, we compare the difference between original and processed images, and find it is the loss of high-frequency components in the input image that leads to accuracy drop of the classifier. Based on this finding, RAIN enhances the input's high-frequency details to retain the CNN's high prediction accuracy. Concretely, RAIN consists of two novel randomization modules: randomized small circular shift (RdmSCS) and randomized down-upsampling (RdmDU). The RdmDU module randomly downsamples the input image, and then the RdmSCS module circularly shifts the input image along a randomly chosen direction by a small but random number of pixels. Finally, the RdmDU module performs upsampling with a detail-enhancement model, such as deep super-resolution networks. We conduct extensive experiments on the STL10 and ImageNet datasets to verify the effectiveness of RAIN against various types of adversarial attacks.
△ Less
Submitted 4 November, 2020; v1 submitted 23 April, 2020;
originally announced April 2020.
-
Community Detection and Matrix Completion with Social and Item Similarity Graphs
Authors:
Qiaosheng Zhang,
Vincent Y. F. Tan,
Changho Suh
Abstract:
We consider the problem of recovering a binary rating matrix as well as clusters of users and items based on a partially observed matrix together with side-information in the form of social and item similarity graphs. These two graphs are both generated according to the celebrated stochastic block model (SBM). We develop lower and upper bounds on sample complexity that match for various scenarios.…
▽ More
We consider the problem of recovering a binary rating matrix as well as clusters of users and items based on a partially observed matrix together with side-information in the form of social and item similarity graphs. These two graphs are both generated according to the celebrated stochastic block model (SBM). We develop lower and upper bounds on sample complexity that match for various scenarios. Our information-theoretic results quantify the benefits of the availability of the social and item similarity graphs. Further analysis reveals that under certain scenarios, the social and item similarity graphs produce an interesting synergistic effect. This means that observing two graphs is strictly better than observing just one in terms of reducing the sample complexity.
△ Less
Submitted 13 January, 2021; v1 submitted 6 December, 2019;
originally announced December 2019.
-
Economy Statistical Recurrent Units For Inferring Nonlinear Granger Causality
Authors:
Saurabh Khanna,
Vincent Y. F. Tan
Abstract:
Granger causality is a widely-used criterion for analyzing interactions in large-scale networks. As most physical interactions are inherently nonlinear, we consider the problem of inferring the existence of pairwise Granger causality between nonlinearly interacting stochastic processes from their time series measurements. Our proposed approach relies on modeling the embedded nonlinearities in the…
▽ More
Granger causality is a widely-used criterion for analyzing interactions in large-scale networks. As most physical interactions are inherently nonlinear, we consider the problem of inferring the existence of pairwise Granger causality between nonlinearly interacting stochastic processes from their time series measurements. Our proposed approach relies on modeling the embedded nonlinearities in the measurements using a component-wise time series prediction model based on Statistical Recurrent Units (SRUs). We make a case that the network topology of Granger causal relations is directly inferrable from a structured sparse estimate of the internal parameters of the SRU networks trained to predict the processes$'$ time series measurements. We propose a variant of SRU, called economy-SRU, which, by design has considerably fewer trainable parameters, and therefore less prone to overfitting. The economy-SRU computes a low-dimensional sketch of its high-dimensional hidden state in the form of random projections to generate the feedback for its recurrent processing. Additionally, the internal weight parameters of the economy-SRU are strategically regularized in a group-wise manner to facilitate the proposed network in extracting meaningful predictive features that are highly time-localized to mimic real-world causal events. Extensive experiments are carried out to demonstrate that the proposed economy-SRU based time series prediction model outperforms the MLP, LSTM and attention-gated CNN-based time series models considered previously for inferring Granger causality.
△ Less
Submitted 13 January, 2020; v1 submitted 22 November, 2019;
originally announced November 2019.
-
Unsupervised Image Noise Modeling with Self-Consistent GAN
Authors:
Hanshu Yan,
Xuan Chen,
Vincent Y. F. Tan,
Wenhan Yang,
Joe Wu,
Jiashi Feng
Abstract:
Noise modeling lies in the heart of many image processing tasks. However, existing deep learning methods for noise modeling generally require clean and noisy image pairs for model training; these image pairs are difficult to obtain in many realistic scenarios. To ameliorate this problem, we propose a self-consistent GAN (SCGAN), that can directly extract noise maps from noisy images, thus enabling…
▽ More
Noise modeling lies in the heart of many image processing tasks. However, existing deep learning methods for noise modeling generally require clean and noisy image pairs for model training; these image pairs are difficult to obtain in many realistic scenarios. To ameliorate this problem, we propose a self-consistent GAN (SCGAN), that can directly extract noise maps from noisy images, thus enabling unsupervised noise modeling. In particular, the SCGAN introduces three novel self-consistent constraints that are complementary to one another, viz.: the noise model should produce a zero response over a clean input; the noise model should return the same output when fed with a specific pure noise input; and the noise model also should re-extract a pure noise map if the map is added to a clean image. These three constraints are simple yet effective. They jointly facilitate unsupervised learning of a noise model for various noise types. To demonstrate its wide applicability, we deploy the SCGAN on three image processing tasks including blind image denoising, rain streak removal, and noisy image super-resolution. The results demonstrate the effectiveness and superiority of our method over the state-of-the-art methods on a variety of benchmark datasets, even though the noise types vary significantly and paired clean images are not available.
△ Less
Submitted 4 June, 2020; v1 submitted 13 June, 2019;
originally announced June 2019.
-
A Ranking Model Motivated by Nonnegative Matrix Factorization with Applications to Tennis Tournaments
Authors:
Rui Xia,
Vincent Y. F. Tan,
Louis Filstroff,
Cédric Févotte
Abstract:
We propose a novel ranking model that combines the Bradley-Terry-Luce probability model with a nonnegative matrix factorization framework to model and uncover the presence of latent variables that influence the performance of top tennis players. We derive an efficient, provably convergent, and numerically stable majorization-minimization-based algorithm to maximize the likelihood of datasets under…
▽ More
We propose a novel ranking model that combines the Bradley-Terry-Luce probability model with a nonnegative matrix factorization framework to model and uncover the presence of latent variables that influence the performance of top tennis players. We derive an efficient, provably convergent, and numerically stable majorization-minimization-based algorithm to maximize the likelihood of datasets under the proposed statistical model. The model is tested on datasets involving the outcomes of matches between 20 top male and female tennis players over 14 major tournaments for men (including the Grand Slams and the ATP Masters 1000) and 16 major tournaments for women over the past 10 years. Our model automatically infers that the surface of the court (e.g., clay or hard court) is a key determinant of the performances of male players, but less so for females. Top players on various surfaces over this longitudinal period are also identified in an objective manner.
△ Less
Submitted 12 June, 2019; v1 submitted 15 March, 2019;
originally announced March 2019.