-
Widely Linear Matched Filter: A Lynchpin towards the Interpretability of Complex-valued CNNs
Authors:
Qingchen Wang,
Zhe Li,
Zdenka Babic,
Wei Deng,
Ljubiša Stanković,
Danilo P. Mandic
Abstract:
A recent study on the interpretability of real-valued convolutional neural networks (CNNs) {Stankovic_Mandic_2023CNN} has revealed a direct and physically meaningful link with the task of finding features in data through matched filters. However, applying this paradigm to illuminate the interpretability of complex-valued CNNs meets a formidable obstacle: the extension of matched filtering to a gen…
▽ More
A recent study on the interpretability of real-valued convolutional neural networks (CNNs) {Stankovic_Mandic_2023CNN} has revealed a direct and physically meaningful link with the task of finding features in data through matched filters. However, applying this paradigm to illuminate the interpretability of complex-valued CNNs meets a formidable obstacle: the extension of matched filtering to a general class of noncircular complex-valued data, referred to here as the widely linear matched filter (WLMF), has been only implicit in the literature. To this end, to establish the interpretability of the operation of complex-valued CNNs, we introduce a general WLMF paradigm, provide its solution and undertake analysis of its performance. For rigor, our WLMF solution is derived without imposing any assumption on the probability density of noise. The theoretical advantages of the WLMF over its standard strictly linear counterpart (SLMF) are provided in terms of their output signal-to-noise-ratios (SNRs), with WLMF consistently exhibiting enhanced SNR. Moreover, the lower bound on the SNR gain of WLMF is derived, together with condition to attain this bound. This serves to revisit the convolution-activation-pooling chain in complex-valued CNNs through the lens of matched filtering, which reveals the potential of WLMFs to provide physical interpretability and enhance explainability of general complex-valued CNNs. Simulations demonstrate the agreement between the theoretical and numerical results.
△ Less
Submitted 31 January, 2024; v1 submitted 29 January, 2024;
originally announced January 2024.
-
Manifold-based Shapley for SAR Recognization Network Explanation
Authors:
Xuran Hu,
Mingzhe Zhu,
Yuan**g Liu,
Zhenpeng Feng,
LJubisa Stankovic
Abstract:
Explainable artificial intelligence (XAI) holds immense significance in enhancing the deep neural network's transparency and credibility, particularly in some risky and high-cost scenarios, like synthetic aperture radar (SAR). Shapley is a game-based explanation technique with robust mathematical foundations. However, Shapley assumes that model's features are independent, rendering Shapley explana…
▽ More
Explainable artificial intelligence (XAI) holds immense significance in enhancing the deep neural network's transparency and credibility, particularly in some risky and high-cost scenarios, like synthetic aperture radar (SAR). Shapley is a game-based explanation technique with robust mathematical foundations. However, Shapley assumes that model's features are independent, rendering Shapley explanation invalid for high dimensional models. This study introduces a manifold-based Shapley method by projecting high-dimensional features into low-dimensional manifold features and subsequently obtaining Fusion-Shap, which aims at (1) addressing the issue of erroneous explanations encountered by traditional Shap; (2) resolving the challenge of interpretability that traditional Shap faces in complex scenarios.
△ Less
Submitted 6 January, 2024;
originally announced January 2024.
-
SAR Despeckling via Regional Denoising Diffusion Probabilistic Model
Authors:
Xuran Hu,
Ziqiang Xu,
Zhihan Chen,
Zhengpeng Feng,
Mingzhe Zhu,
LJubisa Stankovic
Abstract:
Speckle noise poses a significant challenge in maintaining the quality of synthetic aperture radar (SAR) images, so SAR despeckling techniques have drawn increasing attention. Despite the tremendous advancements of deep learning in fixed-scale SAR image despeckling, these methods still struggle to deal with large-scale SAR images. To address this problem, this paper introduces a novel despeckling…
▽ More
Speckle noise poses a significant challenge in maintaining the quality of synthetic aperture radar (SAR) images, so SAR despeckling techniques have drawn increasing attention. Despite the tremendous advancements of deep learning in fixed-scale SAR image despeckling, these methods still struggle to deal with large-scale SAR images. To address this problem, this paper introduces a novel despeckling approach termed Region Denoising Diffusion Probabilistic Model (R-DDPM) based on generative models. R-DDPM enables versatile despeckling of SAR images across various scales, accomplished within a single training session. Moreover, The artifacts in the fused SAR images can be avoided effectively with the utilization of region-guided inverse sampling. Experiments of our proposed R-DDPM on Sentinel-1 data demonstrates superior performance to existing methods.
△ Less
Submitted 5 January, 2024;
originally announced January 2024.
-
PIPO-Net: A Penalty-based Independent Parameters Optimization Deep Unfolding Network
Authors:
Xiumei Li,
Zhijie Zhang,
Huang Bai,
Ljubiša Stanković,
Junpeng Hao,
Junmei Sun
Abstract:
Compressive sensing (CS) has been widely applied in signal and image processing fields. Traditional CS reconstruction algorithms have a complete theoretical foundation but suffer from the high computational complexity, while fashionable deep network-based methods can achieve high-accuracy reconstruction of CS but are short of interpretability. These facts motivate us to develop a deep unfolding ne…
▽ More
Compressive sensing (CS) has been widely applied in signal and image processing fields. Traditional CS reconstruction algorithms have a complete theoretical foundation but suffer from the high computational complexity, while fashionable deep network-based methods can achieve high-accuracy reconstruction of CS but are short of interpretability. These facts motivate us to develop a deep unfolding network named the penalty-based independent parameters optimization network (PIPO-Net) to combine the merits of the above mentioned two kinds of CS methods. Each module of PIPO-Net can be viewed separately as an optimization problem with respective penalty function. The main characteristic of PIPO-Net is that, in each round of training, the learnable parameters in one module are updated independently from those of other modules. This makes the network more flexible to find the optimal solutions of the corresponding problems. Moreover, the mean-subtraction sampling and the high-frequency complementary blocks are developed to improve the performance of PIPO-Net. Experiments on reconstructing CS images demonstrate the effectiveness of the proposed PIPO-Net.
△ Less
Submitted 4 November, 2023;
originally announced November 2023.
-
Fourier Analysis of Signals on Directed Acyclic Graphs (DAG) Using Graph Zero-Padding
Authors:
Ljubisa Stankovic,
Milos Dakovic,
Ali Bagheri Bardi,
Milos Brajovic,
Isidora Stankovic
Abstract:
Directed acyclic graphs (DAGs) are used for modeling causal relationships, dependencies, and flows in various systems. However, spectral analysis becomes impractical in this setting because the eigen-decomposition of the adjacency matrix yields all eigenvalues equal to zero. This inherent property of DAGs results in an inability to differentiate between frequency components of signals on such grap…
▽ More
Directed acyclic graphs (DAGs) are used for modeling causal relationships, dependencies, and flows in various systems. However, spectral analysis becomes impractical in this setting because the eigen-decomposition of the adjacency matrix yields all eigenvalues equal to zero. This inherent property of DAGs results in an inability to differentiate between frequency components of signals on such graphs. This problem can be addressed by alternating the Fourier basis or adding edges in a DAG. However, these approaches change the physics of the considered problem. To address this limitation, we propose a graph zero-padding approach. This approach involves augmenting the original DAG with additional vertices that are connected to the existing structure. The added vertices are characterized by signal values set to zero. The proposed technique enables the spectral evaluation of system outputs on DAGs (in almost all cases), that is the computation of vertex-domain convolution without the adverse effects of aliasing due to changes in a graph structure, with the ultimate goal of preserving the output of the system on a graph as if the changes in the graph structure were not done.
△ Less
Submitted 13 November, 2023; v1 submitted 2 November, 2023;
originally announced November 2023.
-
Eigenvalues of Symmetric Non-normalized Discrete Trigonometric Transforms
Authors:
Ali Bagheri Bardi,
Milos Dakovic,
Taher Yazdanpanah,
Ljubisa Stankovic
Abstract:
A comprehensive approach to the spectrum characterization (derivation of eigenvalues and the corresponding multiplicities) for non-normalized, symmetric discrete trigonometric transforms (DTT) is presented in the paper. Eight types of the DTT are analyzed. New explicit analytic expressions for the eigenvalues, together with their multiplicities, for the cases of three DTT (DCT$_{(1)}$, DCT…
▽ More
A comprehensive approach to the spectrum characterization (derivation of eigenvalues and the corresponding multiplicities) for non-normalized, symmetric discrete trigonometric transforms (DTT) is presented in the paper. Eight types of the DTT are analyzed. New explicit analytic expressions for the eigenvalues, together with their multiplicities, for the cases of three DTT (DCT$_{(1)}$, DCT$_{(5)}$, and DST$_{(8)}$), are the main contribution of this paper. Moreover, the presented theory is supplemented by new, original derivations for the closed-form expressions of the square and the trace of analyzed DTT matrices.
△ Less
Submitted 16 February, 2023;
originally announced February 2023.
-
Cluster-CAM: Cluster-Weighted Visual Interpretation of CNNs' Decision in Image Classification
Authors:
Zhenpeng Feng,
Hongbing Ji,
Milos Dakovic,
Xiyang Cui,
Mingzhe Zhu,
Ljubisa Stankovic
Abstract:
Despite the tremendous success of convolutional neural networks (CNNs) in computer vision, the mechanism of CNNs still lacks clear interpretation. Currently, class activation map** (CAM), a famous visualization technique to interpret CNN's decision, has drawn increasing attention. Gradient-based CAMs are efficient while the performance is heavily affected by gradient vanishing and exploding. In…
▽ More
Despite the tremendous success of convolutional neural networks (CNNs) in computer vision, the mechanism of CNNs still lacks clear interpretation. Currently, class activation map** (CAM), a famous visualization technique to interpret CNN's decision, has drawn increasing attention. Gradient-based CAMs are efficient while the performance is heavily affected by gradient vanishing and exploding. In contrast, gradient-free CAMs can avoid computing gradients to produce more understandable results. However, existing gradient-free CAMs are quite time-consuming because hundreds of forward interference per image are required. In this paper, we proposed Cluster-CAM, an effective and efficient gradient-free CNN interpretation algorithm. Cluster-CAM can significantly reduce the times of forward propagation by splitting the feature maps into clusters in an unsupervised manner. Furthermore, we propose an artful strategy to forge a cognition-base map and cognition-scissors from clustered feature maps. The final salience heatmap will be computed by merging the above cognition maps. Qualitative results conspicuously show that Cluster-CAM can produce heatmaps where the highlighted regions match the human's cognition more precisely than existing CAMs. The quantitative evaluation further demonstrates the superiority of Cluster-CAM in both effectiveness and efficiency.
△ Less
Submitted 3 February, 2023;
originally announced February 2023.
-
Fair and skill-diverse student group formation via constrained k-way graph partitioning
Authors:
Alexander Jenkins,
Imad Jaimoukha,
Ljubisa Stankovic,
Danilo Mandic
Abstract:
Forming the right combination of students in a group promises to enable a powerful and effective environment for learning and collaboration. However, defining a group of students is a complex task which has to satisfy multiple constraints. This work introduces an unsupervised algorithm for fair and skill-diverse student group formation. This is achieved by taking account of student course marks an…
▽ More
Forming the right combination of students in a group promises to enable a powerful and effective environment for learning and collaboration. However, defining a group of students is a complex task which has to satisfy multiple constraints. This work introduces an unsupervised algorithm for fair and skill-diverse student group formation. This is achieved by taking account of student course marks and sensitive attributes provided by the education office. The skill sets of students are determined using unsupervised dimensionality reduction of course mark data via the Laplacian eigenmap. The problem is formulated as a constrained graph partitioning problem, whereby the diversity of skill sets in each group are maximised, group sizes are upper and lower bounded according to available resources, and `balance' of a sensitive attribute is lower bounded to enforce fairness in group formation. This optimisation problem is solved using integer programming and its effectiveness is demonstrated on a dataset of student course marks from Imperial College London.
△ Less
Submitted 12 January, 2023;
originally announced January 2023.
-
Demystifying CNNs for Images by Matched Filters
Authors:
Shengxi Li,
Xinyi Zhao,
Ljubisa Stankovic,
Danilo Mandic
Abstract:
The success of convolution neural networks (CNN) has been revolutionising the way we approach and use intelligent machines in the Big Data era. Despite success, CNNs have been consistently put under scrutiny owing to their \textit{black-box} nature, an \textit{ad hoc} manner of their construction, together with the lack of theoretical support and physical meanings of their operation. This has been…
▽ More
The success of convolution neural networks (CNN) has been revolutionising the way we approach and use intelligent machines in the Big Data era. Despite success, CNNs have been consistently put under scrutiny owing to their \textit{black-box} nature, an \textit{ad hoc} manner of their construction, together with the lack of theoretical support and physical meanings of their operation. This has been prohibitive to both the quantitative and qualitative understanding of CNNs, and their application in more sensitive areas such as AI for health. We set out to address these issues, and in this way demystify the operation of CNNs, by employing the perspective of matched filtering. We first illuminate that the convolution operation, the very core of CNNs, represents a matched filter which aims to identify the presence of features in input data. This then serves as a vehicle to interpret the convolution-activation-pooling chain in CNNs under the theoretical umbrella of matched filtering, a common operation in signal processing. We further provide extensive examples and experiments to illustrate this connection, whereby the learning in CNNs is shown to also perform matched filtering, which further sheds light onto physical meaning of learnt parameters and layers. It is our hope that this material will provide new insights into the understanding, constructing and analysing of CNNs, as well as paving the way for develo** new methods and architectures of CNNs.
△ Less
Submitted 16 October, 2022;
originally announced October 2022.
-
VS-CAM: Vertex Semantic Class Activation Map** to Interpret Vision Graph Neural Network
Authors:
Zhenpeng Feng,
Xiyang Cui,
Hongbing Ji,
Mingzhe Zhu,
Ljubisa Stankovic
Abstract:
Graph convolutional neural network (GCN) has drawn increasing attention and attained good performance in various computer vision tasks, however, there lacks a clear interpretation of GCN's inner mechanism. For standard convolutional neural networks (CNNs), class activation map** (CAM) methods are commonly used to visualize the connection between CNN's decision and image region by generating a he…
▽ More
Graph convolutional neural network (GCN) has drawn increasing attention and attained good performance in various computer vision tasks, however, there lacks a clear interpretation of GCN's inner mechanism. For standard convolutional neural networks (CNNs), class activation map** (CAM) methods are commonly used to visualize the connection between CNN's decision and image region by generating a heatmap. Nonetheless, such heatmap usually exhibits semantic-chaos when these CAMs are applied to GCN directly. In this paper, we proposed a novel visualization method particularly applicable to GCN, Vertex Semantic Class Activation Map** (VS-CAM). VS-CAM includes two independent pipelines to produce a set of semantic-probe maps and a semantic-base map, respectively. Semantic-probe maps are used to detect the semantic information from semantic-base map to aggregate a semantic-aware heatmap. Qualitative results show that VS-CAM can obtain heatmaps where the highlighted regions match the objects much more precisely than CNN-based CAM. The quantitative evaluation further demonstrates the superiority of VS-CAM.
△ Less
Submitted 15 September, 2022;
originally announced September 2022.
-
Analytical Interpretation of Latent Codes in InfoGAN with SAR Images
Authors:
Zhenpeng Feng,
Milos Dakovic,
Hongbing Ji,
Mingzhe Zhu,
Ljubisa Stankovic
Abstract:
Generative Adversarial Networks (GANs) can synthesize abundant photo-realistic synthetic aperture radar (SAR) images. Some recent GANs (e.g., InfoGAN), are even able to edit specific properties of the synthesized images by introducing latent codes. It is crucial for SAR image synthesis since the targets in real SAR images are with different properties due to the imaging mechanism. Despite the succ…
▽ More
Generative Adversarial Networks (GANs) can synthesize abundant photo-realistic synthetic aperture radar (SAR) images. Some recent GANs (e.g., InfoGAN), are even able to edit specific properties of the synthesized images by introducing latent codes. It is crucial for SAR image synthesis since the targets in real SAR images are with different properties due to the imaging mechanism. Despite the success of InfoGAN in manipulating properties, there still lacks a clear explanation of how these latent codes affect synthesized properties, thus editing specific properties usually relies on empirical trials, unreliable and time-consuming. In this paper, we show that latent codes are disentangled to affect the properties of SAR images in a non-linear manner. By introducing some property estimators for latent codes, we are able to provide a completely analytical nonlinear model to decompose the entangled causality between latent codes and different properties. The qualitative and quantitative experimental results further reveal that the properties can be calculated by latent codes, inversely, the satisfying latent codes can be estimated given desired properties. In this case, properties can be manipulated by latent codes as we expect.
△ Less
Submitted 26 May, 2022;
originally announced May 2022.
-
How likely is a random graph shift-enabled?
Authors:
Liyan Chen,
Samuel Cheng,
Vladimir Stankovic,
Lina Stankovic
Abstract:
The shift-enabled property of an underlying graph is essential in designing distributed filters. This article discusses when a random graph is shift-enabled. In particular, popular graph models ER, WS, BA random graph are used, weighted and unweighted, as well as signed graphs. Our results show that the considered unweighted connected random graphs are shift-enabled with high probability when the…
▽ More
The shift-enabled property of an underlying graph is essential in designing distributed filters. This article discusses when a random graph is shift-enabled. In particular, popular graph models ER, WS, BA random graph are used, weighted and unweighted, as well as signed graphs. Our results show that the considered unweighted connected random graphs are shift-enabled with high probability when the number of edges is moderately high. However, very dense graphs, as well as fully connected graphs, are not shift-enabled. Interestingly, this behaviour is not observed for weighted connected graphs, which are always shift-enabled unless the number of edges in the graph is very low.
△ Less
Submitted 28 August, 2021;
originally announced September 2021.
-
Convolutional Neural Networks Demystified: A Matched Filtering Perspective Based Tutorial
Authors:
Ljubisa Stankovic,
Danilo Mandic
Abstract:
Deep Neural Networks (DNN) and especially Convolutional Neural Networks (CNN) are a de-facto standard for the analysis of large volumes of signals and images. Yet, their development and underlying principles have been largely performed in an ad-hoc and black box fashion. To help demystify CNNs, we revisit their operation from first principles and a matched filtering perspective. We establish that…
▽ More
Deep Neural Networks (DNN) and especially Convolutional Neural Networks (CNN) are a de-facto standard for the analysis of large volumes of signals and images. Yet, their development and underlying principles have been largely performed in an ad-hoc and black box fashion. To help demystify CNNs, we revisit their operation from first principles and a matched filtering perspective. We establish that the convolution operation within CNNs, their very backbone, represents a matched filter which examines the input signal/image for the presence of pre-defined features. This perspective is shown to be physically meaningful, and serves as a basis for a step-by-step tutorial on the operation of CNNs, including pooling, zero padding, various ways of dimensionality reduction. Starting from first principles, both the feed-forward pass and the learning stage (via back-propagation) are illuminated in detail, both through a worked-out numerical example and the corresponding visualizations. It is our hope that this tutorial will help shed new light and physical intuition into the understanding and further development of deep neural networks.
△ Less
Submitted 22 March, 2022; v1 submitted 26 August, 2021;
originally announced August 2021.
-
Understanding the Basis of Graph Convolutional Neural Networks via an Intuitive Matched Filtering Approach
Authors:
Ljubisa Stankovic,
Danilo Mandic
Abstract:
Graph Convolutional Neural Networks (GCNN) are becoming a preferred model for data processing on irregular domains, yet their analysis and principles of operation are rarely examined due to the black box nature of NNs. To this end, we revisit the operation of GCNNs and show that their convolution layers effectively perform matched filtering of input data with the chosen patterns (features). This a…
▽ More
Graph Convolutional Neural Networks (GCNN) are becoming a preferred model for data processing on irregular domains, yet their analysis and principles of operation are rarely examined due to the black box nature of NNs. To this end, we revisit the operation of GCNNs and show that their convolution layers effectively perform matched filtering of input data with the chosen patterns (features). This allows us to provide a unifying account of GCNNs through a matched filter perspective, whereby the nonlinear ReLU and max-pooling layers are also discussed within the matched filtering framework. This is followed by a step-by-step guide on information propagation and learning in GCNNs. It is also shown that standard CNNs and fully connected NNs can be obtained as a special case of GCNNs. A carefully chosen numerical example guides the reader through the various steps of GCNN operation and learning both visually and numerically.
△ Less
Submitted 23 August, 2021;
originally announced August 2021.
-
Dynamic Portfolio Cuts: A Spectral Approach to Graph-Theoretic Diversification
Authors:
Alvaro Arroyo,
Bruno Scalzo,
Ljubisa Stankovic,
Danilo P. Mandic
Abstract:
Stock market returns are typically analyzed using standard regression, yet they reside on irregular domains which is a natural scenario for graph signal processing. To this end, we consider a market graph as an intuitive way to represent the relationships between financial assets. Traditional methods for estimating asset-return covariance operate under the assumption of statistical time-invariance…
▽ More
Stock market returns are typically analyzed using standard regression, yet they reside on irregular domains which is a natural scenario for graph signal processing. To this end, we consider a market graph as an intuitive way to represent the relationships between financial assets. Traditional methods for estimating asset-return covariance operate under the assumption of statistical time-invariance, and are thus unable to appropriately infer the underlying true structure of the market graph. This work introduces a class of graph spectral estimators which cater for the nonstationarity inherent to asset price movements, and serve as a basis to represent the time-varying interactions between assets through a dynamic spectral market graph. Such an account of the time-varying nature of the asset-return covariance allows us to introduce the notion of dynamic spectral portfolio cuts, whereby the graph is partitioned into time-evolving clusters, allowing for online and robust asset allocation. The advantages of the proposed framework over traditional methods are demonstrated through numerical case studies using real-world price data.
△ Less
Submitted 7 June, 2021;
originally announced June 2021.
-
On the Sparsity Bound for the Existence of a Unique Solution in Compressive Sensing by the Gershgorin Theorem
Authors:
Ljubisa Stankovic
Abstract:
Since compressive sensing deals with a signal reconstruction using a reduced set of measurements, the existence of a unique solution is of crucial importance. The most important approach to this problem is based on the restricted isometry property which is computationally unfeasible. The coherence index-based uniqueness criteria are computationally efficient, however, they are pessimistic. An appr…
▽ More
Since compressive sensing deals with a signal reconstruction using a reduced set of measurements, the existence of a unique solution is of crucial importance. The most important approach to this problem is based on the restricted isometry property which is computationally unfeasible. The coherence index-based uniqueness criteria are computationally efficient, however, they are pessimistic. An approach to alleviate this problem has been recently introduced by relaxing the coherence index condition for the unique signal reconstruction using the orthogonal matching pursuit approach. This approach can be further relaxed and the sparsity bound improved if we consider only the solution existence rather than its reconstruction. One such improved bound for the sparsity limit is derived in this paper using the Gershgorin disk theorem.
△ Less
Submitted 27 May, 2021;
originally announced May 2021.
-
Improved Coherence Index-Based Bound in Compressive Sensing
Authors:
Ljubisa Stankovic,
Milos Brajovic,
Danilo Mandic,
Isidora Stankovic,
Milos Dakovic
Abstract:
Within the Compressive Sensing (CS) paradigm, sparse signals can be reconstructed based on a reduced set of measurements. Reliability of the solution is determined by the uniqueness condition. With its mathematically tractable and feasible calculation, coherence index is one of very few CS metrics with a considerable practical importance. In this paper, we propose an improvement of the coherence b…
▽ More
Within the Compressive Sensing (CS) paradigm, sparse signals can be reconstructed based on a reduced set of measurements. Reliability of the solution is determined by the uniqueness condition. With its mathematically tractable and feasible calculation, coherence index is one of very few CS metrics with a considerable practical importance. In this paper, we propose an improvement of the coherence based uniqueness relation for the matching pursuit algorithms. Starting from a simple and intuitive derivation of the standard uniqueness condition based on the coherence index, we derive a less conservative coherence index-based lower bound for signal sparsity. The results are generalized to the uniqueness condition of the $l_0$-norm minimization for a signal represented in two orthonormal bases.
△ Less
Submitted 11 March, 2021;
originally announced March 2021.
-
Nonstationary Portfolios: Diversification in the Spectral Domain
Authors:
Bruno Scalzo,
Alvaro Arroyo,
Ljubisa Stankovic,
Danilo P. Mandic
Abstract:
Classical portfolio optimization methods typically determine an optimal capital allocation through the implicit, yet critical, assumption of statistical time-invariance. Such models are inadequate for real-world markets as they employ standard time-averaging based estimators which suffer significant information loss if the market observables are non-stationary. To this end, we reformulate the port…
▽ More
Classical portfolio optimization methods typically determine an optimal capital allocation through the implicit, yet critical, assumption of statistical time-invariance. Such models are inadequate for real-world markets as they employ standard time-averaging based estimators which suffer significant information loss if the market observables are non-stationary. To this end, we reformulate the portfolio optimization problem in the spectral domain to cater for the nonstationarity inherent to asset price movements and, in this way, allow for optimal capital allocations to be time-varying. Unlike existing spectral portfolio techniques, the proposed framework employs augmented complex statistics in order to exploit the interactions between the real and imaginary parts of the complex spectral variables, which in turn allows for the modelling of both harmonics and cyclostationarity in the time domain. The advantages of the proposed framework over traditional methods are demonstrated through numerical simulations using real-world price data.
△ Less
Submitted 31 January, 2021;
originally announced February 2021.
-
A Probabilistic Spectral Analysis of Multivariate Real-Valued Nonstationary Signals
Authors:
Bruno Scalzo,
Ljubisa Stankovic,
Danilo P. Mandic
Abstract:
A class of multivariate spectral representations for real-valued nonstationary random variables is introduced, which is characterised by a general complex Gaussian distribution. In this way, the temporal signal properties -- harmonicity, wide-sense stationarity and cyclostationarity -- are designated respectively by the mean, Hermitian variance and pseudo-variance of the associated time-frequency…
▽ More
A class of multivariate spectral representations for real-valued nonstationary random variables is introduced, which is characterised by a general complex Gaussian distribution. In this way, the temporal signal properties -- harmonicity, wide-sense stationarity and cyclostationarity -- are designated respectively by the mean, Hermitian variance and pseudo-variance of the associated time-frequency representation (TFR). For rigour, the estimators of the TFR distribution parameters are derived within a maximum likelihood framework and are shown to be statistically consistent, owing to the statistical identifiability of the proposed distribution parametrization. By virtue of the assumed probabilistic model, a generalised likelihood ratio test (GLRT) for nonstationarity detection is also proposed. Intuitive examples demonstrate the utility of the derived probabilistic framework for spectral analysis in low-SNR environments.
△ Less
Submitted 27 July, 2020;
originally announced July 2020.
-
The Support Uncertainty Principle and the Graph Rihaczek Distribution: Revisited and Improved
Authors:
Ljubisa Stankovic
Abstract:
The classical support uncertainty principle states that the signal and its discrete Fourier transform (DFT) cannot be localized simultaneously in an arbitrary small area in the time and the frequency domain. The product of the number of nonzero samples in the time domain and the frequency domain is greater or equal to the total number of signal samples. The support uncertainty principle has been e…
▽ More
The classical support uncertainty principle states that the signal and its discrete Fourier transform (DFT) cannot be localized simultaneously in an arbitrary small area in the time and the frequency domain. The product of the number of nonzero samples in the time domain and the frequency domain is greater or equal to the total number of signal samples. The support uncertainty principle has been extended to the arbitrary orthogonal pairs of signal basis and the graph signals, stating that the product of supports in the vertex domain and the spectral domain is greater than the reciprocal squared maximum absolute value of the basis functions. This form is then used in compressive sensing and sparse signal processing to define the reconstruction conditions. In this paper, we will revisit the graph signal uncertainty principle using the graph Rihaczek distribution as an analysis tool and derive an improved bound for the support uncertainty principle of graph signals.
△ Less
Submitted 19 June, 2020; v1 submitted 16 May, 2020;
originally announced May 2020.
-
RANSAC-Based Signal Denoising Using Compressive Sensing
Authors:
Ljubisa Stankovic,
Milos Brajovic,
Isidora Stankovic,
Jonatan Lerga,
Milos Dakovic
Abstract:
In this paper, we present an approach to the reconstruction of signals exhibiting sparsity in a transformation domain, having some heavily disturbed samples. This sparsity-driven signal recovery exploits a carefully suited random sampling consensus (RANSAC) methodology for the selection of an inlier subset of samples. To this aim, two fundamental properties are used: a signal sample represents a l…
▽ More
In this paper, we present an approach to the reconstruction of signals exhibiting sparsity in a transformation domain, having some heavily disturbed samples. This sparsity-driven signal recovery exploits a carefully suited random sampling consensus (RANSAC) methodology for the selection of an inlier subset of samples. To this aim, two fundamental properties are used: a signal sample represents a linear combination of the sparse coefficients, whereas the disturbance degrade original signal sparsity. The properly selected samples are further used as measurements in the sparse signal reconstruction, performed using algorithms from the compressive sensing framework. Besides the fact that the disturbance degrades signal sparsity in the transformation domain, no other disturbance-related assumptions are made -- there are no special requirements regarding its statistical behavior or the range of its values. As a case study, the discrete Fourier transform (DFT) is considered as a domain of signal sparsity, owing to its significance in signal processing theory and applications. Numerical results strongly support the presented theory. In addition, exact relation for the signal-to-noise ratio (SNR) of the reconstructed signal is also presented. This simple result, which conveniently characterizes the RANSAC-based reconstruction performance, is numerically confirmed by a set of statistical examples.
△ Less
Submitted 27 March, 2020;
originally announced March 2020.
-
Graph Signal Processing -- Part III: Machine Learning on Graphs, from Graph Topology to Applications
Authors:
Ljubisa Stankovic,
Danilo Mandic,
Milos Dakovic,
Milos Brajovic,
Bruno Scalzo,
Shengxi Li,
Anthony G. Constantinides
Abstract:
Many modern data analytics applications on graphs operate on domains where graph topology is not known a priori, and hence its determination becomes part of the problem definition, rather than serving as prior knowledge which aids the problem solution. Part III of this monograph starts by addressing ways to learn graph topology, from the case where the physics of the problem already suggest a poss…
▽ More
Many modern data analytics applications on graphs operate on domains where graph topology is not known a priori, and hence its determination becomes part of the problem definition, rather than serving as prior knowledge which aids the problem solution. Part III of this monograph starts by addressing ways to learn graph topology, from the case where the physics of the problem already suggest a possible topology, through to most general cases where the graph topology is learned from the data. A particular emphasis is on graph topology definition based on the correlation and precision matrices of the observed data, combined with additional prior knowledge and structural conditions, such as the smoothness or sparsity of graph connections. For learning sparse graphs (with small number of edges), the least absolute shrinkage and selection operator, known as LASSO is employed, along with its graph specific variant, graphical LASSO. For completeness, both variants of LASSO are derived in an intuitive way, and explained. An in-depth elaboration of the graph topology learning paradigm is provided through several examples on physically well defined graphs, such as electric circuits, linear heat transfer, social and computer networks, and spring-mass systems. As many graph neural networks (GNN) and convolutional graph networks (GCN) are emerging, we have also reviewed the main trends in GNNs and GCNs, from the perspective of graph signal filtering. Tensor representation of lattice-structured graphs is next considered, and it is shown that tensors (multidimensional data arrays) are a special class of graph signals, whereby the graph vertices reside on a high-dimensional regular lattice structure. This part of monograph concludes with two emerging applications in financial data processing and underground transportation networks modeling.
△ Less
Submitted 2 January, 2020;
originally announced January 2020.
-
Robust Deep Graph Based Learning for Binary Classification
Authors:
Minxiang Ye,
Vladimir Stankovic,
Lina Stankovic,
Gene Cheung
Abstract:
Convolutional neural network (CNN)-based feature learning has become state of the art, since given sufficient training data, CNN can significantly outperform traditional methods for various classification tasks. However, feature learning becomes more difficult if some training labels are noisy. With traditional regularization techniques, CNN often overfits to the noisy training labels, resulting i…
▽ More
Convolutional neural network (CNN)-based feature learning has become state of the art, since given sufficient training data, CNN can significantly outperform traditional methods for various classification tasks. However, feature learning becomes more difficult if some training labels are noisy. With traditional regularization techniques, CNN often overfits to the noisy training labels, resulting in sub-par classification performance. In this paper, we propose a robust binary classifier, based on CNNs, to learn deep metric functions, which are then used to construct an optimal underlying graph structure used to clean noisy labels via graph Laplacian regularization (GLR). GLR is posed as a convex maximum a posteriori (MAP) problem solved via convex quadratic programming (QP). To penalize samples around the decision boundary, we propose two regularized loss functions for semi-supervised learning. The binary classification experiments on three datasets, varying in number and type of features, demonstrate that given a noisy training dataset, our proposed networks outperform several state-of-the-art classifiers, including label-noise robust support vector machine, CNNs with three different robust loss functions, model-based GLR, and dynamic graph CNN classifiers.
△ Less
Submitted 6 December, 2019;
originally announced December 2019.
-
Portfolio Cuts: A Graph-Theoretic Framework to Diversification
Authors:
Bruno Scalzo Dees,
Ljubisa Stankovic,
Anthony G. Constantinides,
Danilo P. Mandic
Abstract:
Investment returns naturally reside on irregular domains, however, standard multivariate portfolio optimization methods are agnostic to data structure. To this end, we investigate ways for domain knowledge to be conveniently incorporated into the analysis, by means of graphs. Next, to relax the assumption of the completeness of graph topology and to equip the graph model with practically relevant…
▽ More
Investment returns naturally reside on irregular domains, however, standard multivariate portfolio optimization methods are agnostic to data structure. To this end, we investigate ways for domain knowledge to be conveniently incorporated into the analysis, by means of graphs. Next, to relax the assumption of the completeness of graph topology and to equip the graph model with practically relevant physical intuition, we introduce the portfolio cut paradigm. Such a graph-theoretic portfolio partitioning technique is shown to allow the investor to devise robust and tractable asset allocation schemes, by virtue of a rigorous graph framework for considering smaller, computationally feasible, and economically meaningful clusters of assets, based on graph cuts. In turn, this makes it possible to fully utilize the asset returns covariance matrix for constructing the portfolio, even without the requirement for its inversion. The advantages of the proposed framework over traditional methods are demonstrated through numerical simulations based on real-world price data.
△ Less
Submitted 16 October, 2019; v1 submitted 12 October, 2019;
originally announced October 2019.
-
Graph Signal Processing -- Part II: Processing and Analyzing Signals on Graphs
Authors:
Ljubisa Stankovic,
Danilo Mandic,
Milos Dakovic,
Milos Brajovic,
Bruno Scalzo,
Anthony G. Constantinides
Abstract:
The focus of Part I of this monograph has been on both the fundamental properties, graph topologies, and spectral representations of graphs. Part II embarks on these concepts to address the algorithmic and practical issues centered round data/signal processing on graphs, that is, the focus is on the analysis and estimation of both deterministic and random data on graphs. The fundamental ideas rela…
▽ More
The focus of Part I of this monograph has been on both the fundamental properties, graph topologies, and spectral representations of graphs. Part II embarks on these concepts to address the algorithmic and practical issues centered round data/signal processing on graphs, that is, the focus is on the analysis and estimation of both deterministic and random data on graphs. The fundamental ideas related to graph signals are introduced through a simple and intuitive, yet illustrative and general enough case study of multisensor temperature field estimation. The concept of systems on graph is defined using graph signal shift operators, which generalize the corresponding principles from traditional learning systems. At the core of the spectral domain representation of graph signals and systems is the Graph Discrete Fourier Transform (GDFT). The spectral domain representations are then used as the basis to introduce graph signal filtering concepts and address their design, including Chebyshev polynomial approximation series. Ideas related to the sampling of graph signals are presented and further linked with compressive sensing. Localized graph signal analysis in the joint vertex-spectral domain is referred to as the vertex-frequency analysis, since it can be considered as an extension of classical time-frequency analysis to the graph domain of a signal. Important topics related to the local graph Fourier transform (LGFT) are covered, together with its various forms including the graph spectral and vertex domain windows and the inversion conditions and relations. A link between the LGFT with spectral varying window and the spectral graph wavelet transform (SGWT) is also established. Realizations of the LGFT and SGWT using polynomial (Chebyshev) approximations of the spectral functions are further considered. Finally, energy versions of the vertex-frequency representations are introduced.
△ Less
Submitted 23 September, 2019;
originally announced September 2019.
-
Unitary Shift Operators on a Graph
Authors:
Bruno Scalzo Dees,
Ljubisa Stankovic,
Milos Dakovic,
Anthony G. Constantinides,
Danilo P. Mandic
Abstract:
A unitary shift operator (GSO) for signals on a graph is introduced, which exhibits the desired property of energy preservation over both backward and forward graph shifts. For rigour, the graph differential operator is also derived in an analytical form. The commutativity relation of the shift operator with the Fourier transform is next explored in conjunction with the proposed GSO to introduce a…
▽ More
A unitary shift operator (GSO) for signals on a graph is introduced, which exhibits the desired property of energy preservation over both backward and forward graph shifts. For rigour, the graph differential operator is also derived in an analytical form. The commutativity relation of the shift operator with the Fourier transform is next explored in conjunction with the proposed GSO to introduce a graph discrete Fourier transform (GDFT) which, unlike existing approaches, ensures the orthogonality of GDFT bases and admits a natural frequency-domain interpretation. The proposed GDFT is shown to allow for a coherent definition of the graph discrete Hilbert transform (GDHT) and the graph analytic signal. The advantages of the proposed GSO are demonstrated through illustrative examples.
△ Less
Submitted 17 September, 2019; v1 submitted 12 September, 2019;
originally announced September 2019.
-
A Class of Doubly Stochastic Shift Operators for Random Graph Signals and their Boundedness
Authors:
Bruno Scalzo Dees,
Ljubisa Stankovic,
Milos Dakovic,
Anthony G. Constantinides,
Danilo P. Mandic
Abstract:
A class of doubly stochastic graph shift operators (GSO) is proposed, which is shown to exhibit: (i) lower and upper $L_{2}$-boundedness for locally stationary random graph signals; (ii) $L_{2}$-isometry for \textit{i.i.d.} random graph signals with the asymptotic increase in the incoming neighbourhood size of vertices; and (iii) preservation of the mean of any graph signal. These properties are o…
▽ More
A class of doubly stochastic graph shift operators (GSO) is proposed, which is shown to exhibit: (i) lower and upper $L_{2}$-boundedness for locally stationary random graph signals; (ii) $L_{2}$-isometry for \textit{i.i.d.} random graph signals with the asymptotic increase in the incoming neighbourhood size of vertices; and (iii) preservation of the mean of any graph signal. These properties are obtained through a statistical consistency analysis of the graph shift, and by exploiting the dual role of the doubly stochastic GSO as a Markov (diffusion) matrix and as an unbiased expectation operator. Practical utility of the class of doubly stochastic GSOs is demonstrated in a real-world multi-sensor signal filtering setting.
△ Less
Submitted 7 February, 2020; v1 submitted 5 August, 2019;
originally announced August 2019.
-
Vertex-Frequency Graph Signal Processing: A review
Authors:
Ljubisa Stankovic,
Danilo P. Mandic,
Milos Dakovic,
Bruno Scalzo,
Milos Brajovic,
Ervin Sejdic,
Anthony G. Constantinides
Abstract:
Graph signal processing deals with signals which are observed on an irregular graph domain. While many approaches have been developed in classical graph theory to cluster vertices and segment large graphs in a signal independent way, signal localization based approaches to the analysis of data on graph represent a new research direction which is also a key to big data analytics on graphs. To this…
▽ More
Graph signal processing deals with signals which are observed on an irregular graph domain. While many approaches have been developed in classical graph theory to cluster vertices and segment large graphs in a signal independent way, signal localization based approaches to the analysis of data on graph represent a new research direction which is also a key to big data analytics on graphs. To this end, after an overview of the basic definitions in graphs and graph signals, we present and discuss a localized form of the graph Fourier transform. To establish an analogy with classical signal processing, spectral- and vertex-domain definitions of the localization window are given next. The spectral and vertex localization kernels are then related to the wavelet transform, followed by a study of filtering and inversion of the localized graph Fourier transform. For rigour, the analysis of energy representation and frames in the localized graph Fourier transform is extended to the energy forms of vertex-frequency distributions, which operate even without the need to apply localization windows. Another link with classical signal processing is established through the concept of local smoothness, which is subsequently related to the particular paradigm of signal smoothness on graphs. This all represents a comprehensive account of the relation of general vertex-frequency analysis with classical time-frequency analysis, and important but missing link for more advanced applications of graphs signal processing. The theory is supported by illustrative and practically relevant examples.
△ Less
Submitted 26 December, 2019; v1 submitted 8 July, 2019;
originally announced July 2019.
-
Graph Signal Processing -- Part I: Graphs, Graph Spectra, and Spectral Clustering
Authors:
Ljubisa Stankovic,
Danilo Mandic,
Milos Dakovic,
Milos Brajovic,
Bruno Scalzo,
Tony Constantinides
Abstract:
The area of Data Analytics on graphs promises a paradigm shift as we approach information processing of classes of data, which are typically acquired on irregular but structured domains (social networks, various ad-hoc sensor networks). Yet, despite its long history, current approaches mostly focus on the optimization of graphs themselves, rather than on directly inferring learning strategies, suc…
▽ More
The area of Data Analytics on graphs promises a paradigm shift as we approach information processing of classes of data, which are typically acquired on irregular but structured domains (social networks, various ad-hoc sensor networks). Yet, despite its long history, current approaches mostly focus on the optimization of graphs themselves, rather than on directly inferring learning strategies, such as detection, estimation, statistical and probabilistic inference, clustering and separation from signals and data acquired on graphs. To fill this void, we first revisit graph topologies from a Data Analytics point of view, and establish a taxonomy of graph networks through a linear algebraic formalism of graph topology (vertices, connections, directivity). This serves as a basis for spectral analysis of graphs, whereby the eigenvalues and eigenvectors of graph Laplacian and adjacency matrices are shown to convey physical meaning related to both graph topology and higher-order graph properties, such as cuts, walks, paths, and neighborhoods. Next, to illustrate estimation strategies performed on graph signals, spectral analysis of graphs is introduced through eigenanalysis of mathematical descriptors of graphs and in a generic way. Finally, a framework for vertex clustering and graph segmentation is established based on graph spectral representation (eigenanalysis) which illustrates the power of graphs in various data association tasks. The supporting examples demonstrate the promise of Graph Data Analytics in modeling structural and functional/semantic inferences. At the same time, Part I serves as a basis for Part II and Part III which deal with theory, methods and applications of processing Data on Graphs and Graph Topology Learning from data.
△ Less
Submitted 12 August, 2019; v1 submitted 8 July, 2019;
originally announced July 2019.
-
Quantization in Compressive Sensing: A Signal Processing Approach
Authors:
Isidora Stankovic,
Milos Brajovic,
Milos Dakovic,
Cornel Ioana,
Ljubisa Stankovic
Abstract:
Influence of the finite-length registers and quantization effects on the reconstruction of sparse and approximately sparse signals is analyzed in this paper. For the nonquantized measurements, the compressive sensing (CS) framework provides highly accurate reconstruction algorithms that produce negligible errors when the reconstruction conditions are met. However, hardware implementations of signa…
▽ More
Influence of the finite-length registers and quantization effects on the reconstruction of sparse and approximately sparse signals is analyzed in this paper. For the nonquantized measurements, the compressive sensing (CS) framework provides highly accurate reconstruction algorithms that produce negligible errors when the reconstruction conditions are met. However, hardware implementations of signal processing algorithms involve the finite-length registers and quantization of the measurements. An analysis of the effects related to the measurements quantization with an arbitrary number of bits is the topic of this paper. A unified mathematical model for the analysis of the quantization noise and the signal nonsparsity on the CS reconstruction is presented. An exact formula for the expected energy of error in the CS-based reconstructed signal is derived. The theory is validated through various numerical examples with quantized measurements, including the cases of approximately sparse signals, noise folding, and floating-point arithmetics.
△ Less
Submitted 1 July, 2019;
originally announced July 2019.
-
Distributed Intelligent Illumination Control in the Context of Probabilistic Graphical Models
Authors:
M. Cosovic,
T. Devaja,
D. Bajovic,
J. Machaj,
G. McCutcheon,
V. Stankovic,
L. Stankovic,
D. Vukobratovic
Abstract:
Lighting systems based on light-emitting diodes (LEDs) possess many benefits over their incandescent counterparts including longer lifespans, lower energy costs, better quality of light and no toxic elements, all without sacrificing consumer satisfaction. Their lifespan is not affected by switching frequency allowing for better illumination control and system efficiency. In this paper, we present…
▽ More
Lighting systems based on light-emitting diodes (LEDs) possess many benefits over their incandescent counterparts including longer lifespans, lower energy costs, better quality of light and no toxic elements, all without sacrificing consumer satisfaction. Their lifespan is not affected by switching frequency allowing for better illumination control and system efficiency. In this paper, we present a fully distributed energy-saving illumination dimming control strategy for the system of a lighting network which consists of a group of LEDs and user-associated devices. In order to solve the optimization problem, we are using a distributed approach that utilizes factor graphs and the belief propagation algorithm. Using probabilistic graphical models to represent and solve the system model provides for a natural description of the problem structure, where user devices and LED controllers exchange data via line-of-sight communication.
△ Less
Submitted 12 June, 2019;
originally announced June 2019.
-
On the Decomposition of Multivariate Nonstationary Multicomponent Signals
Authors:
Ljubisa Stankovic,
Milos Brajovic,
Milos Dakovic,
Danilo Mandic
Abstract:
With their ability to handle an increased amount of information, multivariate and multichannel signals can be used to solve problems normally not solvable with signals obtained from a single source. One such problem is the decomposition signals with several components whose domains of support significantly overlap in both the time and the frequency domain, including the joint time-frequency domain…
▽ More
With their ability to handle an increased amount of information, multivariate and multichannel signals can be used to solve problems normally not solvable with signals obtained from a single source. One such problem is the decomposition signals with several components whose domains of support significantly overlap in both the time and the frequency domain, including the joint time-frequency domain. Initially, we proposed a solution to this problem based on the Wigner distribution of multivariate signals, which requires the attenuation of the cross-terms. In this paper, an advanced solution based on an eigenvalue analysis of the multivariate signal autocorrelation matrix, followed by their time-frequency concentration measure minimization, is presented. This analysis provides less restrictive conditions for the signal decomposition than in the case of Wigner distribution. The algorithm for the components separation is based on the concentration measures of the eigenvector time-frequency representation, that are linear combinations of the overlap** signal components. With an increased number of sensors/channels, the robustness of the decomposition process to additive noise is also achieved. The theory is supported by numerical examples. The required channel dissimilarity is statistically investigated as well.
△ Less
Submitted 31 March, 2019;
originally announced April 2019.
-
An Example-Driven Introduction to Data Analytics on Graphs
Authors:
Ljubisa Stankovic,
Danilo Mandic,
Milos Dakovic,
Ilya Kisil,
Ervin Sejdic,
Anthony G. Constantinides
Abstract:
Graphs are irregular structures which naturally account for data integrity, however, traditional approaches have been established outside Signal Processing, and largely focus on analyzing the underlying graphs rather than signals on graphs. Given the rapidly increasing availability of multisensor and multinode measurements, likely recorded on irregular or ad-hoc grids, it would be extremely advant…
▽ More
Graphs are irregular structures which naturally account for data integrity, however, traditional approaches have been established outside Signal Processing, and largely focus on analyzing the underlying graphs rather than signals on graphs. Given the rapidly increasing availability of multisensor and multinode measurements, likely recorded on irregular or ad-hoc grids, it would be extremely advantageous to analyze such structured data as graph signals and thus benefit from the ability of graphs to incorporate spatial awareness of the sensing locations, sensor importance, and local versus global sensor association. The aim of this lecture note is therefore to establish a common language between graph signals, defined on irregular signal domains, and some of the most fundamental paradigms in DSP, such as spectral analysis of multichannel signals, system transfer function, digital filter design, parameter estimation, and optimal filter design. This is achieved through a physically meaningful and intuitive real-world example of geographically distributed multisensor temperature estimation. A similar spatial multisensor arrangement is already widely used in Signal Processing curricula to introduce minimum variance estimators and Kalman filters \cite{HM}, and by adopting this framework we facilitate a seamless integration of graph theory into the curriculum of existing DSP courses. By bridging the gap between standard approaches and graph signal processing, we also show that standard methods can be thought of as special cases of their graph counterparts, evaluated on linear graphs. It is hoped that our approach would not only help to demystify graph theoretic approaches in education and research but it would also empower practitioners to explore a whole host of otherwise prohibitive modern applications.
△ Less
Submitted 12 May, 2019; v1 submitted 26 March, 2019;
originally announced March 2019.
-
An Intuitive Derivation of the Coherence Index Relation in Compressive Sensing
Authors:
Ljubisa Stankovic,
Danilo Mandic,
Milos Dakovic,
Ilya Kisil
Abstract:
The existence and uniqueness conditions are a prerequisite for reliable reconstruction of sparse signals from reduced sets of measurements within the Compressive Sensing (CS) paradigm. However, despite their underpinning role for practical applications, existing uniqueness relations are either computationally prohibitive to implement (Restricted Isometry Property), or involve mathematical tools th…
▽ More
The existence and uniqueness conditions are a prerequisite for reliable reconstruction of sparse signals from reduced sets of measurements within the Compressive Sensing (CS) paradigm. However, despite their underpinning role for practical applications, existing uniqueness relations are either computationally prohibitive to implement (Restricted Isometry Property), or involve mathematical tools that are beyond the standard background of engineering graduates (Coherence Index). This can introduce conceptual and computational obstacles in the development of engineering intuition, the design of suboptimal practical solutions, or understanding of limitations. To this end, we introduce a simple but rigorous derivation of the coherence index condition, based on standard linear algebra, with the aim to empower signal processing practitioners with intuition in the design and ease in implementation of CS systems. Given that the coherence index is one of very few CS metrics that admits mathematically tractable and computationally feasible calculation, it is our hope that this work will help bridge the gap between the theory and applications of compressive sensing.
△ Less
Submitted 26 March, 2019;
originally announced March 2019.
-
Undirected graphs: is the shift-enabled condition trivial or necessary?
Authors:
Liyan Chen,
Samuel Cheng,
Kanghang He,
Lina Stankovic,
Vladimir Stankovic
Abstract:
It has recently been shown that, contrary to the wide belief that a shift-enabled condition (necessary for any shift-invariant filter to be representable by a graph shift matrix) can be ignored because any non-shift-enabled matrix can be converted to a shift-enabled matrix, such a conversion in general may not hold for a directed graph with non-symmetric shift matrix. This letter extends this prio…
▽ More
It has recently been shown that, contrary to the wide belief that a shift-enabled condition (necessary for any shift-invariant filter to be representable by a graph shift matrix) can be ignored because any non-shift-enabled matrix can be converted to a shift-enabled matrix, such a conversion in general may not hold for a directed graph with non-symmetric shift matrix. This letter extends this prior work, focusing on undirected graphs where the shift matrix is generally symmetric. We show that while, in this case, the shift matrix can be converted to satisfy the original shift-enabled condition, the converted matrix is not associated with the original graph, that is, it does not capture anymore the structure of the graph signal. We show via a counterexample, that a non-shift-enabled matrix cannot be converted to a shift-enabled one and still maintain the topological structure of the underlying graph, which is necessary to facilitate localized signal processing.
△ Less
Submitted 22 May, 2019; v1 submitted 30 October, 2018;
originally announced October 2018.
-
Optimal detection and error exponents for hidden multi-state processes via random duration model approach
Authors:
Dragana Bajović,
Kanghang He,
Lina Stanković,
Dejan Vukobratović,
Vladimir Stanković
Abstract:
We study detection of random signals corrupted by noise that over time switch their values (states) from a finite set of possible values, where the switchings occur at unknown points in time. We model such signals by means of a random duration model that to each possible state assigns a probability mass function which controls the statistics of durations of that state occurrences. Assuming two pos…
▽ More
We study detection of random signals corrupted by noise that over time switch their values (states) from a finite set of possible values, where the switchings occur at unknown points in time. We model such signals by means of a random duration model that to each possible state assigns a probability mass function which controls the statistics of durations of that state occurrences. Assuming two possible signal states and Gaussian noise, we derive optimal likelihood ratio test and show that it has a computationally tractable form of a matrix product, with the number of matrices involved in the product being the number of process observations. Each matrix involved in the product is of dimension equal to the sum of durations spreads of the two states, and it can be decomposed as a product of a diagonal random matrix controlled by the process observations and a sparse constant matrix which governs transitions in the sequence of states. Using this result, we show that the Neyman-Pearson error exponent is equal to the top Lyapunov exponent for the corresponding random matrices. Using theory of large deviations, we derive a lower bound on the error exponent. Finally, we show that this bound is tight by means of numerical simulations.
△ Less
Submitted 25 December, 2017;
originally announced December 2017.
-
Shift-enabled graphs: Graphs where shift-invariant filters are representable as polynomials of shift operations
Authors:
Liyan Chen,
Samuel Cheng,
Vlandimir Stankovic,
Lina Stankovic
Abstract:
In digital signal processing, shift-invariant filters can be represented as a polynomial expansion of a shift operation,that is, the Z-transform representation. When extended to graph signal processing (GSP), this would mean that a shift-invariant graph filter can be represented as a polynomial of the adjacency (shift) matrix of the graph. However, the characteristic and minimum polynomials of the…
▽ More
In digital signal processing, shift-invariant filters can be represented as a polynomial expansion of a shift operation,that is, the Z-transform representation. When extended to graph signal processing (GSP), this would mean that a shift-invariant graph filter can be represented as a polynomial of the adjacency (shift) matrix of the graph. However, the characteristic and minimum polynomials of the adjacency matrix must be identical for the property to hold. While it has been suggested that this condition might be ignored as it is always possible to find a polynomial transform to represent the original adjacency matrix by another adjacency matrix that satisfies the condition, this letter shows that a filter that is shift invariant in terms of the original graph may not be shift invariant anymore under the modified graph and vice versa. We introduce the notion of "shift-enabled graph" for graphs that satisfy the aforementioned condition, and present a concrete example of a graph that is not "shift-enabled" and a shift-invariant filter that is not a polynomial of the shift operation matrix. The result provides a deeper understanding of shift-invariant filters when applied in GSP and shows that further investigation of shift-enabled graphs is needed to make it applicable to practical scenarios.
△ Less
Submitted 10 June, 2018; v1 submitted 28 October, 2017;
originally announced October 2017.
-
Conditions for Unique Reconstruction of Sparse Signals Using Compressive Sensing Methods
Authors:
Ljubisa Stankovic,
Milos Dakovic,
Srdjan Stankovic,
Irena Orovic
Abstract:
A signal is sparse in one of its representation domain if the number of nonzero coefficients in that domain is much smaller than the total number of coefficients. Sparse signals can be reconstructed from a very reduced set of measurements/observations. The topic of this paper are conditions for the unique reconstruction of sparse signals from a reduced set of observations. After the basic definiti…
▽ More
A signal is sparse in one of its representation domain if the number of nonzero coefficients in that domain is much smaller than the total number of coefficients. Sparse signals can be reconstructed from a very reduced set of measurements/observations. The topic of this paper are conditions for the unique reconstruction of sparse signals from a reduced set of observations. After the basic definitions are introduced, the unique reconstruction conditions are reviewed using the spark, restricted isometry, and coherence of the measurement matrix. Uniqueness of the reconstruction of signals sparse in the discrete Fourier domain (DFT), as the most important signal transformation domain, is considered as well.
△ Less
Submitted 16 June, 2017;
originally announced June 2017.
-
Reconstruction of Sparse and Nonsparse Signals from a Reduced Set of Samples
Authors:
Ljubisa Stankovic,
Isidora Stankovic
Abstract:
Signals sparse in a transformation domain can be recovered from a reduced set of randomly positioned samples by using compressive sensing algorithms. Simple re- construction algorithms are presented in the first part of the paper. The missing samples manifest themselves as a noise in this reconstruction. Once the reconstruction conditions for a sparse signal are met and the reconstruction is achie…
▽ More
Signals sparse in a transformation domain can be recovered from a reduced set of randomly positioned samples by using compressive sensing algorithms. Simple re- construction algorithms are presented in the first part of the paper. The missing samples manifest themselves as a noise in this reconstruction. Once the reconstruction conditions for a sparse signal are met and the reconstruction is achieved, the noise due to missing samples does not influence the results in a direct way. It influences the possibility to recover a signal only. Additive input noise will remain in the resulting reconstructed signal. The accuracy of the recovery results is related to the additive input noise. Simple derivation of this relation is presented. If a reconstruction algorithm for a sparse signal is used in the reconstruction of a nonsparse signal then the noise due to missing samples will remain and behave as an additive input noise. An exact relation for the mean square error of this error is derived for the partial DFT matrix case in this paper and presented in form of a theorem. It takes into account very important fact that if all samples are available then the error will be zero, for both sparse and nonsparse recovered signals. Theory is illustrated and checked on statistical examples.
△ Less
Submitted 6 December, 2015;
originally announced December 2015.
-
Reconstruction of Randomly Sampled Sparse Signals Using an Adaptive Gradient Algorithm
Authors:
Ljubisa Stankovic,
Milos Dakovic
Abstract:
Sparse signals can be recovered from a reduced set of samples by using compressive sensing algorithms. In common methods the signal is recovered in the sparse domain. A method for the reconstruction of sparse signal which reconstructs the remaining missing samples/measurements is recently proposed. The available samples are fixed, while the missing samples are considered as minimization variables.…
▽ More
Sparse signals can be recovered from a reduced set of samples by using compressive sensing algorithms. In common methods the signal is recovered in the sparse domain. A method for the reconstruction of sparse signal which reconstructs the remaining missing samples/measurements is recently proposed. The available samples are fixed, while the missing samples are considered as minimization variables. Recovery of missing samples/measurements is done using an adaptive gradient-based algorithm in the time domain. A new criterion for the parameter adaptation in this algorithm, based on the gradient direction angles, is proposed. It improves the algorithm computational efficiency. A theorem for the uniqueness of the recovered signal for given set of missing samples (reconstruction variables) is presented. The case when available samples are a random subset of a uniformly or nonuniformly sampled signal is considered in this paper. A recalculation procedure is used to reconstruct the nonuniformly sampled signal. The methods are illustrated on statistical examples.
△ Less
Submitted 26 April, 2015; v1 submitted 1 December, 2014;
originally announced December 2014.
-
Polynomial Fourier Domain as a Domain of Signal Sparsity
Authors:
Srdjan Stankovic,
Irena Orovic,
Ljubisa Stankovic
Abstract:
A compressive sensing (CS) reconstruction method for polynomial phase signals is proposed in this paper. It relies on the Polynomial Fourier transform, which is used to establish a relationship between the observation and sparsity domain. Polynomial phase signals are not sparse in commonly used domains such as Fourier or wavelet domain. Therefore, for polynomial phase signals standard CS algorithm…
▽ More
A compressive sensing (CS) reconstruction method for polynomial phase signals is proposed in this paper. It relies on the Polynomial Fourier transform, which is used to establish a relationship between the observation and sparsity domain. Polynomial phase signals are not sparse in commonly used domains such as Fourier or wavelet domain. Therefore, for polynomial phase signals standard CS algorithms applied in these transformation domains cannot provide satisfactory results. In that sense, the Polynomial Fourier transform is used to ensure sparsity. The proposed approach is generalized using time-frequency representations obtained by the Local Polynomial Fourier transform (LPFT). In particular, the first-order LPFT can produce linear time-frequency representation for chirps. It provides revealing signal local behavior, which leads to sparse representation. The theory is illustrated on examples.
△ Less
Submitted 1 March, 2015; v1 submitted 24 October, 2014;
originally announced November 2014.
-
On the ISAR Image Analysis and Recovery with Unavailable or Heavily Corrupted Data
Authors:
Ljubisa Stankovic
Abstract:
Common ISAR radar images and signals can be reconstructed from much fewer samples than the sampling theorem requires since they are usually sparse. Unavailable randomly positioned samples can result from heavily corrupted parts of the signal. Since these samples can be omitted and declared as unavailable, the application of the compressive sensing methods in the recovery of heavily corrupted signa…
▽ More
Common ISAR radar images and signals can be reconstructed from much fewer samples than the sampling theorem requires since they are usually sparse. Unavailable randomly positioned samples can result from heavily corrupted parts of the signal. Since these samples can be omitted and declared as unavailable, the application of the compressive sensing methods in the recovery of heavily corrupted signal and radar images is possible. A\ simple direct method for the recovery of unavailable signal samples and the calculation of the restored ISAR image is reviewed. An analysis of the noise influence is performed. For fast maneuvering ISAR targets the sparsity property is lost since the ISAR image is blurred. A nonparametric quadratic time-frequency representations based method is used to restore the ISAR image sparsity. However, the linear relation between the signal and the sparsity domain transformation is lost. A recently proposed gradient recovery algorithm is adapted for this kind of analysis. It does not require the linear relation of the signal and its sparsity domain transformation in the process of unavailable data recovery. The presented methods and results are tested on several numerical examples proving the expected accuracy and improvements.
△ Less
Submitted 1 June, 2014;
originally announced June 2014.
-
Adaptive Variable Step Algorithm for Missing Samples Recovery in Sparse Signals
Authors:
Ljubisa Stankovic,
Milos Dakovic,
Stefan Vujovic
Abstract:
Recovery of arbitrarily positioned samples that are missing in sparse signals recently attracted significant research interest. Sparse signals with heavily corrupted arbitrary positioned samples could be analyzed in the same way as compressive sensed signals by omitting the corrupted samples and considering them as unavailable during the recovery process. The reconstruction of missing samples is d…
▽ More
Recovery of arbitrarily positioned samples that are missing in sparse signals recently attracted significant research interest. Sparse signals with heavily corrupted arbitrary positioned samples could be analyzed in the same way as compressive sensed signals by omitting the corrupted samples and considering them as unavailable during the recovery process. The reconstruction of missing samples is done by using one of the well known reconstruction algorithms. In this paper we will propose a very simple and efficient adaptive variable step algorithm, applied directly to the concentration measures, without reformulating the reconstruction problem within the standard linear programming form. Direct application of the gradient approach to the nondifferentiable forms of measures lead us to introduce a variable step size algorithm. A criterion for changing adaptive algorithm parameters is presented. The results are illustrated on the examples with sparse signals, including approximately sparse signals and noisy sparse signals.
△ Less
Submitted 23 September, 2013;
originally announced September 2013.
