-
Bregman Plug-and-Play Priors
Authors:
Abdullah H. Al-Shabili,
Xiaojian Xu,
Ivan Selesnick,
Ulugbek S. Kamilov
Abstract:
The past few years have seen a surge of activity around integration of deep learning networks and optimization algorithms for solving inverse problems. Recent work on plug-and-play priors (PnP), regularization by denoising (RED), and deep unfolding has shown the state-of-the-art performance of such integration in a variety of applications. However, the current paradigm for designing such algorithm…
▽ More
The past few years have seen a surge of activity around integration of deep learning networks and optimization algorithms for solving inverse problems. Recent work on plug-and-play priors (PnP), regularization by denoising (RED), and deep unfolding has shown the state-of-the-art performance of such integration in a variety of applications. However, the current paradigm for designing such algorithms is inherently Euclidean, due to the usage of the quadratic norm within the projection and proximal operators. We propose to broaden this perspective by considering a non-Euclidean setting based on the more general Bregman distance. Our new Bregman Proximal Gradient Method variant of PnP (PnP-BPGM) and Bregman Steepest Descent variant of RED (RED-BSD) replace the traditional updates in PnP and RED from the quadratic norms to more general Bregman distance. We present a theoretical convergence result for PnP-BPGM and demonstrate the effectiveness of our algorithms on Poisson linear inverse problems.
△ Less
Submitted 4 February, 2022;
originally announced February 2022.
-
Image Fusion via Sparse Regularization with Non-Convex Penalties
Authors:
Nantheera Anantrasirichai,
Rencheng Zheng,
Ivan Selesnick,
Alin Achim
Abstract:
The L1 norm regularized least squares method is often used for finding sparse approximate solutions and is widely used in 1-D signal restoration. Basis pursuit denoising (BPD) performs noise reduction in this way. However, the shortcoming of using L1 norm regularization is the underestimation of the true solution. Recently, a class of non-convex penalties have been proposed to improve this situati…
▽ More
The L1 norm regularized least squares method is often used for finding sparse approximate solutions and is widely used in 1-D signal restoration. Basis pursuit denoising (BPD) performs noise reduction in this way. However, the shortcoming of using L1 norm regularization is the underestimation of the true solution. Recently, a class of non-convex penalties have been proposed to improve this situation. This kind of penalty function is non-convex itself, but preserves the convexity property of the whole cost function. This approach has been confirmed to offer good performance in 1-D signal denoising. This paper demonstrates the aforementioned method to 2-D signals (images) and applies it to multisensor image fusion. The problem is posed as an inverse one and a corresponding cost function is judiciously designed to include two data attachment terms. The whole cost function is proved to be convex upon suitably choosing the non-convex penalty, so that the cost function minimization can be tackled by convex optimization approaches, which comprise simple computations. The performance of the proposed method is benchmarked against a number of state-of-the-art image fusion techniques and superior performance is demonstrated both visually and in terms of various assessment measures.
△ Less
Submitted 29 January, 2020; v1 submitted 23 May, 2019;
originally announced May 2019.
-
Improved Sparse Low-Rank Matrix Estimation
Authors:
Ankit Parekh,
Ivan W. Selesnick
Abstract:
We address the problem of estimating a sparse low-rank matrix from its noisy observation. We propose an objective function consisting of a data-fidelity term and two parameterized non-convex penalty functions. Further, we show how to set the parameters of the non-convex penalty functions, in order to ensure that the objective function is strictly convex. The proposed objective function better esti…
▽ More
We address the problem of estimating a sparse low-rank matrix from its noisy observation. We propose an objective function consisting of a data-fidelity term and two parameterized non-convex penalty functions. Further, we show how to set the parameters of the non-convex penalty functions, in order to ensure that the objective function is strictly convex. The proposed objective function better estimates sparse low-rank matrices than a convex method which utilizes the sum of the nuclear norm and the $\ell_1$ norm. We derive an algorithm (as an instance of ADMM) to solve the proposed problem, and guarantee its convergence provided the scalar augmented Lagrangian parameter is set appropriately. We demonstrate the proposed method for denoising an audio signal and an adjacency matrix representing protein interactions in the `Escherichia coli' bacteria.
△ Less
Submitted 12 April, 2017; v1 submitted 29 April, 2016;
originally announced May 2016.
-
Repetitive Transients Extraction Algorithm for Detecting Bearing Faults
Authors:
Wangpeng He,
Yin Ding,
Yanyang Zi,
Ivan W. Selesnick
Abstract:
This paper addresses the problem of noise reduction with simultaneous components extrac- tion in vibration signals for faults diagnosis of bearing. The observed vibration signal is modeled as a summation of two components contaminated by noise, and each component composes of repetitive transients. To extract the two components simultaneously, an approach by solving an optimization problem is propo…
▽ More
This paper addresses the problem of noise reduction with simultaneous components extrac- tion in vibration signals for faults diagnosis of bearing. The observed vibration signal is modeled as a summation of two components contaminated by noise, and each component composes of repetitive transients. To extract the two components simultaneously, an approach by solving an optimization problem is proposed in this paper. The problem adopts convex sparsity-based regularization scheme for decomposition, and non-convex regularization is used to further promote the sparsity but preserving the global convexity. A synthetic example is presented to illustrate the performance of the proposed approach for repetitive feature extraction. The performance and effectiveness of the proposed method are further demonstrated by applying to compound faults and single fault diagnosis of a locomotive bearing. The results show the proposed approach can effectively extract the features of outer and inner race defects.
△ Less
Submitted 6 August, 2016; v1 submitted 11 January, 2016;
originally announced January 2016.
-
Enhanced Low-Rank Matrix Approximation
Authors:
Ankit Parekh,
Ivan W. Selesnick
Abstract:
This letter proposes to estimate low-rank matrices by formulating a convex optimization problem with non-convex regularization. We employ parameterized non-convex penalty functions to estimate the non-zero singular values more accurately than the nuclear norm. A closed-form solution for the global optimum of the proposed objective function (sum of data fidelity and the non-convex regularizer) is a…
▽ More
This letter proposes to estimate low-rank matrices by formulating a convex optimization problem with non-convex regularization. We employ parameterized non-convex penalty functions to estimate the non-zero singular values more accurately than the nuclear norm. A closed-form solution for the global optimum of the proposed objective function (sum of data fidelity and the non-convex regularizer) is also derived. The solution reduces to singular value thresholding method as a special case. The proposed method is demonstrated for image denoising.
△ Less
Submitted 12 April, 2016; v1 submitted 5 November, 2015;
originally announced November 2015.
-
Detection of Faults in Rotating Machinery Using Periodic Time-Frequency Sparsity
Authors:
Yin Ding,
Wangpeng He,
Binqiang Chen,
Yanyang Zi,
Ivan W. Selesnick
Abstract:
This paper addresses the problem of extracting periodic oscillatory features in vibration sig- nals for detecting faults in rotating machinery. To extract the feature, we propose an approach in the short-time Fourier transform (STFT) domain where the periodic oscillatory feature man- ifests itself as a relatively sparse grid. To estimate the sparse grid, we formulate an optimization problem using…
▽ More
This paper addresses the problem of extracting periodic oscillatory features in vibration sig- nals for detecting faults in rotating machinery. To extract the feature, we propose an approach in the short-time Fourier transform (STFT) domain where the periodic oscillatory feature man- ifests itself as a relatively sparse grid. To estimate the sparse grid, we formulate an optimization problem using customized binary weights in the regularizer, where the weights are formulated to promote periodicity. In order to solve the proposed optimization problem, we develop an algorithm called augmented Lagrangian majorization-minimization algorithm, which combines the split augmented Lagrangian shrinkage algorithm (SALSA) with majorization-minimization (MM), and is guaranteed to converge for both convex and non-convex formulation. As examples, the proposed approach is applied to simulated data, and used as a tool for diagnosing faults in bearings and gearboxes for real data, and compared to some state-of-the-art methods. The results show the proposed approach can effectively detect and extract the periodical oscillatory features.
△ Less
Submitted 30 July, 2016; v1 submitted 2 November, 2015;
originally announced November 2015.
-
Sparsity-based Algorithm for Detecting Faults in Rotating Machines
Authors:
Wangpeng He,
Yin Ding,
Yanyang Zi,
Ivan W. Selesnick
Abstract:
This paper addresses the detection of periodic transients in vibration signals for detecting faults in rotating machines. For this purpose, we present a method to estimate periodic-group-sparse signals in noise. The method is based on the formulation of a convex optimization problem. A fast iterative algorithm is given for its solution. A simulated signal is formulated to verify the performance of…
▽ More
This paper addresses the detection of periodic transients in vibration signals for detecting faults in rotating machines. For this purpose, we present a method to estimate periodic-group-sparse signals in noise. The method is based on the formulation of a convex optimization problem. A fast iterative algorithm is given for its solution. A simulated signal is formulated to verify the performance of the proposed approach for periodic feature extraction. The detection performance of comparative methods is compared with that of the proposed approach via RMSE values and receiver operating characteristic (ROC) curves. Finally, the proposed approach is applied to compound faults diagnosis of motor bearings. The non-stationary vibration data were acquired from a SpectraQuest's machinery fault simulator. The processed results show the proposed approach can effectively detect and extract the useful features of bearing outer race and inner race defect.
△ Less
Submitted 30 October, 2015;
originally announced November 2015.
-
Sparsity-based Correction of Exponential Artifacts
Authors:
Yin Ding,
Ivan W. Selesnick
Abstract:
This paper describes an exponential transient excision algorithm (ETEA). In biomedical time series analysis, e.g., in vivo neural recording and electrocorticography (ECoG), some measurement artifacts take the form of piecewise exponential transients. The proposed method is formulated as an unconstrained convex optimization problem, regularized by smoothed l1-norm penalty function, which can be sol…
▽ More
This paper describes an exponential transient excision algorithm (ETEA). In biomedical time series analysis, e.g., in vivo neural recording and electrocorticography (ECoG), some measurement artifacts take the form of piecewise exponential transients. The proposed method is formulated as an unconstrained convex optimization problem, regularized by smoothed l1-norm penalty function, which can be solved by majorization-minimization (MM) method. With a slight modification of the regularizer, ETEA can also suppress more irregular piecewise smooth artifacts, especially, ocular artifacts (OA) in electroencephalog- raphy (EEG) data. Examples of synthetic signal, EEG data, and ECoG data are presented to illustrate the proposed algorithms.
△ Less
Submitted 24 September, 2015;
originally announced September 2015.
-
Convex Denoising using Non-Convex Tight Frame Regularization
Authors:
Ankit Parekh,
Ivan W. Selesnick
Abstract:
This paper considers the problem of signal denoising using a sparse tight-frame analysis prior. The L1 norm has been extensively used as a regularizer to promote sparsity; however, it tends to under-estimate non-zero values of the underlying signal. To more accurately estimate non-zero values, we propose the use of a non-convex regularizer, chosen so as to ensure convexity of the objective functio…
▽ More
This paper considers the problem of signal denoising using a sparse tight-frame analysis prior. The L1 norm has been extensively used as a regularizer to promote sparsity; however, it tends to under-estimate non-zero values of the underlying signal. To more accurately estimate non-zero values, we propose the use of a non-convex regularizer, chosen so as to ensure convexity of the objective function. The convexity of the objective function is ensured by constraining the parameter of the non-convex penalty. We use ADMM to obtain a solution and show how to guarantee that ADMM converges to the global optimum of the objective function. We illustrate the proposed method for 1D and 2D signal denoising.
△ Less
Submitted 3 June, 2015; v1 submitted 3 April, 2015;
originally announced April 2015.
-
Image Restoration using Total Variation with Overlap** Group Sparsity
Authors:
Jun Liu,
Ting-Zhu Huang,
Ivan W. Selesnick,
Xiao-Guang Lv,
Po-Yu Chen
Abstract:
Image restoration is one of the most fundamental issues in imaging science. Total variation (TV) regularization is widely used in image restoration problems for its capability to preserve edges. In the literature, however, it is also well known for producing staircase-like artifacts. Usually, the high-order total variation (HTV) regularizer is an good option except its over-smoothing property. In…
▽ More
Image restoration is one of the most fundamental issues in imaging science. Total variation (TV) regularization is widely used in image restoration problems for its capability to preserve edges. In the literature, however, it is also well known for producing staircase-like artifacts. Usually, the high-order total variation (HTV) regularizer is an good option except its over-smoothing property. In this work, we study a minimization problem where the objective includes an usual $l_2$ data-fidelity term and an overlap** group sparsity total variation regularizer which can avoid staircase effect and allow edges preserving in the restored image. We also proposed a fast algorithm for solving the corresponding minimization problem and compare our method with the state-of-the-art TV based methods and HTV based method. The numerical experiments illustrate the efficiency and effectiveness of the proposed method in terms of PSNR, relative error and computing time.
△ Less
Submitted 19 October, 2013; v1 submitted 13 October, 2013;
originally announced October 2013.
-
Group-Sparse Signal Denoising: Non-Convex Regularization, Convex Optimization
Authors:
Po-Yu Chen,
Ivan W. Selesnick
Abstract:
Convex optimization with sparsity-promoting convex regularization is a standard approach for estimating sparse signals in noise. In order to promote sparsity more strongly than convex regularization, it is also standard practice to employ non-convex optimization. In this paper, we take a third approach. We utilize a non-convex regularization term chosen such that the total cost function (consistin…
▽ More
Convex optimization with sparsity-promoting convex regularization is a standard approach for estimating sparse signals in noise. In order to promote sparsity more strongly than convex regularization, it is also standard practice to employ non-convex optimization. In this paper, we take a third approach. We utilize a non-convex regularization term chosen such that the total cost function (consisting of data consistency and regularization terms) is convex. Therefore, sparsity is more strongly promoted than in the standard convex formulation, but without sacrificing the attractive aspects of convex optimization (unique minimum, robust algorithms, etc.). We use this idea to improve the recently developed 'overlap** group shrinkage' (OGS) algorithm for the denoising of group-sparse signals. The algorithm is applied to the problem of speech enhancement with favorable results in terms of both SNR and perceptual quality.
△ Less
Submitted 30 November, 2013; v1 submitted 22 August, 2013;
originally announced August 2013.
-
Translation-Invariant Shrinkage/Thresholding of Group Sparse Signals
Authors:
Po-Yu Chen,
Ivan W. Selesnick
Abstract:
This paper addresses signal denoising when large-amplitude coefficients form clusters (groups). The L1-norm and other separable sparsity models do not capture the tendency of coefficients to cluster (group sparsity). This work develops an algorithm, called 'overlap** group shrinkage' (OGS), based on the minimization of a convex cost function involving a group-sparsity promoting penalty function.…
▽ More
This paper addresses signal denoising when large-amplitude coefficients form clusters (groups). The L1-norm and other separable sparsity models do not capture the tendency of coefficients to cluster (group sparsity). This work develops an algorithm, called 'overlap** group shrinkage' (OGS), based on the minimization of a convex cost function involving a group-sparsity promoting penalty function. The groups are fully overlap** so the denoising method is translation-invariant and blocking artifacts are avoided. Based on the principle of majorization-minimization (MM), we derive a simple iterative minimization algorithm that reduces the cost function monotonically. A procedure for setting the regularization parameter, based on attenuating the noise to a specified level, is also described. The proposed approach is illustrated on speech enhancement, wherein the OGS approach is applied in the short-time Fourier transform (STFT) domain. The denoised speech produced by OGS does not suffer from musical noise.
△ Less
Submitted 29 March, 2013;
originally announced April 2013.
-
Sparse Frequency Analysis with Sparse-Derivative Instantaneous Amplitude and Phase Functions
Authors:
Yin Ding,
Ivan W. Selesnick
Abstract:
This paper addresses the problem of expressing a signal as a sum of frequency components (sinusoids) wherein each sinusoid may exhibit abrupt changes in its amplitude and/or phase. The Fourier transform of a narrow-band signal, with a discontinuous amplitude and/or phase function, exhibits spectral and temporal spreading. The proposed method aims to avoid such spreading by explicitly modeling the…
▽ More
This paper addresses the problem of expressing a signal as a sum of frequency components (sinusoids) wherein each sinusoid may exhibit abrupt changes in its amplitude and/or phase. The Fourier transform of a narrow-band signal, with a discontinuous amplitude and/or phase function, exhibits spectral and temporal spreading. The proposed method aims to avoid such spreading by explicitly modeling the signal of interest as a sum of sinusoids with time-varying amplitudes. So as to accommodate abrupt changes, it is further assumed that the amplitude/phase functions are approximately piecewise constant (i.e., their time-derivatives are sparse). The proposed method is based on a convex variational (optimization) approach wherein the total variation (TV) of the amplitude functions are regularized subject to a perfect (or approximate) reconstruction constraint. A computationally efficient algorithm is derived based on convex optimization techniques. The proposed technique can be used to perform band-pass filtering that is relatively insensitive to narrow-band amplitude/phase jumps present in data, which normally pose a challenge (due to transients, leakage, etc.). The method is illustrated using both synthetic signals and human EEG data for the purpose of band-pass filtering and the estimation of phase synchrony indexes.
△ Less
Submitted 26 February, 2013;
originally announced February 2013.
-
Sparse Signal Estimation by Maximally Sparse Convex Optimization
Authors:
Ivan W. Selesnick,
Ilker Bayram
Abstract:
This paper addresses the problem of sparsity penalized least squares for applications in sparse signal processing, e.g. sparse deconvolution. This paper aims to induce sparsity more strongly than L1 norm regularization, while avoiding non-convex optimization. For this purpose, this paper describes the design and use of non-convex penalty functions (regularizers) constrained so as to ensure the con…
▽ More
This paper addresses the problem of sparsity penalized least squares for applications in sparse signal processing, e.g. sparse deconvolution. This paper aims to induce sparsity more strongly than L1 norm regularization, while avoiding non-convex optimization. For this purpose, this paper describes the design and use of non-convex penalty functions (regularizers) constrained so as to ensure the convexity of the total cost function, F, to be minimized. The method is based on parametric penalty functions, the parameters of which are constrained to ensure convexity of F. It is shown that optimal parameters can be obtained by semidefinite programming (SDP). This maximally sparse convex (MSC) approach yields maximally non-convex sparsity-inducing penalty functions constrained such that the total cost function, F, is convex. It is demonstrated that iterative MSC (IMSC) can yield solutions substantially more sparse than the standard convex sparsity-inducing approach, i.e., L1 norm minimization.
△ Less
Submitted 3 January, 2014; v1 submitted 22 February, 2013;
originally announced February 2013.
-
Compressed Sensing for Moving Imagery in Medical Imaging
Authors:
Cagdas Bilen,
Yao Wang,
Ivan Selesnick
Abstract:
Numerous applications in signal processing have benefited from the theory of compressed sensing which shows that it is possible to reconstruct signals sampled below the Nyquist rate when certain conditions are satisfied. One of these conditions is that there exists a known transform that represents the signal with a sufficiently small number of non-zero coefficients. However when the signal to be…
▽ More
Numerous applications in signal processing have benefited from the theory of compressed sensing which shows that it is possible to reconstruct signals sampled below the Nyquist rate when certain conditions are satisfied. One of these conditions is that there exists a known transform that represents the signal with a sufficiently small number of non-zero coefficients. However when the signal to be reconstructed is composed of moving images or volumes, it is challenging to form such regularization constraints with traditional transforms such as wavelets. In this paper, we present a motion compensating prior for such signals that is derived directly from the optical flow constraint and can utilize the motion information during compressed sensing reconstruction. Proposed regularization method can be used in a wide variety of applications involving compressed sensing and images or volumes of moving and deforming objects. It is also shown that it is possible to estimate the signal and the motion jointly or separately. Practical examples from magnetic resonance imaging has been presented to demonstrate the benefit of the proposed method.
△ Less
Submitted 26 March, 2012;
originally announced March 2012.
-
High Speed Compressed Sensing Reconstruction in Dynamic Parallel MRI Using Augmented Lagrangian and Parallel Processing
Authors:
Cagdas Bilen,
Yao Wang,
Ivan Selesnick
Abstract:
Magnetic Resonance Imaging (MRI) is one of the fields that the compressed sensing theory is well utilized to reduce the scan time significantly leading to faster imaging or higher resolution images. It has been shown that a small fraction of the overall measurements are sufficient to reconstruct images with the combination of compressed sensing and parallel imaging. Various reconstruction algorith…
▽ More
Magnetic Resonance Imaging (MRI) is one of the fields that the compressed sensing theory is well utilized to reduce the scan time significantly leading to faster imaging or higher resolution images. It has been shown that a small fraction of the overall measurements are sufficient to reconstruct images with the combination of compressed sensing and parallel imaging. Various reconstruction algorithms has been proposed for compressed sensing, among which Augmented Lagrangian based methods have been shown to often perform better than others for many different applications. In this paper, we propose new Augmented Lagrangian based solutions to the compressed sensing reconstruction problem with analysis and synthesis prior formulations. We also propose a computational method which makes use of properties of the sampling pattern to significantly improve the speed of the reconstruction for the proposed algorithms in Cartesian sampled MRI. The proposed algorithms are shown to outperform earlier methods especially for the case of dynamic MRI for which the transfer function tends to be a very large matrix and significantly ill conditioned. It is also demonstrated that the proposed algorithm can be accelerated much further than other methods in case of a parallel implementation with graphics processing units (GPUs).
△ Less
Submitted 20 March, 2012;
originally announced March 2012.