-
Analysis of Sparse Recovery Algorithms via the Replica Method
Authors:
Ali Bereyhi,
Ralf R. Müller,
Hermann Schulz-Baldes
Abstract:
This manuscript goes through the fundamental connections between statistical mechanics and estimation theory by focusing on the particular problem of compressive sensing. We first show that the asymptotic analysis of a sparse recovery algorithm is mathematically equivalent to the problem of calculating the free energy of a spin glass in the thermodynamic limit. We then use the replica method from…
▽ More
This manuscript goes through the fundamental connections between statistical mechanics and estimation theory by focusing on the particular problem of compressive sensing. We first show that the asymptotic analysis of a sparse recovery algorithm is mathematically equivalent to the problem of calculating the free energy of a spin glass in the thermodynamic limit. We then use the replica method from statistical mechanics to evaluate the performance in the asymptotic regime. The asymptotic results have several applications in communications and signal processing. We briefly go through two instances of these applications: Characterization of joint sparse recovery algorithms used in distributed compressive sensing, and tuning of receivers employed for detection of spatially modulated signals.
△ Less
Submitted 19 December, 2022;
originally announced December 2022.
-
Secure Coding via Gaussian Random Fields
Authors:
Ali Bereyhi,
Bruno Loureiro,
Florent Krzakala,
Ralf R. Müller,
Hermann Schulz-Baldes
Abstract:
Inverse probability problems whose generative models are given by strictly nonlinear Gaussian random fields show the all-or-nothing behavior: There exists a critical rate at which Bayesian inference exhibits a phase transition. Below this rate, the optimal Bayesian estimator recovers the data perfectly, and above it the recovered data becomes uncorrelated. This study uses the replica method from t…
▽ More
Inverse probability problems whose generative models are given by strictly nonlinear Gaussian random fields show the all-or-nothing behavior: There exists a critical rate at which Bayesian inference exhibits a phase transition. Below this rate, the optimal Bayesian estimator recovers the data perfectly, and above it the recovered data becomes uncorrelated. This study uses the replica method from the theory of spin glasses to show that this critical rate is the channel capacity. This interesting finding has a particular application to the problem of secure transmission: A strictly nonlinear Gaussian random field along with random binning can be used to securely encode a confidential message in a wiretap channel. Our large-system characterization demonstrates that this secure coding scheme asymptotically achieves the secrecy capacity of the Gaussian wiretap channel.
△ Less
Submitted 18 May, 2022;
originally announced May 2022.
-
Bayesian Inference with Nonlinear Generative Models: Comments on Secure Learning
Authors:
Ali Bereyhi,
Bruno Loureiro,
Florent Krzakala,
Ralf R. Müller,
Hermann Schulz-Baldes
Abstract:
Unlike the classical linear model, nonlinear generative models have been addressed sparsely in the literature of statistical learning. This work aims to bringing attention to these models and their secrecy potential. To this end, we invoke the replica method to derive the asymptotic normalized cross entropy in an inverse probability problem whose generative model is described by a Gaussian random…
▽ More
Unlike the classical linear model, nonlinear generative models have been addressed sparsely in the literature of statistical learning. This work aims to bringing attention to these models and their secrecy potential. To this end, we invoke the replica method to derive the asymptotic normalized cross entropy in an inverse probability problem whose generative model is described by a Gaussian random field with a generic covariance function. Our derivations further demonstrate the asymptotic statistical decoupling of the Bayesian estimator and specify the decoupled setting for a given nonlinear model.
The replica solution depicts that strictly nonlinear models establish an all-or-nothing phase transition: There exists a critical load at which the optimal Bayesian inference changes from perfect to an uncorrelated learning. Based on this finding, we design a new secure coding scheme which achieves the secrecy capacity of the wiretap channel. This interesting result implies that strictly nonlinear generative models are perfectly secured without any secure coding. We justify this latter statement through the analysis of an illustrative model for perfectly secure and reliable inference.
△ Less
Submitted 13 July, 2022; v1 submitted 19 January, 2022;
originally announced January 2022.
-
Replica Symmetry Breaking in Compressive Sensing
Authors:
Ali Bereyhi,
Ralf Müller,
Hermann Schulz-Baldes
Abstract:
For noisy compressive sensing systems, the asymptotic distortion with respect to an arbitrary distortion function is determined when a general class of least-square based reconstruction schemes is employed. The sampling matrix is considered to belong to a large ensemble of random matrices including i.i.d. and projector matrices, and the source vector is assumed to be i.i.d. with a desired distribu…
▽ More
For noisy compressive sensing systems, the asymptotic distortion with respect to an arbitrary distortion function is determined when a general class of least-square based reconstruction schemes is employed. The sampling matrix is considered to belong to a large ensemble of random matrices including i.i.d. and projector matrices, and the source vector is assumed to be i.i.d. with a desired distribution. We take a statistical mechanical approach by representing the asymptotic distortion as a macroscopic parameter of a spin glass and employing the replica method for the large-system analysis. In contrast to earlier studies, we evaluate the general replica ansatz which includes the RS ansatz as well as RSB. The generality of the solution enables us to study the impact of symmetry breaking. Our numerical investigations depict that for the reconstruction scheme with the "zero-norm" penalty function, the RS fails to predict the asymptotic distortion for relatively large compression rates; however, the one-step RSB ansatz gives a valid prediction of the performance within a larger regime of compression rates.
△ Less
Submitted 26 April, 2017;
originally announced April 2017.
-
Statistical Mechanics of MAP Estimation: General Replica Ansatz
Authors:
Ali Bereyhi,
Ralf R. Müller,
Hermann Schulz-Baldes
Abstract:
The large-system performance of MAP estimation is studied considering a general distortion function when the observation vector is received through a linear system with additive white Gaussian noise. The analysis considers the system matrix to be chosen from the large class of rotationally invariant random matrices. We take a statistical mechanical approach by introducing a spin glass correspondin…
▽ More
The large-system performance of MAP estimation is studied considering a general distortion function when the observation vector is received through a linear system with additive white Gaussian noise. The analysis considers the system matrix to be chosen from the large class of rotationally invariant random matrices. We take a statistical mechanical approach by introducing a spin glass corresponding to the estimator, and employing the replica method for the large-system analysis. In contrast to earlier replica based studies, our analysis evaluates the general replica ansatz of the corresponding spin glass and determines the asymptotic distortion of the estimator for any structure of the replica correlation matrix. Consequently, the replica symmetric as well as the Replica Symmetry (RS) breaking ansatz with $b$ steps of breaking is deduced from the given general replica ansatz. The generality of our distortion function lets us derive a more general form of the MAP decoupling principle. Based on the general replica ansatz, we show that for any structure of the replica correlation matrix, the vector-valued system decouples into a bank of equivalent decoupled linear systems followed by MAP estimators. The structure of the decoupled linear system is further studied under both the RS and the Replica Symmetry Breaking (RSB) assumptions. For $b$ steps of RSB, the decoupled system is found to be an additive system with a noise term given as the sum of an independent Gaussian random variable with $b$ correlated impairment terms. As an application of our study, we investigate large compressive sensing systems by considering the $\ell_p$ minimization recovery schemes. Our numerical investigations show that the replica symmetric ansatz for $\ell_0$ norm recovery fails to give an accurate approximation of the mean square error as the compression rate grows, and therefore, the RSB ansätze are needed.
△ Less
Submitted 23 October, 2017; v1 submitted 6 December, 2016;
originally announced December 2016.
-
RSB Decoupling Property of MAP Estimators
Authors:
Ali Bereyhi,
Ralf R. Müller,
Hermann Schulz-Baldes
Abstract:
The large-system decoupling property of a MAP estimator is studied when it estimates the i.i.d. vector $\boldsymbol{x}$ from the observation $\boldsymbol{y}=\mathbf{A}\boldsymbol{x}+\boldsymbol{z}$ with $\mathbf{A}$ being chosen from a wide range of matrix ensembles, and the noise vector $\boldsymbol{z}$ being i.i.d. and Gaussian. Using the replica method, we show that the marginal joint distribut…
▽ More
The large-system decoupling property of a MAP estimator is studied when it estimates the i.i.d. vector $\boldsymbol{x}$ from the observation $\boldsymbol{y}=\mathbf{A}\boldsymbol{x}+\boldsymbol{z}$ with $\mathbf{A}$ being chosen from a wide range of matrix ensembles, and the noise vector $\boldsymbol{z}$ being i.i.d. and Gaussian. Using the replica method, we show that the marginal joint distribution of any two corresponding input and output symbols converges to a deterministic distribution which describes the input-output distribution of a single user system followed by a MAP estimator. Under the $b$RSB assumption, the single user system is a scalar channel with additive noise where the noise term is given by the sum of an independent Gaussian random variable and $b$ correlated interference terms. As the $b$RSB assumption reduces to RS, the interference terms vanish which results in the formerly studied RS decoupling principle.
△ Less
Submitted 8 November, 2016;
originally announced November 2016.
