Showing 1–27 of 27 results for author: Saab, R

Tuning In: Analysis of Audio Classifier Performance in Clinical Settings with Limited Data

Authors: Hamza Mahdi, Eptehal Nashnoush, Rami Saab, Arjun Balachandar, Rishit Dagli, Lucas X. Perri, Houman Khosravani

Abstract: This study assesses deep learning models for audio classification in a clinical setting with the constraint of small datasets reflecting real-world prospective data collection. We analyze CNNs, including DenseNet and ConvNeXt, alongside transformer models like ViT, SWIN, and AST, and compare them against pre-trained audio models such as YAMNet and VGGish. Our method highlights the benefits of pre-… ▽ More This study assesses deep learning models for audio classification in a clinical setting with the constraint of small datasets reflecting real-world prospective data collection. We analyze CNNs, including DenseNet and ConvNeXt, alongside transformer models like ViT, SWIN, and AST, and compare them against pre-trained audio models such as YAMNet and VGGish. Our method highlights the benefits of pre-training on large datasets before fine-tuning on specific clinical data. We prospectively collected two first-of-their-kind patient audio datasets from stroke patients. We investigated various preprocessing techniques, finding that RGB and grayscale spectrogram transformations affect model performance differently based on the priors they learn from pre-training. Our findings indicate CNNs can match or exceed transformer models in small dataset contexts, with DenseNet-Contrastive and AST models showing notable performance. This study highlights the significance of incremental marginal gains through model selection, pre-training, and preprocessing in sound classification; this offers valuable insights for clinical diagnostics that rely on audio classification. △ Less

Submitted 5 April, 2024; v1 submitted 7 February, 2024; originally announced February 2024.

Comments: CHIL 2024

SPFQ: A Stochastic Algorithm and Its Error Analysis for Neural Network Quantization

Authors: **jie Zhang, Rayan Saab

Abstract: Quantization is a widely used compression method that effectively reduces redundancies in over-parameterized neural networks. However, existing quantization techniques for deep neural networks often lack a comprehensive error analysis due to the presence of non-convex loss functions and nonlinear activations. In this paper, we propose a fast stochastic algorithm for quantizing the weights of fully… ▽ More Quantization is a widely used compression method that effectively reduces redundancies in over-parameterized neural networks. However, existing quantization techniques for deep neural networks often lack a comprehensive error analysis due to the presence of non-convex loss functions and nonlinear activations. In this paper, we propose a fast stochastic algorithm for quantizing the weights of fully trained neural networks. Our approach leverages a greedy path-following mechanism in combination with a stochastic quantizer. Its computational complexity scales only linearly with the number of weights in the network, thereby enabling the efficient quantization of large networks. Importantly, we establish, for the first time, full-network error bounds, under an infinite alphabet condition and minimal assumptions on the weights and input data. As an application of this result, we prove that when quantizing a multi-layer network having Gaussian weights, the relative square quantization error exhibits a linear decay as the degree of over-parametrization increases. Furthermore, we demonstrate that it is possible to achieve error bounds equivalent to those obtained in the infinite alphabet case, using on the order of a mere $\log\log N$ bits per weight, where $N$ represents the largest number of neurons in a layer. △ Less

Submitted 19 September, 2023; originally announced September 2023.

A simple approach for quantizing neural networks

Authors: Johannes Maly, Rayan Saab

Abstract: In this short note, we propose a new method for quantizing the weights of a fully trained neural network. A simple deterministic pre-processing step allows us to quantize network layers via memoryless scalar quantization while preserving the network performance on given training data. On one hand, the computational complexity of this pre-processing slightly exceeds that of state-of-the-art algorit… ▽ More In this short note, we propose a new method for quantizing the weights of a fully trained neural network. A simple deterministic pre-processing step allows us to quantize network layers via memoryless scalar quantization while preserving the network performance on given training data. On one hand, the computational complexity of this pre-processing slightly exceeds that of state-of-the-art algorithms in the literature. On the other hand, our approach does not require any hyper-parameter tuning and, in contrast to previous methods, allows a plain analysis. We provide rigorous theoretical guarantees in the case of quantizing single network layers and show that the relative error decays with the number of parameters in the network if the training data behaves well, e.g., if it is sampled from suitable random distributions. The developed method also readily allows the quantization of deep networks by consecutive application to single layers. △ Less

Submitted 4 April, 2023; v1 submitted 7 September, 2022; originally announced September 2022.

Spectrally Adaptive Common Spatial Patterns

Authors: Mahta Mousavi, Eric Lybrand, Shuangquan Feng, Shuai Tang, Rayan Saab, Virginia de Sa

Abstract: The method of Common Spatial Patterns (CSP) is widely used for feature extraction of electroencephalography (EEG) data, such as in motor imagery brain-computer interface (BCI) systems. It is a data-driven method estimating a set of spatial filters so that the power of the filtered EEG signal is maximized for one motor imagery class and minimized for the other. This method, however, is prone to ove… ▽ More The method of Common Spatial Patterns (CSP) is widely used for feature extraction of electroencephalography (EEG) data, such as in motor imagery brain-computer interface (BCI) systems. It is a data-driven method estimating a set of spatial filters so that the power of the filtered EEG signal is maximized for one motor imagery class and minimized for the other. This method, however, is prone to overfitting and is known to suffer from poor generalization especially with limited calibration data. Additionally, due to the high heterogeneity in brain data and the non-stationarity of brain activity, CSP is usually trained for each user separately resulting in long calibration sessions or frequent re-calibrations that are tiring for the user. In this work, we propose a novel algorithm called Spectrally Adaptive Common Spatial Patterns (SACSP) that improves CSP by learning a temporal/spectral filter for each spatial filter so that the spatial filters are concentrated on the most relevant temporal frequencies for each user. We show the efficacy of SACSP in providing better generalizability and higher classification accuracy from calibration to online control compared to existing methods. Furthermore, we show that SACSP provides neurophysiologically relevant information about the temporal frequencies of the filtered signals. Our results highlight the differences in the motor imagery signal among BCI users as well as spectral differences in the signals generated for each class, and show the importance of learning robust user-specific features in a data-driven manner. △ Less

Submitted 9 February, 2022; originally announced February 2022.

Post-training Quantization for Neural Networks with Provable Guarantees

Authors: **jie Zhang, Yixuan Zhou, Rayan Saab

Abstract: While neural networks have been remarkably successful in a wide array of applications, implementing them in resource-constrained hardware remains an area of intense research. By replacing the weights of a neural network with quantized (e.g., 4-bit, or binary) counterparts, massive savings in computation cost, memory, and power consumption are attained. To that end, we generalize a post-training ne… ▽ More While neural networks have been remarkably successful in a wide array of applications, implementing them in resource-constrained hardware remains an area of intense research. By replacing the weights of a neural network with quantized (e.g., 4-bit, or binary) counterparts, massive savings in computation cost, memory, and power consumption are attained. To that end, we generalize a post-training neural-network quantization method, GPFQ, that is based on a greedy path-following mechanism. Among other things, we propose modifications to promote sparsity of the weights, and rigorously analyze the associated error. Additionally, our error analysis expands the results of previous work on GPFQ to handle general quantization alphabets, showing that for quantizing a single-layer network, the relative square error essentially decays linearly in the number of weights -- i.e., level of over-parametrization. Our result holds across a range of input distributions and for both fully-connected and convolutional architectures thereby also extending previous results. To empirically evaluate the method, we quantize several common architectures with few bits per weight, and test them on ImageNet, showing only minor loss of accuracy compared to unquantized models. We also demonstrate that standard modifications, such as bias correction and mixed precision quantization, further improve accuracy. △ Less

Submitted 15 January, 2023; v1 submitted 26 January, 2022; originally announced January 2022.

Comments: This paper was accepted by SIAM Journal on Mathematics of Data Science (SIMODS)

On the Number of Faces and Radii of Cells Induced by Gaussian Spherical Tessellations

Authors: Eric Lybrand, Anna Ma, Rayan Saab

Abstract: We study a geometric property related to spherical hyperplane tessellations in $\mathbb{R}^{d}$. We first consider a fixed $x$ on the Euclidean sphere and tessellations with $M \gg d$ hyperplanes passing through the origin having normal vectors distributed according to a Gaussian distribution. We show that with high probability there exists a subset of the hyperplanes whose cardinality is on the o… ▽ More We study a geometric property related to spherical hyperplane tessellations in $\mathbb{R}^{d}$. We first consider a fixed $x$ on the Euclidean sphere and tessellations with $M \gg d$ hyperplanes passing through the origin having normal vectors distributed according to a Gaussian distribution. We show that with high probability there exists a subset of the hyperplanes whose cardinality is on the order of $d\log(d)\log(M)$ such that the radius of the cell containing $x$ induced by these hyperplanes is bounded above by, up to constants, $d\log(d)\log(M)/M$. We extend this result to hold for all cells in the tessellation with high probability. Up to logarithmic terms, this upper bound matches the previously established lower bound of Goyal et al. (IEEE T. Inform. Theory 44(1):16-31, 1998). △ Less

Submitted 30 August, 2021; originally announced August 2021.

Sigma-Delta and Distributed Noise-Sha** Quantization Methods for Random Fourier Features

Authors: **jie Zhang, Harish Kannan, Alexander Cloninger, Rayan Saab

Abstract: We propose the use of low bit-depth Sigma-Delta and distributed noise-sha** methods for quantizing the Random Fourier features (RFFs) associated with shift-invariant kernels. We prove that our quantized RFFs -- even in the case of $1$-bit quantization -- allow a high accuracy approximation of the underlying kernels, and the approximation error decays at least polynomially fast as the dimension o… ▽ More We propose the use of low bit-depth Sigma-Delta and distributed noise-sha** methods for quantizing the Random Fourier features (RFFs) associated with shift-invariant kernels. We prove that our quantized RFFs -- even in the case of $1$-bit quantization -- allow a high accuracy approximation of the underlying kernels, and the approximation error decays at least polynomially fast as the dimension of the RFFs increases. We also show that the quantized RFFs can be further compressed, yielding an excellent trade-off between memory use and accuracy. Namely, the approximation error now decays exponentially as a function of the bits used. Moreover, we empirically show by testing the performance of our methods on several machine learning tasks that our method compares favorably to other state of the art quantization methods in this context. △ Less

Submitted 12 April, 2022; v1 submitted 4 June, 2021; originally announced June 2021.

A Greedy Algorithm for Quantizing Neural Networks

Authors: Eric Lybrand, Rayan Saab

Abstract: We propose a new computationally efficient method for quantizing the weights of pre- trained neural networks that is general enough to handle both multi-layer perceptrons and convolutional neural networks. Our method deterministically quantizes layers in an iterative fashion with no complicated re-training required. Specifically, we quantize each neuron, or hidden unit, using a greedy path-followi… ▽ More We propose a new computationally efficient method for quantizing the weights of pre- trained neural networks that is general enough to handle both multi-layer perceptrons and convolutional neural networks. Our method deterministically quantizes layers in an iterative fashion with no complicated re-training required. Specifically, we quantize each neuron, or hidden unit, using a greedy path-following algorithm. This simple algorithm is equivalent to running a dynamical system, which we prove is stable for quantizing a single-layer neural network (or, alternatively, for quantizing the first layer of a multi-layer network) when the training data are Gaussian. We show that under these assumptions, the quantization error decays with the width of the layer, i.e., its level of over-parametrization. We provide numerical experiments, on multi-layer networks, to illustrate the performance of our methods on MNIST and CIFAR10 data, as well as for quantizing the VGG16 network using ImageNet data. △ Less

Submitted 15 August, 2021; v1 submitted 29 October, 2020; originally announced October 2020.

Faster Binary Embeddings for Preserving Euclidean Distances

Authors: **jie Zhang, Rayan Saab

Abstract: We propose a fast, distance-preserving, binary embedding algorithm to transform a high-dimensional dataset $\mathcal{T}\subseteq\mathbb{R}^n$ into binary sequences in the cube $\{\pm 1\}^m$. When $\mathcal{T}$ consists of well-spread (i.e., non-sparse) vectors, our embedding method applies a stable noise-sha** quantization scheme to $A x$ where $A\in\mathbb{R}^{m\times n}$ is a sparse Gaussian r… ▽ More We propose a fast, distance-preserving, binary embedding algorithm to transform a high-dimensional dataset $\mathcal{T}\subseteq\mathbb{R}^n$ into binary sequences in the cube $\{\pm 1\}^m$. When $\mathcal{T}$ consists of well-spread (i.e., non-sparse) vectors, our embedding method applies a stable noise-sha** quantization scheme to $A x$ where $A\in\mathbb{R}^{m\times n}$ is a sparse Gaussian random matrix. This contrasts with most binary embedding methods, which usually use $x\mapsto \mathrm{sign}(Ax)$ for the embedding. Moreover, we show that Euclidean distances among the elements of $\mathcal{T}$ are approximated by the $\ell_1$ norm on the images of $\{\pm 1\}^m$ under a fast linear transformation. This again contrasts with standard methods, where the Hamming distance is used instead. Our method is both fast and memory efficient, with time complexity $O(m)$ and space complexity $O(m)$. Further, we prove that the method is accurate and its associated error is comparable to that of a continuous valued Johnson-Lindenstrauss embedding plus a quantization error that admits a polynomial decay as the embedding dimension $m$ increases. Thus the length of the binary codes required to achieve a desired accuracy is quite small, and we show it can even be compressed further without compromising the accuracy. To illustrate our results, we test the proposed method on natural images and show that it achieves strong performance. △ Less

Submitted 9 March, 2021; v1 submitted 1 October, 2020; originally announced October 2020.

Comments: Published as a conference paper at ICLR 2021

Random Vector Functional Link Networks for Function Approximation on Manifolds

Authors: Deanna Needell, Aaron A. Nelson, Rayan Saab, Palina Salanevich, Olov Schavemaker

Abstract: The learning speed of feed-forward neural networks is notoriously slow and has presented a bottleneck in deep learning applications for several decades. For instance, gradient-based learning algorithms, which are used extensively to train neural networks, tend to work slowly when all of the network parameters must be iteratively tuned. To counter this, both researchers and practitioners have tried… ▽ More The learning speed of feed-forward neural networks is notoriously slow and has presented a bottleneck in deep learning applications for several decades. For instance, gradient-based learning algorithms, which are used extensively to train neural networks, tend to work slowly when all of the network parameters must be iteratively tuned. To counter this, both researchers and practitioners have tried introducing randomness to reduce the learning requirement. Based on the original construction of Igelnik and Pao, single layer neural-networks with random input-to-hidden layer weights and biases have seen success in practice, but the necessary theoretical justification is lacking. In this paper, we begin to fill this theoretical gap. We provide a (corrected) rigorous proof that the Igelnik and Pao construction is a universal approximator for continuous functions on compact domains, with approximation error decaying asymptotically like $O(1/\sqrt{n})$ for the number $n$ of network nodes. We then extend this result to the non-asymptotic setting, proving that one can achieve any desired approximation error with high probability provided $n$ is sufficiently large. We further adapt this randomized neural network architecture to approximate functions on smooth, compact submanifolds of Euclidean space, providing theoretical guarantees in both the asymptotic and non-asymptotic forms. Finally, we illustrate our results on manifolds with numerical experiments. △ Less

Submitted 28 March, 2024; v1 submitted 30 July, 2020; originally announced July 2020.

Comments: 37 pages, 1 figure

MSC Class: 62M45

New Algorithms and Improved Guarantees for One-Bit Compressed Sensing on Manifolds

Authors: Mark Iwen, Eric Lybrand, Aaron Nelson, Rayan Saab

Abstract: We study the problem of approximately recovering signals on a manifold from one-bit linear measurements drawn from either a Gaussian ensemble, partial circulant ensemble, or bounded orthonormal ensemble and quantized using Sigma-Delta or distributed noise sha** schemes. We assume we are given a Geometric Multi-Resolution Analysis, which approximates the manifold, and we propose a convex optimiza… ▽ More We study the problem of approximately recovering signals on a manifold from one-bit linear measurements drawn from either a Gaussian ensemble, partial circulant ensemble, or bounded orthonormal ensemble and quantized using Sigma-Delta or distributed noise sha** schemes. We assume we are given a Geometric Multi-Resolution Analysis, which approximates the manifold, and we propose a convex optimization algorithm for signal recovery. We prove an upper bound on the recovery error which outperforms prior works that use memoryless scalar quantization, requires a simpler analysis, and extends the class of measurements beyond Gaussians. Finally, we illustrate our results with numerical experiments. △ Less

Submitted 24 April, 2019; v1 submitted 10 February, 2019; originally announced February 2019.

Fast binary embeddings, and quantized compressed sensing with structured matrices

Authors: Thang Huynh, Rayan Saab

Abstract: This paper deals with two related problems, namely distance-preserving binary embeddings and quantization for compressed sensing . First, we propose fast methods to replace points from a subset $\mathcal{X} \subset \mathbb{R}^n$, associated with the Euclidean metric, with points in the cube $\{\pm 1\}^m$ and we associate the cube with a pseudo-metric that approximates Euclidean distance among poin… ▽ More This paper deals with two related problems, namely distance-preserving binary embeddings and quantization for compressed sensing . First, we propose fast methods to replace points from a subset $\mathcal{X} \subset \mathbb{R}^n$, associated with the Euclidean metric, with points in the cube $\{\pm 1\}^m$ and we associate the cube with a pseudo-metric that approximates Euclidean distance among points in $\mathcal{X}$. Our methods rely on quantizing fast Johnson-Lindenstrauss embeddings based on bounded orthonormal systems and partial circulant ensembles, both of which admit fast transforms. Our quantization methods utilize noise-sha**, and include Sigma-Delta schemes and distributed noise-sha** schemes. The resulting approximation errors decay polynomially and exponentially fast in $m$, depending on the embedding method. This dramatically outperforms the current decay rates associated with binary embeddings and Hamming distances. Additionally, it is the first such binary embedding result that applies to fast Johnson-Lindenstrauss maps while preserving $\ell_2$ norms. Second, we again consider noise-sha** schemes, albeit this time to quantize compressed sensing measurements arising from bounded orthonormal ensembles and partial circulant matrices. We show that these methods yield a reconstruction error that again decays with the number of measurements (and bits), when using convex optimization for reconstruction. Specifically, for Sigma-Delta schemes, the error decays polynomially in the number of measurements, and it decays exponentially for distributed noise-sha** schemes based on beta encoding. These results are near optimal and the first of their kind dealing with bounded orthonormal systems. △ Less

Submitted 18 July, 2018; v1 submitted 25 January, 2018; originally announced January 2018.

Quantization for Low-Rank Matrix Recovery

Authors: Eric Lybrand, Rayan Saab

Abstract: We study Sigma-Delta quantization methods coupled with appropriate reconstruction algorithms for digitizing randomly sampled low-rank matrices. We show that the reconstruction error associated with our methods decays polynomially with the oversampling factor, and we leverage our results to obtain root-exponential accuracy by optimizing over the choice of quantization scheme. Additionally, we show… ▽ More We study Sigma-Delta quantization methods coupled with appropriate reconstruction algorithms for digitizing randomly sampled low-rank matrices. We show that the reconstruction error associated with our methods decays polynomially with the oversampling factor, and we leverage our results to obtain root-exponential accuracy by optimizing over the choice of quantization scheme. Additionally, we show that a random encoding scheme, applied to the quantized measurements, yields a near-optimal exponential bit-rate. As an added benefit, our schemes are robust both to noise and to deviations from the low-rank assumption. In short, we provide a full generalization of analogous results, obtained in the classical setup of bandlimited function acquisition, and more recently, in the finite frame and compressed sensing setups to the case of low-rank matrices sampled with sub-Gaussian linear operators. Finally, we believe our techniques for generalizing results from the compressed sensing setup to the analogous low-rank matrix setup is applicable to other quantization schemes. △ Less

Submitted 17 April, 2018; v1 submitted 28 September, 2017; originally announced September 2017.

Simple Classification using Binary Data

Authors: Deanna Needell, Rayan Saab, Tina Woolf

Abstract: Binary, or one-bit, representations of data arise naturally in many applications, and are appealing in both hardware implementations and algorithm design. In this work, we study the problem of data classification from binary data and propose a framework with low computation and resource costs. We illustrate the utility of the proposed approach through stylized and realistic numerical experiments,… ▽ More Binary, or one-bit, representations of data arise naturally in many applications, and are appealing in both hardware implementations and algorithm design. In this work, we study the problem of data classification from binary data and propose a framework with low computation and resource costs. We illustrate the utility of the proposed approach through stylized and realistic numerical experiments, and provide a theoretical analysis for a simple case. We hope that our framework and analysis will serve as a foundation for studying similar types of approaches. △ Less

Submitted 6 July, 2017; originally announced July 2017.

Quantized Compressed Sensing for Partial Random Circulant Matrices

Authors: Joe-Mei Feng, Felix Krahmer, Rayan Saab

Abstract: We provide the first analysis of a non-trivial quantization scheme for compressed sensing measurements arising from structured measurements. Specifically, our analysis studies compressed sensing matrices consisting of rows selected at random, without replacement, from a circulant matrix generated by a random subgaussian vector. We quantize the measurements using stable, possibly one-bit, Sigma-Del… ▽ More We provide the first analysis of a non-trivial quantization scheme for compressed sensing measurements arising from structured measurements. Specifically, our analysis studies compressed sensing matrices consisting of rows selected at random, without replacement, from a circulant matrix generated by a random subgaussian vector. We quantize the measurements using stable, possibly one-bit, Sigma-Delta schemes, and use a reconstruction method based on convex optimization. We show that the part of the reconstruction error due to quantization decays polynomially in the number of measurements. This is in line with analogous results on Sigma-Delta quantization associated with random Gaussian or subgaussian matrices, and significantly better than results associated with the widely assumed memoryless scalar quantization. Moreover, we prove that our approach is stable and robust; i.e., the reconstruction error degrades gracefully in the presence of non-quantization noise and when the underlying signal is not strictly sparse. The analysis relies on results concerning subgaussian chaos processes as well as a variation of McDiarmid's inequality. △ Less

Submitted 15 February, 2017; originally announced February 2017.

Comments: 15 pages

MSC Class: 94A12; 94A20; 41A25; 15B05; 15B52

Weighted $\ell_1$-Minimization for Sparse Recovery under Arbitrary Prior Information

Authors: Deanna Needell, Rayan Saab, Tina Woolf

Abstract: Weighted $\ell_1$-minimization has been studied as a technique for the reconstruction of a sparse signal from compressively sampled measurements when prior information about the signal, in the form of a support estimate, is available. In this work, we study the recovery conditions and the associated recovery guarantees of weighted $\ell_1$-minimization when arbitrarily many distinct weights are pe… ▽ More Weighted $\ell_1$-minimization has been studied as a technique for the reconstruction of a sparse signal from compressively sampled measurements when prior information about the signal, in the form of a support estimate, is available. In this work, we study the recovery conditions and the associated recovery guarantees of weighted $\ell_1$-minimization when arbitrarily many distinct weights are permitted. For example, such a setup might be used when one has multiple estimates for the support of a signal, and these estimates have varying degrees of accuracy. Our analysis yields an extension to existing works that assume only a single support estimate set upon which a constant weight is applied. We include numerical experiments, with both synthetic signals and real video data, that demonstrate the benefits of allowing non-uniform weights in the reconstruction procedure. △ Less

Submitted 7 December, 2016; v1 submitted 3 June, 2016; originally announced June 2016.

Comments: 25 pages, 6 figures

From compressed sensing to compressed bit-streams: practical encoders, tractable decoders

Authors: Rayan Saab, Rongrong Wang, Ozgur Yilmaz

Abstract: Compressed sensing is now established as an effective method for dimension reduction when the underlying signals are sparse or compressible with respect to some suitable basis or frame. One important, yet under-addressed problem regarding the compressive acquisition of analog signals is how to perform quantization. This is directly related to the important issues of how "compressed" compressed sen… ▽ More Compressed sensing is now established as an effective method for dimension reduction when the underlying signals are sparse or compressible with respect to some suitable basis or frame. One important, yet under-addressed problem regarding the compressive acquisition of analog signals is how to perform quantization. This is directly related to the important issues of how "compressed" compressed sensing is (in terms of the total number of bits one ends up using after acquiring the signal) and ultimately whether compressed sensing can be used to obtain compressed representations of suitable signals. Building on our recent work, we propose a concrete and practicable method for performing "analog-to-information conversion". Following a compressive signal acquisition stage, the proposed method consists of a quantization stage, based on $ΣΔ$ (sigma-delta) quantization, and a subsequent encoding (compression) stage that fits within the framework of compressed sensing seamlessly. We prove that, using this method, we can convert analog compressive samples to compressed digital bitstreams and decode using tractable algorithms based on convex optimization. We prove that the proposed AIC provides a nearly optimal encoding of sparse and compressible signals. Finally, we present numerical experiments illustrating the effectiveness of the proposed analog-to-information converter. △ Less

Submitted 3 April, 2016; originally announced April 2016.

Comments: 32 pages, 4 figures

A Deterministic Analysis of Decimation for Sigma-Delta Quantization of Bandlimited Functions

Authors: Ingrid Daubechies, Rayan Saab

Abstract: We study Sigma-Delta ($ΣΔ$) quantization of oversampled bandlimited functions. We prove that digitally integrating blocks of bits and then down-sampling, a process known as decimation, can efficiently encode the associated $ΣΔ$ bit-stream. It allows a large reduction in the bit-rate while still permitting good approximation of the underlying bandlimited function via an appropriate reconstruction k… ▽ More We study Sigma-Delta ($ΣΔ$) quantization of oversampled bandlimited functions. We prove that digitally integrating blocks of bits and then down-sampling, a process known as decimation, can efficiently encode the associated $ΣΔ$ bit-stream. It allows a large reduction in the bit-rate while still permitting good approximation of the underlying bandlimited function via an appropriate reconstruction kernel. Specifically, in the case of stable $r$th order $ΣΔ$ schemes we show that the reconstruction error decays exponentially in the bit-rate. For example, this result applies to the 1-bit, greedy, first-order $ΣΔ$ scheme. △ Less

Submitted 30 May, 2015; originally announced June 2015.

Quantization of compressive samples with stable and robust recovery

Authors: Rayan Saab, Rongrong Wang, Ozgur Yilmaz

Abstract: In this paper we study the quantization stage that is implicit in any compressed sensing signal acquisition paradigm. We propose using Sigma-Delta quantization and a subsequent reconstruction scheme based on convex optimization. We prove that the reconstruction error due to quantization decays polynomially in the number of measurements. Our results apply to arbitrary signals, including compressibl… ▽ More In this paper we study the quantization stage that is implicit in any compressed sensing signal acquisition paradigm. We propose using Sigma-Delta quantization and a subsequent reconstruction scheme based on convex optimization. We prove that the reconstruction error due to quantization decays polynomially in the number of measurements. Our results apply to arbitrary signals, including compressible ones, and account for measurement noise. Additionally, they hold for sub-Gaussian (including Gaussian and Bernoulli) random compressed sensing measurements, as well as for both high bit-depth and coarse quantizers, and they extend to 1-bit quantization. In the noise-free case, when the signal is strictly sparse we prove that by optimizing the order of the quantization scheme one can obtain root-exponential decay in the reconstruction error due to quantization. △ Less

Submitted 31 March, 2015; originally announced April 2015.

Noise-sha** Quantization Methods for Frame-based and Compressive Sampling Systems

Authors: Evan Chou, C. Sinan Güntürk, Felix Krahmer, Rayan Saab, Özgür Yılmaz

Abstract: Noise sha** refers to an analog-to-digital conversion methodology in which quantization error is arranged to lie mostly outside the signal spectrum by means of oversampling and feedback. Recently it has been successfully applied to more general redundant linear sampling and reconstruction systems associated with frames as well as non-linear systems associated with compressive sampling. This chap… ▽ More Noise sha** refers to an analog-to-digital conversion methodology in which quantization error is arranged to lie mostly outside the signal spectrum by means of oversampling and feedback. Recently it has been successfully applied to more general redundant linear sampling and reconstruction systems associated with frames as well as non-linear systems associated with compressive sampling. This chapter reviews some of the recent progress in this subject. △ Less

Submitted 20 February, 2015; originally announced February 2015.

Comments: 18 pages, 2 figures. chapter in the SAMPTA 2013 book

Recovery Analysis for Weighted $\ell_1$-Minimization Using a Null Space Property

Authors: Hassan Mansour, Rayan Saab

Abstract: We study the recovery of sparse signals from underdetermined linear measurements when a potentially erroneous support estimate is available. Our results are twofold. First, we derive necessary and sufficient conditions for signal recovery from compressively sampled measurements using weighted $\ell_1$-norm minimization. These conditions, which depend on the choice of weights as well as the size an… ▽ More We study the recovery of sparse signals from underdetermined linear measurements when a potentially erroneous support estimate is available. Our results are twofold. First, we derive necessary and sufficient conditions for signal recovery from compressively sampled measurements using weighted $\ell_1$-norm minimization. These conditions, which depend on the choice of weights as well as the size and accuracy of the support estimate, are on the null space of the measurement matrix. They can guarantee recovery even when standard $\ell_1$ minimization fails. Second, we derive bounds on the number of Gaussian measurements for these conditions to be satisfied, i.e., for weighted $\ell_1$ minimization to successfully recover all sparse signals whose support has been estimated sufficiently accurately. Our bounds show that weighted $\ell_1$ minimization requires significantly fewer measurements than standard $\ell_1$ minimization when the support estimate is relatively accurate. △ Less

Submitted 4 December, 2014; originally announced December 2014.

Comments: 16 pages, 2 figures

Quantization and Compressive Sensing

Authors: Petros T. Boufounos, Laurent Jacques, Felix Krahmer, Rayan Saab

Abstract: Quantization is an essential step in digitizing signals, and, therefore, an indispensable component of any modern acquisition system. This book chapter explores the interaction of quantization and compressive sensing and examines practical quantization strategies for compressive acquisition systems. Specifically, we first provide a brief overview of quantization and examine fundamental performance… ▽ More Quantization is an essential step in digitizing signals, and, therefore, an indispensable component of any modern acquisition system. This book chapter explores the interaction of quantization and compressive sensing and examines practical quantization strategies for compressive acquisition systems. Specifically, we first provide a brief overview of quantization and examine fundamental performance bounds applicable to any quantization approach. Next, we consider several forms of scalar quantizers, namely uniform, non-uniform, and 1-bit. We provide performance bounds and fundamental analysis, as well as practical quantizer designs and reconstruction algorithms that account for quantization. Furthermore, we provide an overview of Sigma-Delta ($ΣΔ$) quantization in the compressed sensing context, and also discuss implementation issues, recovery algorithms and performance bounds. As we demonstrate, proper accounting for quantization and careful quantizer design has significant impact in the performance of a compressive acquisition system. △ Less

Submitted 24 November, 2014; v1 submitted 6 May, 2014; originally announced May 2014.

Comments: 35 pages, 20 figures, to appear in Springer book "Compressed Sensing and Its Applications", 2014

Near-Optimal Encoding for Sigma-Delta Quantization of Finite Frame Expansions

Authors: Mark Iwen, Rayan Saab

Abstract: In this paper we investigate encoding the bit-stream resulting from coarse Sigma-Delta quantization of finite frame expansions (i.e., overdetermined representations) of vectors. We show that for a wide range of finite-frames, including random frames and piecewise smooth frames, there exists a simple encoding algorithm ---acting only on the Sigma-Delta bit stream--- and an associated decoding algor… ▽ More In this paper we investigate encoding the bit-stream resulting from coarse Sigma-Delta quantization of finite frame expansions (i.e., overdetermined representations) of vectors. We show that for a wide range of finite-frames, including random frames and piecewise smooth frames, there exists a simple encoding algorithm ---acting only on the Sigma-Delta bit stream--- and an associated decoding algorithm that together yield an approximation error which decays exponentially in the number of bits used. The encoding strategy consists of applying a discrete random operator to the Sigma-Delta bit stream and assigning a binary codeword to the result. The reconstruction procedure is essentially linear and equivalent to solving a least squares minimization problem. △ Less

Submitted 8 July, 2013; originally announced July 2013.

Sigma-Delta quantization of sub-Gaussian frame expansions and its application to compressed sensing

Authors: Felix Krahmer, Rayan Saab, Özgür Yılmaz

Abstract: Suppose that the collection $\{e_i\}_{i=1}^m$ forms a frame for $\R^k$, where each entry of the vector $e_i$ is a sub-Gaussian random variable. We consider expansions in such a frame, which are then quantized using a Sigma-Delta scheme. We show that an arbitrary signal in $\R^k$ can be recovered from its quantized frame coefficients up to an error which decays root-exponentially in the oversamplin… ▽ More Suppose that the collection $\{e_i\}_{i=1}^m$ forms a frame for $\R^k$, where each entry of the vector $e_i$ is a sub-Gaussian random variable. We consider expansions in such a frame, which are then quantized using a Sigma-Delta scheme. We show that an arbitrary signal in $\R^k$ can be recovered from its quantized frame coefficients up to an error which decays root-exponentially in the oversampling rate $m/k$. Here the quantization scheme is assumed to be chosen appropriately depending on the oversampling rate and the quantization alphabet can be coarse. The result holds with high probability on the draw of the frame uniformly for all signals. The crux of the argument is a bound on the extreme singular values of the product of a deterministic matrix and a sub-Gaussian frame. For fine quantization alphabets, we leverage this bound to show polynomial error decay in the context of compressed sensing. Our results extend previous results for structured deterministic frame expansions and Gaussian compressed sensing measurements. △ Less

Submitted 19 June, 2013; originally announced June 2013.

Comments: 22 pages

MSC Class: 94A12; 94A20; 41A25; 15B52

Recovering Compressively Sampled Signals Using Partial Support Information

Authors: Michael P. Friedlander, Hassan Mansour, Rayan Saab, Ozgur Yilmaz

Abstract: In this paper we study recovery conditions of weighted $\ell_1$ minimization for signal reconstruction from compressed sensing measurements when partial support information is available. We show that if at least 50% of the (partial) support information is accurate, then weighted $\ell_1$ minimization is stable and robust under weaker conditions than the analogous conditions for standard $\ell_1$ m… ▽ More In this paper we study recovery conditions of weighted $\ell_1$ minimization for signal reconstruction from compressed sensing measurements when partial support information is available. We show that if at least 50% of the (partial) support information is accurate, then weighted $\ell_1$ minimization is stable and robust under weaker conditions than the analogous conditions for standard $\ell_1$ minimization. Moreover, weighted $\ell_1$ minimization provides better bounds on the reconstruction error in terms of the measurement noise and the compressibility of the signal to be recovered. We illustrate our results with extensive numerical experiments on synthetic data and real audio and video signals. △ Less

Submitted 22 July, 2011; v1 submitted 22 October, 2010; originally announced October 2010.

Comments: 22 pages, 10 figures

Sobolev Duals for Random Frames and Sigma-Delta Quantization of Compressed Sensing Measurements

Authors: S. Güntürk, A. Powell, R. Saab, Ö. Yılmaz

Abstract: Quantization of compressed sensing measurements is typically justified by the robust recovery results of Candès, Romberg and Tao, and of Donoho. These results guarantee that if a uniform quantizer of step size $δ$ is used to quantize $m$ measurements $y = Φx$ of a $k$-sparse signal $x \in \R^N$, where $Φ$ satisfies the restricted isometry property, then the approximate recovery $x^#$ via… ▽ More Quantization of compressed sensing measurements is typically justified by the robust recovery results of Candès, Romberg and Tao, and of Donoho. These results guarantee that if a uniform quantizer of step size $δ$ is used to quantize $m$ measurements $y = Φx$ of a $k$-sparse signal $x \in \R^N$, where $Φ$ satisfies the restricted isometry property, then the approximate recovery $x^#$ via $\ell_1$-minimization is within $O(δ)$ of $x$. The simplest and commonly assumed approach is to quantize each measurement independently. In this paper, we show that if instead an $r$th order $ΣΔ$ quantization scheme with the same output alphabet is used to quantize $y$, then there is an alternative recovery method via Sobolev dual frames which guarantees a reduction of the approximation error by a factor of $(m/k)^{(r-1/2)α}$ for any $0 < α< 1$, if $m \gtrsim_r k (\log N)^{1/(1-α)}$. The result holds with high probability on the initial draw of the measurement matrix $Φ$ from the Gaussian distribution, and uniformly for all $k$-sparse signals $x$ that satisfy a mild size condition on their supports. △ Less

Submitted 1 February, 2010; originally announced February 2010.

Sparse Recovery by Non-convex Optimization -- Instance Optimality

Authors: Rayan Saab, Ozgur Yilmaz

Abstract: In this note, we address the theoretical properties of $Δ_p$, a class of compressed sensing decoders that rely on $\ell^p$ minimization with 0<p<1 to recover estimates of sparse and compressible signals from incomplete and inaccurate measurements. In particular, we extend the results of Candes, Romberg and Tao, and Wojtaszczyk regarding the decoder $Δ_1$, based on $\ell^1$ minimization, to… ▽ More In this note, we address the theoretical properties of $Δ_p$, a class of compressed sensing decoders that rely on $\ell^p$ minimization with 0<p<1 to recover estimates of sparse and compressible signals from incomplete and inaccurate measurements. In particular, we extend the results of Candes, Romberg and Tao, and Wojtaszczyk regarding the decoder $Δ_1$, based on $\ell^1$ minimization, to $Δ_p$ with 0<p<1. Our results are two-fold. First, we show that under certain sufficient conditions that are weaker than the analogous sufficient conditions for $Δ_1$ the decoders $Δ_p$ are robust to noise and stable in the sense that they are (2,p) instance optimal for a large class of encoders. Second, we extend the results of Wojtaszczyk to show that, like $Δ_1$, the decoders $Δ_p$ are (2,2) instance optimal in probability provided the measurement matrix is drawn from an appropriate distribution. △ Less

Submitted 10 August, 2009; v1 submitted 4 September, 2008; originally announced September 2008.

Comments: 32 pages, 4 figures v2