-
Stochastic Markov Gradient Descent and Training Low-Bit Neural Networks
Authors:
Jonathan Ashbrock,
Alexander M. Powell
Abstract:
The massive size of modern neural networks has motivated substantial recent interest in neural network quantization. We introduce Stochastic Markov Gradient Descent (SMGD), a discrete optimization method applicable to training quantized neural networks. The SMGD algorithm is designed for settings where memory is highly constrained during training. We provide theoretical guarantees of algorithm per…
▽ More
The massive size of modern neural networks has motivated substantial recent interest in neural network quantization. We introduce Stochastic Markov Gradient Descent (SMGD), a discrete optimization method applicable to training quantized neural networks. The SMGD algorithm is designed for settings where memory is highly constrained during training. We provide theoretical guarantees of algorithm performance as well as encouraging numerical results.
△ Less
Submitted 22 December, 2020; v1 submitted 25 August, 2020;
originally announced August 2020.
-
High order low-bit Sigma-Delta quantization for fusion frames
Authors:
Zhen Gao,
Felix Krahmer,
Alexander M. Powell
Abstract:
We construct high order low-bit Sigma-Delta $(ΣΔ)$ quantizers for the vector-valued setting of fusion frames. We prove that these $ΣΔ$ quantizers can be stably implemented to quantize fusion frame measurements on subspaces $W_n$ using $\log_2( {\rm dim}(W_n)+1)$ bits per measurement. Signal reconstruction is performed using a version of Sobolev duals for fusion frames, and numerical experiments ar…
▽ More
We construct high order low-bit Sigma-Delta $(ΣΔ)$ quantizers for the vector-valued setting of fusion frames. We prove that these $ΣΔ$ quantizers can be stably implemented to quantize fusion frame measurements on subspaces $W_n$ using $\log_2( {\rm dim}(W_n)+1)$ bits per measurement. Signal reconstruction is performed using a version of Sobolev duals for fusion frames, and numerical experiments are given to validate the overall performance.
△ Less
Submitted 3 October, 2020; v1 submitted 17 June, 2020;
originally announced June 2020.
-
A Schauder basis for $L_2$ consisting of non-negative functions
Authors:
Daniel Freeman,
Alexander M. Powell,
Mitchell A. Taylor
Abstract:
We prove that $L_2(\mathbb{R})$ contains a Schauder basis of non-negative functions. Similarly, $L_p(\mathbb{R})$ contains a Schauder basic sequence of non-negative functions such that $L_p(\mathbb{R})$ embeds into the closed span of the sequence. We prove as well that if $X$ is a separable Banach space with the bounded approximation property, then any set in $X$ with dense span contains a quasi-b…
▽ More
We prove that $L_2(\mathbb{R})$ contains a Schauder basis of non-negative functions. Similarly, $L_p(\mathbb{R})$ contains a Schauder basic sequence of non-negative functions such that $L_p(\mathbb{R})$ embeds into the closed span of the sequence. We prove as well that if $X$ is a separable Banach space with the bounded approximation property, then any set in $X$ with dense span contains a quasi-basis (Schauder frame) for $X$. Furthermore, if $X$ is a separable Banach lattice with a bibasis then any set in $X$ with dense span contains a u-frame.
△ Less
Submitted 21 March, 2020;
originally announced March 2020.
-
A Sharp Balian-Low Uncertainty Principle for Shift-Invariant Spaces
Authors:
Douglas P. Hardin,
Michael C. Northington V.,
Alexander M. Powell
Abstract:
A sharp version of the Balian-Low theorem is proven for the generators of finitely generated shift-invariant spaces. If generators $\{f_k\}_{k=1}^K \subset L^2(\mathbb{R}^d)$ are translated along a lattice to form a frame or Riesz basis for a shift-invariant space $V$, and if $V$ has extra invariance by a suitable finer lattice, then one of the generators $f_k$ must satisfy…
▽ More
A sharp version of the Balian-Low theorem is proven for the generators of finitely generated shift-invariant spaces. If generators $\{f_k\}_{k=1}^K \subset L^2(\mathbb{R}^d)$ are translated along a lattice to form a frame or Riesz basis for a shift-invariant space $V$, and if $V$ has extra invariance by a suitable finer lattice, then one of the generators $f_k$ must satisfy $\int_{\mathbb{R}^d} |x| |f_k(x)|^2 dx = \infty$, namely, $\widehat{f_k} \notin H^{1/2}(\mathbb{R}^d)$. Similar results are proven for frames of translates that are not Riesz bases without the assumption of extra lattice invariance. The best previously existing results in the literature give a notably weaker conclusion using the Sobolev space $H^{d/2+ε}(\mathbb{R}^d)$; our results provide an absolutely sharp improvement with $H^{1/2}(\mathbb{R}^d)$. Our results are sharp in the sense that $H^{1/2}(\mathbb{R}^d)$ cannot be replaced by $H^s(\mathbb{R}^d)$ for any $s<1/2$.
△ Less
Submitted 16 October, 2015;
originally announced October 2015.
-
Error bounds for consistent reconstruction: random polytopes and coverage processes
Authors:
Alexander M. Powell,
J. Tyler Whitehouse
Abstract:
Consistent reconstruction is a method for producing an estimate $\widetilde{x} \in \mathbb{R}^d$ of a signal $x\in \mathbb{R}^d$ if one is given a collection of $N$ noisy linear measurements $q_n = \langle x, \varphi_n \rangle + ε_n$, $1 \leq n \leq N$, that have been corrupted by i.i.d. uniform noise $\{ε_n\}_{n=1}^N$. We prove mean squared error bounds for consistent reconstruction when the meas…
▽ More
Consistent reconstruction is a method for producing an estimate $\widetilde{x} \in \mathbb{R}^d$ of a signal $x\in \mathbb{R}^d$ if one is given a collection of $N$ noisy linear measurements $q_n = \langle x, \varphi_n \rangle + ε_n$, $1 \leq n \leq N$, that have been corrupted by i.i.d. uniform noise $\{ε_n\}_{n=1}^N$. We prove mean squared error bounds for consistent reconstruction when the measurement vectors $\{\varphi_n\}_{n=1}^N\subset \mathbb{R}^d$ are drawn independently at random from a suitable distribution on the unit-sphere $\mathbb{S}^{d-1}$. Our main results prove that the mean squared error (MSE) for consistent reconstruction is of the optimal order $\mathbb{E}\|x - \widetilde{x}\|^2 \leq Kδ^2/N^2$ under general conditions on the measurement vectors. We also prove refined MSE bounds when the measurement vectors are i.i.d. uniformly distributed on the unit-sphere $\mathbb{S}^{d-1}$ and, in particular, show that in this case the constant $K$ is dominated by $d^3$, the cube of the ambient dimension. The proofs involve an analysis of random polytopes using coverage processes on the sphere.
△ Less
Submitted 27 May, 2014;
originally announced May 2014.
-
Time-frequency concentration of generating systems
Authors:
Philippe Jaming,
Alexander M. Powell
Abstract:
Uncertainty principles for generating systems $\{e_n\}_{n=1}^{\infty} \subset \ltwo$ are proven and quantify the interplay between $\ell^r(\N)$ coefficient stability properties and time-frequency localization with respect to $|t|^p$ power weight dispersions. As a sample result, it is proven that if the unit-norm system $\{e_n\}_{n=1}^{\infty}$ is a Schauder basis or frame for $\ltwo$ then the tw…
▽ More
Uncertainty principles for generating systems $\{e_n\}_{n=1}^{\infty} \subset \ltwo$ are proven and quantify the interplay between $\ell^r(\N)$ coefficient stability properties and time-frequency localization with respect to $|t|^p$ power weight dispersions. As a sample result, it is proven that if the unit-norm system $\{e_n\}_{n=1}^{\infty}$ is a Schauder basis or frame for $\ltwo$ then the two dispersion sequences $Δ(e_n)$, $Δ(\bar{e_n})$ and the one mean sequence $μ(e_n)$ cannot all be bounded. On the other hand, it is constructively proven that there exists a unit-norm exact system $\{f_n\}_{n=1}^{\infty}$ in $\ltwo$ for which all four of the sequences $Δ(f_n)$, $Δ(\bar{f_n})$, $μ(f_n)$, $μ(\bar{f_n})$ are bounded.
△ Less
Submitted 22 February, 2010;
originally announced February 2010.
-
Uncertainty principles for orthonormal sequences
Authors:
Philippe Jaming,
Alexander M. Powell
Abstract:
The aim of this paper is to provide complementary quantitative extensions of two results of H.S. Shapiro on the time-frequency concentration of orthonormal sequences in $L^2 (\R)$. More precisely, Shapiro proved that if the elements of an orthonormal sequence and their Fourier transforms are all pointwise bounded by a fixed function in $L^2(\R)$ then the sequence is finite. In a related result,…
▽ More
The aim of this paper is to provide complementary quantitative extensions of two results of H.S. Shapiro on the time-frequency concentration of orthonormal sequences in $L^2 (\R)$. More precisely, Shapiro proved that if the elements of an orthonormal sequence and their Fourier transforms are all pointwise bounded by a fixed function in $L^2(\R)$ then the sequence is finite. In a related result, Shapiro also proved that if the elements of an orthonormal sequence and their Fourier transforms have uniformly bounded means and dispersions then the sequence is finite. This paper gives quantitative bounds on the size of the finite orthonormal sequences in Shapiro's uncertainty principles. The bounds are obtained by using prolate spheroïdal wave functions and combinatorial estimates on the number of elements in a spherical code. Extensions for Riesz bases and different measures of time-frequency concentration are also given.
△ Less
Submitted 16 June, 2006;
originally announced June 2006.
