Showing 1–45 of 45 results for author: Iwen, M

Tensor Deli: Tensor Completion for Low CP-Rank Tensors via Random Sampling

Authors: Cullen Haselby, Mark Iwen, Santhosh Karnik, Rongrong Wang

Abstract: We propose two provably accurate methods for low CP-rank tensor completion - one using adaptive sampling and one using nonadaptive sampling. Both of our algorithms combine matrix completion techniques for a small number of slices along with Jennrich's algorithm to learn the factors corresponding to the first two modes, and then solve systems of linear equations to learn the factors corresponding t… ▽ More We propose two provably accurate methods for low CP-rank tensor completion - one using adaptive sampling and one using nonadaptive sampling. Both of our algorithms combine matrix completion techniques for a small number of slices along with Jennrich's algorithm to learn the factors corresponding to the first two modes, and then solve systems of linear equations to learn the factors corresponding to the remaining modes. For order-$3$ tensors, our algorithms follow a "sandwich" sampling strategy that more densely samples a few outer slices (the bread), and then more sparsely samples additional inner slices (the bbq-braised tofu) for the final completion. For an order-$d$, CP-rank $r$ tensor of size $n \times \cdots \times n$ that satisfies mild assumptions, our adaptive sampling algorithm recovers the CP-decomposition with high probability while using at most $O(nr\log r + dnr)$ samples and $O(n^2r^2+dnr^2)$ operations. Our nonadaptive sampling algorithm recovers the CP-decomposition with high probability while using at most $O(dnr^2\log n + nr\log^2 n)$ samples and runs in polynomial time. Numerical experiments demonstrate that both of our methods work well on noisy synthetic data as well as on real world data. △ Less

Submitted 14 March, 2024; originally announced March 2024.

Comments: 25 pages, 8 figures

MSC Class: 65F55; 15A83; 15A69

On Outer Bi-Lipschitz Extensions of Linear Johnson-Lindenstrauss Embeddings of Subsets of $\mathbb{R}^N$

Authors: Rafael Chiclana, Mark A. Iwen, Mark Philip Roach

Abstract: The celebrated Johnson-Lindenstrauss lemma states that for all $\varepsilon \in (0,1)$ and finite sets $X \subseteq \mathbb{R}^N$ with $n>1$ elements, there exists a matrix $Φ\in \mathbb{R}^{m \times N}$ with $m=\mathcal{O}(\varepsilon^{-2}\log n)$ such that \[ (1 - \varepsilon) \|x-y\|_2 \leq \|Φx-Φy\|_2 \leq (1+\varepsilon)\| x- y\|_2 \quad \forall\, x, y \in X.\] Herein we consider terminal emb… ▽ More The celebrated Johnson-Lindenstrauss lemma states that for all $\varepsilon \in (0,1)$ and finite sets $X \subseteq \mathbb{R}^N$ with $n>1$ elements, there exists a matrix $Φ\in \mathbb{R}^{m \times N}$ with $m=\mathcal{O}(\varepsilon^{-2}\log n)$ such that \[ (1 - \varepsilon) \|x-y\|_2 \leq \|Φx-Φy\|_2 \leq (1+\varepsilon)\| x- y\|_2 \quad \forall\, x, y \in X.\] Herein we consider terminal embedding results which have recently been introduced in the computer science literature as stronger extensions of the Johnson-Lindenstrauss lemma for finite sets. After a short survey of this relatively recent line of work, we extend the theory of terminal embeddings to hold for arbitrary (e.g., infinite) subsets $X \subseteq \mathbb{R}^N$, and then specialize our generalized results to the case where $X$ is a low-dimensional compact submanifold of $\mathbb{R}^N$. In particular, we prove the following generalization of the Johnson-Lindenstrauss lemma: For all $\varepsilon \in (0,1)$ and $X\subseteq\mathbb{R}^N$, there exists a terminal embedding $f: \mathbb{R}^N \longrightarrow \mathbb{R}^{m}$ such that $$(1 - \varepsilon) \| x - y \|_2 \leq \left\| f(x) - f(y) \right\|_2 \leq (1 + \varepsilon) \| x - y \|_2 \quad \forall \, x \in X ~{\rm and}~ \forall \, y \in \mathbb{R}^N.$$ Crucially, we show that the dimension $m$ of the range of $f$ above is optimal up to multiplicative constants, satisfying $m=\mathcal{O}(\varepsilon^{-2} ω^2(S_X))$, where $ω(S_X)$ is the Gaussian width of the set of unit secants of $X$, $S_X=\overline{\{(x-y)/\|x-y\|_2 \colon x \neq y \in X\}}$. Furthermore, our proofs are constructive and yield algorithms for computing a general class of terminal embeddings $f$, an instance of which is demonstrated herein to allow for more accurate compressive nearest neighbor classification than standard linear Johnson-Lindenstrauss embeddings do in practice. △ Less

Submitted 6 March, 2024; originally announced March 2024.

Comments: 16 pages, 4 figures. arXiv admin note: substantial text overlap with arXiv:2206.03376

MSC Class: 51F30; 65D18; 68R12

Fast and Low-Memory Compressive Sensing Algorithms for Low Tucker-Rank Tensor Approximation from Streamed Measurements

Authors: Cullen Haselby, Mark A. Iwen, Deanna Needell, Elizaveta Rebrova, William Swartworth

Abstract: In this paper we consider the problem of recovering a low-rank Tucker approximation to a massive tensor based solely on structured random compressive measurements. Crucially, the proposed random measurement ensembles are both designed to be compactly represented (i.e., low-memory), and can also be efficiently computed in one-pass over the tensor. Thus, the proposed compressive sensing approach may… ▽ More In this paper we consider the problem of recovering a low-rank Tucker approximation to a massive tensor based solely on structured random compressive measurements. Crucially, the proposed random measurement ensembles are both designed to be compactly represented (i.e., low-memory), and can also be efficiently computed in one-pass over the tensor. Thus, the proposed compressive sensing approach may be used to produce a low-rank factorization of a huge tensor that is too large to store in memory with a total memory footprint on the order of the much smaller desired low-rank factorization. In addition, the compressive sensing recovery algorithm itself (which takes the compressive measurements as input, and then outputs a low-rank factorization) also runs in a time which principally depends only on the size of the sought factorization, making its runtime sub-linear in the size of the large tensor one is approximating. Finally, unlike prior works related to (streaming) algorithms for low-rank tensor approximation from such compressive measurements, we present a unified analysis of both Kronecker and Khatri-Rao structured measurement ensembles culminating in error guarantees comparing the error of our recovery algorithm's approximation of the input tensor to the best possible low-rank Tucker approximation error achievable for the tensor by any possible algorithm. We further include an empirical study of the proposed approach that verifies our theoretical findings and explores various trade-offs of parameters of interest. △ Less

Submitted 25 August, 2023; originally announced August 2023.

Comments: 59 pages, 8 figures

MSC Class: 65F55

Tensor Sandwich: Tensor Completion for Low CP-Rank Tensors via Adaptive Random Sampling

Authors: Cullen Haselby, Santhosh Karnik, Mark Iwen

Abstract: We propose an adaptive and provably accurate tensor completion approach based on combining matrix completion techniques (see, e.g., arXiv:0805.4471, arXiv:1407.3619, arXiv:1306.2979) for a small number of slices with a modified noise robust version of Jennrich's algorithm. In the simplest case, this leads to a sampling strategy that more densely samples two outer slices (the bread), and then more… ▽ More We propose an adaptive and provably accurate tensor completion approach based on combining matrix completion techniques (see, e.g., arXiv:0805.4471, arXiv:1407.3619, arXiv:1306.2979) for a small number of slices with a modified noise robust version of Jennrich's algorithm. In the simplest case, this leads to a sampling strategy that more densely samples two outer slices (the bread), and then more sparsely samples additional inner slices (the bbq-braised tofu) for the final completion. Under mild assumptions on the factor matrices, the proposed algorithm completes an $n \times n \times n$ tensor with CP-rank $r$ with high probability while using at most $\mathcal{O}(nr\log^2 r)$ adaptively chosen samples. Empirical experiments further verify that the proposed approach works well in practice, including as a low-rank approximation method in the presence of additive noise. △ Less

Submitted 3 July, 2023; originally announced July 2023.

Comments: 6 pages, 5 figures. Sampling Theory and Applications Conference 2023

MSC Class: 15A83 (Primary) 68Q25 (Secondary)

Sparse Spectral Methods for Solving High-Dimensional and Multiscale Elliptic PDEs

Authors: Craig Gross, Mark Iwen

Abstract: In his monograph Chebyshev and Fourier Spectral Methods, John Boyd claimed that, regarding Fourier spectral methods for solving differential equations, ``[t]he virtues of the Fast Fourier Transform will continue to improve as the relentless march to larger and larger [bandwidths] continues''. This paper attempts to further the virtue of the Fast Fourier Transform (FFT) as not only bandwidth is pus… ▽ More In his monograph Chebyshev and Fourier Spectral Methods, John Boyd claimed that, regarding Fourier spectral methods for solving differential equations, ``[t]he virtues of the Fast Fourier Transform will continue to improve as the relentless march to larger and larger [bandwidths] continues''. This paper attempts to further the virtue of the Fast Fourier Transform (FFT) as not only bandwidth is pushed to its limits, but also the dimension of the problem. Instead of using the traditional FFT however, we make a key substitution: a high-dimensional, sparse Fourier transform (SFT) paired with randomized rank-1 lattice methods. The resulting sparse spectral method rapidly and automatically determines a set of Fourier basis functions whose span is guaranteed to contain an accurate approximation of the solution of a given elliptic PDE. This much smaller, near-optimal Fourier basis is then used to efficiently solve the given PDE in a runtime which only depends on the PDE's data compressibility and ellipticity properties, while breaking the curse of dimensionality and relieving linear dependence on any multiscale structure in the original problem. Theoretical performance of the method is established herein with convergence analysis in the Sobolev norm for a general class of non-constant diffusion equations, as well as pointers to technical extensions of the convergence analysis to more general advection-diffusion-reaction equations. Numerical experiments demonstrate good empirical performance on several multiscale and high-dimensional example problems, further showcasing the promise of the proposed methods in practice. △ Less

Submitted 1 February, 2023; originally announced February 2023.

MSC Class: 65N35; 65T40; 35J15; 65D40; 35J05

Modewise Johnson-Lindenstrauss Embeddings for Nuclear Many-Body Theory

Authors: A. Zare, R. Wirth, C. A. Haselby, H. Hergert, M. Iwen

Abstract: In this work, we explore modewise Johnson-Lindenstrauss embeddings (JLEs) as a tool to reduce the computational cost and memory requirements of nuclear many-body methods. JLEs are randomized projections of high-dimensional data tensors onto low-dimensional subspaces that preserve key structural features. Such embeddings allow for the oblivious and incremental compression of large tensors, e.g., th… ▽ More In this work, we explore modewise Johnson-Lindenstrauss embeddings (JLEs) as a tool to reduce the computational cost and memory requirements of nuclear many-body methods. JLEs are randomized projections of high-dimensional data tensors onto low-dimensional subspaces that preserve key structural features. Such embeddings allow for the oblivious and incremental compression of large tensors, e.g., the nuclear Hamiltonian, into significantly smaller random sketches that still allow for the accurate calculation of ground-state energies and other observables. Their oblivious character makes it possible to compress a tensor without knowing in advance exactly what observables one might want to approximate at a later time. This opens the door for the use of tensors that are much too large to store in memory, e.g., complete two-plus three-nucleon Hamiltonians in large, symmetry-unrestricted bases. Such compressed Hamiltonians can be stored and used later on with relative ease. As a first step, we analyze the JLE's impact on the second-order Many-Body Perturbation Theory (MBPT) corrections for nuclear ground-state observables. Numerical experiments for a wide range of closed-shell nuclei, model spaces and state-of-the-art nuclear interactions demonstrate the validity and potential of the proposed approach: We can compress nuclear Hamiltonians hundred- to thousand-fold while only incurring mean relative errors of 1\% or less in ground-state observables. Importantly, we show that JLEs capture the relevant physical information contained in the highly structured Hamiltonian tensor despite their random characteristics. In addition to the significant storage savings, the achieved compressions imply multiple order-of magnitude reductions in computational effort when the compressed Hamiltonians are used in higher-order MBPT or nonperturbative many-body methods. △ Less

Submitted 2 November, 2022; originally announced November 2022.

Comments: 23 pages, 14 figures

Journal ref: Eur. Phys. J A 59, 95 (2023)

Neural Network Approximation of Continuous Functions in High Dimensions with Applications to Inverse Problems

Authors: Santhosh Karnik, Rongrong Wang, Mark Iwen

Abstract: The remarkable successes of neural networks in a huge variety of inverse problems have fueled their adoption in disciplines ranging from medical imaging to seismic analysis over the past decade. However, the high dimensionality of such inverse problems has simultaneously left current theory, which predicts that networks should scale exponentially in the dimension of the problem, unable to explain… ▽ More The remarkable successes of neural networks in a huge variety of inverse problems have fueled their adoption in disciplines ranging from medical imaging to seismic analysis over the past decade. However, the high dimensionality of such inverse problems has simultaneously left current theory, which predicts that networks should scale exponentially in the dimension of the problem, unable to explain why the seemingly small networks used in these settings work as well as they do in practice. To reduce this gap between theory and practice, we provide a general method for bounding the complexity required for a neural network to approximate a Hölder (or uniformly) continuous function defined on a high-dimensional set with a low-complexity structure. The approach is based on the observation that the existence of a Johnson-Lindenstrauss embedding $A\in\mathbb{R}^{d\times D}$ of a given high-dimensional set $S\subset\mathbb{R}^D$ into a low dimensional cube $[-M,M]^d$ implies that for any Hölder (or uniformly) continuous function $f:S\to\mathbb{R}^p$, there exists a Hölder (or uniformly) continuous function $g:[-M,M]^d\to\mathbb{R}^p$ such that $g(Ax)=f(x)$ for all $x\in S$. Hence, if one has a neural network which approximates $g:[-M,M]^d\to\mathbb{R}^p$, then a layer can be added that implements the JL embedding $A$ to obtain a neural network that approximates $f:S\to\mathbb{R}^p$. By pairing JL embedding results along with results on approximation of Hölder (or uniformly) continuous functions by neural networks, one then obtains results which bound the complexity required for a neural network to approximate Hölder (or uniformly) continuous functions on high dimensional sets. The end result is a general theoretical framework which can then be used to better explain the observed empirical successes of smaller networks in a wider variety of inverse problems than current theory allows. △ Less

Submitted 10 October, 2023; v1 submitted 28 August, 2022; originally announced August 2022.

Comments: 26 pages, 1 figure

MSC Class: 68T07

On Outer Bi-Lipschitz Extensions of Linear Johnson-Lindenstrauss Embeddings of Low-Dimensional Submanifolds of $\mathbb{R}^N$

Authors: Mark A. Iwen, Mark Philip Roach

Abstract: Let $\mathcal{M}$ be a compact $d$-dimensional submanifold of $\mathbb{R}^N$ with reach $τ$ and volume $V_{\mathcal M}$. Fix $ε\in (0,1)$. In this paper we prove that a nonlinear function $f: \mathbb{R}^N \rightarrow \mathbb{R}^{m}$ exists with $m \leq C \left(d / ε^2 \right) \log \left(\frac{\sqrt[d]{V_{\mathcal M}}}τ \right)$ such that… ▽ More Let $\mathcal{M}$ be a compact $d$-dimensional submanifold of $\mathbb{R}^N$ with reach $τ$ and volume $V_{\mathcal M}$. Fix $ε\in (0,1)$. In this paper we prove that a nonlinear function $f: \mathbb{R}^N \rightarrow \mathbb{R}^{m}$ exists with $m \leq C \left(d / ε^2 \right) \log \left(\frac{\sqrt[d]{V_{\mathcal M}}}τ \right)$ such that $$(1 - ε) \| {\bf x} - {\bf y} \|_2 \leq \left\| f({\bf x}) - f({\bf y}) \right\|_2 \leq (1 + ε) \| {\bf x} - {\bf y} \|_2$$ holds for all ${\bf x} \in \mathcal{M}$ and ${\bf y} \in \mathbb{R}^N$. In effect, $f$ not only serves as a bi-Lipschitz function from $\mathcal{M}$ into $\mathbb{R}^{m}$ with bi-Lipschitz constants close to one, but also approximately preserves all distances from points not in $\mathcal{M}$ to all points in $\mathcal{M}$ in its image. Furthermore, the proof is constructive and yields an algorithm which works well in practice. In particular, it is empirically demonstrated herein that such nonlinear functions allow for more accurate compressive nearest neighbor classification than standard linear Johnson-Lindenstrauss embeddings do in practice. △ Less

Submitted 7 June, 2022; originally announced June 2022.

MSC Class: 51F30; 65D18; 68R12

Characterizing unit spheres in Euclidean spaces via reach and volume

Authors: Mark Iwen, Benjamin Schmidt, Arman Tavakoli

Abstract: Let $M$ be a smooth, connected, compact submanifold of $\mathbb{R}^n$ without boundary and of dimension $k\geq 2$. Let $\mathbb{S}^k \subset \mathbb{R}^{k+1}\subset \mathbb{R}^n$ denote the $k$-dimesnional unit sphere. We show if $M$ has reach equal to one, then its volume satisfies $\text{vol}(M)\geq \text{vol}(\mathbb{S}^k)$ with equality holding only if $M$ is congruent to $\mathbb{S}^k$. Let $M$ be a smooth, connected, compact submanifold of $\mathbb{R}^n$ without boundary and of dimension $k\geq 2$. Let $\mathbb{S}^k \subset \mathbb{R}^{k+1}\subset \mathbb{R}^n$ denote the $k$-dimesnional unit sphere. We show if $M$ has reach equal to one, then its volume satisfies $\text{vol}(M)\geq \text{vol}(\mathbb{S}^k)$ with equality holding only if $M$ is congruent to $\mathbb{S}^k$. △ Less

Submitted 12 February, 2022; originally announced February 2022.

MSC Class: 53Z50

Toward Fast and Provably Accurate Near-field Ptychographic Phase Retrieval

Authors: Mark Iwen, Michael Perlmutter, Mark Philip Roach

Abstract: Ptychography is an imaging technique which involves a sample being illuminated by a coherent, localized probe of illumination. When the probe interacts with the sample, the light is diffracted and a diffraction pattern is detected. Then the sample (or probe) is shifted laterally in space to illuminate a new area of the sample whilst ensuring sufficient overlap. Near-field Ptychography (NFP) occurs… ▽ More Ptychography is an imaging technique which involves a sample being illuminated by a coherent, localized probe of illumination. When the probe interacts with the sample, the light is diffracted and a diffraction pattern is detected. Then the sample (or probe) is shifted laterally in space to illuminate a new area of the sample whilst ensuring sufficient overlap. Near-field Ptychography (NFP) occurs when the sample is placed at a short defocus distance having a large Fresnel number. In this paper, we prove that certain NFP measurements are robustly invertible (up to an unavoidable global phase ambiguity) by constructing a point spread function and physical mask which leads to a well-conditioned lifted linear system. We then apply a block phase retrieval algorithm using weighted angular synchronization and prove that the proposed approach accurately recovers the measured sample. Finally, we also propose using a Wirtinger Flow for NFP problems and numerically evaluate that alternate approach both against our main proposed approach, as well as with NFP measurements for which our main approach does not apply. △ Less

Submitted 30 January, 2023; v1 submitted 20 December, 2021; originally announced December 2021.

MSC Class: 65T99; 65Z05; 94A12 ACM Class: G.1.0

A Hybrid Scattering Transform for Signals with Isolated Singularities

Authors: Michael Perlmutter, Jieqian He, Mark Iwen, Matthew Hirn

Abstract: The scattering transform is a wavelet-based model of Convolutional Neural Networks originally introduced by S. Mallat. Mallat's analysis shows that this network has desirable stability and invariance guarantees and therefore helps explain the observation that the filters learned by early layers of a Convolutional Neural Network typically resemble wavelets. Our aim is to understand what sort of fil… ▽ More The scattering transform is a wavelet-based model of Convolutional Neural Networks originally introduced by S. Mallat. Mallat's analysis shows that this network has desirable stability and invariance guarantees and therefore helps explain the observation that the filters learned by early layers of a Convolutional Neural Network typically resemble wavelets. Our aim is to understand what sort of filters should be used in the later layers of the network. Towards this end, we propose a two-layer hybrid scattering transform. In our first layer, we convolve the input signal with a wavelet filter transform to promote sparsity, and, in the second layer, we convolve with a Gabor filter to leverage the sparsity created by the first layer. We show that these measurements characterize information about signals with isolated singularities. We also show that the Gabor measurements used in the second layer can be used to synthesize sparse signals such as those produced by the first layer. △ Less

Submitted 10 October, 2021; originally announced October 2021.

MSC Class: 94A12

On Fast Johnson-Lindenstrauss Embeddings of Compact Submanifolds of $\mathbb{R}^N$ with Boundary

Authors: Mark A. Iwen, Benjamin Schmidt, Arman Tavakoli

Abstract: Let $\mathcal{M}$ be a smooth $d$-dimensional submanifold of $\mathbb{R}^N$ with boundary that's equipped with the Euclidean (chordal) metric, and choose $m \leq N$. In this paper we consider the probability that a random matrix $A \in \mathbb{R}^{m \times N}$ will serve as a bi-Lipschitz function $A: \mathcal{M} \rightarrow \mathbb{R}^m$ with bi-Lipschitz constants close to one for three differen… ▽ More Let $\mathcal{M}$ be a smooth $d$-dimensional submanifold of $\mathbb{R}^N$ with boundary that's equipped with the Euclidean (chordal) metric, and choose $m \leq N$. In this paper we consider the probability that a random matrix $A \in \mathbb{R}^{m \times N}$ will serve as a bi-Lipschitz function $A: \mathcal{M} \rightarrow \mathbb{R}^m$ with bi-Lipschitz constants close to one for three different types of distributions on the $m \times N$ matrices $A$, including two whose realizations are guaranteed to have fast matrix-vector multiplies. In doing so we generalize prior randomized metric space embedding results of this type for submanifolds of $\mathbb{R}^N$ by allowing for the presence of boundary while also retaining, and in some cases improving, prior lower bounds on the achievable embedding dimensions $m$ for which one can expect small distortion with high probability. In particular, motivated by recent modewise embedding constructions for tensor data, herein we present a new class of highly structured distributions on matrices which outperform prior structured matrix distributions for embedding sufficiently low-dimensional submanifolds of $\mathbb{R}^N$ (with $d \lesssim \sqrt{N}$) with respect to both achievable embedding dimension, and computationally efficient realizations. As a consequence we are able to present, for example, a general new class of Johnson-Lindenstrauss embedding matrices for $\mathcal{O}(\log^c N)$-dimensional submanifolds of $\mathbb{R}^N$ which enjoy $\mathcal{O}(N \log (\log N))$-time matrix vector multiplications. △ Less

Submitted 21 May, 2022; v1 submitted 8 October, 2021; originally announced October 2021.

Comments: Revision comments: Fixed typos in the text and lemma 2.1. Renumbered theorems by section

Modewise Operators, the Tensor Restricted Isometry Property, and Low-Rank Tensor Recovery

Authors: Mark A. Iwen, Deanna Needell, Michael Perlmutter, Elizaveta Rebrova

Abstract: Recovery of sparse vectors and low-rank matrices from a small number of linear measurements is well-known to be possible under various model assumptions on the measurements. The key requirement on the measurement matrices is typically the restricted isometry property, that is, approximate orthonormality when acting on the subspace to be recovered. Among the most widely used random matrix measureme… ▽ More Recovery of sparse vectors and low-rank matrices from a small number of linear measurements is well-known to be possible under various model assumptions on the measurements. The key requirement on the measurement matrices is typically the restricted isometry property, that is, approximate orthonormality when acting on the subspace to be recovered. Among the most widely used random matrix measurement models are (a) independent sub-gaussian models and (b) randomized Fourier-based models, allowing for the efficient computation of the measurements. For the now ubiquitous tensor data, direct application of the known recovery algorithms to the vectorized or matricized tensor is awkward and memory-heavy because of the huge measurement matrices to be constructed and stored. In this paper, we propose modewise measurement schemes based on sub-gaussian and randomized Fourier measurements. These modewise operators act on the pairs or other small subsets of the tensor modes separately. They require significantly less memory than the measurements working on the vectorized tensor, provably satisfy the tensor restricted isometry property and experimentally can recover the tensor data from fewer measurements and do not require impractical storage. △ Less

Submitted 21 September, 2021; originally announced September 2021.

MSC Class: 15B52; 15A69; 15A83; 97N40

Phase Retrieval for $L^2([-π,π])$ via the Provably Accurate and Noise Robust Numerical Inversion of Spectrogram Measurements

Authors: Mark Iwen, Michael Perlmutter, Nada Sissouno, Aditya Viswanathan

Abstract: In this paper, we focus on the approximation of smooth functions $f: [-π, π] \rightarrow \mathbb{C}$, up to an unresolvable global phase ambiguity, from a finite set of Short Time Fourier Transform (STFT) magnitude (i.e., spectrogram) measurements. Two algorithms are developed for approximately inverting such measurements, each with theoretical error guarantees establishing their correctness. A de… ▽ More In this paper, we focus on the approximation of smooth functions $f: [-π, π] \rightarrow \mathbb{C}$, up to an unresolvable global phase ambiguity, from a finite set of Short Time Fourier Transform (STFT) magnitude (i.e., spectrogram) measurements. Two algorithms are developed for approximately inverting such measurements, each with theoretical error guarantees establishing their correctness. A detailed numerical study also demonstrates that both algorithms work well in practice and have good numerical convergence behavior. △ Less

Submitted 4 June, 2021; originally announced June 2021.

MSC Class: 65T60; 65Z05; 94A12

Lower Bounds on the Low-Distortion Embedding Dimension of Submanifolds of $\mathbb{R}^n$

Authors: Mark Iwen, Arman Tavakoli, Benjamin Schmidt

Abstract: Let $\mathcal{M}$ be a smooth submanifold of $\mathbb{R}^n$ equipped with the Euclidean (chordal) metric. This note considers the smallest dimension $m$ for which there exists a bi-Lipschitz function $f: \mathcal{M} \mapsto \mathbb{R}^m$ with bi-Lipschitz constants close to one. The main result bounds the embedding dimension $m$ below in terms of the bi-Lipschitz constants of $f$ and the reach, vo… ▽ More Let $\mathcal{M}$ be a smooth submanifold of $\mathbb{R}^n$ equipped with the Euclidean (chordal) metric. This note considers the smallest dimension $m$ for which there exists a bi-Lipschitz function $f: \mathcal{M} \mapsto \mathbb{R}^m$ with bi-Lipschitz constants close to one. The main result bounds the embedding dimension $m$ below in terms of the bi-Lipschitz constants of $f$ and the reach, volume, diameter, and dimension of $\mathcal{M}$. This new lower bound is applied to show that prior upper bounds by Eftekhari and Wakin (arXiv:1306.4748) on the minimal low-distortion embedding dimension of such manifolds using random matrices achieve near-optimal dependence on both reach and volume. This supports random linear maps as being nearly as efficient as the best possible nonlinear maps at reducing the ambient dimension for manifold data. In the process of proving our main result, we also prove similar results concerning the impossibility of achieving better nonlinear measurement maps with the Restricted Isometry Property (RIP) in compressive sensing applications. △ Less

Submitted 27 May, 2021; originally announced May 2021.

On the $\ell^\infty$-norms of the Singular Vectors of Arbitrary Powers of a Difference Matrix with Applications to Sigma-Delta Quantization

Authors: Theodore Faust, Mark Iwen, Rayan Saab, Rongrong Wang

Abstract: Let $\| A \|_{\max} := \max_{i,j} |A_{i,j}|$ denote the maximum magnitude of entries of a given matrix $A$. In this paper we show that $$\max \left\{ \|U_r \|_{\max},\|V_r\|_{\max} \right\} \le \frac{(Cr)^{6r}}{\sqrt{N}},$$ where $U_r$ and $V_r$ are the matrices whose columns are, respectively, the left and right singular vectors of the $r$-th order finite difference matrix $D^{r}$ with… ▽ More Let $\| A \|_{\max} := \max_{i,j} |A_{i,j}|$ denote the maximum magnitude of entries of a given matrix $A$. In this paper we show that $$\max \left\{ \|U_r \|_{\max},\|V_r\|_{\max} \right\} \le \frac{(Cr)^{6r}}{\sqrt{N}},$$ where $U_r$ and $V_r$ are the matrices whose columns are, respectively, the left and right singular vectors of the $r$-th order finite difference matrix $D^{r}$ with $r \geq 2$, and where $D$ is the $N\times N$ finite difference matrix with $1$ on the diagonal, $-1$ on the sub-diagonal, and $0$ elsewhere. Here $C$ is a universal constant that is independent of both $N$ and $r$. Among other things, this establishes that both the right and left singular vectors of such finite difference matrices are Bounded Orthonormal Systems (BOSs) with known upper bounds on their BOS constants, objects of general interest in classical compressive sensing theory. Such finite difference matrices are also fundamental to standard $r^{\rm th}$ order Sigma-Delta quantization schemes more specifically, and as a result the new bounds provided herein on the maximum $\ell^{\infty}$-norms of their $\ell^2$-normalized singular vectors allow for several previous Sigma-Delta quantization results to be generalized and improved. △ Less

Submitted 24 March, 2021; originally announced March 2021.

Comments: 24 pages not including appendices, 54 pages with appendices

MSC Class: 15A42 (Primary) 15A18; 94A12; 39A22 (Secondary)

Sparse Fourier Transforms on Rank-1 Lattices for the Rapid and Low-Memory Approximation of Functions of Many Variables

Authors: Craig Gross, Mark Iwen, Lutz Kämmerer, Toni Volkmer

Abstract: We consider fast, provably accurate algorithms for approximating functions on the $d$-dimensional torus, $f: \mathbb{ T }^d \rightarrow \mathbb{C}$, that are sparse (or compressible) in the Fourier basis. In particular, suppose that the Fourier coefficients of $f$, $\{c_{\bf k} (f) \}_{{\bf k} \in \mathbb{Z}^d}$, are concentrated in a finite set $I \subset \mathbb{Z}^d$ so that… ▽ More We consider fast, provably accurate algorithms for approximating functions on the $d$-dimensional torus, $f: \mathbb{ T }^d \rightarrow \mathbb{C}$, that are sparse (or compressible) in the Fourier basis. In particular, suppose that the Fourier coefficients of $f$, $\{c_{\bf k} (f) \}_{{\bf k} \in \mathbb{Z}^d}$, are concentrated in a finite set $I \subset \mathbb{Z}^d$ so that $$\min_{Ω\subset I s.t. |Ω| =s } \left\| f - \sum_{{\bf k} \in Ω} c_{\bf k} (f) e^{ -2 πi {\bf k} \cdot \circ} \right\|_2 < ε\|f \|_2$$ holds for $s \ll |I|$ and $ε\in (0,1)$. We aim to identify a near-minimizing subset $Ω\subset I$ and accurately approximate the associated Fourier coefficients $\{ c_{\bf k} (f) \}_{{\bf k} \in Ω}$ as rapidly as possible. We present both deterministic as well as randomized algorithms using $O(s^2 d \log^c (|I|))$-time/memory and $O(s d \log^c (|I|))$-time/memory, respectively. Most crucially, all of the methods proposed herein achieve these runtimes while satisfying theoretical best $s$-term approximation guarantees which guarantee their numerical accuracy and robustness to noise for general functions. These are achieved by modifying several one-dimensional Sparse Fourier Transform (SFT) methods to subsample a function along a reconstructing rank-1 lattice for the given frequency set $I$ to rapidly identify a near-minimizing subset $Ω\subset I$ without using anything about the lattice beyond its generating vector. This requires new fast and low-memory frequency identification techniques capable of rapidly recovering vector-valued frequencies in $\mathbb{Z}^d$ as opposed to simple integer frequencies in the univariate setting. Two different strategies are proposed and analyzed, each with different accuracy versus computational speed and memory tradeoffs. △ Less

Submitted 17 December, 2020; originally announced December 2020.

MSC Class: 65T40; 65D15; 42B05; 65Y20; 65T50

A Deterministic Algorithm for Constructing Multiple Rank-1 Lattices of Near-Optimal Size

Authors: Craig Gross, Mark A. Iwen, Lutz Kämmerer, Toni Volkmer

Abstract: In this paper we present the first known deterministic algorithm for the construction of multiple rank-1 lattices for the approximation of periodic functions of many variables. The algorithm works by converting a potentially large reconstructing single rank-1 lattice for some $ d $-dimensional frequency set $ I \subset [N]^d $ into a collection of much smaller rank-1 lattices which allow for accur… ▽ More In this paper we present the first known deterministic algorithm for the construction of multiple rank-1 lattices for the approximation of periodic functions of many variables. The algorithm works by converting a potentially large reconstructing single rank-1 lattice for some $ d $-dimensional frequency set $ I \subset [N]^d $ into a collection of much smaller rank-1 lattices which allow for accurate and efficient reconstruction of trigonometric polynomials with coefficients in $ I $ (and, therefore, for the approximation of multivariate periodic functions). The total number of sampling points in the resulting multiple rank-1 lattices is theoretically shown to be less than $ \mathcal{O}\left( |I| \log^{ 2 }(N |I|) \right) $ with constants independent of $d$, and by performing one-dimensional fast Fourier transforms on samples of trigonometric polynomials with Fourier support in $ I $ at these points, we obtain exact reconstruction of all Fourier coefficients in fewer than $ \mathcal{O}\left(d\,|I|\log^4 (N|I|)\right) $ total operations. Additionally, we present a second multiple rank-1 lattice construction algorithm which constructs lattices with even fewer sampling points at the cost of only being able to reconstruct exact trigonometric polynomials rather than having additional theoretical approximation. Both algorithms are tested numerically and surpass the theoretical bounds. Notably, we observe that the oversampling factors #samples$/|I|$ appear to grow only logarithmically in $ |I| $ for the first algorithm and appear near-optimally bounded by four in the second algorithm. △ Less

Submitted 21 March, 2020; originally announced March 2020.

MSC Class: 65T40; 42B05; 68Q17; 68Q25; 42B35; 65T50; 65Y20; 65D30; 65D32

Lower Memory Oblivious (Tensor) Subspace Embeddings with Fewer Random Bits: Modewise Methods for Least Squares

Authors: M. A. Iwen, D. Needell, E. Rebrova, A. Zare

Abstract: In this paper new general modewise Johnson-Lindenstrauss (JL) subspace embeddings are proposed that are both considerably faster to generate and easier to store than traditional JL embeddings when working with extremely large vectors and/or tensors. Corresponding embedding results are then proven for two different types of low-dimensional (tensor) subspaces. The first of these new subspace embed… ▽ More In this paper new general modewise Johnson-Lindenstrauss (JL) subspace embeddings are proposed that are both considerably faster to generate and easier to store than traditional JL embeddings when working with extremely large vectors and/or tensors. Corresponding embedding results are then proven for two different types of low-dimensional (tensor) subspaces. The first of these new subspace embedding results produces improved space complexity bounds for embeddings of rank-$r$ tensors whose CP decompositions are contained in the span of a fixed (but unknown) set of $r$ rank-one basis tensors. In the traditional vector setting this first result yields new and very general near-optimal oblivious subspace embedding constructions that require fewer random bits to generate than standard JL embeddings when embedding subspaces of $\mathbb{C}^N$ spanned by basis vectors with special Kronecker structure. The second result proven herein provides new fast JL embeddings of arbitrary $r$-dimensional subspaces $\mathcal{S} \subset \mathbb{C}^N$ which also require fewer random bits (and so are easier to store - i.e., require less space) than standard fast JL embedding methods in order to achieve small $ε$-distortions. These new oblivious subspace embedding results work by $(i)$ effectively folding any given vector in $\mathcal{S}$ into a (not necessarily low-rank) tensor, and then $(ii)$ embedding the resulting tensor into $\mathbb{C}^m$ for $m \leq C r \log^c(N) / ε^2$. Applications related to compression and fast compressed least squares solution methods are also considered, including those used for fitting low-rank CP decompositions, and the proposed JL embedding results are shown to work well numerically in both settings. △ Less

Submitted 16 December, 2020; v1 submitted 17 December, 2019; originally announced December 2019.

Sparse Harmonic Transforms II: Best $s$-Term Approximation Guarantees for Bounded Orthonormal Product Bases in Sublinear-Time

Authors: Bosu Choi, Mark Iwen, Toni Volkmer

Abstract: In this paper, we develop a sublinear-time compressive sensing algorithm for approximating functions of many variables which are compressible in a given Bounded Orthonormal Product Basis (BOPB). The resulting algorithm is shown to both have an associated best $s$-term recovery guarantee in the given BOPB, and also to work well numerically for solving sparse approximation problems involving functio… ▽ More In this paper, we develop a sublinear-time compressive sensing algorithm for approximating functions of many variables which are compressible in a given Bounded Orthonormal Product Basis (BOPB). The resulting algorithm is shown to both have an associated best $s$-term recovery guarantee in the given BOPB, and also to work well numerically for solving sparse approximation problems involving functions contained in the span of fairly general sets of as many as $\sim10^{230}$ orthonormal basis functions. All code is made publicly available. As part of the proof of the main recovery guarantee new variants of the well known CoSaMP algorithm are proposed which can utilize any sufficiently accurate support identification procedure satisfying a {Support Identification Property (SIP)} in order to obtain strong sparse approximation guarantees. These new CoSaMP variants are then proven to have both runtime and recovery error behavior which are largely determined by the associated runtime and error behavior of the chosen support identification method. The main theoretical results of the paper are then shown by develo** a sublinear-time support identification algorithm for general BOPB sets which is robust to arbitrary additive errors. Using this new support identification method to create a new CoSaMP variant then results in a new robust sublinear-time compressive sensing algorithm for BOPB-compressible functions of many variables. △ Less

Submitted 20 September, 2019; originally announced September 2019.

MSC Class: 65T40; 68W25

Inverting Spectrogram Measurements via Aliased Wigner Distribution Deconvolution and Angular Synchronization

Authors: Michael Perlmutter, Sami Merhi, Aditya Viswanathan, Mark Iwen

Abstract: We propose a two-step approach for reconstructing a signal ${\bf x}\in\mathbb{C}^d$ from subsampled short-time Fourier transform magnitude (spectogram) measurements: First, we use an aliased Wigner distribution deconvolution approach to solve for a portion of the rank-one matrix ${\bf \widehat{\bf x}}{\bf \widehat{\bf x}}^{*}.$ Second, we use angular syncrhonization to solve for… ▽ More We propose a two-step approach for reconstructing a signal ${\bf x}\in\mathbb{C}^d$ from subsampled short-time Fourier transform magnitude (spectogram) measurements: First, we use an aliased Wigner distribution deconvolution approach to solve for a portion of the rank-one matrix ${\bf \widehat{\bf x}}{\bf \widehat{\bf x}}^{*}.$ Second, we use angular syncrhonization to solve for ${\bf \widehat{\bf x}}$ (and then for ${\bf x}$ by Fourier inversion). Using this method, we produce two new efficient phase retrieval algorithms that perform well numerically in comparison to standard approaches and also prove two theorems, one which guarantees the recovery of discrete, bandlimited signals ${\bf x}\in\mathbb{C}^{d}$ from fewer than $d$ STFT magnitude measurements and another which establishes a new class of deterministic coded diffraction pattern measurements which are guaranteed to allow efficient and noise robust recovery. △ Less

Submitted 24 July, 2019; originally announced July 2019.

MSC Class: 65T60; 65Z05; 94A12

A direct solver for the phase retrieval problem in ptychographic imaging

Authors: Nada Sissouno, Florian Boßmann, Frank Filbir, Mark Iwen, Maik Kahnt, Rayan Saab, Christian Schroer, Wolfgang zu Castell

Abstract: Measurements achieved with ptychographic imaging are a special case of diffraction measurements. They are generated by illuminating small parts of a sample with, e.g., a focused X-ray beam. By shifting the sample, a set of far-field diffraction patterns of the whole sample are then obtained. From a mathematical point of view those measurements are the squared modulus of the windowed Fourier transf… ▽ More Measurements achieved with ptychographic imaging are a special case of diffraction measurements. They are generated by illuminating small parts of a sample with, e.g., a focused X-ray beam. By shifting the sample, a set of far-field diffraction patterns of the whole sample are then obtained. From a mathematical point of view those measurements are the squared modulus of the windowed Fourier transform of the sample. Thus, we have a phase retrieval problem for local Fourier measurements. A direct solver for this problem was introduced by Iwen, Viswanathan and Wang in 2016 and improved by Iwen, Preskitt, Saab and Viswanathan in 2018. Motivated by the applied perspective of ptychographic imaging, we present a generalization of this method and compare the different versions in numerical experiments. The new method proposed herein turns out to be more stable, particularly in the case of missing data. △ Less

Submitted 16 April, 2019; originally announced April 2019.

New Algorithms and Improved Guarantees for One-Bit Compressed Sensing on Manifolds

Authors: Mark Iwen, Eric Lybrand, Aaron Nelson, Rayan Saab

Abstract: We study the problem of approximately recovering signals on a manifold from one-bit linear measurements drawn from either a Gaussian ensemble, partial circulant ensemble, or bounded orthonormal ensemble and quantized using Sigma-Delta or distributed noise sha** schemes. We assume we are given a Geometric Multi-Resolution Analysis, which approximates the manifold, and we propose a convex optimiza… ▽ More We study the problem of approximately recovering signals on a manifold from one-bit linear measurements drawn from either a Gaussian ensemble, partial circulant ensemble, or bounded orthonormal ensemble and quantized using Sigma-Delta or distributed noise sha** schemes. We assume we are given a Geometric Multi-Resolution Analysis, which approximates the manifold, and we propose a convex optimization algorithm for signal recovery. We prove an upper bound on the recovery error which outperforms prior works that use memoryless scalar quantization, requires a simpler analysis, and extends the class of measurements beyond Gaussians. Finally, we illustrate our results with numerical experiments. △ Less

Submitted 24 April, 2019; v1 submitted 10 February, 2019; originally announced February 2019.

Sparse Harmonic Transforms: A New Class of Sublinear-time Algorithms for Learning Functions of Many Variables

Authors: Bosu Choi, Mark Iwen, Felix Krahmer

Abstract: We develop fast and memory efficient numerical methods for learning functions of many variables that admit sparse representations in terms of general bounded orthonormal tensor product bases. Such functions appear in many applications including, e.g., various Uncertainty Quantification(UQ) problems involving the solution of parametric PDE that are approximately sparse in Chebyshev or Legendre prod… ▽ More We develop fast and memory efficient numerical methods for learning functions of many variables that admit sparse representations in terms of general bounded orthonormal tensor product bases. Such functions appear in many applications including, e.g., various Uncertainty Quantification(UQ) problems involving the solution of parametric PDE that are approximately sparse in Chebyshev or Legendre product bases. We expect that our results provide a starting point for a new line of research on sublinear-time solution techniques for UQ applications of the type above which will eventually be able to scale to significantly higher-dimensional problems than what are currently computationally feasible. More concretely, let $B$ be a finite Bounded Orthonormal Product Basis (BOPB) of cardinality $|B|=N$. We will develop methods that approximate any function $f$ that is sparse in the BOPB, that is, $f:\mathcal{D}\subset R^D\rightarrow C$ of the form $f(\mathbf{x})=\sum_{b\in S}c_b\cdot b(\mathbf{x})$ with $S\subset B$ of cardinality $|S| =s\ll N$. Our method has a runtime of just $(s\log N)^{O(1)}$, uses only $(s\log N)^{O(1)}$ function evaluations on a fixed and nonadaptive grid, and not more than $(s\log N)^{O(1)}$ bits of memory. For $s\ll N$, the runtime $(s\log N)^{O(1)}$ will be less than what is required to simply enumerate the elements of the basis $B$; thus our method is the first approach applicable in a general BOPB framework that falls into the class referred to as "sublinear-time". This and the similarly reduced sample and memory requirements set our algorithm apart from previous works based on standard compressive sensing algorithms such as basis pursuit which typically store and utilize full intermediate basis representations of size $Ω(N)$. △ Less

Submitted 7 May, 2020; v1 submitted 14 August, 2018; originally announced August 2018.

On Recovery Guarantees for One-Bit Compressed Sensing on Manifolds

Authors: Mark A. Iwen, Felix Krahmer, Sara Krause-Solberg, Johannes Maly

Abstract: This paper studies the problem of recovering a signal from one-bit compressed sensing measurements under a manifold model; that is, assuming that the signal lies on or near a manifold of low intrinsic dimension. We provide a convex recovery method based on the Geometric Multi-Resolution Analysis and prove recovery guarantees with a near-optimal scaling in the intrinsic manifold dimension. Our meth… ▽ More This paper studies the problem of recovering a signal from one-bit compressed sensing measurements under a manifold model; that is, assuming that the signal lies on or near a manifold of low intrinsic dimension. We provide a convex recovery method based on the Geometric Multi-Resolution Analysis and prove recovery guarantees with a near-optimal scaling in the intrinsic manifold dimension. Our method is the first tractable algorithm with such guarantees for this setting. The results are complemented by numerical experiments confirming the validity of our approach. △ Less

Submitted 23 July, 2020; v1 submitted 17 July, 2018; originally announced July 2018.

Lower Lipschitz Bounds for Phase Retrieval from Locally Supported Measurements

Authors: Mark A. Iwen, Sami Merhi, Michael Perlmutter

Abstract: In this short note, we consider the worst case noise robustness of any phase retrieval algorithm which aims to reconstruct all nonvanishing vectors $\mathbf{x} \in \mathbb{C}^d$ (up to a single global phase multiple) from the magnitudes of an arbitrary collection of local correlation measurements. Examples of such measurements include both spectrogram measurements of $\mathbf{x}$ using locally sup… ▽ More In this short note, we consider the worst case noise robustness of any phase retrieval algorithm which aims to reconstruct all nonvanishing vectors $\mathbf{x} \in \mathbb{C}^d$ (up to a single global phase multiple) from the magnitudes of an arbitrary collection of local correlation measurements. Examples of such measurements include both spectrogram measurements of $\mathbf{x}$ using locally supported windows and masked Fourier transform intensity measurements of $\mathbf{x}$ using bandlimited masks. As a result, the robustness results considered herein apply to a wide range of both ptychographic and Fourier ptychographic imaging scenarios. In particular, the main results imply that the accurate recovery of high-resolution images of extremely large samples using highly localized probes is likely to require an extremely large number of measurements in order to be robust to worst case measurement noise, independent of the recovery algorithm employed. Furthermore, recent pushes to achieve high-speed and high-resolution ptychographic imaging of integrated circuits for process verification and failure analysis will likely need to carefully balance probe design (e.g., their effective time-frequency support) against the total number of measurements acquired in order for their imaging techniques to be stable to measurement noise, no matter what reconstruction algorithms are applied. △ Less

Submitted 21 June, 2018; originally announced June 2018.

MSC Class: 94A20 (Primary); 94A12 (Secondary); 65Z05 (Secondary); 65T50 (Secondary)

Extension of PCA to Higher Order Data Structures: An Introduction to Tensors, Tensor Decompositions, and Tensor PCA

Authors: Ali Zare, Alp Ozdemir, Mark A. Iwen, Selin Aviyente

Abstract: The widespread use of multisensor technology and the emergence of big data sets have brought the necessity to develop more versatile tools to represent higher-order data with multiple aspects and high dimensionality. Data in the form of multidimensional arrays, also referred to as tensors, arises in a variety of applications including chemometrics, hyperspectral imaging, high resolution videos, ne… ▽ More The widespread use of multisensor technology and the emergence of big data sets have brought the necessity to develop more versatile tools to represent higher-order data with multiple aspects and high dimensionality. Data in the form of multidimensional arrays, also referred to as tensors, arises in a variety of applications including chemometrics, hyperspectral imaging, high resolution videos, neuroimaging, biometrics, and social network analysis. Early multiway data analysis approaches reformatted such tensor data as large vectors or matrices and then resorted to dimensionality reduction methods developed for classical two-way analysis such as PCA. However, one cannot discover hidden components within multiway data using conventional PCA. To this end, tensor decomposition methods which are flexible in the choice of the constraints and that extract more general latent components have been proposed. In this paper, we review the major tensor decomposition methods with a focus on problems targeted by classical PCA. In particular, we present tensor methods that aim to solve three important challenges typically addressed by PCA: dimensionality reduction, i.e. low-rank tensor approximation, supervised learning, i.e. learning linear subspaces for feature extraction, and robust low-rank tensor recovery. We also provide experimental results to compare different tensor models for both dimensionality reduction and supervised learning applications. △ Less

Submitted 25 June, 2018; v1 submitted 1 March, 2018; originally announced March 2018.

A New Class of Fully Discrete Sparse Fourier Transforms: Faster Stable Implementations with Guarantees

Authors: Sami Merhi, Ruochuan Zhang, Mark A. Iwen, Andrew Christlieb

Abstract: In this paper we consider Sparse Fourier Transform (SFT) algorithms for approximately computing the best $s$-term approximation of the Discrete Fourier Transform (DFT) $\mathbf{\hat{f}} \in \mathbb{C}^N$ of any given input vector $\mathbf{f} \in \mathbb{C}^N$ in just $\left( s \log N\right)^{\mathcal{O}(1)}$-time using only a similarly small number of entries of $\mathbf{f}$. In particular, we pre… ▽ More In this paper we consider Sparse Fourier Transform (SFT) algorithms for approximately computing the best $s$-term approximation of the Discrete Fourier Transform (DFT) $\mathbf{\hat{f}} \in \mathbb{C}^N$ of any given input vector $\mathbf{f} \in \mathbb{C}^N$ in just $\left( s \log N\right)^{\mathcal{O}(1)}$-time using only a similarly small number of entries of $\mathbf{f}$. In particular, we present a deterministic SFT algorithm which is guaranteed to always recover a near best $s$-term approximation of the DFT of any given input vector $\mathbf{f} \in \mathbb{C}^N$ in $\mathcal{O} \left( s^2 \log ^{\frac{11}{2}} (N) \right)$-time. Unlike previous deterministic results of this kind, our deterministic result holds for both arbitrary vectors $\mathbf{f} \in \mathbb{C}^N$ and vector lengths $N$. In addition to these deterministic SFT results, we also develop several new publicly available randomized SFT implementations for approximately computing $\mathbf{\hat{f}}$ from $\mathbf{f}$ using the same general techniques. The best of these new implementations is shown to outperform existing discrete sparse Fourier transform methods with respect to both runtime and noise robustness for large vector lengths $N$. △ Less

Submitted 8 June, 2017; originally announced June 2017.

MSC Class: 65T40 (Primary); 65T50 (Secondary)

Recovery of Compactly Supported Functions from Spectrogram Measurements via Lifting

Authors: Sami Merhi, Aditya Viswanathan, Mark Iwen

Abstract: A novel phase retrieval method, motivated by ptychographic imaging, is proposed for the approximate recovery of a compactly supported specimen function $f:\mathbb{R}\rightarrow\mathbb{C}$ from its continuous short time Fourier transform (STFT) spectrogram measurements. The method, partially inspired by the well known PhaseLift algorithm, is based on a lifted formulation of the infinite dimensional… ▽ More A novel phase retrieval method, motivated by ptychographic imaging, is proposed for the approximate recovery of a compactly supported specimen function $f:\mathbb{R}\rightarrow\mathbb{C}$ from its continuous short time Fourier transform (STFT) spectrogram measurements. The method, partially inspired by the well known PhaseLift algorithm, is based on a lifted formulation of the infinite dimensional problem which is then later truncated for the sake of computation. Numerical experiments demonstrate the promise of the proposed approach. △ Less

Submitted 6 June, 2017; originally announced June 2017.

Comments: 2017 International Conference on Sampling Theory and Applications (SampTA2017)

MSC Class: 65T99

A Deterministic Sparse FFT for Functions with Structured Fourier Sparsity

Authors: Sina Bittens, Ruochuan Zhang, Mark A. Iwen

Abstract: In this paper a deterministic sparse Fourier transform algorithm is presented which breaks the quadratic-in-sparsity runtime bottleneck for a large class of periodic functions exhibiting structured frequency support. These functions include, e.g., the oft-considered set of block frequency sparse functions of the form… ▽ More In this paper a deterministic sparse Fourier transform algorithm is presented which breaks the quadratic-in-sparsity runtime bottleneck for a large class of periodic functions exhibiting structured frequency support. These functions include, e.g., the oft-considered set of block frequency sparse functions of the form $$f(x) = \sum^{n}_{j=1} \sum^{B-1}_{k=0} c_{ω_j + k} e^{i(ω_j + k)x},~~\{ ω_1, \dots, ω_n \} \subset \left(-\left\lceil \frac{N}{2}\right\rceil, \left\lfloor \frac{N}{2}\right\rfloor\right]\cap\mathbb{Z}$$ as a simple subclass. Theoretical error bounds in combination with numerical experiments demonstrate that the newly proposed algorithms are both fast and robust to noise. In particular, they outperform standard sparse Fourier transforms in the rapid recovery of block frequency sparse functions of the type above. △ Less

Submitted 20 November, 2017; v1 submitted 15 May, 2017; originally announced May 2017.

Comments: 39 pages, 5 figures

Multiscale Analysis for Higher-order Tensors

Authors: Alp Ozdemir, Ali Zare, Mark A. Iwen, Selin Aviyente

Abstract: The widespread use of multisensor technology and the emergence of big datasets have created the need to develop tools to reduce, approximate, and classify large and multimodal data such as higher-order tensors. While early approaches focused on matrix and vector based methods to represent these higher-order data, more recently it has been shown that tensor decomposition methods are better equipped… ▽ More The widespread use of multisensor technology and the emergence of big datasets have created the need to develop tools to reduce, approximate, and classify large and multimodal data such as higher-order tensors. While early approaches focused on matrix and vector based methods to represent these higher-order data, more recently it has been shown that tensor decomposition methods are better equipped to capture couplings across their different modes. For these reasons, tensor decomposition methods have found applications in many different signal processing problems including dimensionality reduction, signal separation, linear regression, feature extraction, and classification. However, most of the existing tensor decomposition methods are based on the principle of finding a low-rank approximation in a linear subspace structure, where the definition of the rank may change depending on the particular decomposition. Since many datasets are not necessarily low-rank in a linear subspace, this often results in high approximation errors or low compression rates. In this paper, we introduce a new adaptive, multi-scale tensor decomposition method for higher order data inspired by hybrid linear modeling and subspace clustering techniques. In particular, we develop a multi-scale higher-order singular value decomposition (MS-HoSVD) approach where a given tensor is first permuted and then partitioned into several sub-tensors each of which can be represented as a low-rank tensor with increased representational efficiency. The proposed approach is evaluated for dimensionality reduction and classification for several different real-life tensor signals with promising results. △ Less

Submitted 29 June, 2018; v1 submitted 27 April, 2017; originally announced April 2017.

Phase Retrieval from Local Measurements: Improved Robustness via Eigenvector-Based Angular Synchronization

Authors: Mark A. Iwen, Brian Preskitt, Rayan Saab, Aditya Viswanathan

Abstract: We improve a phase retrieval approach that uses correlation-based measurements with compactly supported measurement masks [27]. The improved algorithm admits deterministic measurement constructions together with a robust, fast recovery algorithm that consists of solving a system of linear equations in a lifted space, followed by finding an eigenvector (e.g., via an inverse power iteration). Theore… ▽ More We improve a phase retrieval approach that uses correlation-based measurements with compactly supported measurement masks [27]. The improved algorithm admits deterministic measurement constructions together with a robust, fast recovery algorithm that consists of solving a system of linear equations in a lifted space, followed by finding an eigenvector (e.g., via an inverse power iteration). Theoretical reconstruction error guarantees from [27] are improved as a result for the new and more robust reconstruction approach proposed herein. Numerical experiments demonstrate robustness and computational efficiency that outperforms competing approaches on large problems. Finally, we show that this approach also trivially extends to phase retrieval problems based on windowed Fourier measurements. △ Less

Submitted 6 December, 2016; v1 submitted 4 December, 2016; originally announced December 2016.

Technical Report: Improved Fourier Reconstruction using Jump Information with Applications to MRI

Authors: Jade Larriva-Latt, Angela Morrison, Alison Radgowski, Joseph Tobin, Aditya Viswanathan, Mark Iwen

Abstract: Certain applications such as Magnetic Resonance Imaging (MRI) require the reconstruction of functions from Fourier spectral data. When the underlying functions are piecewise-smooth, standard Fourier approximation methods suffer from the Gibbs phenomenon - with associated oscillatory artifacts in the vicinity of edges and an overall reduced order of convergence in the approximation. This paper prop… ▽ More Certain applications such as Magnetic Resonance Imaging (MRI) require the reconstruction of functions from Fourier spectral data. When the underlying functions are piecewise-smooth, standard Fourier approximation methods suffer from the Gibbs phenomenon - with associated oscillatory artifacts in the vicinity of edges and an overall reduced order of convergence in the approximation. This paper proposes an edge-augmented Fourier reconstruction procedure which uses only the first few Fourier coefficients of an underlying piecewise-smooth function to accurately estimate jump information and then incorporate it into a Fourier partial sum approximation. We provide both theoretical and empirical results showing the improved accuracy of the proposed method, as well as comparisons demonstrating superior performance over existing state-of-the-art sparse optimization-based methods. Extensions of the proposed techniques to functions of several variables are also addressed preliminarily. All code used to generate the results in this report are made publicly available. △ Less

Submitted 12 October, 2016; originally announced October 2016.

A Distributed and Incremental SVD Algorithm for Agglomerative Data Analysis on Large Networks

Authors: M. A. Iwen, B. W. Ong

Abstract: In this paper, we show that the SVD of a matrix can be constructed efficiently in a hierarchical approach. Our algorithm is proven to recover the singular values and left singular vectors if the rank of the input matrix $A$ is known. Further, the hierarchical algorithm can be used to recover the $d$ largest singular values and left singular vectors with bounded error. We also show that the propose… ▽ More In this paper, we show that the SVD of a matrix can be constructed efficiently in a hierarchical approach. Our algorithm is proven to recover the singular values and left singular vectors if the rank of the input matrix $A$ is known. Further, the hierarchical algorithm can be used to recover the $d$ largest singular values and left singular vectors with bounded error. We also show that the proposed method is stable with respect to roundoff errors or corruption of the original matrix entries. Numerical experiments validate the proposed algorithms and parallel cost analysis. △ Less

Submitted 29 June, 2016; v1 submitted 26 January, 2016; originally announced January 2016.

Rapidly Computing Sparse Legendre Expansions via Sparse Fourier Transforms

Authors: Xianfeng Hu, Mark Iwen, Hye** Kim

Abstract: In this paper we propose a general strategy for rapidly computing sparse Legendre expansions. The resulting methods yield a new class of fast algorithms capable of approximating a given function $f:[-1,1] \rightarrow \mathbb{R}$ with a near-optimal linear combination of $s$ Legendre polynomials of degree $\leq N$ in just $(s \log N)^{\mathcal{O}(1)}$-time. When $s \ll N$ these algorithms exhibit s… ▽ More In this paper we propose a general strategy for rapidly computing sparse Legendre expansions. The resulting methods yield a new class of fast algorithms capable of approximating a given function $f:[-1,1] \rightarrow \mathbb{R}$ with a near-optimal linear combination of $s$ Legendre polynomials of degree $\leq N$ in just $(s \log N)^{\mathcal{O}(1)}$-time. When $s \ll N$ these algorithms exhibit sublinear runtime complexities in $N$, as opposed to traditional $Ω(N \log N)$-time methods for computing all of the first $N$ Legendre coefficients of $f$. Theoretical as well as numerical results demonstrate the promise of the proposed approach. △ Less

Submitted 26 March, 2016; v1 submitted 19 August, 2015; originally announced August 2015.

MSC Class: 65D05; 42A10; 68W25

Fast Phase Retrieval from Local Correlation Measurements

Authors: Mark Iwen, Aditya Viswanathan, Yang Wang

Abstract: We develop a fast phase retrieval method which can utilize a large class of local phaseless correlation-based measurements in order to recover a given signal ${\bf x} \in \mathbb{C}^d$ (up to an unknown global phase) in near-linear $\mathcal{O} \left( d \log^4 d \right)$-time. Accompanying theoretical analysis proves that the proposed algorithm is guaranteed to deterministically recover all signal… ▽ More We develop a fast phase retrieval method which can utilize a large class of local phaseless correlation-based measurements in order to recover a given signal ${\bf x} \in \mathbb{C}^d$ (up to an unknown global phase) in near-linear $\mathcal{O} \left( d \log^4 d \right)$-time. Accompanying theoretical analysis proves that the proposed algorithm is guaranteed to deterministically recover all signals ${\bf x}$ satisfying a natural flatness (i.e., non-sparsity) condition for a particular choice of deterministic correlation-based measurements. A randomized version of these same measurements is then shown to provide nonuniform probabilistic recovery guarantees for arbitrary signals ${\bf x} \in \mathbb{C}^d$. Numerical experiments demonstrate the method's speed, accuracy, and robustness in practice -- all code is made publicly available. Finally, we conclude by develo** an extension of the proposed method to the sparse phase retrieval problem; specifically, we demonstrate a sublinear-time compressive phase retrieval algorithm which is guaranteed to recover a given $s$-sparse vector ${\bf x} \in \mathbb{C}^d$ with high probability in just $\mathcal{O}(s \log^5 s \cdot \log d)$-time using only $\mathcal{O}(s \log^4 s \cdot \log d)$ magnitude measurements. In doing so we demonstrate the existence of compressive phase retrieval algorithms with near-optimal linear-in-sparsity runtime complexities. △ Less

Submitted 9 July, 2016; v1 submitted 10 January, 2015; originally announced January 2015.

Comments: added more empirical evaluations/performance comparisons, clarifications/additions to introduction/abstract

Robust Sparse Phase Retrieval Made Easy

Authors: Mark Iwen, Aditya Viswanathan, Yang Wang

Abstract: In this short note we propose a simple two-stage sparse phase retrieval strategy that uses a near-optimal number of measurements, and is both computationally efficient and robust to measurement noise. In addition, the proposed strategy is fairly general, allowing for a large number of new measurement constructions and recovery algorithms to be designed with minimal effort. In this short note we propose a simple two-stage sparse phase retrieval strategy that uses a near-optimal number of measurements, and is both computationally efficient and robust to measurement noise. In addition, the proposed strategy is fairly general, allowing for a large number of new measurement constructions and recovery algorithms to be designed with minimal effort. △ Less

Submitted 24 April, 2015; v1 submitted 20 October, 2014; originally announced October 2014.

Fast Subspace Approximation via Greedy Least-Squares

Authors: Mark Iwen, Felix Krahmer

Abstract: In this note, we develop fast and deterministic dimensionality reduction techniques for a family of subspace approximation problems. Let $P\subset \mathbbm{R}^N$ be a given set of $M$ points. The techniques developed herein find an $O(n \log M)$-dimensional subspace that is guaranteed to always contain a near-best fit $n$-dimensional hyperplane $\mathcal{H}$ for $P$ with respect to the cumulative… ▽ More In this note, we develop fast and deterministic dimensionality reduction techniques for a family of subspace approximation problems. Let $P\subset \mathbbm{R}^N$ be a given set of $M$ points. The techniques developed herein find an $O(n \log M)$-dimensional subspace that is guaranteed to always contain a near-best fit $n$-dimensional hyperplane $\mathcal{H}$ for $P$ with respect to the cumulative projection error $(\sum_{{\bf x} \in P} \| {\bf x} - Π_\mathcal{H} {\bf x} \|^p_2)^{1/p}$, for any chosen $p > 2$. The deterministic algorithm runs in $\tilde{O} (MN^2)$-time, and can be randomized to run in only $\tilde{O} (MNn)$-time while maintaining its error guarantees with high probability. In the case $p = \infty$ the dimensionality reduction techniques can be combined with efficient algorithms for computing the John ellipsoid of a data set in order to produce an $n$-dimensional subspace whose maximum $\ell_2$-distance to any point in the convex hull of $P$ is minimized. The resulting algorithm remains $\tilde{O} (MNn)$-time. In addition, the dimensionality reduction techniques developed herein can also be combined with other existing subspace approximation algorithms for $2 < p \leq \infty$ - including more accurate algorithms based on convex programming relaxations - in order to reduce their runtimes. △ Less

Submitted 4 December, 2013; originally announced December 2013.

Near-Optimal Encoding for Sigma-Delta Quantization of Finite Frame Expansions

Authors: Mark Iwen, Rayan Saab

Abstract: In this paper we investigate encoding the bit-stream resulting from coarse Sigma-Delta quantization of finite frame expansions (i.e., overdetermined representations) of vectors. We show that for a wide range of finite-frames, including random frames and piecewise smooth frames, there exists a simple encoding algorithm ---acting only on the Sigma-Delta bit stream--- and an associated decoding algor… ▽ More In this paper we investigate encoding the bit-stream resulting from coarse Sigma-Delta quantization of finite frame expansions (i.e., overdetermined representations) of vectors. We show that for a wide range of finite-frames, including random frames and piecewise smooth frames, there exists a simple encoding algorithm ---acting only on the Sigma-Delta bit stream--- and an associated decoding algorithm that together yield an approximation error which decays exponentially in the number of bits used. The encoding strategy consists of applying a discrete random operator to the Sigma-Delta bit stream and assigning a binary codeword to the result. The reconstruction procedure is essentially linear and equivalent to solving a least squares minimization problem. △ Less

Submitted 8 July, 2013; originally announced July 2013.

Compressed Sensing with Sparse Binary Matrices: Instance Optimal Error Guarantees in Near-Optimal Time

Authors: M. A. Iwen

Abstract: A compressed sensing method consists of a rectangular measurement matrix, $M \in \mathbbm{R}^{m \times N}$ with $m \ll N$, together with an associated recovery algorithm, $\mathcal{A}: \mathbbm{R}^m \rightarrow \mathbbm{R}^N$. Compressed sensing methods aim to construct a high quality approximation to any given input vector ${\bf x} \in \mathbbm{R}^N$ using only $M {\bf x} \in \mathbbm{R}^m$ as in… ▽ More A compressed sensing method consists of a rectangular measurement matrix, $M \in \mathbbm{R}^{m \times N}$ with $m \ll N$, together with an associated recovery algorithm, $\mathcal{A}: \mathbbm{R}^m \rightarrow \mathbbm{R}^N$. Compressed sensing methods aim to construct a high quality approximation to any given input vector ${\bf x} \in \mathbbm{R}^N$ using only $M {\bf x} \in \mathbbm{R}^m$ as input. In particular, we focus herein on instance optimal nonlinear approximation error bounds for $M$ and $\mathcal{A}$ of the form $ \| {\bf x} - \mathcal{A} (M {\bf x}) \|_p \leq \| {\bf x} - {\bf x}^{\rm opt}_k \|_p + C k^{1/p - 1/q} \| {\bf x} - {\bf x}^{\rm opt}_k \|_q$ for ${\bf x} \in \mathbbm{R}^N$, where ${\bf x}^{\rm opt}_k$ is the best possible $k$-term approximation to ${\bf x}$. In this paper we develop a compressed sensing method whose associated recovery algorithm, $\mathcal{A}$, runs in $O((k \log k) \log N)$-time, matching a lower bound up to a $O(\log k)$ factor. This runtime is obtained by using a new class of sparse binary compressed sensing matrices of near optimal size in combination with sublinear-time recovery techniques motivated by sketching algorithms for high-volume data streams. The new class of matrices is constructed by randomly subsampling rows from well-chosen incoherent matrix constructions which already have a sub-linear number of rows. As a consequence, fewer random bits than previously required are needed in order to select the rows utilized by the fast reconstruction algorithms considered herein. △ Less

Submitted 24 February, 2013; originally announced February 2013.

A symbol-based algorithm for decoding bar codes

Authors: Mark Iwen, Fadil Santosa, Rachel Ward

Abstract: We investigate the problem of decoding a bar code from a signal measured with a hand-held laser-based scanner. Rather than formulating the inverse problem as one of binary image reconstruction, we instead incorporate the symbology of the bar code into the reconstruction algorithm directly, and search for a sparse representation of the UPC bar code with respect to this known dictionary. Our approac… ▽ More We investigate the problem of decoding a bar code from a signal measured with a hand-held laser-based scanner. Rather than formulating the inverse problem as one of binary image reconstruction, we instead incorporate the symbology of the bar code into the reconstruction algorithm directly, and search for a sparse representation of the UPC bar code with respect to this known dictionary. Our approach significantly reduces the degrees of freedom in the problem, allowing for accurate reconstruction that is robust to noise and unknown parameters in the scanning device. We propose a greedy reconstruction algorithm and provide robust reconstruction guarantees. Numerical examples illustrate the insensitivity of our symbology-based reconstruction to both imprecise model parameters and noise on the scanned measurements. △ Less

Submitted 25 October, 2012; originally announced October 2012.

Comments: 24 pages, 12 figures

MSC Class: 44A35; 94A12; 65K05; 94A08; 68U10

Approximation of Points on Low-Dimensional Manifolds Via Random Linear Projections

Authors: Mark A. Iwen, Mauro Maggioni

Abstract: This paper considers the approximate reconstruction of points, x \in R^D, which are close to a given compact d-dimensional submanifold, M, of R^D using a small number of linear measurements of x. In particular, it is shown that a number of measurements of x which is independent of the extrinsic dimension D suffices for highly accurate reconstruction of a given x with high probability. Furthermore,… ▽ More This paper considers the approximate reconstruction of points, x \in R^D, which are close to a given compact d-dimensional submanifold, M, of R^D using a small number of linear measurements of x. In particular, it is shown that a number of measurements of x which is independent of the extrinsic dimension D suffices for highly accurate reconstruction of a given x with high probability. Furthermore, it is also proven that all vectors, x, which are sufficiently close to M can be reconstructed with uniform approximation guarantees when the number of linear measurements of x depends logarithmically on D. Finally, the proofs of these facts are constructive: A practical algorithm for manifold-based signal recovery is presented in the process of proving the two main results mentioned above. △ Less

Submitted 15 April, 2012; originally announced April 2012.

On the Design of Deterministic Matrices for Fast Recovery of Fourier Compressible Functions

Authors: J. Bailey, M. A. Iwen, C. V. Spencer

Abstract: We present a general class of compressed sensing matrices which are then demonstrated to have associated sublinear-time sparse approximation algorithms. We then develop methods for constructing specialized matrices from this class which are sparse when multiplied with a discrete Fourier transform matrix. Ultimately, these considerations improve previous sampling requirements for deterministic spar… ▽ More We present a general class of compressed sensing matrices which are then demonstrated to have associated sublinear-time sparse approximation algorithms. We then develop methods for constructing specialized matrices from this class which are sparse when multiplied with a discrete Fourier transform matrix. Ultimately, these considerations improve previous sampling requirements for deterministic sparse Fourier transform methods. △ Less

Submitted 30 May, 2011; originally announced May 2011.

Improved Approximation Guarantees for Sublinear-Time Fourier Algorithms

Authors: M. A. Iwen

Abstract: In this paper modified variants of the sparse Fourier transform algorithms from [14] are presented which improve on the approximation error bounds of the original algorithms. In addition, simple methods for extending the improved sparse Fourier transforms to higher dimensional settings are developed. As a consequence, approximate Fourier transforms are obtained which will identify a near-optimal k… ▽ More In this paper modified variants of the sparse Fourier transform algorithms from [14] are presented which improve on the approximation error bounds of the original algorithms. In addition, simple methods for extending the improved sparse Fourier transforms to higher dimensional settings are developed. As a consequence, approximate Fourier transforms are obtained which will identify a near-optimal k-term Fourier series for any given input function, $f : [0, 2 pi] -> C, in O(k^2 \cdot D^4)$ time (neglecting logarithmic factors). Faster randomized Fourier algorithm variants with runtime complexities that scale linearly in the sparsity parameter k are also presented. △ Less

Submitted 30 September, 2010; originally announced October 2010.

ACM Class: G.1.2

A Deterministic Sub-linear Time Sparse Fourier Algorithm via Non-adaptive Compressed Sensing Methods

Authors: M. A. Iwen

Abstract: We study the problem of estimating the best B term Fourier representation for a given frequency-sparse signal (i.e., vector) $\textbf{A}$ of length $N \gg B$. More explicitly, we investigate how to deterministically identify B of the largest magnitude frequencies of $\hat{\textbf{A}}$, and estimate their coefficients, in polynomial$(B,\log N)$ time. Randomized sub-linear time algorithms which ha… ▽ More We study the problem of estimating the best B term Fourier representation for a given frequency-sparse signal (i.e., vector) $\textbf{A}$ of length $N \gg B$. More explicitly, we investigate how to deterministically identify B of the largest magnitude frequencies of $\hat{\textbf{A}}$, and estimate their coefficients, in polynomial$(B,\log N)$ time. Randomized sub-linear time algorithms which have a small (controllable) probability of failure for each processed signal exist for solving this problem. However, for failure intolerant applications such as those involving mission-critical hardware designed to process many signals over a long lifetime, deterministic algorithms with no probability of failure are highly desirable. In this paper we build on the deterministic Compressed Sensing results of Cormode and Muthukrishnan (CM) \cite{CMDetCS3,CMDetCS1,CMDetCS2} in order to develop the first known deterministic sub-linear time sparse Fourier Transform algorithm suitable for failure intolerant applications. Furthermore, in the process of develo** our new Fourier algorithm, we present a simplified deterministic Compressed Sensing algorithm which improves on CM's algebraic compressibility results while simultaneously maintaining their results concerning exponential decay. △ Less

Submitted 9 August, 2007; originally announced August 2007.

Comments: 16 pages total, 10 in paper, 6 in appended