-
Efficient Detection of Long Consistent Cycles and its Application to Distributed Synchronization
Authors:
Shaohan Li,
Yunpeng Shi,
Gilad Lerman
Abstract:
Group synchronization plays a crucial role in global pipelines for Structure from Motion (SfM). Its formulation is nonconvex and it is faced with highly corrupted measurements. Cycle consistency has been effective in addressing these challenges. However, computationally efficient solutions are needed for cycles longer than three, especially in practical scenarios where 3-cycles are unavailable. To…
▽ More
Group synchronization plays a crucial role in global pipelines for Structure from Motion (SfM). Its formulation is nonconvex and it is faced with highly corrupted measurements. Cycle consistency has been effective in addressing these challenges. However, computationally efficient solutions are needed for cycles longer than three, especially in practical scenarios where 3-cycles are unavailable. To overcome this computational bottleneck, we propose an algorithm for group synchronization that leverages information from cycles of lengths ranging from three to six with a time complexity of order $O(n^3)$ (or $O(n^{2.373})$ when using a faster matrix multiplication algorithm). We establish non-trivial theory for this and related methods that achieves competitive sample complexity, assuming the uniform corruption model. To advocate the practical need for our method, we consider distributed group synchronization, which requires at least 4-cycles, and we illustrate state-of-the-art performance by our method in this context.
△ Less
Submitted 5 July, 2024;
originally announced July 2024.
-
Artificial Intelligence and Algorithmic Price Collusion in Two-sided Markets
Authors:
Cristian Chica,
Yinglong Guo,
Gilad Lerman
Abstract:
Algorithmic price collusion facilitated by artificial intelligence (AI) algorithms raises significant concerns. We examine how AI agents using Q-learning engage in tacit collusion in two-sided markets. Our experiments reveal that AI-driven platforms achieve higher collusion levels compared to Bertrand competition. Increased network externalities significantly enhance collusion, suggesting AI algor…
▽ More
Algorithmic price collusion facilitated by artificial intelligence (AI) algorithms raises significant concerns. We examine how AI agents using Q-learning engage in tacit collusion in two-sided markets. Our experiments reveal that AI-driven platforms achieve higher collusion levels compared to Bertrand competition. Increased network externalities significantly enhance collusion, suggesting AI algorithms exploit them to maximize profits. Higher user heterogeneity or greater utility from outside options generally reduce collusion, while higher discount rates increase it. Tacit collusion remains feasible even at low discount rates. To mitigate collusive behavior and inform potential regulatory measures, we propose incorporating a penalty term in the Q-learning algorithm.
△ Less
Submitted 4 July, 2024;
originally announced July 2024.
-
A Subspace-Constrained Tyler's Estimator and its Applications to Structure from Motion
Authors:
Feng Yu,
Teng Zhang,
Gilad Lerman
Abstract:
We present the subspace-constrained Tyler's estimator (STE) designed for recovering a low-dimensional subspace within a dataset that may be highly corrupted with outliers. STE is a fusion of the Tyler's M-estimator (TME) and a variant of the fast median subspace. Our theoretical analysis suggests that, under a common inlier-outlier model, STE can effectively recover the underlying subspace, even w…
▽ More
We present the subspace-constrained Tyler's estimator (STE) designed for recovering a low-dimensional subspace within a dataset that may be highly corrupted with outliers. STE is a fusion of the Tyler's M-estimator (TME) and a variant of the fast median subspace. Our theoretical analysis suggests that, under a common inlier-outlier model, STE can effectively recover the underlying subspace, even when it contains a smaller fraction of inliers relative to other methods in the field of robust subspace recovery. We apply STE in the context of Structure from Motion (SfM) in two ways: for robust estimation of the fundamental matrix and for the removal of outlying cameras, enhancing the robustness of the SfM pipeline. Numerical experiments confirm the state-of-the-art performance of our method in these applications. This research makes significant contributions to the field of robust subspace recovery, particularly in the context of computer vision and 3D reconstruction.
△ Less
Submitted 7 May, 2024; v1 submitted 17 April, 2024;
originally announced April 2024.
-
Theoretical Guarantees for the Subspace-Constrained Tyler's Estimator
Authors:
Gilad Lerman,
Feng Yu,
Teng Zhang
Abstract:
This work analyzes the subspace-constrained Tyler's estimator (STE) designed for recovering a low-dimensional subspace within a dataset that may be highly corrupted with outliers. It assumes a weak inlier-outlier model and allows the fraction of inliers to be smaller than a fraction that leads to computational hardness of the robust subspace recovery problem. It shows that in this setting, if the…
▽ More
This work analyzes the subspace-constrained Tyler's estimator (STE) designed for recovering a low-dimensional subspace within a dataset that may be highly corrupted with outliers. It assumes a weak inlier-outlier model and allows the fraction of inliers to be smaller than a fraction that leads to computational hardness of the robust subspace recovery problem. It shows that in this setting, if the initialization of STE, which is an iterative algorithm, satisfies a certain condition, then STE can effectively recover the underlying subspace. It further shows that under the generalized haystack model, STE initialized by the Tyler's M-estimator (TME), can recover the subspace when the fraction of iniliers is too small for TME to handle.
△ Less
Submitted 12 April, 2024; v1 submitted 27 March, 2024;
originally announced March 2024.
-
The effect of Leaky ReLUs on the training and generalization of overparameterized networks
Authors:
Yinglong Guo,
Shaohan Li,
Gilad Lerman
Abstract:
We investigate the training and generalization errors of overparameterized neural networks (NNs) with a wide class of leaky rectified linear unit (ReLU) functions. More specifically, we carefully upper bound both the convergence rate of the training error and the generalization error of such NNs and investigate the dependence of these bounds on the Leaky ReLU parameter, $α$. We show that $α=-1$, w…
▽ More
We investigate the training and generalization errors of overparameterized neural networks (NNs) with a wide class of leaky rectified linear unit (ReLU) functions. More specifically, we carefully upper bound both the convergence rate of the training error and the generalization error of such NNs and investigate the dependence of these bounds on the Leaky ReLU parameter, $α$. We show that $α=-1$, which corresponds to the absolute value activation function, is optimal for the training error bound. Furthermore, in special settings, it is also optimal for the generalization error bound. Numerical experiments empirically support the practical choices guided by the theory.
△ Less
Submitted 25 February, 2024; v1 submitted 19 February, 2024;
originally announced February 2024.
-
Improved Convergence Rates of Windowed Anderson Acceleration for Symmetric Fixed-Point Iterations
Authors:
Casey Garner,
Gilad Lerman,
Teng Zhang
Abstract:
This paper studies the commonly utilized windowed Anderson acceleration (AA) algorithm for fixed-point methods, $x^{(k+1)}=q(x^{(k)})$. It provides the first proof that when the operator $q$ is linear and symmetric the windowed AA, which uses a sliding window of prior iterates, improves the root-linear convergence factor over the fixed-point iterations. When $q$ is nonlinear, yet has a symmetric J…
▽ More
This paper studies the commonly utilized windowed Anderson acceleration (AA) algorithm for fixed-point methods, $x^{(k+1)}=q(x^{(k)})$. It provides the first proof that when the operator $q$ is linear and symmetric the windowed AA, which uses a sliding window of prior iterates, improves the root-linear convergence factor over the fixed-point iterations. When $q$ is nonlinear, yet has a symmetric Jacobian at a fixed point, a slightly modified AA algorithm is proved to have an analogous root-linear convergence factor improvement over fixed-point iterations. Simulations verify our observations. Furthermore, experiments with different data models demonstrate AA is significantly superior to the standard fixed-point methods for Tyler's M-estimation.
△ Less
Submitted 8 March, 2024; v1 submitted 4 November, 2023;
originally announced November 2023.
-
Monotone Generative Modeling via a Gromov-Monge Embedding
Authors:
Wonjun Lee,
Yifei Yang,
Dongmian Zou,
Gilad Lerman
Abstract:
Generative adversarial networks (GANs) are popular for generative tasks; however, they often require careful architecture selection, extensive empirical tuning, and are prone to mode collapse. To overcome these challenges, we propose a novel model that identifies the low-dimensional structure of the underlying data distribution, maps it into a low-dimensional latent space while preserving the unde…
▽ More
Generative adversarial networks (GANs) are popular for generative tasks; however, they often require careful architecture selection, extensive empirical tuning, and are prone to mode collapse. To overcome these challenges, we propose a novel model that identifies the low-dimensional structure of the underlying data distribution, maps it into a low-dimensional latent space while preserving the underlying geometry, and then optimally transports a reference measure to the embedded distribution. We prove three key properties of our method: 1) The encoder preserves the geometry of the underlying data; 2) The generator is $c$-cyclically monotone, where $c$ is an intrinsic embedding cost employed by the encoder; and 3) The discriminator's modulus of continuity improves with the geometric preservation of the data. Numerical experiments demonstrate the effectiveness of our approach in generating high-quality images and exhibiting robustness to both mode collapse and training instability.
△ Less
Submitted 3 July, 2024; v1 submitted 2 November, 2023;
originally announced November 2023.
-
Spectrally Constrained Optimization
Authors:
Casey Garner,
Gilad Lerman,
Shuzhong Zhang
Abstract:
We investigate how to solve smooth matrix optimization problems with general linear inequality constraints on the eigenvalues of a symmetric matrix. We present solution methods to obtain exact global minima for linear objective functions, i.e., $F(X) = \langle C, X \rangle$, and perform exact projections onto the eigenvalue constraint set. Two first-order algorithms are developed to obtain first-o…
▽ More
We investigate how to solve smooth matrix optimization problems with general linear inequality constraints on the eigenvalues of a symmetric matrix. We present solution methods to obtain exact global minima for linear objective functions, i.e., $F(X) = \langle C, X \rangle$, and perform exact projections onto the eigenvalue constraint set. Two first-order algorithms are developed to obtain first-order stationary points for general non-convex objective functions. Both methods are proven to converge sublinearly when the constraint set is convex. Numerical experiments demonstrate the applicability of both the model and the methods.
△ Less
Submitted 12 July, 2023; v1 submitted 8 July, 2023;
originally announced July 2023.
-
Cubic-Regularized Newton for Spectral Constrained Matrix Optimization and its Application to Fairness
Authors:
Casey Garner,
Gilad Lerman,
Shuzhong Zhang
Abstract:
Matrix functions are utilized to rewrite smooth spectral constrained matrix optimization problems as smooth unconstrained problems over the set of symmetric matrices which are then solved via the cubic-regularized Newton method. A second-order chain rule identity for matrix functions is proven to compute the higher-order derivatives to implement cubic-regularized Newton, and a new convergence anal…
▽ More
Matrix functions are utilized to rewrite smooth spectral constrained matrix optimization problems as smooth unconstrained problems over the set of symmetric matrices which are then solved via the cubic-regularized Newton method. A second-order chain rule identity for matrix functions is proven to compute the higher-order derivatives to implement cubic-regularized Newton, and a new convergence analysis is provided for cubic-regularized Newton for matrix vector spaces. We demonstrate the applicability of our approach by conducting numerical experiments on both synthetic and real datasets. In our experiments, we formulate a new model for estimating fair and robust covariance matrices in the spirit of the Tyler's M-estimator (TME) model and demonstrate its advantage.
△ Less
Submitted 2 September, 2022;
originally announced September 2022.
-
Robust Group Synchronization via Quadratic Programming
Authors:
Yunpeng Shi,
Cole Wyeth,
Gilad Lerman
Abstract:
We propose a novel quadratic programming formulation for estimating the corruption levels in group synchronization, and use these estimates to solve this problem. Our objective function exploits the cycle consistency of the group and we thus refer to our method as detection and estimation of structural consistency (DESC). This general framework can be extended to other algebraic and geometric stru…
▽ More
We propose a novel quadratic programming formulation for estimating the corruption levels in group synchronization, and use these estimates to solve this problem. Our objective function exploits the cycle consistency of the group and we thus refer to our method as detection and estimation of structural consistency (DESC). This general framework can be extended to other algebraic and geometric structures. Our formulation has the following advantages: it can tolerate corruption as high as the information-theoretic bound, it does not require a good initialization for the estimates of group elements, it has a simple interpretation, and under some mild conditions the global minimum of our objective function exactly recovers the corruption levels. We demonstrate the competitive accuracy of our approach on both synthetic and real data experiments of rotation averaging.
△ Less
Submitted 17 June, 2022;
originally announced June 2022.
-
An Unpooling Layer for Graph Generation
Authors:
Yinglong Guo,
Dongmian Zou,
Gilad Lerman
Abstract:
We propose a novel and trainable graph unpooling layer for effective graph generation. Given a graph with features, the unpooling layer enlarges this graph and learns its desired new structure and features. Since this unpooling layer is trainable, it can be applied to graph generation either in the decoder of a variational autoencoder or in the generator of a generative adversarial network (GAN).…
▽ More
We propose a novel and trainable graph unpooling layer for effective graph generation. Given a graph with features, the unpooling layer enlarges this graph and learns its desired new structure and features. Since this unpooling layer is trainable, it can be applied to graph generation either in the decoder of a variational autoencoder or in the generator of a generative adversarial network (GAN). We prove that the unpooled graph remains connected and any connected graph can be sequentially unpooled from a 3-nodes graph. We apply the unpooling layer within the GAN generator. Since the most studied instance of graph generation is molecular generation, we test our ideas in this context. Using the QM9 and ZINC datasets, we demonstrate the improvement obtained by using the unpooling layer instead of an adjacency-matrix-based approach.
△ Less
Submitted 5 March, 2023; v1 submitted 3 June, 2022;
originally announced June 2022.
-
Fast, Accurate and Memory-Efficient Partial Permutation Synchronization
Authors:
Shaohan Li,
Yunpeng Shi,
Gilad Lerman
Abstract:
Previous partial permutation synchronization (PPS) algorithms, which are commonly used for multi-object matching, often involve computation-intensive and memory-demanding matrix operations. These operations become intractable for large scale structure-from-motion datasets. For pure permutation synchronization, the recent Cycle-Edge Message Passing (CEMP) framework suggests a memory-efficient and f…
▽ More
Previous partial permutation synchronization (PPS) algorithms, which are commonly used for multi-object matching, often involve computation-intensive and memory-demanding matrix operations. These operations become intractable for large scale structure-from-motion datasets. For pure permutation synchronization, the recent Cycle-Edge Message Passing (CEMP) framework suggests a memory-efficient and fast solution. Here we overcome the restriction of CEMP to compact groups and propose an improved algorithm, CEMP-Partial, for estimating the corruption levels of the observed partial permutations. It allows us to subsequently implement a nonconvex weighted projected power method without the need of spectral initialization. The resulting new PPS algorithm, MatchFAME (Fast, Accurate and Memory-Efficient Matching), only involves sparse matrix operations, and thus enjoys lower time and space complexities in comparison to previous PPS algorithms. We prove that under adversarial corruption, though without additive noise and with certain assumptions, CEMP-Partial is able to exactly classify corrupted and clean partial permutations. We demonstrate the state-of-the-art accuracy, speed and memory efficiency of our method on both synthetic and real datasets.
△ Less
Submitted 31 March, 2022; v1 submitted 30 March, 2022;
originally announced March 2022.
-
Stochastic and Private Nonconvex Outlier-Robust PCA
Authors:
Tyler Maunu,
Chenyu Yu,
Gilad Lerman
Abstract:
We develop theoretically guaranteed stochastic methods for outlier-robust PCA. Outlier-robust PCA seeks an underlying low-dimensional linear subspace from a dataset that is corrupted with outliers. We are able to show that our methods, which involve stochastic geodesic gradient descent over the Grassmannian manifold, converge and recover an underlying subspace in various regimes through the develo…
▽ More
We develop theoretically guaranteed stochastic methods for outlier-robust PCA. Outlier-robust PCA seeks an underlying low-dimensional linear subspace from a dataset that is corrupted with outliers. We are able to show that our methods, which involve stochastic geodesic gradient descent over the Grassmannian manifold, converge and recover an underlying subspace in various regimes through the development of a novel convergence analysis. The main application of this method is an effective differentially private algorithm for outlier-robust PCA that uses a Gaussian noise mechanism within the stochastic gradient method. Our results emphasize the advantages of the nonconvex methods over another convex approach to solving this problem in the differentially private setting. Experiments on synthetic and stylized data verify these results.
△ Less
Submitted 17 March, 2022;
originally announced March 2022.
-
Robust Vector Quantized-Variational Autoencoder
Authors:
Chieh-Hsin Lai,
Dongmian Zou,
Gilad Lerman
Abstract:
Image generative models can learn the distributions of the training data and consequently generate examples by sampling from these distributions. However, when the training dataset is corrupted with outliers, generative models will likely produce examples that are also similar to the outliers. In fact, a small portion of outliers may induce state-of-the-art generative models, such as Vector Quanti…
▽ More
Image generative models can learn the distributions of the training data and consequently generate examples by sampling from these distributions. However, when the training dataset is corrupted with outliers, generative models will likely produce examples that are also similar to the outliers. In fact, a small portion of outliers may induce state-of-the-art generative models, such as Vector Quantized-Variational AutoEncoder (VQ-VAE), to learn a significant mode from the outliers. To mitigate this problem, we propose a robust generative model based on VQ-VAE, which we name Robust VQ-VAE (RVQ-VAE). In order to achieve robustness, RVQ-VAE uses two separate codebooks for the inliers and outliers. To ensure the codebooks embed the correct components, we iteratively update the sets of inliers and outliers during each training epoch. To ensure that the encoded data points are matched to the correct codebooks, we quantize using a weighted Euclidean distance, whose weights are determined by directional variances of the codebooks. Both codebooks, together with the encoder and decoder, are trained jointly according to the reconstruction loss and the quantization loss. We experimentally demonstrate that RVQ-VAE is able to generate examples from inliers even if a large portion of the training data points are corrupted.
△ Less
Submitted 20 September, 2022; v1 submitted 4 February, 2022;
originally announced February 2022.
-
Scalable Cluster-Consistency Statistics for Robust Multi-Object Matching
Authors:
Yunpeng Shi,
Shaohan Li,
Tyler Maunu,
Gilad Lerman
Abstract:
We develop new statistics for robustly filtering corrupted keypoint matches in the structure from motion pipeline. The statistics are based on consistency constraints that arise within the clustered structure of the graph of keypoint matches. The statistics are designed to give smaller values to corrupted matches and than uncorrupted matches. These new statistics are combined with an iterative rew…
▽ More
We develop new statistics for robustly filtering corrupted keypoint matches in the structure from motion pipeline. The statistics are based on consistency constraints that arise within the clustered structure of the graph of keypoint matches. The statistics are designed to give smaller values to corrupted matches and than uncorrupted matches. These new statistics are combined with an iterative reweighting scheme to filter keypoints, which can then be fed into any standard structure from motion pipeline. This filtering method can be efficiently implemented and scaled to massive datasets as it only requires sparse matrix multiplication. We demonstrate the efficacy of this method on synthetic and real structure from motion datasets and show that it achieves state-of-the-art accuracy and speed in these tasks.
△ Less
Submitted 13 January, 2022;
originally announced January 2022.
-
Ensemble Riemannian Data Assimilation over the Wasserstein Space
Authors:
Sagar K. Tamang,
Ardeshir Ebtehaj,
Peter J. Van Leeuwen,
Dongmian Zou,
Gilad Lerman
Abstract:
In this paper, we present an ensemble data assimilation paradigm over a Riemannian manifold equipped with the Wasserstein metric. Unlike the Eulerian penalization of error in the Euclidean space, the Wasserstein metric can capture translation and difference between the shapes of square-integrable probability distributions of the background state and observations -- enabling to formally penalize ge…
▽ More
In this paper, we present an ensemble data assimilation paradigm over a Riemannian manifold equipped with the Wasserstein metric. Unlike the Eulerian penalization of error in the Euclidean space, the Wasserstein metric can capture translation and difference between the shapes of square-integrable probability distributions of the background state and observations -- enabling to formally penalize geophysical biases in state-space with non-Gaussian distributions. The new approach is applied to dissipative and chaotic evolutionary dynamics and its potential advantages and limitations are highlighted compared to the classic variational and filtering data assimilation approaches under systematic and random errors.
△ Less
Submitted 24 March, 2021; v1 submitted 7 September, 2020;
originally announced September 2020.
-
Message Passing Least Squares Framework and its Application to Rotation Synchronization
Authors:
Yunpeng Shi,
Gilad Lerman
Abstract:
We propose an efficient algorithm for solving group synchronization under high levels of corruption and noise, while we focus on rotation synchronization. We first describe our recent theoretically guaranteed message passing algorithm that estimates the corruption levels of the measured group ratios. We then propose a novel reweighted least squares method to estimate the group elements, where the…
▽ More
We propose an efficient algorithm for solving group synchronization under high levels of corruption and noise, while we focus on rotation synchronization. We first describe our recent theoretically guaranteed message passing algorithm that estimates the corruption levels of the measured group ratios. We then propose a novel reweighted least squares method to estimate the group elements, where the weights are initialized and iteratively updated using the estimated corruption levels. We demonstrate the superior performance of our algorithm over state-of-the-art methods for rotation synchronization using both synthetic and real data.
△ Less
Submitted 14 August, 2020; v1 submitted 27 July, 2020;
originally announced July 2020.
-
You Need to Calm Down: Calmness Regularity for a Class of Seminorm Optimization Problems
Authors:
Alex Gutierrez,
Gilad Lerman,
Sam Stewart
Abstract:
Compressed sensing involves solving a minimization problem with objective function $Ω(\boldsymbol{x}) = \|\boldsymbol{x}\|_1$ and linear constraints $\boldsymbol{A} \boldsymbol{x} = \boldsymbol{b}$. Previous work has explored robustness to errors in $\boldsymbol{A}$ and $\boldsymbol{b}$ under special assumptions. Motivated by these results, we explore robustness to errors in $\boldsymbol{A}$ for a…
▽ More
Compressed sensing involves solving a minimization problem with objective function $Ω(\boldsymbol{x}) = \|\boldsymbol{x}\|_1$ and linear constraints $\boldsymbol{A} \boldsymbol{x} = \boldsymbol{b}$. Previous work has explored robustness to errors in $\boldsymbol{A}$ and $\boldsymbol{b}$ under special assumptions. Motivated by these results, we explore robustness to errors in $\boldsymbol{A}$ for a wider class of objective functions $Ω$ and for a more general setting, where the solution may not be unique. Similar results for errors in $\boldsymbol{b}$ are known and easier to prove. More precisely, for a seminorm $Ω(\boldsymbol{x})$ with a polyhedral unit ball, we prove that the set-valued map $S(\boldsymbol{A}) = \arg \min_{\boldsymbol{A} \boldsymbol{x} = \boldsymbol{b}} Ω(\boldsymbol{x})$ is calm in $\boldsymbol{A}$, where calmness is a kind of local Lipschitz regularity.
△ Less
Submitted 17 July, 2020; v1 submitted 11 July, 2020;
originally announced July 2020.
-
Robust Multi-object Matching via Iterative Reweighting of the Graph Connection Laplacian
Authors:
Yunpeng Shi,
Shaohan Li,
Gilad Lerman
Abstract:
We propose an efficient and robust iterative solution to the multi-object matching problem. We first clarify serious limitations of current methods as well as the inappropriateness of the standard iteratively reweighted least squares procedure. In view of these limitations, we suggest a novel and more reliable iterative reweighting strategy that incorporates information from higher-order neighborh…
▽ More
We propose an efficient and robust iterative solution to the multi-object matching problem. We first clarify serious limitations of current methods as well as the inappropriateness of the standard iteratively reweighted least squares procedure. In view of these limitations, we suggest a novel and more reliable iterative reweighting strategy that incorporates information from higher-order neighborhoods by exploiting the graph connection Laplacian. We demonstrate the superior performance of our procedure over state-of-the-art methods using both synthetic and real datasets.
△ Less
Submitted 24 October, 2020; v1 submitted 11 June, 2020;
originally announced June 2020.
-
Novelty Detection via Robust Variational Autoencoding
Authors:
Chieh-Hsin Lai,
Dongmian Zou,
Gilad Lerman
Abstract:
We propose a new method for novelty detection that can tolerate high corruption of the training points, whereas previous works assumed either no or very low corruption. Our method trains a robust variational autoencoder (VAE), which aims to generate a model for the uncorrupted training points. To gain robustness to high corruption, we incorporate the following four changes to the common VAE: 1. Ex…
▽ More
We propose a new method for novelty detection that can tolerate high corruption of the training points, whereas previous works assumed either no or very low corruption. Our method trains a robust variational autoencoder (VAE), which aims to generate a model for the uncorrupted training points. To gain robustness to high corruption, we incorporate the following four changes to the common VAE: 1. Extracting crucial features of the latent code by a carefully designed dimension reduction component for distributions; 2. Modeling the latent distribution as a mixture of Gaussian low-rank inliers and full-rank outliers, where the testing only uses the inlier model; 3. Applying the Wasserstein-1 metric for regularization, instead of the Kullback-Leibler (KL) divergence; and 4. Using a robust error for reconstruction. We establish both robustness to outliers and suitability to low-rank modeling of the Wasserstein metric as opposed to the KL divergence. We illustrate state-of-the-art results on standard benchmarks.
△ Less
Submitted 1 March, 2023; v1 submitted 9 June, 2020;
originally announced June 2020.
-
Regularized Variational Data Assimilation for Bias Treatment using the Wasserstein Metric
Authors:
Sagar K. Tamang,
Ardeshir Ebtehaj,
Dongmian Zou,
Gilad Lerman
Abstract:
This paper presents a new variational data assimilation (VDA) approach for the formal treatment of bias in both model outputs and observations. This approach relies on the Wasserstein metric stemming from the theory of optimal mass transport to penalize the distance between the probability histograms of the analysis state and an a priori reference dataset, which is likely to be more uncertain but…
▽ More
This paper presents a new variational data assimilation (VDA) approach for the formal treatment of bias in both model outputs and observations. This approach relies on the Wasserstein metric stemming from the theory of optimal mass transport to penalize the distance between the probability histograms of the analysis state and an a priori reference dataset, which is likely to be more uncertain but less biased than both model and observations. Unlike previous bias-aware VDA approaches, the new Wasserstein metric VDA (WM-VDA) dynamically treats systematic biases of unknown magnitude and sign in both model and observations through assimilation of the reference data in the probability domain and can fully recover the probability histogram of the analysis state. The performance of WM-VDA is compared with the classic three-dimensional VDA (3D-Var) scheme on first-order linear dynamics and the chaotic Lorenz attractor. Under positive systematic biases in both model and observations, we consistently demonstrate a significant reduction in the forecast bias and unbiased root mean squared error.
△ Less
Submitted 4 March, 2020;
originally announced March 2020.
-
Depth Descent Synchronization in $\mathrm{SO}(D)$
Authors:
Tyler Maunu,
Gilad Lerman
Abstract:
We give robust recovery results for synchronization on the rotation group, $\mathrm{SO}(D)$. In particular, we consider an adversarial corruption setting, where a limited percentage of the observations are arbitrarily corrupted. We give a novel algorithm that exploits Tukey depth in the tangent space, which exactly recovers the underlying rotations up to an outlier percentage of $1/(D(D-1)+2)$. Th…
▽ More
We give robust recovery results for synchronization on the rotation group, $\mathrm{SO}(D)$. In particular, we consider an adversarial corruption setting, where a limited percentage of the observations are arbitrarily corrupted. We give a novel algorithm that exploits Tukey depth in the tangent space, which exactly recovers the underlying rotations up to an outlier percentage of $1/(D(D-1)+2)$. This corresponds to an outlier fraction of $1/4$ for $\mathrm{SO}(2)$ and $1/8$ for $\mathrm{SO}(3)$. In the case of $D=2$, we demonstrate that a variant of this algorithm converges linearly to the ground truth rotations. We finish by discussing this result in relation to a simpler nonconvex energy minimization framework based on least absolute deviations, which exhibits spurious fixed points.
△ Less
Submitted 17 March, 2022; v1 submitted 12 February, 2020;
originally announced February 2020.
-
Robust Group Synchronization via Cycle-Edge Message Passing
Authors:
Gilad Lerman,
Yunpeng Shi
Abstract:
We propose a general framework for solving the group synchronization problem, where we focus on the setting of adversarial or uniform corruption and sufficiently small noise. Specifically, we apply a novel message passing procedure that uses cycle consistency information in order to estimate the corruption levels of group ratios and consequently solve the synchronization problem in our setting. We…
▽ More
We propose a general framework for solving the group synchronization problem, where we focus on the setting of adversarial or uniform corruption and sufficiently small noise. Specifically, we apply a novel message passing procedure that uses cycle consistency information in order to estimate the corruption levels of group ratios and consequently solve the synchronization problem in our setting. We first explain why the group cycle consistency information is essential for effectively solving group synchronization problems. We then establish exact recovery and linear convergence guarantees for the proposed message passing procedure under a deterministic setting with adversarial corruption. These guarantees hold as long as the ratio of corrupted cycles per edge is bounded by a reasonable constant. We also establish the stability of the proposed procedure to sub-Gaussian noise. We further establish exact recovery with high probability under a common uniform corruption model.
△ Less
Submitted 27 July, 2021; v1 submitted 24 December, 2019;
originally announced December 2019.
-
Robust Subspace Recovery with Adversarial Outliers
Authors:
Tyler Maunu,
Gilad Lerman
Abstract:
We study the problem of robust subspace recovery (RSR) in the presence of adversarial outliers. That is, we seek a subspace that contains a large portion of a dataset when some fraction of the data points are arbitrarily corrupted. We first examine a theoretical estimator that is intractable to calculate and use it to derive information-theoretic bounds of exact recovery. We then propose two tract…
▽ More
We study the problem of robust subspace recovery (RSR) in the presence of adversarial outliers. That is, we seek a subspace that contains a large portion of a dataset when some fraction of the data points are arbitrarily corrupted. We first examine a theoretical estimator that is intractable to calculate and use it to derive information-theoretic bounds of exact recovery. We then propose two tractable estimators: a variant of RANSAC and a simple relaxation of the theoretical estimator. The two estimators are fast to compute and achieve state-of-the-art theoretical performance in a noiseless RSR setting with adversarial outliers. The former estimator achieves better theoretical guarantees in the noiseless case, while the latter estimator is robust to small noise, and its guarantees significantly improve with non-adversarial models of outliers. We give a complete comparison of guarantees for the adversarial RSR problem, as well as a short discussion on the estimation of affine subspaces.
△ Less
Submitted 5 April, 2019;
originally announced April 2019.
-
Robust Subspace Recovery Layer for Unsupervised Anomaly Detection
Authors:
Chieh-Hsin Lai,
Dongmian Zou,
Gilad Lerman
Abstract:
We propose a neural network for unsupervised anomaly detection with a novel robust subspace recovery layer (RSR layer). This layer seeks to extract the underlying subspace from a latent representation of the given data and removes outliers that lie away from this subspace. It is used within an autoencoder. The encoder maps the data into a latent space, from which the RSR layer extracts the subspac…
▽ More
We propose a neural network for unsupervised anomaly detection with a novel robust subspace recovery layer (RSR layer). This layer seeks to extract the underlying subspace from a latent representation of the given data and removes outliers that lie away from this subspace. It is used within an autoencoder. The encoder maps the data into a latent space, from which the RSR layer extracts the subspace. The decoder then smoothly maps back the underlying subspace to a "manifold" close to the original inliers. Inliers and outliers are distinguished according to the distances between the original and mapped positions (small for inliers and large for outliers). Extensive numerical experiments with both image and document datasets demonstrate state-of-the-art precision and recall.
△ Less
Submitted 24 December, 2019; v1 submitted 30 March, 2019;
originally announced April 2019.
-
Analysis and algorithms for $\ell_p$-based semi-supervised learning on graphs
Authors:
Mauricio Flores,
Jeff Calder,
Gilad Lerman
Abstract:
This paper addresses theory and applications of $\ell_p$-based Laplacian regularization in semi-supervised learning. The graph $p$-Laplacian for $p>2$ has been proposed recently as a replacement for the standard ($p=2$) graph Laplacian in semi-supervised learning problems with very few labels, where Laplacian learning is degenerate.
In the first part of the paper we prove new discrete to continu…
▽ More
This paper addresses theory and applications of $\ell_p$-based Laplacian regularization in semi-supervised learning. The graph $p$-Laplacian for $p>2$ has been proposed recently as a replacement for the standard ($p=2$) graph Laplacian in semi-supervised learning problems with very few labels, where Laplacian learning is degenerate.
In the first part of the paper we prove new discrete to continuum convergence results for $p$-Laplace problems on $k$-nearest neighbor ($k$-NN) graphs, which are more commonly used in practice than random geometric graphs. Our analysis shows that, on $k$-NN graphs, the $p$-Laplacian retains information about the data distribution as $p\to \infty$ and Lipschitz learning ($p=\infty$) is sensitive to the data distribution. This situation can be contrasted with random geometric graphs, where the $p$-Laplacian forgets the data distribution as $p\to \infty$. We also present a general framework for proving discrete to continuum convergence results in graph-based learning that only requires pointwise consistency and monotonicity.
In the second part of the paper, we develop fast algorithms for solving the variational and game-theoretic $p$-Laplace equations on weighted graphs for $p>2$. We present several efficient and scalable algorithms for both formulations, and present numerical results on synthetic data indicating their convergence properties. Finally, we conduct extensive numerical experiments on the MNIST, FashionMNIST and EMNIST datasets that illustrate the effectiveness of the $p$-Laplacian formulation for semi-supervised learning with few labels. In particular, we find that Lipschitz learning ($p=\infty$) performs well with very few labels on $k$-NN graphs, which experimentally validates our theoretical findings that Lipschitz learning retains information about the data distribution (the unlabeled data) on $k$-NN graphs.
△ Less
Submitted 27 January, 2022; v1 submitted 15 January, 2019;
originally announced January 2019.
-
Solving Jigsaw Puzzles By the Graph Connection Laplacian
Authors:
Vahan Huroyan,
Gilad Lerman,
Hau-Tieng Wu
Abstract:
We propose a novel mathematical framework to address the problem of automatically solving large jigsaw puzzles. This problem assumes a large image, which is cut into equal square pieces that are arbitrarily rotated and shuffled, and asks to recover the original image given the transformed pieces. The main contribution of this work is a method for recovering the rotations of the pieces when both sh…
▽ More
We propose a novel mathematical framework to address the problem of automatically solving large jigsaw puzzles. This problem assumes a large image, which is cut into equal square pieces that are arbitrarily rotated and shuffled, and asks to recover the original image given the transformed pieces. The main contribution of this work is a method for recovering the rotations of the pieces when both shuffles and rotations are unknown. A major challenge of this procedure is estimating the graph connection Laplacian without the knowledge of shuffles. A careful combination of our proposed method for estimating rotations with any existing method for estimating shuffles results in a practical solution for the jigsaw puzzle problem. Our theory guarantees, in a clean setting, that our basic idea of recovering rotations is robust to some corruption of the connection graph. Numerical experiments demonstrate the competitive accuracy of this solution, its robustness to corruption and, its computational advantage for large puzzles.
△ Less
Submitted 1 November, 2020; v1 submitted 7 November, 2018;
originally announced November 2018.
-
Encoding Robust Representation for Graph Generation
Authors:
Dongmian Zou,
Gilad Lerman
Abstract:
Generative networks have made it possible to generate meaningful signals such as images and texts from simple noise. Recently, generative methods based on GAN and VAE were developed for graphs and graph signals. However, the mathematical properties of these methods are unclear, and training good generative models is difficult. This work proposes a graph generation model that uses a recent adaptati…
▽ More
Generative networks have made it possible to generate meaningful signals such as images and texts from simple noise. Recently, generative methods based on GAN and VAE were developed for graphs and graph signals. However, the mathematical properties of these methods are unclear, and training good generative models is difficult. This work proposes a graph generation model that uses a recent adaptation of Mallat's scattering transform to graphs. The proposed model is naturally composed of an encoder and a decoder. The encoder is a Gaussianized graph scattering transform, which is robust to signal and graph manipulation. The decoder is a simple fully connected network that is adapted to specific tasks, such as link prediction, signal generation on graphs and full graph and signal generation. The training of our proposed system is efficient since it is only applied to the decoder and the hardware requirements are moderate. Numerical results demonstrate state-of-the-art performance of the proposed system for both link prediction and graph and signal generation.
△ Less
Submitted 15 January, 2019; v1 submitted 28 September, 2018;
originally announced September 2018.
-
Phase transition in random tensors with multiple independent spikes
Authors:
Wei-Kuo Chen,
Madeline Handschy,
Gilad Lerman
Abstract:
Consider a spiked random tensor obtained as a mixture of two components: noise in the form of a symmetric Gaussian $p$-tensor for $p\geq 3$ and signal in the form of a symmetric low-rank random tensor. The latter is defined as a linear combination of $k$ independent symmetric rank-one random tensors, referred to as spikes, with weights referred to as signal-to-noise ratios (SNRs). The entries of t…
▽ More
Consider a spiked random tensor obtained as a mixture of two components: noise in the form of a symmetric Gaussian $p$-tensor for $p\geq 3$ and signal in the form of a symmetric low-rank random tensor. The latter is defined as a linear combination of $k$ independent symmetric rank-one random tensors, referred to as spikes, with weights referred to as signal-to-noise ratios (SNRs). The entries of the vectors that determine the spikes are i.i.d. sampled from general probability distributions supported on bounded subsets of $\mathbb{R}$. This work focuses on the problem of detecting the presence of these spikes, and establishes the phase transition of this detection problem for any fixed $k \geq 1$. In particular, it shows that for a set of relatively low SNRs it is impossible to distinguish between the spiked and non-spiked Gaussian tensors. Furthermore, in the interior of the complement of this set, where at least one of the $k$ SNRs is relatively high, these two tensors are distinguishable by the likelihood ratio test. In addition, when the total number of low-rank components, $k$, of the $p$-tensor of size $N$ grows in the order $o(N^{(p-2)/4})$ as $N$ tends to infinity, the problem exhibits an analogous phase transition. This theory for spike detection is also shown to imply that recovery of the spikes by the minimum mean square error exhibits the same phase transition. The main methods used in this work arise from the study of mean field spin glass models, where the phase transition thresholds are identified as the critical inverse temperatures distinguishing the high and low-temperature regimes of the free energies. In particular, our result formulates the first full characterization of the high temperature regime for vector-valued spin glass models with independent coordinates.
△ Less
Submitted 9 November, 2020; v1 submitted 18 September, 2018;
originally announced September 2018.
-
Estimation of Camera Locations in Highly Corrupted Scenarios: All About that Base, No Shape Trouble
Authors:
Yunpeng Shi,
Gilad Lerman
Abstract:
We propose a strategy for improving camera location estimation in structure from motion. Our setting assumes highly corrupted pairwise directions (i.e., normalized relative location vectors), so there is a clear room for improving current state-of-the-art solutions for this problem. Our strategy identifies severely corrupted pairwise directions by using a geometric consistency condition. It then s…
▽ More
We propose a strategy for improving camera location estimation in structure from motion. Our setting assumes highly corrupted pairwise directions (i.e., normalized relative location vectors), so there is a clear room for improving current state-of-the-art solutions for this problem. Our strategy identifies severely corrupted pairwise directions by using a geometric consistency condition. It then selects a cleaner set of pairwise directions as a preprocessing step for common solvers. We theoretically guarantee the successful performance of a basic version of our strategy under a synthetic corruption model. Numerical results on artificial and real data demonstrate the significant improvement obtained by our strategy.
△ Less
Submitted 7 April, 2018;
originally announced April 2018.
-
Graph Convolutional Neural Networks via Scattering
Authors:
Dongmian Zou,
Gilad Lerman
Abstract:
We generalize the scattering transform to graphs and consequently construct a convolutional neural network on graphs. We show that under certain conditions, any feature generated by such a network is approximately invariant to permutations and stable to graph manipulations. Numerical results demonstrate competitive performance on relevant datasets.
We generalize the scattering transform to graphs and consequently construct a convolutional neural network on graphs. We show that under certain conditions, any feature generated by such a network is approximately invariant to permutations and stable to graph manipulations. Numerical results demonstrate competitive performance on relevant datasets.
△ Less
Submitted 18 November, 2018; v1 submitted 30 March, 2018;
originally announced April 2018.
-
An Overview of Robust Subspace Recovery
Authors:
Gilad Lerman,
Tyler Maunu
Abstract:
This paper will serve as an introduction to the body of work on robust subspace recovery. Robust subspace recovery involves finding an underlying low-dimensional subspace in a dataset that is possibly corrupted with outliers. While this problem is easy to state, it has been difficult to develop optimal algorithms due to its underlying nonconvexity. This work emphasizes advantages and disadvantages…
▽ More
This paper will serve as an introduction to the body of work on robust subspace recovery. Robust subspace recovery involves finding an underlying low-dimensional subspace in a dataset that is possibly corrupted with outliers. While this problem is easy to state, it has been difficult to develop optimal algorithms due to its underlying nonconvexity. This work emphasizes advantages and disadvantages of proposed approaches and unsolved problems in the area.
△ Less
Submitted 5 July, 2018; v1 submitted 2 March, 2018;
originally announced March 2018.
-
Exact Camera Location Recovery by Least Unsquared Deviations
Authors:
Gilad Lerman,
Yunpeng Shi,
Teng Zhang
Abstract:
We establish exact recovery for the Least Unsquared Deviations (LUD) algorithm of Ozyesil and Singer. More precisely, we show that for sufficiently many cameras with given corrupted pairwise directions, where both camera locations and pairwise directions are generated by a special probabilistic model, the LUD algorithm exactly recovers the camera locations with high probability. A similar exact re…
▽ More
We establish exact recovery for the Least Unsquared Deviations (LUD) algorithm of Ozyesil and Singer. More precisely, we show that for sufficiently many cameras with given corrupted pairwise directions, where both camera locations and pairwise directions are generated by a special probabilistic model, the LUD algorithm exactly recovers the camera locations with high probability. A similar exact recovery guarantee was established for the ShapeFit algorithm by Hand, Lee and Voroninski, but with typically less corruption.
△ Less
Submitted 9 September, 2018; v1 submitted 27 September, 2017;
originally announced September 2017.
-
Robust Sparse Covariance Estimation by Thresholding Tyler's M-Estimator
Authors:
John Goes,
Gilad Lerman,
Boaz Nadler
Abstract:
Estimating a high-dimensional sparse covariance matrix from a limited number of samples is a fundamental problem in contemporary data analysis. Most proposals to date, however, are not robust to outliers or heavy tails. Towards bridging this gap, in this work we consider estimating a sparse shape matrix from $n$ samples following a possibly heavy tailed elliptical distribution. We propose estimato…
▽ More
Estimating a high-dimensional sparse covariance matrix from a limited number of samples is a fundamental problem in contemporary data analysis. Most proposals to date, however, are not robust to outliers or heavy tails. Towards bridging this gap, in this work we consider estimating a sparse shape matrix from $n$ samples following a possibly heavy tailed elliptical distribution. We propose estimators based on thresholding either Tyler's M-estimator or its regularized variant. We derive bounds on the difference in spectral norm between our estimators and the shape matrix in the joint limit as the dimension $p$ and sample size $n$ tend to infinity with $p/n\toγ>0$. These bounds are minimax rate-optimal. Results on simulated data support our theoretical analysis.
△ Less
Submitted 19 September, 2018; v1 submitted 24 June, 2017;
originally announced June 2017.
-
A Well-Tempered Landscape for Non-convex Robust Subspace Recovery
Authors:
Tyler Maunu,
Teng Zhang,
Gilad Lerman
Abstract:
We present a mathematical analysis of a non-convex energy landscape for robust subspace recovery. We prove that an underlying subspace is the only stationary point and local minimizer in a specified neighborhood under a deterministic condition on a dataset. If the deterministic condition is satisfied, we further show that a geodesic gradient descent method over the Grassmannian manifold can exactl…
▽ More
We present a mathematical analysis of a non-convex energy landscape for robust subspace recovery. We prove that an underlying subspace is the only stationary point and local minimizer in a specified neighborhood under a deterministic condition on a dataset. If the deterministic condition is satisfied, we further show that a geodesic gradient descent method over the Grassmannian manifold can exactly recover the underlying subspace when the method is properly initialized. Proper initialization by principal component analysis is guaranteed with a simple deterministic condition. Under slightly stronger assumptions, the gradient descent method with a piecewise constant step-size scheme achieves linear convergence. The practicality of the deterministic condition is demonstrated on some statistical models of data, and the method achieves almost state-of-the-art recovery guarantees on the Haystack Model for different regimes of sample size and ambient dimension. In particular, when the ambient dimension is fixed and the sample size is large enough, we show that our gradient method can exactly recover the underlying subspace for any fixed fraction of outliers (less than 1).
△ Less
Submitted 28 February, 2019; v1 submitted 12 June, 2017;
originally announced June 2017.
-
Distributed Robust Subspace Recovery
Authors:
Vahan Huroyan,
Gilad Lerman
Abstract:
We propose distributed solutions to the problem of Robust Subspace Recovery (RSR). Our setting assumes a huge dataset in an ad hoc network without a central processor, where each node has access only to one chunk of the dataset. Furthermore, part of the whole dataset lies around a low-dimensional subspace and the other part is composed of outliers that lie away from that subspace. The goal is to r…
▽ More
We propose distributed solutions to the problem of Robust Subspace Recovery (RSR). Our setting assumes a huge dataset in an ad hoc network without a central processor, where each node has access only to one chunk of the dataset. Furthermore, part of the whole dataset lies around a low-dimensional subspace and the other part is composed of outliers that lie away from that subspace. The goal is to recover the underlying subspace for the whole dataset, without transferring the data itself between the nodes. We first apply the Consensus-Based Gradient method to the Geometric Median Subspace algorithm for RSR. For this purpose, we propose an iterative solution for the local dual minimization problem and establish its r-linear convergence. We then explain how to distributedly implement the Reaper and Fast Median Subspace algorithms for RSR. The proposed algorithms display competitive performance on both synthetic and real data.
△ Less
Submitted 4 July, 2018; v1 submitted 25 May, 2017;
originally announced May 2017.
-
On the energy landscape of the mixed even $p$-spin model
Authors:
Wei-Kuo Chen,
Madeline Handschy,
Gilad Lerman
Abstract:
We investigate the energy landscape of the mixed even $p$-spin model with Ising spin configurations. We show that for any given energy level between zero and the maximal energy, with overwhelming probability there exist exponentially many distinct spin configurations such that their energies stay near this energy level. Furthermore, their magnetizations and overlaps are concentrated around some fi…
▽ More
We investigate the energy landscape of the mixed even $p$-spin model with Ising spin configurations. We show that for any given energy level between zero and the maximal energy, with overwhelming probability there exist exponentially many distinct spin configurations such that their energies stay near this energy level. Furthermore, their magnetizations and overlaps are concentrated around some fixed constants. In particular, at the level of maximal energy, we prove that the Hamiltonian exhibits exponentially many orthogonal peaks. This improves the results of Chatterjee and Ding-Eldan-Zhai, where the former established a logarithmic size of the number of the orthogonal peaks, while the latter proved a polynomial size. Our second main result obtains disorder chaos at zero temperature and at any external field. As a byproduct, this implies that the fluctuation of the maximal energy is superconcentrated when the external field vanishes and obeys a Gaussian limit law when the external field is present.
△ Less
Submitted 24 March, 2017; v1 submitted 14 September, 2016;
originally announced September 2016.
-
Enhancing Feature Tracking With Gyro Regularization
Authors:
Bryan Poling,
Gilad Lerman
Abstract:
We present a deeply integrated method of exploiting low-cost gyroscopes to improve general purpose feature tracking. Most previous methods use gyroscopes to initialize and bound the search for features. In contrast, we use them to regularize the tracking energy function so that they can directly assist in the tracking of ambiguous and poor-quality features. We demonstrate that our simple technique…
▽ More
We present a deeply integrated method of exploiting low-cost gyroscopes to improve general purpose feature tracking. Most previous methods use gyroscopes to initialize and bound the search for features. In contrast, we use them to regularize the tracking energy function so that they can directly assist in the tracking of ambiguous and poor-quality features. We demonstrate that our simple technique offers significant improvements in performance over conventional template-based tracking methods, and is in fact competitive with more complex and computationally expensive state-of-the-art trackers, but at a fraction of the computational cost. Additionally, we show that the practice of initializing template-based feature trackers like KLT (Kanade-Lucas-Tomasi) using gyro-predicted optical flow offers no advantage over using a careful optical-only initialization method, suggesting that some deeper level of integration, like the method we propose, is needed in order to realize a genuine improvement in tracking performance from these inertial sensors.
△ Less
Submitted 4 November, 2015;
originally announced November 2015.
-
Fast Landmark Subspace Clustering
Authors:
Xu Wang,
Gilad Lerman
Abstract:
Kernel methods obtain superb performance in terms of accuracy for various machine learning tasks since they can effectively extract nonlinear relations. However, their time complexity can be rather large especially for clustering tasks. In this paper we define a general class of kernels that can be easily approximated by randomization. These kernels appear in various applications, in particular, t…
▽ More
Kernel methods obtain superb performance in terms of accuracy for various machine learning tasks since they can effectively extract nonlinear relations. However, their time complexity can be rather large especially for clustering tasks. In this paper we define a general class of kernels that can be easily approximated by randomization. These kernels appear in various applications, in particular, traditional spectral clustering, landmark-based spectral clustering and landmark-based subspace clustering. We show that for $n$ data points from $K$ clusters with $D$ landmarks, the randomization procedure results in an algorithm of complexity $O(KnD)$. Furthermore, we bound the error between the original clustering scheme and its randomization. To illustrate the power of this framework, we propose a new fast landmark subspace (FLS) clustering algorithm. Experiments over synthetic and real datasets demonstrate the superior performance of FLS in accelerating subspace clustering with marginal sacrifice of accuracy.
△ Less
Submitted 28 October, 2015;
originally announced October 2015.
-
Nonparametric Bayesian Regression on Manifolds via Brownian Motion
Authors:
Xu Wang,
Gilad Lerman
Abstract:
This paper proposes a novel framework for manifold-valued regression and establishes its consistency as well as its contraction rate. It assumes a predictor with values in the interval $[0,1]$ and response with values in a compact Riemannian manifold $M$. This setting is useful for applications such as modeling dynamic scenes or shape deformations, where the visual scene or the deformed objects ca…
▽ More
This paper proposes a novel framework for manifold-valued regression and establishes its consistency as well as its contraction rate. It assumes a predictor with values in the interval $[0,1]$ and response with values in a compact Riemannian manifold $M$. This setting is useful for applications such as modeling dynamic scenes or shape deformations, where the visual scene or the deformed objects can be modeled by a manifold. The proposed framework is nonparametric and uses the heat kernel (and its associated Brownian motion) on manifolds as an averaging procedure. It directly generalizes the use of the Gaussian kernel (as a natural model of additive noise) in vector-valued regression problems. In order to avoid explicit dependence on estimates of the heat kernel, we follow a Bayesian setting, where Brownian motion on $M$ induces a prior distribution on the space of continuous functions $C([0,1], M)$. For the case of discretized Brownian motion, we establish the consistency of the posterior distribution in terms of the $L_{q}$ distances for any $1 \leq q < \infty$. Most importantly, we establish contraction rate of order $O(n^{-1/4+ε})$ for any fixed $ε>0$, where $n$ is the number of observations. For the continuous Brownian motion we establish weak consistency.
△ Less
Submitted 23 July, 2015;
originally announced July 2015.
-
Riemannian Multi-Manifold Modeling
Authors:
Xu Wang,
Konstantinos Slavakis,
Gilad Lerman
Abstract:
This paper advocates a novel framework for segmenting a dataset in a Riemannian manifold $M$ into clusters lying around low-dimensional submanifolds of $M$. Important examples of $M$, for which the proposed clustering algorithm is computationally efficient, are the sphere, the set of positive definite matrices, and the Grassmannian. The clustering problem with these examples of $M$ is already usef…
▽ More
This paper advocates a novel framework for segmenting a dataset in a Riemannian manifold $M$ into clusters lying around low-dimensional submanifolds of $M$. Important examples of $M$, for which the proposed clustering algorithm is computationally efficient, are the sphere, the set of positive definite matrices, and the Grassmannian. The clustering problem with these examples of $M$ is already useful for numerous application domains such as action identification in video sequences, dynamic texture clustering, brain fiber segmentation in medical imaging, and clustering of deformed images. The proposed clustering algorithm constructs a data-affinity matrix by thoroughly exploiting the intrinsic geometry and then applies spectral clustering. The intrinsic local geometry is encoded by local sparse coding and more importantly by directional information of local tangent spaces and geodesics. Theoretical guarantees are established for a simplified variant of the algorithm even when the clusters intersect. To avoid complication, these guarantees assume that the underlying submanifolds are geodesic. Extensive validation on synthetic and real data demonstrates the resiliency of the proposed method against deviations from the theoretical model as well as its superior performance over state-of-the-art techniques.
△ Less
Submitted 30 September, 2014;
originally announced October 2014.
-
Compressive Earth Observatory: An Insight from AIRS/AMSU Retrievals
Authors:
Ardeshir Mohammad Ebtehaj,
Efi Foufoula-Georgiou,
Gilad Lerman,
Rafael Luis Bras
Abstract:
We demonstrate that the global fields of temperature, humidity and geopotential heights admit a nearly sparse representation in the wavelet domain, offering a viable path forward to explore new paradigms of sparsity-promoting data assimilation and compressive recovery of land surface-atmospheric states from space. We illustrate this idea using retrieval products of the Atmospheric Infrared Sounder…
▽ More
We demonstrate that the global fields of temperature, humidity and geopotential heights admit a nearly sparse representation in the wavelet domain, offering a viable path forward to explore new paradigms of sparsity-promoting data assimilation and compressive recovery of land surface-atmospheric states from space. We illustrate this idea using retrieval products of the Atmospheric Infrared Sounder (AIRS) and Advanced Microwave Sounding Unit (AMSU) on board the Aqua satellite. The results reveal that the sparsity of the fields of temperature is relatively pressure-independent while atmospheric humidity and geopotential heights are typically sparser at lower and higher pressure levels, respectively. We provide evidence that these land-atmospheric states can be accurately estimated using a small set of measurements by taking advantage of their sparsity prior.
△ Less
Submitted 30 December, 2014; v1 submitted 17 September, 2014;
originally announced September 2014.
-
Fast, Robust and Non-convex Subspace Recovery
Authors:
Gilad Lerman,
Tyler Maunu
Abstract:
This work presents a fast and non-convex algorithm for robust subspace recovery. The data sets considered include inliers drawn around a low-dimensional subspace of a higher dimensional ambient space, and a possibly large portion of outliers that do not lie nearby this subspace. The proposed algorithm, which we refer to as Fast Median Subspace (FMS), is designed to robustly determine the underlyin…
▽ More
This work presents a fast and non-convex algorithm for robust subspace recovery. The data sets considered include inliers drawn around a low-dimensional subspace of a higher dimensional ambient space, and a possibly large portion of outliers that do not lie nearby this subspace. The proposed algorithm, which we refer to as Fast Median Subspace (FMS), is designed to robustly determine the underlying subspace of such data sets, while having lower computational complexity than existing methods. We prove convergence of the FMS iterates to a stationary point. Further, under a special model of data, FMS converges to a point which is near to the global minimum with overwhelming probability. Under this model, we show that the iteration complexity is globally bounded and locally $r$-linear. The latter theorem holds for any fixed fraction of outliers (less than 1) and any fixed positive distance between the limit point and the global minimum. Numerical experiments on synthetic and real data demonstrate its competitive speed and accuracy.
△ Less
Submitted 9 June, 2016; v1 submitted 24 June, 2014;
originally announced June 2014.
-
Better Feature Tracking Through Subspace Constraints
Authors:
Bryan Poling,
Gilad Lerman,
Arthur Szlam
Abstract:
Feature tracking in video is a crucial task in computer vision. Usually, the tracking problem is handled one feature at a time, using a single-feature tracker like the Kanade-Lucas-Tomasi algorithm, or one of its derivatives. While this approach works quite well when dealing with high-quality video and "strong" features, it often falters when faced with dark and noisy video containing low-quality…
▽ More
Feature tracking in video is a crucial task in computer vision. Usually, the tracking problem is handled one feature at a time, using a single-feature tracker like the Kanade-Lucas-Tomasi algorithm, or one of its derivatives. While this approach works quite well when dealing with high-quality video and "strong" features, it often falters when faced with dark and noisy video containing low-quality features. We present a framework for jointly tracking a set of features, which enables sharing information between the different features in the scene. We show that our method can be employed to track features for both rigid and nonrigid motions (possibly of few moving bodies) even when some features are occluded. Furthermore, it can be used to significantly improve tracking results in poorly-lit scenes (where there is a mix of good and bad features). Our approach does not require direct modeling of the structure or the motion of the scene, and runs in real time on a single CPU core.
△ Less
Submitted 9 May, 2014;
originally announced May 2014.
-
Variational Data Assimilation via Sparse Regularization
Authors:
A. M. Ebtehaj,
M. Zupanski,
G. Lerman,
E. Foufoula-Georgiou
Abstract:
This paper studies the role of sparse regularization in a properly chosen basis for variational data assimilation (VDA) problems. Specifically, it focuses on data assimilation of noisy and down-sampled observations while the state variable of interest exhibits sparsity in the real or transformed domain. We show that in the presence of sparsity, the $\ell_{1}$-norm regularization produces more accu…
▽ More
This paper studies the role of sparse regularization in a properly chosen basis for variational data assimilation (VDA) problems. Specifically, it focuses on data assimilation of noisy and down-sampled observations while the state variable of interest exhibits sparsity in the real or transformed domain. We show that in the presence of sparsity, the $\ell_{1}$-norm regularization produces more accurate and stable solutions than the classic data assimilation methods. To motivate further developments of the proposed methodology, assimilation experiments are conducted in the wavelet and spectral domain using the linear advection-diffusion equation.
△ Less
Submitted 6 June, 2013;
originally announced June 2013.
-
A New Approach To Two-View Motion Segmentation Using Global Dimension Minimization
Authors:
Bryan Poling,
Gilad Lerman
Abstract:
We present a new approach to rigid-body motion segmentation from two views. We use a previously developed nonlinear embedding of two-view point correspondences into a 9-dimensional space and identify the different motions by segmenting lower-dimensional subspaces. In order to overcome nonuniform distributions along the subspaces, whose dimensions are unknown, we suggest the novel concept of global…
▽ More
We present a new approach to rigid-body motion segmentation from two views. We use a previously developed nonlinear embedding of two-view point correspondences into a 9-dimensional space and identify the different motions by segmenting lower-dimensional subspaces. In order to overcome nonuniform distributions along the subspaces, whose dimensions are unknown, we suggest the novel concept of global dimension and its minimization for clustering subspaces with some theoretical motivation. We propose a fast projected gradient algorithm for minimizing global dimension and thus segmenting motions from 2-views. We develop an outlier detection framework around the proposed method, and we present state-of-the-art results on outlier-free and outlier-corrupted two-view data for segmenting motion.
△ Less
Submitted 7 January, 2014; v1 submitted 10 April, 2013;
originally announced April 2013.
-
Spectral Clustering Based on Local PCA
Authors:
Ery Arias-Castro,
Gilad Lerman,
Teng Zhang
Abstract:
We propose a spectral clustering method based on local principal components analysis (PCA). After performing local PCA in selected neighborhoods, the algorithm builds a nearest neighbor graph weighted according to a discrepancy between the principal subspaces in the neighborhoods, and then applies spectral clustering. As opposed to standard spectral methods based solely on pairwise distances betwe…
▽ More
We propose a spectral clustering method based on local principal components analysis (PCA). After performing local PCA in selected neighborhoods, the algorithm builds a nearest neighbor graph weighted according to a discrepancy between the principal subspaces in the neighborhoods, and then applies spectral clustering. As opposed to standard spectral methods based solely on pairwise distances between points, our algorithm is able to resolve intersections. We establish theoretical guarantees for simpler variants within a prototypical mathematical framework for multi-manifold clustering, and evaluate our algorithm on various simulated data sets.
△ Less
Submitted 9 January, 2013;
originally announced January 2013.
-
Robust computation of linear models by convex relaxation
Authors:
Gilad Lerman,
Michael McCoy,
Joel A. Tropp,
Teng Zhang
Abstract:
Consider a dataset of vector-valued observations that consists of noisy inliers, which are explained well by a low-dimensional subspace, along with some number of outliers. This work describes a convex optimization problem, called REAPER, that can reliably fit a low-dimensional model to this type of data. This approach parameterizes linear subspaces using orthogonal projectors, and it uses a relax…
▽ More
Consider a dataset of vector-valued observations that consists of noisy inliers, which are explained well by a low-dimensional subspace, along with some number of outliers. This work describes a convex optimization problem, called REAPER, that can reliably fit a low-dimensional model to this type of data. This approach parameterizes linear subspaces using orthogonal projectors, and it uses a relaxation of the set of orthogonal projectors to reach the convex formulation. The paper provides an efficient algorithm for solving the REAPER problem, and it documents numerical experiments which confirm that REAPER can dependably find linear structure in synthetic and natural data. In addition, when the inliers lie near a low-dimensional subspace, there is a rigorous theory that describes when REAPER can approximate this subspace.
△ Less
Submitted 11 August, 2014; v1 submitted 17 February, 2012;
originally announced February 2012.
-
A Novel M-Estimator for Robust PCA
Authors:
Teng Zhang,
Gilad Lerman
Abstract:
We study the basic problem of robust subspace recovery. That is, we assume a data set that some of its points are sampled around a fixed subspace and the rest of them are spread in the whole ambient space, and we aim to recover the fixed underlying subspace. We first estimate "robust inverse sample covariance" by solving a convex minimization procedure; we then recover the subspace by the bottom e…
▽ More
We study the basic problem of robust subspace recovery. That is, we assume a data set that some of its points are sampled around a fixed subspace and the rest of them are spread in the whole ambient space, and we aim to recover the fixed underlying subspace. We first estimate "robust inverse sample covariance" by solving a convex minimization procedure; we then recover the subspace by the bottom eigenvectors of this matrix (their number correspond to the number of eigenvalues close to 0). We guarantee exact subspace recovery under some conditions on the underlying data. Furthermore, we propose a fast iterative algorithm, which linearly converges to the matrix minimizing the convex problem. We also quantify the effect of noise and regularization and discuss many other practical and theoretical issues for improving the subspace recovery in various settings. When replacing the sum of terms in the convex energy function (that we minimize) with the sum of squares of terms, we obtain that the new minimizer is a scaled version of the inverse sample covariance (when exists). We thus interpret our minimizer and its subspace (spanned by its bottom eigenvectors) as robust versions of the empirical inverse covariance and the PCA subspace respectively. We compare our method with many other algorithms for robust PCA on synthetic and real data sets and demonstrate state-of-the-art speed and accuracy.
△ Less
Submitted 23 June, 2014; v1 submitted 20 December, 2011;
originally announced December 2011.
-
Robust recovery of multiple subspaces by geometric l_p minimization
Authors:
Gilad Lerman,
Teng Zhang
Abstract:
We assume i.i.d. data sampled from a mixture distribution with K components along fixed d-dimensional linear subspaces and an additional outlier component. For p>0, we study the simultaneous recovery of the K fixed subspaces by minimizing the l_p-averaged distances of the sampled data points from any K subspaces. Under some conditions, we show that if $0<p\leq1$, then all underlying subspaces can…
▽ More
We assume i.i.d. data sampled from a mixture distribution with K components along fixed d-dimensional linear subspaces and an additional outlier component. For p>0, we study the simultaneous recovery of the K fixed subspaces by minimizing the l_p-averaged distances of the sampled data points from any K subspaces. Under some conditions, we show that if $0<p\leq1$, then all underlying subspaces can be precisely recovered by l_p minimization with overwhelming probability. On the other hand, if K>1 and p>1, then the underlying subspaces cannot be recovered or even nearly recovered by l_p minimization. The results of this paper partially explain the successes and failures of the basic approach of l_p energy minimization for modeling data by multiple subspaces.
△ Less
Submitted 1 February, 2012; v1 submitted 19 April, 2011;
originally announced April 2011.