-
Deep Autoencoders: From Understanding to Generalization Guarantees
Authors:
Romain Cosentino,
Randall Balestriero,
Richard Baraniuk,
Behnaam Aazhang
Abstract:
A big mystery in deep learning continues to be the ability of methods to generalize when the number of model parameters is larger than the number of training examples. In this work, we take a step towards a better understanding of the underlying phenomena of Deep Autoencoders (AEs), a mainstream deep learning solution for learning compressed, interpretable, and structured data representations. In…
▽ More
A big mystery in deep learning continues to be the ability of methods to generalize when the number of model parameters is larger than the number of training examples. In this work, we take a step towards a better understanding of the underlying phenomena of Deep Autoencoders (AEs), a mainstream deep learning solution for learning compressed, interpretable, and structured data representations. In particular, we interpret how AEs approximate the data manifold by exploiting their continuous piecewise affine structure. Our reformulation of AEs provides new insights into their map**, reconstruction guarantees, as well as an interpretation of commonly used regularization techniques. We leverage these findings to derive two new regularizations that enable AEs to capture the inherent symmetry in the data. Our regularizations leverage recent advances in the group of transformation learning to enable AEs to better approximate the data manifold without explicitly defining the group underlying the manifold. Under the assumption that the symmetry of the data can be explained by a Lie group, we prove that the regularizations ensure the generalization of the corresponding AEs. A range of experimental evaluations demonstrate that our methods outperform other state-of-the-art regularization techniques.
△ Less
Submitted 24 November, 2021; v1 submitted 20 September, 2020;
originally announced September 2020.
-
The Geometry of Deep Networks: Power Diagram Subdivision
Authors:
Randall Balestriero,
Romain Cosentino,
Behnaam Aazhang,
Richard Baraniuk
Abstract:
We study the geometry of deep (neural) networks (DNs) with piecewise affine and convex nonlinearities. The layers of such DNs have been shown to be {\em max-affine spline operators} (MASOs) that partition their input space and apply a region-dependent affine map** to their input to produce their output. We demonstrate that each MASO layer's input space partitioning corresponds to a {\em power di…
▽ More
We study the geometry of deep (neural) networks (DNs) with piecewise affine and convex nonlinearities. The layers of such DNs have been shown to be {\em max-affine spline operators} (MASOs) that partition their input space and apply a region-dependent affine map** to their input to produce their output. We demonstrate that each MASO layer's input space partitioning corresponds to a {\em power diagram} (an extension of the classical Voronoi tiling) with a number of regions that grows exponentially with respect to the number of units (neurons). We further show that a composition of MASO layers (e.g., the entire DN) produces a progressively subdivided power diagram and provide its analytical form. The subdivision process constrains the affine maps on the (exponentially many) power diagram regions to greatly reduce their complexity. For classification problems, we obtain a formula for a MASO DN's decision boundary in the input space plus a measure of its curvature that depends on the DN's nonlinearities, weights, and architecture. Numerous numerical experiments support and extend our theoretical results.
△ Less
Submitted 21 May, 2019;
originally announced May 2019.
-
Data-Driven Estimation Of Mutual Information Between Dependent Data
Authors:
Rakesh Malladi,
Don H Johnson,
Behnaam Aazhang
Abstract:
We consider the problem of estimating mutual information between dependent data, an important problem in many science and engineering applications. We propose a data-driven, non-parametric estimator of mutual information in this paper. The main novelty of our solution lies in transforming the data to frequency domain to make the problem tractable. We define a novel metric--mutual information in fr…
▽ More
We consider the problem of estimating mutual information between dependent data, an important problem in many science and engineering applications. We propose a data-driven, non-parametric estimator of mutual information in this paper. The main novelty of our solution lies in transforming the data to frequency domain to make the problem tractable. We define a novel metric--mutual information in frequency--to detect and quantify the dependence between two random processes across frequency using Cramér's spectral representation. Our solution calculates mutual information as a function of frequency to estimate the mutual information between the dependent data over time. We validate its performance on linear and nonlinear models. In addition, mutual information in frequency estimated as a part of our solution can also be used to infer cross-frequency coupling in the data.
△ Less
Submitted 7 March, 2017;
originally announced March 2017.
-
Robust Unsupervised Transient Detection With Invariant Representation based on the Scattering Network
Authors:
Randall Balestriero,
Behnaam Aazhang
Abstract:
We present a sparse and invariant representation with low asymptotic complexity for robust unsupervised transient and onset zone detection in noisy environments. This unsupervised approach is based on wavelet transforms and leverages the scattering network from Mallat et al. by deriving frequency invariance. This frequency invariance is a key concept to enforce robust representations of transients…
▽ More
We present a sparse and invariant representation with low asymptotic complexity for robust unsupervised transient and onset zone detection in noisy environments. This unsupervised approach is based on wavelet transforms and leverages the scattering network from Mallat et al. by deriving frequency invariance. This frequency invariance is a key concept to enforce robust representations of transients in presence of possible frequency shifts and perturbations occurring in the original signal. Implementation details as well as complexity analysis are provided in addition of the theoretical framework and the invariance properties. In this work, our primary application consists of predicting the onset of seizure in epileptic patients from subdural recordings as well as detecting inter-ictal spikes.
△ Less
Submitted 23 November, 2016;
originally announced November 2016.
-
Identifying Seizure Onset Zone from the Causal Connectivity Inferred Using Directed Information
Authors:
Rakesh Malladi,
Giridhar Kalamangalam,
Nitin Tandon,
Behnaam Aazhang
Abstract:
In this paper, we developed a model-based and a data-driven estimator for directed information (DI) to infer the causal connectivity graph between electrocorticographic (ECoG) signals recorded from brain and to identify the seizure onset zone (SOZ) in epileptic patients. Directed information, an information theoretic quantity, is a general metric to infer causal connectivity between time-series an…
▽ More
In this paper, we developed a model-based and a data-driven estimator for directed information (DI) to infer the causal connectivity graph between electrocorticographic (ECoG) signals recorded from brain and to identify the seizure onset zone (SOZ) in epileptic patients. Directed information, an information theoretic quantity, is a general metric to infer causal connectivity between time-series and is not restricted to a particular class of models unlike the popular metrics based on Granger causality or transfer entropy. The proposed estimators are shown to be almost surely convergent. Causal connectivity between ECoG electrodes in five epileptic patients is inferred using the proposed DI estimators, after validating their performance on simulated data. We then proposed a model-based and a data-driven SOZ identification algorithm to identify SOZ from the causal connectivity inferred using model-based and data-driven DI estimators respectively. The data-driven SOZ identification outperforms the model-based SOZ identification algorithm when benchmarked against visual analysis by neurologist, the current clinical gold standard. The causal connectivity analysis presented here is the first step towards develo** novel non-surgical treatments for epilepsy.
△ Less
Submitted 16 August, 2016; v1 submitted 27 December, 2015;
originally announced December 2015.
