-
A Methodology for Exploring Deep Convolutional Features in Relation to Hand-Crafted Features with an Application to Music Audio Modeling
Authors:
Anna K. Yanchenko,
Mohammadreza Soltani,
Robert J. Ravier,
Sayan Mukherjee,
Vahid Tarokh
Abstract:
Understanding the features learned by deep models is important from a model trust perspective, especially as deep systems are deployed in the real world. Most recent approaches for deep feature understanding or model explanation focus on highlighting input data features that are relevant for classification decisions. In this work, we instead take the perspective of relating deep features to well-s…
▽ More
Understanding the features learned by deep models is important from a model trust perspective, especially as deep systems are deployed in the real world. Most recent approaches for deep feature understanding or model explanation focus on highlighting input data features that are relevant for classification decisions. In this work, we instead take the perspective of relating deep features to well-studied, hand-crafted features that are meaningful for the application of interest. We propose a methodology and set of systematic experiments for exploring deep features in this setting, where input feature importance approaches for deep feature understanding do not apply. Our experiments focus on understanding which hand-crafted and deep features are useful for the classification task of interest, how robust these features are for related tasks and how similar the deep features are to the meaningful hand-crafted features. Our proposed method is general to many application areas and we demonstrate its utility on orchestral music audio data.
△ Less
Submitted 9 October, 2021; v1 submitted 31 May, 2021;
originally announced June 2021.
-
Improved Automated Machine Learning from Transfer Learning
Authors:
Cat P. Le,
Mohammadreza Soltani,
Robert Ravier,
Vahid Tarokh
Abstract:
In this paper, we propose a neural architecture search framework based on a similarity measure between some baseline tasks and a target task. We first define the notion of the task similarity based on the log-determinant of the Fisher Information matrix. Next, we compute the task similarity from each of the baseline tasks to the target task. By utilizing the relation between a target and a set of…
▽ More
In this paper, we propose a neural architecture search framework based on a similarity measure between some baseline tasks and a target task. We first define the notion of the task similarity based on the log-determinant of the Fisher Information matrix. Next, we compute the task similarity from each of the baseline tasks to the target task. By utilizing the relation between a target and a set of learned baseline tasks, the search space of architectures for the target task can be significantly reduced, making the discovery of the best candidates in the set of possible architectures tractable and efficient, in terms of GPU days. This method eliminates the requirement for training the networks from scratch for a given target task as well as introducing the bias in the initialization of the search space from the human domain.
△ Less
Submitted 29 January, 2022; v1 submitted 27 February, 2021;
originally announced March 2021.
-
Task-Aware Neural Architecture Search
Authors:
Cat P. Le,
Mohammadreza Soltani,
Robert Ravier,
Vahid Tarokh
Abstract:
The design of handcrafted neural networks requires a lot of time and resources. Recent techniques in Neural Architecture Search (NAS) have proven to be competitive or better than traditional handcrafted design, although they require domain knowledge and have generally used limited search spaces. In this paper, we propose a novel framework for neural architecture search, utilizing a dictionary of m…
▽ More
The design of handcrafted neural networks requires a lot of time and resources. Recent techniques in Neural Architecture Search (NAS) have proven to be competitive or better than traditional handcrafted design, although they require domain knowledge and have generally used limited search spaces. In this paper, we propose a novel framework for neural architecture search, utilizing a dictionary of models of base tasks and the similarity between the target task and the atoms of the dictionary; hence, generating an adaptive search space based on the base models of the dictionary. By introducing a gradient-based search algorithm, we can evaluate and discover the best architecture in the search space without fully training the networks. The experimental results show the efficacy of our proposed task-aware approach.
△ Less
Submitted 15 March, 2021; v1 submitted 26 October, 2020;
originally announced October 2020.
-
Approximating the Riemannian Metric from Point Clouds via Manifold Moving Least Squares
Authors:
Barak Sober,
Robert Ravier,
Ingrid Daubechies
Abstract:
The approximation of both geodesic distances and shortest paths on point cloud sampled from an embedded submanifold $\mathcal{M}$ of Euclidean space has been a long-standing challenge in computational geometry. Given a sampling resolution parameter $ h $, state-of-the-art discrete methods yield $ O(h) $ provable approximations. In this paper, we investigate the convergence of such approximations m…
▽ More
The approximation of both geodesic distances and shortest paths on point cloud sampled from an embedded submanifold $\mathcal{M}$ of Euclidean space has been a long-standing challenge in computational geometry. Given a sampling resolution parameter $ h $, state-of-the-art discrete methods yield $ O(h) $ provable approximations. In this paper, we investigate the convergence of such approximations made by Manifold Moving Least-Squares (Manifold-MLS), a method that constructs an approximating manifold $\mathcal{M}^h$ using information from a given point cloud that was developed by Sober \& Levin in 2019. In this paper, we show that provided that $\mathcal{M}\in C^{k}$ and closed (i.e. $\mathcal{M}$ is a compact manifold without boundary) the Riemannian metric of $ \mathcal{M}^h $ approximates the Riemannian metric of $ \mathcal{M}, $. Explicitly, given points $ p_1, p_2 \in \mathcal{M}$ with geodesic distance $ ρ_{\mathcal{M}}(p_1, p_2) $, we show that their corresponding points $ p_1^h, p_2^h \in \mathcal{M}^h$ have a geodesic distance of $ ρ_{\mathcal{M}^h}(p_1^h,p_2^h) = ρ_{\mathcal{M}}(p_1, p_2)(1 + O(h^{k-1})) $ (i.e., the Manifold-MLS is nearly an isometry). We then use this result, as well as the fact that $ \mathcal{M}^h $ can be sampled with any desired resolution, to devise a naive algorithm that yields approximate geodesic distances with a rate of convergence $ O(h^{k-1}) $. We show the potential and the robustness to noise of the proposed method on some numerical simulations.
△ Less
Submitted 20 November, 2020; v1 submitted 20 July, 2020;
originally announced July 2020.
-
GeoStat Representations of Time Series for Fast Classification
Authors:
Robert J. Ravier,
Mohammadreza Soltani,
Miguel Simões,
Denis Garagic,
Vahid Tarokh
Abstract:
Recent advances in time series classification have largely focused on methods that either employ deep learning or utilize other machine learning models for feature extraction. Though successful, their power often comes at the requirement of computational complexity. In this paper, we introduce GeoStat representations for time series. GeoStat representations are based off of a generalization of rec…
▽ More
Recent advances in time series classification have largely focused on methods that either employ deep learning or utilize other machine learning models for feature extraction. Though successful, their power often comes at the requirement of computational complexity. In this paper, we introduce GeoStat representations for time series. GeoStat representations are based off of a generalization of recent methods for trajectory classification, and summarize the information of a time series in terms of comprehensive statistics of (possibly windowed) distributions of easy to compute differential geometric quantities, requiring no dynamic time war**. The features used are intuitive and require minimal parameter tuning. We perform an exhaustive evaluation of GeoStat on a number of real datasets, showing that simple KNN and SVM classifiers trained on these representations exhibit surprising performance relative to modern single model methods requiring significant computational power, achieving state of the art results in many cases. In particular, we show that this methodology achieves good performance on a challenging dataset involving the classification of fishing vessels, where our methods achieve good performance relative to the state of the art despite only having access to approximately two percent of the dataset used in training and evaluating this state of the art.
△ Less
Submitted 11 January, 2021; v1 submitted 13 July, 2020;
originally announced July 2020.
-
A Distributed Online Convex Optimization Algorithm with Improved Dynamic Regret
Authors:
Yan Zhang,
Robert J. Ravier,
Michael M. Zavlanos,
Vahid Tarokh
Abstract:
In this paper, we consider the problem of distributed online convex optimization, where a network of local agents aim to jointly optimize a convex function over a period of multiple time steps. The agents do not have any information about the future. Existing algorithms have established dynamic regret bounds that have explicit dependence on the number of time steps. In this work, we show that we c…
▽ More
In this paper, we consider the problem of distributed online convex optimization, where a network of local agents aim to jointly optimize a convex function over a period of multiple time steps. The agents do not have any information about the future. Existing algorithms have established dynamic regret bounds that have explicit dependence on the number of time steps. In this work, we show that we can remove this dependence assuming that the local objective functions are strongly convex. More precisely, we propose a gradient tracking algorithm where agents jointly communicate and descend based on corrected gradient steps. We verify our theoretical results through numerical experiments.
△ Less
Submitted 12 November, 2019;
originally announced November 2019.
-
Learning Partial Differential Equations from Data Using Neural Networks
Authors:
Ali Hasan,
João M. Pereira,
Robert Ravier,
Sina Farsiu,
Vahid Tarokh
Abstract:
We develop a framework for estimating unknown partial differential equations from noisy data, using a deep learning approach. Given noisy samples of a solution to an unknown PDE, our method interpolates the samples using a neural network, and extracts the PDE by equating derivatives of the neural network approximation. Our method applies to PDEs which are linear combinations of user-defined dictio…
▽ More
We develop a framework for estimating unknown partial differential equations from noisy data, using a deep learning approach. Given noisy samples of a solution to an unknown PDE, our method interpolates the samples using a neural network, and extracts the PDE by equating derivatives of the neural network approximation. Our method applies to PDEs which are linear combinations of user-defined dictionary functions, and generalizes previous methods that only consider parabolic PDEs. We introduce a regularization scheme that prevents the function approximation from overfitting the data and forces it to be a solution of the underlying PDE. We validate the model on simulated data generated by the known PDEs and added Gaussian noise, and we study our method under different levels of noise. We also compare the error of our method with a Cramer-Rao lower bound for an ordinary differential equation. Our results indicate that our method outperforms other methods in estimating PDEs, especially in the low signal-to-noise regime.
△ Less
Submitted 22 October, 2019;
originally announced October 2019.
-
Eyes on the Prize: Improved Biological Surface Registration via Forward Propagation
Authors:
Robert J. Ravier
Abstract:
Many algorithms for surface registration risk producing significant errors if surfaces are significantly nonisometric. Manifold learning has been shown to be effective at improving registration quality, using information from an entire collection of surfaces to correct issues present in pairwise registrations. These methods, however, are not robust to changes in the collection of surfaces, or do n…
▽ More
Many algorithms for surface registration risk producing significant errors if surfaces are significantly nonisometric. Manifold learning has been shown to be effective at improving registration quality, using information from an entire collection of surfaces to correct issues present in pairwise registrations. These methods, however, are not robust to changes in the collection of surfaces, or do not produce accurate registrations at a resolution high enough for subsequent downstream analysis. We propose a novel algorithm for efficiently registering such collections given initial correspondences with varying degrees of accuracy. By combining the initial information with recent developments in manifold learning, we employ a simple metric condition to construct a measure on the space of correspondences between any pair of shapes in our collection, which we then use to distill soft correspondences. We demonstrate that this measure can improve correspondence accuracy between feature points compared to currently employed, less robust methods on a diverse dataset of surfaces from evolutionary biology. We then show how our methods can be used, in combination with recent sampling and interpolation methods, to compute accurate and consistent homeomorphisms between surfaces.
△ Less
Submitted 14 December, 2020; v1 submitted 26 December, 2018;
originally announced December 2018.
