-
Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference
Authors:
Fred Hohman,
Chaoqun Wang,
**mook Lee,
Jochen Görtler,
Dominik Moritz,
Jeffrey P Bigham,
Zhile Ren,
Cecile Foret,
Qi Shan,
Xiaoyi Zhang
Abstract:
On-device machine learning (ML) moves computation from the cloud to personal devices, protecting user privacy and enabling intelligent user experiences. However, fitting models on devices with limited resources presents a major technical challenge: practitioners need to optimize models and balance hardware metrics such as model size, latency, and power. To help practitioners create efficient ML mo…
▽ More
On-device machine learning (ML) moves computation from the cloud to personal devices, protecting user privacy and enabling intelligent user experiences. However, fitting models on devices with limited resources presents a major technical challenge: practitioners need to optimize models and balance hardware metrics such as model size, latency, and power. To help practitioners create efficient ML models, we designed and developed Talaria: a model visualization and optimization system. Talaria enables practitioners to compile models to hardware, interactively visualize model statistics, and simulate optimizations to test the impact on inference metrics. Since its internal deployment two years ago, we have evaluated Talaria using three methodologies: (1) a log analysis highlighting its growth of 800+ practitioners submitting 3,600+ models; (2) a usability survey with 26 users assessing the utility of 20 Talaria features; and (3) a qualitative interview with the 7 most active users about their experience using Talaria.
△ Less
Submitted 3 April, 2024;
originally announced April 2024.
-
Neural Injective Functions for Multisets, Measures and Graphs via a Finite Witness Theorem
Authors:
Tal Amir,
Steven J. Gortler,
Ilai Avni,
Ravina Ravina,
Nadav Dym
Abstract:
Injective multiset functions have a key role in the theoretical study of machine learning on multisets and graphs. Yet, there remains a gap between the provably injective multiset functions considered in theory, which typically rely on polynomial moments, and the multiset functions used in practice, which rely on $\textit{neural moments}$ $\unicode{x2014}$ whose injectivity on multisets has not be…
▽ More
Injective multiset functions have a key role in the theoretical study of machine learning on multisets and graphs. Yet, there remains a gap between the provably injective multiset functions considered in theory, which typically rely on polynomial moments, and the multiset functions used in practice, which rely on $\textit{neural moments}$ $\unicode{x2014}$ whose injectivity on multisets has not been studied to date.
In this paper, we bridge this gap by showing that moments of neural networks do define injective multiset functions, provided that an analytic non-polynomial activation is used. The number of moments required by our theory is optimal essentially up to a multiplicative factor of two. To prove this result, we state and prove a $\textit{finite witness theorem}$, which is of independent interest.
As a corollary to our main theorem, we derive new approximation results for functions on multisets and measures, and new separation results for graph neural networks. We also provide two negative results: (1) moments of piecewise-linear neural networks cannot be injective multiset functions; and (2) even when moment-based multiset functions are injective, they can never be bi-Lipschitz.
△ Less
Submitted 29 October, 2023; v1 submitted 10 June, 2023;
originally announced June 2023.
-
Complete Neural Networks for Complete Euclidean Graphs
Authors:
Snir Hordan,
Tal Amir,
Steven J. Gortler,
Nadav Dym
Abstract:
Neural networks for point clouds, which respect their natural invariance to permutation and rigid motion, have enjoyed recent success in modeling geometric phenomena, from molecular dynamics to recommender systems. Yet, to date, no model with polynomial complexity is known to be complete, that is, able to distinguish between any pair of non-isomorphic point clouds. We fill this theoretical gap by…
▽ More
Neural networks for point clouds, which respect their natural invariance to permutation and rigid motion, have enjoyed recent success in modeling geometric phenomena, from molecular dynamics to recommender systems. Yet, to date, no model with polynomial complexity is known to be complete, that is, able to distinguish between any pair of non-isomorphic point clouds. We fill this theoretical gap by showing that point clouds can be completely determined, up to permutation and rigid motion, by applying the 3-WL graph isomorphism test to the point cloud's centralized Gram matrix. Moreover, we formulate an Euclidean variant of the 2-WL test and show that it is also sufficient to achieve completeness. We then show how our complete Euclidean WL tests can be simulated by an Euclidean graph neural network of moderate size and demonstrate their separation capability on highly symmetrical point clouds.
△ Less
Submitted 9 April, 2024; v1 submitted 31 January, 2023;
originally announced January 2023.
-
Low Dimensional Invariant Embeddings for Universal Geometric Learning
Authors:
Nadav Dym,
Steven J. Gortler
Abstract:
This paper studies separating invariants: map**s on $D$ dimensional domains which are invariant to an appropriate group action, and which separate orbits. The motivation for this study comes from the usefulness of separating invariants in proving universality of equivariant neural network architectures.
We observe that in several cases the cardinality of separating invariants proposed in the m…
▽ More
This paper studies separating invariants: map**s on $D$ dimensional domains which are invariant to an appropriate group action, and which separate orbits. The motivation for this study comes from the usefulness of separating invariants in proving universality of equivariant neural network architectures.
We observe that in several cases the cardinality of separating invariants proposed in the machine learning literature is much larger than the dimension $D$. As a result, the theoretical universal constructions based on these separating invariants is unrealistically large. Our goal in this paper is to resolve this issue.
We show that when a continuous family of semi-algebraic separating invariants is available, separation can be obtained by randomly selecting $2D+1 $ of these invariants. We apply this methodology to obtain an efficient scheme for computing separating invariants for several classical group actions which have been studied in the invariant learning literature. Examples include matrix multiplication actions on point clouds by permutations, rotations, and various other linear groups.
Often the requirement of invariant separation is relaxed and only generic separation is required. In this case, we show that only $D+1$ invariants are required. More importantly, generic invariants are often significantly easier to compute, as we illustrate by discussing generic and full separation for weighted graphs. Finally we outline an approach for proving that separating invariants can be constructed also when the random parameters have finite precision.
△ Less
Submitted 21 November, 2023; v1 submitted 5 May, 2022;
originally announced May 2022.
-
Lattices Without a Big Constant and With Noise
Authors:
Steven J. Gortler,
Louis Theran
Abstract:
We show how Frieze's analysis of subset sum solving using lattices can be done with out any large constants and without flip**. We apply the variant without the large constant to inputs with noise.
We show how Frieze's analysis of subset sum solving using lattices can be done with out any large constants and without flip**. We apply the variant without the large constant to inputs with noise.
△ Less
Submitted 26 April, 2022;
originally announced April 2022.
-
Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels
Authors:
Jochen Görtler,
Fred Hohman,
Dominik Moritz,
Kanit Wongsuphasawat,
Donghao Ren,
Rahul Nair,
Marc Kirchner,
Kayur Patel
Abstract:
The confusion matrix, a ubiquitous visualization for hel** people evaluate machine learning models, is a tabular layout that compares predicted class labels against actual class labels over all data instances. We conduct formative research with machine learning practitioners at Apple and find that conventional confusion matrices do not support more complex data-structures found in modern-day app…
▽ More
The confusion matrix, a ubiquitous visualization for hel** people evaluate machine learning models, is a tabular layout that compares predicted class labels against actual class labels over all data instances. We conduct formative research with machine learning practitioners at Apple and find that conventional confusion matrices do not support more complex data-structures found in modern-day applications, such as hierarchical and multi-output labels. To express such variations of confusion matrices, we design an algebra that models confusion matrices as probability distributions. Based on this algebra, we develop Neo, a visual analytics system that enables practitioners to flexibly author and interact with hierarchical and multi-output confusion matrices, visualize derived metrics, renormalize confusions, and share matrix specifications. Finally, we demonstrate Neo's utility with three model evaluation scenarios that help people better understand model performance and reveal hidden confusions.
△ Less
Submitted 17 February, 2022; v1 submitted 24 October, 2021;
originally announced October 2021.
-
SpEuler: Semantics-preserving Euler Diagrams
Authors:
Rebecca Kehlbeck,
Jochen Görtler,
Yunhai Wang,
Oliver Deussen
Abstract:
Creating comprehensible visualizations of highly overlap** set-typed data is a challenging task due to its complexity. To facilitate insights into set connectivity and to leverage semantic relations between intersections, we propose a fast two-step layout technique for Euler diagrams that are both well-matched and well-formed. Our method conforms to established form guidelines for Euler diagrams…
▽ More
Creating comprehensible visualizations of highly overlap** set-typed data is a challenging task due to its complexity. To facilitate insights into set connectivity and to leverage semantic relations between intersections, we propose a fast two-step layout technique for Euler diagrams that are both well-matched and well-formed. Our method conforms to established form guidelines for Euler diagrams regarding semantics, aesthetics, and readability. First, we establish an initial ordering of the data, which we then use to incrementally create a planar, connected, and monotone dual graph representation. In the next step, the graph is transformed into a circular layout that maintains the semantics and yields simple Euler diagrams with smooth curves. When the data cannot be represented by simple diagrams, our algorithm always falls back to a solution that is not well-formed but still well-matched, whereas previous methods often fail to produce expected results. We show the usefulness of our method for visualizing set-typed data using examples from text analysis and infographics. Furthermore, we discuss the characteristics of our approach and evaluate our method against state-of-the-art methods.
△ Less
Submitted 7 August, 2021;
originally announced August 2021.
-
Unique Geometry and Texture from Corresponding Image Patches
Authors:
Dor Verbin,
Steven J. Gortler,
Todd Zickler
Abstract:
We present a sufficient condition for recovering unique texture and viewpoints from unknown orthographic projections of a flat texture process. We show that four observations are sufficient in general, and we characterize the ambiguous cases. The results are applicable to shape from texture and texture-based structure from motion.
We present a sufficient condition for recovering unique texture and viewpoints from unknown orthographic projections of a flat texture process. We show that four observations are sufficient in general, and we characterize the ambiguous cases. The results are applicable to shape from texture and texture-based structure from motion.
△ Less
Submitted 6 November, 2021; v1 submitted 19 March, 2020;
originally announced March 2020.
-
Uncertainty-Aware Principal Component Analysis
Authors:
Jochen Görtler,
Thilo Spinner,
Dirk Streeb,
Daniel Weiskopf,
Oliver Deussen
Abstract:
We present a technique to perform dimensionality reduction on data that is subject to uncertainty. Our method is a generalization of traditional principal component analysis (PCA) to multivariate probability distributions. In comparison to non-linear methods, linear dimensionality reduction techniques have the advantage that the characteristics of such probability distributions remain intact after…
▽ More
We present a technique to perform dimensionality reduction on data that is subject to uncertainty. Our method is a generalization of traditional principal component analysis (PCA) to multivariate probability distributions. In comparison to non-linear methods, linear dimensionality reduction techniques have the advantage that the characteristics of such probability distributions remain intact after projection. We derive a representation of the PCA sample covariance matrix that respects potential uncertainty in each of the inputs, building the mathematical foundation of our new method: uncertainty-aware PCA. In addition to the accuracy and performance gained by our approach over sampling-based strategies, our formulation allows us to perform sensitivity analysis with regard to the uncertainty in the data. For this, we propose factor traces as a novel visualization that enables to better understand the influence of uncertainty on the chosen principal components. We provide multiple examples of our technique using real-world datasets. As a special case, we show how to propagate multivariate normal distributions through PCA in closed form. Furthermore, we discuss extensions and limitations of our approach.
△ Less
Submitted 1 August, 2019; v1 submitted 3 May, 2019;
originally announced May 2019.
-
A Report on Shape Deformation with a Stretching and Bending Energy
Authors:
Hui Zhao,
Steven J. Gortler
Abstract:
In this report we describe a mesh editing system that we implemented that uses a natural stretching and bending energy defined over smooth surfaces. As such, this energy behaves uniformly under various mesh resolutions. All of the elements of our approach already exist in the literature. We hope that our discussions of these energies helps to shed light on the behaviors of these methods and provid…
▽ More
In this report we describe a mesh editing system that we implemented that uses a natural stretching and bending energy defined over smooth surfaces. As such, this energy behaves uniformly under various mesh resolutions. All of the elements of our approach already exist in the literature. We hope that our discussions of these energies helps to shed light on the behaviors of these methods and provides a unified discussion of these methods.
△ Less
Submitted 22 March, 2016;
originally announced March 2016.
-
On the Embeddability of Delaunay Triangulations in Anisotropic, Normed, and Bregman Spaces
Authors:
Guillermo D. Canas,
Steven J. Gortler
Abstract:
Given a two-dimensional space endowed with a divergence function that is convex in the first argument, continuously differentiable in the second, and satisfies suitable regularity conditions at Voronoi vertices, we show that orphan-freedom (the absence of disconnected Voronoi regions) is sufficient to ensure that Voronoi edges and vertices are also connected, and that the dual is a simple planar g…
▽ More
Given a two-dimensional space endowed with a divergence function that is convex in the first argument, continuously differentiable in the second, and satisfies suitable regularity conditions at Voronoi vertices, we show that orphan-freedom (the absence of disconnected Voronoi regions) is sufficient to ensure that Voronoi edges and vertices are also connected, and that the dual is a simple planar graph. We then prove that the straight-edge dual of an orphan-free Voronoi diagram (with sites as the first argument of the divergence) is always an embedded triangulation.
Among the divergences covered by our proofs are Bregman divergences, anisotropic divergences, as well as all distances derived from strictly convex $\mathcal{C}^1$ norms (including the $L_p$ norms with $1< p < \infty$). While Bregman diagrams of the {first kind} are simply affine diagrams, and their duals ({weighted} Delaunay triangulations) are always embedded, we show that duals of orphan-free Bregman diagrams of the \emph{second kind} are always embedded.
△ Less
Submitted 9 January, 2016; v1 submitted 11 December, 2015;
originally announced December 2015.
-
Low-level Vision by Consensus in a Spatial Hierarchy of Regions
Authors:
Ayan Chakrabarti,
Ying Xiong,
Steven J. Gortler,
Todd Zickler
Abstract:
We introduce a multi-scale framework for low-level vision, where the goal is estimating physical scene values from image data---such as depth from stereo image pairs. The framework uses a dense, overlap** set of image regions at multiple scales and a "local model," such as a slanted-plane model for stereo disparity, that is expected to be valid piecewise across the visual field. Estimation is ca…
▽ More
We introduce a multi-scale framework for low-level vision, where the goal is estimating physical scene values from image data---such as depth from stereo image pairs. The framework uses a dense, overlap** set of image regions at multiple scales and a "local model," such as a slanted-plane model for stereo disparity, that is expected to be valid piecewise across the visual field. Estimation is cast as optimization over a dichotomous mixture of variables, simultaneously determining which regions are inliers with respect to the local model (binary variables) and the correct co-ordinates in the local model space for each inlying region (continuous variables). When the regions are organized into a multi-scale hierarchy, optimization can occur in an efficient and parallel architecture, where distributed computational units iteratively perform calculations and share information through sparse connections between parents and children. The framework performs well on a standard benchmark for binocular stereo, and it produces a distributional scene representation that is appropriate for combining with higher-level reasoning and other low-level cues.
△ Less
Submitted 14 April, 2015; v1 submitted 18 November, 2014;
originally announced November 2014.
-
From Shading to Local Shape
Authors:
Ying Xiong,
Ayan Chakrabarti,
Ronen Basri,
Steven J. Gortler,
David W. Jacobs,
Todd Zickler
Abstract:
We develop a framework for extracting a concise representation of the shape information available from diffuse shading in a small image patch. This produces a mid-level scene descriptor, comprised of local shape distributions that are inferred separately at every image patch across multiple scales. The framework is based on a quadratic representation of local shape that, in the absence of noise, h…
▽ More
We develop a framework for extracting a concise representation of the shape information available from diffuse shading in a small image patch. This produces a mid-level scene descriptor, comprised of local shape distributions that are inferred separately at every image patch across multiple scales. The framework is based on a quadratic representation of local shape that, in the absence of noise, has guarantees on recovering accurate local shape and lighting. And when noise is present, the inferred local shape distributions provide useful shape information without over-committing to any particular image explanation. These local shape distributions naturally encode the fact that some smooth diffuse regions are more informative than others, and they enable efficient and robust reconstruction of object-scale shape. Experimental results show that this approach to surface reconstruction compares well against the state-of-art on both synthetic images and captured photographs.
△ Less
Submitted 7 April, 2014; v1 submitted 10 October, 2013;
originally announced October 2013.
-
Measurement Isomorphism of Graphs
Authors:
Steven J. Gortler,
Dylan P. Thurston
Abstract:
The d-measurement set of a graph is its set of possible squared edge lengths over all d-dimensional embeddings. In this note, we define a new notion of graph isomorphism called d-measurement isomorphism. Two graphs are d-measurement isomorphic if there is agreement in their d-measurement sets. A natural question to ask is "what can be said about two graphs that are d-measurement isomorphic?" In th…
▽ More
The d-measurement set of a graph is its set of possible squared edge lengths over all d-dimensional embeddings. In this note, we define a new notion of graph isomorphism called d-measurement isomorphism. Two graphs are d-measurement isomorphic if there is agreement in their d-measurement sets. A natural question to ask is "what can be said about two graphs that are d-measurement isomorphic?" In this note, we show that this property coincides with the 2-isomorphism property studied by Whitney.
△ Less
Submitted 28 December, 2012;
originally announced December 2012.
-
Duals of Orphan-Free Anisotropic Voronoi Diagrams are Triangulations
Authors:
Guillermo D. Canas,
Steven J. Gortler
Abstract:
We show that, under mild conditions on the underlying metric, duals of appropriately defined anisotropic Voronoi diagrams are embedded triangulations. Furthermore, they always triangulate the convex hull of the vertices, and have other properties that parallel those of ordinary Delaunay triangulations. These results apply to the duals of anisotropic Voronoi diagrams of any set of vertices, so long…
▽ More
We show that, under mild conditions on the underlying metric, duals of appropriately defined anisotropic Voronoi diagrams are embedded triangulations. Furthermore, they always triangulate the convex hull of the vertices, and have other properties that parallel those of ordinary Delaunay triangulations. These results apply to the duals of anisotropic Voronoi diagrams of any set of vertices, so long as the diagram is orphan-free.
△ Less
Submitted 27 March, 2012; v1 submitted 17 February, 2011;
originally announced February 2011.
-
Orphan-Free Anisotropic Voronoi Diagrams
Authors:
Guillermo D. Canas,
Steven J. Gortler
Abstract:
We describe conditions under which an appropriately-defined anisotropic Voronoi diagram of a set of sites in Euclidean space is guaranteed to be composed of connected cells in any number of dimensions. These conditions are natural for problems in optimization and approximation, and algorithms already exist to produce sets of sites that satisfy them.
We describe conditions under which an appropriately-defined anisotropic Voronoi diagram of a set of sites in Euclidean space is guaranteed to be composed of connected cells in any number of dimensions. These conditions are natural for problems in optimization and approximation, and algorithms already exist to produce sets of sites that satisfy them.
△ Less
Submitted 17 February, 2011;
originally announced February 2011.
-
On affine rigidity
Authors:
Steven J. Gortler,
Craig Gotsman,
Ligang Liu,
Dylan P. Thurston
Abstract:
We define the notion of affine rigidity of a hypergraph and prove a variety of fundamental results for this notion. First, we show that affine rigidity can be determined by the rank of a specific matrix which implies that affine rigidity is a generic property of the hypergraph.Then we prove that if a graph is is $(d+1)$-vertex-connected, then it must be "generically neighborhood affinely rigid" in…
▽ More
We define the notion of affine rigidity of a hypergraph and prove a variety of fundamental results for this notion. First, we show that affine rigidity can be determined by the rank of a specific matrix which implies that affine rigidity is a generic property of the hypergraph.Then we prove that if a graph is is $(d+1)$-vertex-connected, then it must be "generically neighborhood affinely rigid" in $d$-dimensional space. This implies that if a graph is $(d+1)$-vertex-connected then any generic framework of its squared graph must be universally rigid.
Our results, and affine rigidity more generally, have natural applications in point registration and localization, as well as connections to manifold learning.
△ Less
Submitted 13 August, 2013; v1 submitted 25 November, 2010;
originally announced November 2010.
