-
An Analysis of the Johnson-Lindenstrauss Lemma with the Bivariate Gamma Distribution
Authors:
Jason Bernstein,
Alec M. Dunton,
Benjamin W. Priest
Abstract:
Probabilistic proofs of the Johnson-Lindenstrauss lemma imply that random projection can reduce the dimension of a data set and approximately preserve pairwise distances. If a distance being approximately preserved is called a success, and the complement of this event is called a failure, then such a random projection likely results in no failures. Assuming a Gaussian random projection, the lemma…
▽ More
Probabilistic proofs of the Johnson-Lindenstrauss lemma imply that random projection can reduce the dimension of a data set and approximately preserve pairwise distances. If a distance being approximately preserved is called a success, and the complement of this event is called a failure, then such a random projection likely results in no failures. Assuming a Gaussian random projection, the lemma is proved by showing that the no-failure probability is positive using a combination of Bonferroni's inequality and Markov's inequality. This paper modifies this proof in two ways to obtain a greater lower bound on the no-failure probability. First, Bonferroni's inequality is applied to pairs of failures instead of individual failures. Second, since a pair of projection errors has a bivariate gamma distribution, the probability of a pair of successes is bounded using an inequality from Jensen (1969). If $n$ is the number of points to be embedded and $μ$ is the probability of a success, then this leads to an increase in the lower bound on the no-failure probability of $\frac{1}{2}\binom{n}{2}(1-μ)^2$ if $\binom{n}{2}$ is even and $\frac{1}{2}\left(\binom{n}{2}-1\right)(1-μ)^2$ if $\binom{n}{2}$ is odd. For example, if $n=10^5$ points are to be embedded in $k=10^4$ dimensions with a tolerance of $ε=0.1$, then the improvement in the lower bound is on the order of $10^{-14}$. We also show that further improvement is possible if the inequality in Jensen (1969) extends to three successes, though we do not have a proof of this result.
△ Less
Submitted 12 July, 2024; v1 submitted 26 May, 2023;
originally announced May 2023.
-
Scalable Gaussian Process Hyperparameter Optimization via Coverage Regularization
Authors:
Killian Wood,
Alec M. Dunton,
Amanda Muyskens,
Benjamin W. Priest
Abstract:
Gaussian processes (GPs) are Bayesian non-parametric models popular in a variety of applications due to their accuracy and native uncertainty quantification (UQ). Tuning GP hyperparameters is critical to ensure the validity of prediction accuracy and uncertainty; uniquely estimating multiple hyperparameters in, e.g. the Matern kernel can also be a significant challenge. Moreover, training GPs on l…
▽ More
Gaussian processes (GPs) are Bayesian non-parametric models popular in a variety of applications due to their accuracy and native uncertainty quantification (UQ). Tuning GP hyperparameters is critical to ensure the validity of prediction accuracy and uncertainty; uniquely estimating multiple hyperparameters in, e.g. the Matern kernel can also be a significant challenge. Moreover, training GPs on large-scale datasets is a highly active area of research: traditional maximum likelihood hyperparameter training requires quadratic memory to form the covariance matrix and has cubic training complexity. To address the scalable hyperparameter tuning problem, we present a novel algorithm which estimates the smoothness and length-scale parameters in the Matern kernel in order to improve robustness of the resulting prediction uncertainties. Using novel loss functions similar to those in conformal prediction algorithms in the computational framework provided by the hyperparameter estimation algorithm MuyGPs, we achieve improved UQ over leave-one-out likelihood maximization while maintaining a high degree of scalability as demonstrated in numerical experiments.
△ Less
Submitted 2 November, 2022; v1 submitted 22 September, 2022;
originally announced September 2022.
-
Light curve completion and forecasting using fast and scalable Gaussian processes (MuyGPs)
Authors:
Imène R. Goumiri,
Alec M. Dunton,
Amanda L. Muyskens,
Benjamin W. Priest,
Robert E. Armstrong
Abstract:
Temporal variations of apparent magnitude, called light curves, are observational statistics of interest captured by telescopes over long periods of time. Light curves afford the exploration of Space Domain Awareness (SDA) objectives such as object identification or pose estimation as latent variable inference problems. Ground-based observations from commercial off the shelf (COTS) cameras remain…
▽ More
Temporal variations of apparent magnitude, called light curves, are observational statistics of interest captured by telescopes over long periods of time. Light curves afford the exploration of Space Domain Awareness (SDA) objectives such as object identification or pose estimation as latent variable inference problems. Ground-based observations from commercial off the shelf (COTS) cameras remain inexpensive compared to higher precision instruments, however, limited sensor availability combined with noisier observations can produce gappy time-series data that can be difficult to model. These external factors confound the automated exploitation of light curves, which makes light curve prediction and extrapolation a crucial problem for applications. Traditionally, image or time-series completion problems have been approached with diffusion-based or exemplar-based methods. More recently, Deep Neural Networks (DNNs) have become the tool of choice due to their empirical success at learning complex nonlinear embeddings. However, DNNs often require large training data that are not necessarily available when looking at unique features of a light curve of a single satellite.
In this paper, we present a novel approach to predicting missing and future data points of light curves using Gaussian Processes (GPs). GPs are non-linear probabilistic models that infer posterior distributions over functions and naturally quantify uncertainty. However, the cubic scaling of GP inference and training is a major barrier to their adoption in applications. In particular, a single light curve can feature hundreds of thousands of observations, which is well beyond the practical realization limits of a conventional GP on a single machine. Consequently, we employ MuyGPs, a scalable framework for hyperparameter estimation of GP models that uses nearest neighbors sparsification and local cross-validation. MuyGPs...
△ Less
Submitted 30 August, 2022;
originally announced August 2022.
-
Fast Gaussian Process Posterior Mean Prediction via Local Cross Validation and Precomputation
Authors:
Alec M. Dunton,
Benjamin W. Priest,
Amanda Muyskens
Abstract:
Gaussian processes (GPs) are Bayesian non-parametric models useful in a myriad of applications. Despite their popularity, the cost of GP predictions (quadratic storage and cubic complexity with respect to the number of training points) remains a hurdle in applying GPs to large data. We present a fast posterior mean prediction algorithm called FastMuyGPs to address this shortcoming. FastMuyGPs is b…
▽ More
Gaussian processes (GPs) are Bayesian non-parametric models useful in a myriad of applications. Despite their popularity, the cost of GP predictions (quadratic storage and cubic complexity with respect to the number of training points) remains a hurdle in applying GPs to large data. We present a fast posterior mean prediction algorithm called FastMuyGPs to address this shortcoming. FastMuyGPs is based upon the MuyGPs hyperparameter estimation algorithm and utilizes a combination of leave-one-out cross-validation, batching, nearest neighbors sparsification, and precomputation to provide scalable, fast GP prediction. We demonstrate several benchmarks wherein FastMuyGPs prediction attains superior accuracy and competitive or superior runtime to both deep neural networks and state-of-the-art scalable GP algorithms.
△ Less
Submitted 22 May, 2022;
originally announced May 2022.
-
TriPoll: Computing Surveys of Triangles in Massive-Scale Temporal Graphs with Metadata
Authors:
Trevor Steil,
Tahsin Reza,
Keita Iwabuchi,
Benjamin W. Priest,
Geoffrey Sanders,
Roger Pearce
Abstract:
Understanding the higher-order interactions within network data is a key objective of network science. Surveys of metadata triangles (or patterned 3-cycles in metadata-enriched graphs) are often of interest in this pursuit. In this work, we develop TriPoll, a prototype distributed HPC system capable of surveying triangles in massive graphs containing metadata on their edges and vertices. We contra…
▽ More
Understanding the higher-order interactions within network data is a key objective of network science. Surveys of metadata triangles (or patterned 3-cycles in metadata-enriched graphs) are often of interest in this pursuit. In this work, we develop TriPoll, a prototype distributed HPC system capable of surveying triangles in massive graphs containing metadata on their edges and vertices. We contrast our approach with much of the prior effort on triangle analysis, which often focuses on simple triangle counting, usually in simple graphs with no metadata. We assess the scalability of TriPoll when surveying triangles involving metadata on real and synthetic graphs with up to hundreds of billions of edges.We utilize communication-reducing optimizations to demonstrate a triangle counting task on a 224 billion edge web graph in approximately half of the time of competing approaches, while additionally supporting metadata-aware capabilities.
△ Less
Submitted 26 July, 2021;
originally announced July 2021.
-
Gaussian Process Classification for Galaxy Blend Identification in LSST
Authors:
James J. Buchanan,
Michael D. Schneider,
Robert E. Armstrong,
Amanda L. Muyskens,
Benjamin W. Priest,
Ryan J. Dana
Abstract:
A significant fraction of observed galaxies in the Rubin Observatory Legacy Survey of Space and Time (LSST) will overlap at least one other galaxy along the same line of sight, in a so-called "blend." The current standard method of assessing blend likelihood in LSST images relies on counting up the number of intensity peaks in the smoothed image of a blend candidate, but the reliability of this pr…
▽ More
A significant fraction of observed galaxies in the Rubin Observatory Legacy Survey of Space and Time (LSST) will overlap at least one other galaxy along the same line of sight, in a so-called "blend." The current standard method of assessing blend likelihood in LSST images relies on counting up the number of intensity peaks in the smoothed image of a blend candidate, but the reliability of this procedure has not yet been comprehensively studied. Here we construct a realistic distribution of blended and unblended galaxies through high-fidelity simulations of LSST-like images, and from this we examine the blend classification accuracy of the standard peak-finding method. Furthermore, we develop a novel Gaussian process blend classifier model, and show that this classifier is competitive with both the peak-finding method as well as with a convolutional neural network model. Finally, whereas the peak-finding method does not naturally assign probabilities to its classification estimates, the Gaussian process model does, and we show that the Gaussian process classification probabilities are generally reliable.
△ Less
Submitted 10 December, 2021; v1 submitted 19 July, 2021;
originally announced July 2021.
-
Star-Galaxy Image Separation with Computationally Efficient Gaussian Process Classification
Authors:
Amanda L. Muyskens,
Imène R. Goumiri,
Benjamin W. Priest,
Michael D. Schneider,
Robert E. Armstrong,
Jason M. Bernstein,
Ryan Dana
Abstract:
We introduce a novel method for discerning optical telescope images of stars from those of galaxies using Gaussian processes (GPs). Although applications of GPs often struggle in high-dimensional data modalities such as optical image classification, we show that a low-dimensional embedding of images into a metric space defined by the principal components of the data suffices to produce high-qualit…
▽ More
We introduce a novel method for discerning optical telescope images of stars from those of galaxies using Gaussian processes (GPs). Although applications of GPs often struggle in high-dimensional data modalities such as optical image classification, we show that a low-dimensional embedding of images into a metric space defined by the principal components of the data suffices to produce high-quality predictions from real large-scale survey data. We develop a novel method of GP classification hyperparameter training that scales approximately linearly in the number of image observations, which allows for application of GP models to large-size Hyper Suprime-Cam (HSC) Subaru Strategic Program data. In our experiments we evaluate the performance of a principal component analysis (PCA) embedded GP predictive model against other machine learning algorithms including a convolutional neural network and an image photometric morphology discriminator. Our analysis shows that our methods compare favorably with current methods in optical image classification while producing posterior distributions from the GP regression that can be used to quantify object classification uncertainty. We further describe how classification uncertainty can be used to efficiently parse large-scale survey imaging data to produce high-confidence object catalogs.
△ Less
Submitted 3 May, 2021;
originally announced May 2021.
-
Star-Galaxy Separation via Gaussian Processes with Model Reduction
Authors:
Imène R. Goumiri,
Amanda L. Muyskens,
Michael D. Schneider,
Benjamin W. Priest,
Robert E. Armstrong
Abstract:
Modern cosmological surveys such as the Hyper Suprime-Cam (HSC) survey produce a huge volume of low-resolution images of both distant galaxies and dim stars in our own galaxy. Being able to automatically classify these images is a long-standing problem in astronomy and critical to a number of different scientific analyses. Recently, the challenge of "star-galaxy" classification has been approached…
▽ More
Modern cosmological surveys such as the Hyper Suprime-Cam (HSC) survey produce a huge volume of low-resolution images of both distant galaxies and dim stars in our own galaxy. Being able to automatically classify these images is a long-standing problem in astronomy and critical to a number of different scientific analyses. Recently, the challenge of "star-galaxy" classification has been approached with Deep Neural Networks (DNNs), which are good at learning complex nonlinear embeddings. However, DNNs are known to overconfidently extrapolate on unseen data and require a large volume of training images that accurately capture the data distribution to be considered reliable. Gaussian Processes (GPs), which infer posterior distributions over functions and naturally quantify uncertainty, haven't been a tool of choice for this task mainly because popular kernels exhibit limited expressivity on complex and high-dimensional data.
In this paper, we present a novel approach to the star-galaxy separation problem that uses GPs and reap their benefits while solving many of the issues traditionally affecting them for classification of high-dimensional celestial image data. After an initial filtering of the raw data of star and galaxy image cutouts, we first reduce the dimensionality of the input images by using a Principal Components Analysis (PCA) before applying GPs using a simple Radial Basis Function (RBF) kernel on the reduced data. Using this method, we greatly improve the accuracy of the classification over a basic application of GPs while improving the computational efficiency and scalability of the method.
△ Less
Submitted 12 October, 2020;
originally announced October 2020.
-
Scaling Graph Clustering with Distributed Sketches
Authors:
Benjamin W. Priest,
Alec Dunton,
Geoffrey Sanders
Abstract:
The unsupervised learning of community structure, in particular the partitioning vertices into clusters or communities, is a canonical and well-studied problem in exploratory graph analysis. However, like most graph analyses the introduction of immense scale presents challenges to traditional methods. Spectral clustering in distributed memory, for example, requires hundreds of expensive bulk-synch…
▽ More
The unsupervised learning of community structure, in particular the partitioning vertices into clusters or communities, is a canonical and well-studied problem in exploratory graph analysis. However, like most graph analyses the introduction of immense scale presents challenges to traditional methods. Spectral clustering in distributed memory, for example, requires hundreds of expensive bulk-synchronous communication rounds to compute an embedding of vertices to a few eigenvectors of a graph associated matrix. Furthermore, the whole computation may need to be repeated if the underlying graph changes some low percentage of edge updates. We present a method inspired by spectral clustering where we instead use matrix sketches derived from random dimension-reducing projections. We show that our method produces embeddings that yield performant clustering results given a fully-dynamic stochastic block model stream using both the fast Johnson-Lindenstrauss and CountSketch transforms. We also discuss the effects of stochastic block model parameters upon the required dimensionality of the subsequent embeddings, and show how random projections could significantly improve the performance of graph clustering in distributed memory.
△ Less
Submitted 24 July, 2020;
originally announced July 2020.
-
Quantum Machine Learning using Gaussian Processes with Performant Quantum Kernels
Authors:
Matthew Otten,
Imène R. Goumiri,
Benjamin W. Priest,
George F. Chapline,
Michael D. Schneider
Abstract:
Quantum computers have the opportunity to be transformative for a variety of computational tasks. Recently, there have been proposals to use the unsimulatably of large quantum devices to perform regression, classification, and other machine learning tasks with quantum advantage by using kernel methods. While unsimulatably is a necessary condition for quantum advantage in machine learning, it is no…
▽ More
Quantum computers have the opportunity to be transformative for a variety of computational tasks. Recently, there have been proposals to use the unsimulatably of large quantum devices to perform regression, classification, and other machine learning tasks with quantum advantage by using kernel methods. While unsimulatably is a necessary condition for quantum advantage in machine learning, it is not sufficient, as not all kernels are equally effective. Here, we study the use of quantum computers to perform the machine learning tasks of one- and multi-dimensional regression, as well as reinforcement learning, using Gaussian Processes. By using approximations of performant classical kernels enhanced with extra quantum resources, we demonstrate that quantum devices, both in simulation and on hardware, can perform machine learning tasks at least as well as, and many times better than, the classical inspiration. Our informed kernel design demonstrates a path towards effectively utilizing quantum devices for machine learning tasks.
△ Less
Submitted 23 April, 2020;
originally announced April 2020.
-
Reinforcement Learning via Gaussian Processes with Neural Network Dual Kernels
Authors:
Imène R. Goumiri,
Benjamin W. Priest,
Michael D. Schneider
Abstract:
While deep neural networks (DNNs) and Gaussian Processes (GPs) are both popularly utilized to solve problems in reinforcement learning, both approaches feature undesirable drawbacks for challenging problems. DNNs learn complex nonlinear embeddings, but do not naturally quantify uncertainty and are often data-inefficient to train. GPs infer posterior distributions over functions, but popular kernel…
▽ More
While deep neural networks (DNNs) and Gaussian Processes (GPs) are both popularly utilized to solve problems in reinforcement learning, both approaches feature undesirable drawbacks for challenging problems. DNNs learn complex nonlinear embeddings, but do not naturally quantify uncertainty and are often data-inefficient to train. GPs infer posterior distributions over functions, but popular kernels exhibit limited expressivity on complex and high-dimensional data. Fortunately, recently discovered conjugate and neural tangent kernel functions encode the behavior of overparameterized neural networks in the kernel domain. We demonstrate that these kernels can be efficiently applied to regression and reinforcement learning problems by analyzing a baseline case study. We apply GPs with neural network dual kernels to solve reinforcement learning tasks for the first time. We demonstrate, using the well-understood mountain-car problem, that GPs empowered with dual kernels perform at least as well as those using the conventional radial basis function kernel. We conjecture that by inheriting the probabilistic rigor of GPs and the powerful embedding properties of DNNs, GPs using NN dual kernels will empower future reinforcement learning models on difficult domains.
△ Less
Submitted 10 April, 2020;
originally announced April 2020.
-
DegreeSketch: Distributed Cardinality Sketches on Massive Graphs with Applications
Authors:
Benjamin W. Priest
Abstract:
We present DegreeSketch, a semi-streaming distributed sketch data structure and demonstrate its utility for estimating local neighborhood sizes and local triangle count heavy hitters on massive graphs. DegreeSketch consists of vertex-centric cardinality sketches distributed across a set of processors that are accumulated in a single pass, and then behaves as a persistent query engine capable of ap…
▽ More
We present DegreeSketch, a semi-streaming distributed sketch data structure and demonstrate its utility for estimating local neighborhood sizes and local triangle count heavy hitters on massive graphs. DegreeSketch consists of vertex-centric cardinality sketches distributed across a set of processors that are accumulated in a single pass, and then behaves as a persistent query engine capable of approximately answering graph queries pertaining to the sizes of adjacency set unions and intersections. The $t$th local neighborhood of a vertex is the number of vertices reachable in $G$ from $v$ by traversing at most $t$ edges, whereas the local triangle count is the number of 3-cycles in which it is included. Both metrics are useful in graph analysis applications, but exact computations scale poorly as graph sizes grow. We present efficient algorithms for estimating both local neighborhood sizes and local triangle count heavy hitters using DegreeSketch. In our experiments we implement DegreeSketch using the celebrated hyperloglog cardinality sketch and utilize the distributed communication tool YGM to achieve state-of-the-art performance in distributed memory.
△ Less
Submitted 8 April, 2020;
originally announced April 2020.
