-
Synaptic Sampling of Neural Networks
Authors:
James B. Aimone,
William Severa,
J. Darby Smith
Abstract:
Probabilistic artificial neural networks offer intriguing prospects for enabling the uncertainty of artificial intelligence methods to be described explicitly in their function; however, the development of techniques that quantify uncertainty by well-understood methods such as Monte Carlo sampling has been limited by the high costs of stochastic sampling on deterministic computing hardware. Emergi…
▽ More
Probabilistic artificial neural networks offer intriguing prospects for enabling the uncertainty of artificial intelligence methods to be described explicitly in their function; however, the development of techniques that quantify uncertainty by well-understood methods such as Monte Carlo sampling has been limited by the high costs of stochastic sampling on deterministic computing hardware. Emerging computing systems that are amenable to hardware-level probabilistic computing, such as those that leverage stochastic devices, may make probabilistic neural networks more feasible in the not-too-distant future. This paper describes the scANN technique -- \textit{sampling (by coinflips) artificial neural networks} -- which enables neural networks to be sampled directly by treating the weights as Bernoulli coin flips. This method is natively well suited for probabilistic computing techniques that focus on tunable stochastic devices, nearly matches fully deterministic performance while also describing the uncertainty of correct and incorrect neural network outputs.
△ Less
Submitted 21 November, 2023;
originally announced November 2023.
-
Probabilistic Neural Circuits leveraging AI-Enhanced Codesign for Random Number Generation
Authors:
Suma G. Cardwell,
Catherine D. Schuman,
J. Darby Smith,
Karan Patel,
Jaesuk Kwon,
Samuel Liu,
Christopher Allemang,
Shashank Misra,
Jean Anne Incorvia,
James B. Aimone
Abstract:
Stochasticity is ubiquitous in the world around us. However, our predominant computing paradigm is deterministic. Random number generation (RNG) can be a computationally inefficient operation in this system especially for larger workloads. Our work leverages the underlying physics of emerging devices to develop probabilistic neural circuits for RNGs from a given distribution. However, codesign for…
▽ More
Stochasticity is ubiquitous in the world around us. However, our predominant computing paradigm is deterministic. Random number generation (RNG) can be a computationally inefficient operation in this system especially for larger workloads. Our work leverages the underlying physics of emerging devices to develop probabilistic neural circuits for RNGs from a given distribution. However, codesign for novel circuits and systems that leverage inherent device stochasticity is a hard problem. This is mostly due to the large design space and complexity of doing so. It requires concurrent input from multiple areas in the design stack from algorithms, architectures, circuits, to devices. In this paper, we present examples of optimal circuits developed leveraging AI-enhanced codesign techniques using constraints from emerging devices and algorithms. Our AI-enhanced codesign approach accelerated design and enabled interactions between experts from different areas of the microelectronics design stack including theory, algorithms, circuits, and devices. We demonstrate optimal probabilistic neural circuits using magnetic tunnel junction and tunnel diode devices that generate an RNG from a given distribution.
△ Less
Submitted 1 December, 2022;
originally announced December 2022.
-
Stochastic Neuromorphic Circuits for Solving MAXCUT
Authors:
Bradley H. Theilman,
Yipu Wang,
Ojas D. Parekh,
William Severa,
J. Darby Smith,
James B. Aimone
Abstract:
Finding the maximum cut of a graph (MAXCUT) is a classic optimization problem that has motivated parallel algorithm development. While approximate algorithms to MAXCUT offer attractive theoretical guarantees and demonstrate compelling empirical performance, such approximation approaches can shift the dominant computational cost to the stochastic sampling operations. Neuromorphic computing, which u…
▽ More
Finding the maximum cut of a graph (MAXCUT) is a classic optimization problem that has motivated parallel algorithm development. While approximate algorithms to MAXCUT offer attractive theoretical guarantees and demonstrate compelling empirical performance, such approximation approaches can shift the dominant computational cost to the stochastic sampling operations. Neuromorphic computing, which uses the organizing principles of the nervous system to inspire new parallel computing architectures, offers a possible solution. One ubiquitous feature of natural brains is stochasticity: the individual elements of biological neural networks possess an intrinsic randomness that serves as a resource enabling their unique computational capacities. By designing circuits and algorithms that make use of randomness similarly to natural brains, we hypothesize that the intrinsic randomness in microelectronics devices could be turned into a valuable component of a neuromorphic architecture enabling more efficient computations. Here, we present neuromorphic circuits that transform the stochastic behavior of a pool of random devices into useful correlations that drive stochastic solutions to MAXCUT. We show that these circuits perform favorably in comparison to software solvers and argue that this neuromorphic hardware implementation provides a path for scaling advantages. This work demonstrates the utility of combining neuromorphic principles with intrinsic randomness as a computational resource for new computational architectures.
△ Less
Submitted 5 October, 2022;
originally announced October 2022.
-
Autonomous Passage Planning for a Polar Vessel
Authors:
Jonathan D. Smith,
Samuel Hall,
George Coombs,
James Byrne,
Michael A. S. Thorne,
J. Alexander Brearley,
Derek Long,
Michael Meredith,
Maria Fox
Abstract:
We introduce a method for long-distance maritime route planning in polar regions, taking into account complex changing environmental conditions. The method allows the construction of optimised routes, describing the three main stages of the process: discrete modelling of the environmental conditions using a non-uniform mesh, the construction of mesh-optimal paths, and path smoothing. In order to a…
▽ More
We introduce a method for long-distance maritime route planning in polar regions, taking into account complex changing environmental conditions. The method allows the construction of optimised routes, describing the three main stages of the process: discrete modelling of the environmental conditions using a non-uniform mesh, the construction of mesh-optimal paths, and path smoothing. In order to account for different vehicle properties we construct a series of data driven functions that can be applied to the environmental mesh to determine the speed limitations and fuel requirements for a given vessel and mesh cell, representing these quantities graphically and geospatially. In describing our results, we demonstrate an example use case for route planning for the polar research ship the RRS Sir David Attenborough (SDA), accounting for ice-performance characteristics and validating the spatial-temporal route construction in the region of the Weddell Sea, Antarctica. We demonstrate the versatility of this route construction method by demonstrating that routes change depending on the seasonal sea ice variability, differences in the route-planning objective functions used, and the presence of other environmental conditions such as currents. To demonstrate the generality of our approach, we present examples in the Arctic Ocean and the Baltic Sea. The techniques outlined in this manuscript are generic and can therefore be applied to vessels with different characteristics. Our approach can have considerable utility beyond just a single vessel planning procedure, and we outline how this workflow is applicable to a wider community, e.g. commercial and passenger ship**.
△ Less
Submitted 13 September, 2022; v1 submitted 17 August, 2022;
originally announced September 2022.
-
Neuromorphic scaling advantages for energy-efficient random walk computation
Authors:
J. Darby Smith,
Aaron J. Hill,
Leah E. Reeder,
Brian C. Franke,
Richard B. Lehoucq,
Ojas Parekh,
William Severa,
James B. Aimone
Abstract:
Computing stands to be radically improved by neuromorphic computing (NMC) approaches inspired by the brain's incredible efficiency and capabilities. Most NMC research, which aims to replicate the brain's computational structure and architecture in man-made hardware, has focused on artificial intelligence; however, less explored is whether this brain-inspired hardware can provide value beyond cogni…
▽ More
Computing stands to be radically improved by neuromorphic computing (NMC) approaches inspired by the brain's incredible efficiency and capabilities. Most NMC research, which aims to replicate the brain's computational structure and architecture in man-made hardware, has focused on artificial intelligence; however, less explored is whether this brain-inspired hardware can provide value beyond cognitive tasks. We demonstrate that high-degree parallelism and configurability of spiking neuromorphic architectures makes them well-suited to implement random walks via discrete time Markov chains. Such random walks are useful in Monte Carlo methods, which represent a fundamental computational tool for solving a wide range of numerical computing tasks. Additionally, we show how the mathematical basis for a probabilistic solution involving a class of stochastic differential equations can leverage those simulations to provide solutions for a range of broadly applicable computational tasks. Despite being in an early development stage, we find that NMC platforms, at a sufficient scale, can drastically reduce the energy demands of high-performance computing (HPC) platforms.
△ Less
Submitted 27 July, 2021;
originally announced July 2021.
-
HypoSVI: Hypocenter inversion with Stein variational inference and Physics Informed Neural Networks
Authors:
Jonathan D. Smith,
Zachary E. Ross,
Kamyar Azizzadenesheli,
Jack B. Muir
Abstract:
We introduce a scheme for probabilistic hypocenter inversion with Stein variational inference. Our approach uses a differentiable forward model in the form of a physics informed neural network, which we train to solve the Eikonal equation. This allows for rapid approximation of the posterior by iteratively optimizing a collection of particles against a kernelized Stein discrepancy. We show that th…
▽ More
We introduce a scheme for probabilistic hypocenter inversion with Stein variational inference. Our approach uses a differentiable forward model in the form of a physics informed neural network, which we train to solve the Eikonal equation. This allows for rapid approximation of the posterior by iteratively optimizing a collection of particles against a kernelized Stein discrepancy. We show that the method is well-equipped to handle highly multimodal posterior distributions, which are common in hypocentral inverse problems. A suite of experiments is performed to examine the influence of the various hyperparameters. Once trained, the method is valid for any seismic network geometry within the study area without the need to build travel time tables. We show that the computational demands scale efficiently with the number of differential times, making it ideal for large-N sensing technologies like Distributed Acoustic Sensing. The techniques outlined in this manuscript have considerable implications beyond just ray-tracing procedures, with the work flow applicable to other fields with computationally expensive inversion procedures such as full waveform inversion.
△ Less
Submitted 17 August, 2022; v1 submitted 8 January, 2021;
originally announced January 2021.
-
Solving a steady-state PDE using spiking networks and neuromorphic hardware
Authors:
J. Darby Smith,
William Severa,
Aaron J. Hill,
Leah Reeder,
Brian Franke,
Richard B. Lehoucq,
Ojas D. Parekh,
James B. Aimone
Abstract:
The widely parallel, spiking neural networks of neuromorphic processors can enable computationally powerful formulations. While recent interest has focused on primarily machine learning tasks, the space of appropriate applications is wide and continually expanding. Here, we leverage the parallel and event-driven structure to solve a steady state heat equation using a random walk method. The random…
▽ More
The widely parallel, spiking neural networks of neuromorphic processors can enable computationally powerful formulations. While recent interest has focused on primarily machine learning tasks, the space of appropriate applications is wide and continually expanding. Here, we leverage the parallel and event-driven structure to solve a steady state heat equation using a random walk method. The random walk can be executed fully within a spiking neural network using stochastic neuron behavior, and we provide results from both IBM TrueNorth and Intel Loihi implementations. Additionally, we position this algorithm as a potential scalable benchmark for neuromorphic systems.
△ Less
Submitted 21 May, 2020;
originally announced May 2020.
-
EikoNet: Solving the Eikonal equation with Deep Neural Networks
Authors:
Jonathan D. Smith,
Kamyar Azizzadenesheli,
Zachary E. Ross
Abstract:
The recent deep learning revolution has created an enormous opportunity for accelerating compute capabilities in the context of physics-based simulations. Here, we propose EikoNet, a deep learning approach to solving the Eikonal equation, which characterizes the first-arrival-time field in heterogeneous 3D velocity structures. Our grid-free approach allows for rapid determination of the travel tim…
▽ More
The recent deep learning revolution has created an enormous opportunity for accelerating compute capabilities in the context of physics-based simulations. Here, we propose EikoNet, a deep learning approach to solving the Eikonal equation, which characterizes the first-arrival-time field in heterogeneous 3D velocity structures. Our grid-free approach allows for rapid determination of the travel time between any two points within a continuous 3D domain. These travel time solutions are allowed to violate the differential equation - which casts the problem as one of optimization - with the goal of finding network parameters that minimize the degree to which the equation is violated. In doing so, the method exploits the differentiability of neural networks to calculate the spatial gradients analytically, meaning the network can be trained on its own without ever needing solutions from a finite difference algorithm. EikoNet is rigorously tested on several velocity models and sampling methods to demonstrate robustness and versatility. Training and inference are highly parallelized, making the approach well-suited for GPUs. EikoNet has low memory overhead, and further avoids the need for travel-time lookup tables. The developed approach has important applications to earthquake hypocenter inversion, ray multi-pathing, and tomographic modeling, as well as to other fields beyond seismology where ray tracing is essential.
△ Less
Submitted 11 August, 2020; v1 submitted 24 March, 2020;
originally announced April 2020.
-
Fast Maximization of Non-Submodular, Monotonic Functions on the Integer Lattice
Authors:
Alan Kuhnle,
J. David Smith,
Victoria G. Crawford,
My T. Thai
Abstract:
The optimization of submodular functions on the integer lattice has received much attention recently, but the objective functions of many applications are non-submodular. We provide two approximation algorithms for maximizing a non-submodular function on the integer lattice subject to a cardinality constraint; these are the first algorithms for this purpose that have polynomial query complexity. W…
▽ More
The optimization of submodular functions on the integer lattice has received much attention recently, but the objective functions of many applications are non-submodular. We provide two approximation algorithms for maximizing a non-submodular function on the integer lattice subject to a cardinality constraint; these are the first algorithms for this purpose that have polynomial query complexity. We propose a general framework for influence maximization on the integer lattice that generalizes prior works on this topic, and we demonstrate the efficiency of our algorithms in this context.
△ Less
Submitted 17 May, 2018;
originally announced May 2018.
-
Deterministic & Adaptive Non-Submodular Maximization via the Primal Curvature
Authors:
J. David Smith,
My T. Thai
Abstract:
While greedy algorithms have long been observed to perform well on a wide variety of problems, up to now approximation ratios have only been known for their application to problems having submodular objective functions $f$. Since many practical problems have non-submodular $f$, there is a critical need to devise new techniques to bound the performance of greedy algorithms in the case of non-submod…
▽ More
While greedy algorithms have long been observed to perform well on a wide variety of problems, up to now approximation ratios have only been known for their application to problems having submodular objective functions $f$. Since many practical problems have non-submodular $f$, there is a critical need to devise new techniques to bound the performance of greedy algorithms in the case of non-submodularity.
Our primary contribution is the introduction of a novel technique for estimating the approximation ratio of the greedy algorithm for maximization of monotone non-decreasing functions based on the curvature of $f$ without relying on the submodularity constraint. We show that this technique reduces to the classical $(1 - 1/e)$ ratio for submodular functions. Furthermore, we develop an extension of this ratio to the adaptive greedy algorithm, which allows applications to non-submodular stochastic maximization problems. This notably extends support to applications modeling incomplete data with uncertainty.
△ Less
Submitted 12 January, 2018; v1 submitted 22 February, 2017;
originally announced February 2017.
-
TipTop: (Almost) Exact Solutions for Influence Maximization in Billion-scale Networks
Authors:
Xiang Li,
J. David Smith,
Thang N. Dinh,
My T. Thai
Abstract:
In this paper, we study the Cost-aware Target Viral Marketing (CTVM) problem, a generalization of Influence Maximization (IM). CTVM asks for the most cost-effective users to influence the most relevant users. In contrast to the vast literature, we attempt to offer exact solutions. As the problem is NP-hard, thus, exact solutions are intractable, we propose TipTop, a $(1-ε)$-optimal solution for ar…
▽ More
In this paper, we study the Cost-aware Target Viral Marketing (CTVM) problem, a generalization of Influence Maximization (IM). CTVM asks for the most cost-effective users to influence the most relevant users. In contrast to the vast literature, we attempt to offer exact solutions. As the problem is NP-hard, thus, exact solutions are intractable, we propose TipTop, a $(1-ε)$-optimal solution for arbitrary $ε>0$ that scales to very large networks such as Twitter. At the heart of TipTop lies an innovative technique that reduces the number of samples as much as possible. This allows us to exactly solve CTVM on a much smaller space of generated samples using Integer Programming. Furthermore, TipTop lends a tool for researchers to benchmark their solutions against the optimal one in large-scale networks, which is currently not available.
△ Less
Submitted 7 February, 2019; v1 submitted 29 January, 2017;
originally announced January 2017.
