Showing 1–17 of 17 results for author: Severa, W

Neuromorphic Co-Design as a Game

Authors: Craig M. Vineyard, William M. Severa, James B. Aimone

Abstract: Co-design is a prominent topic presently in computing, speaking to the mutual benefit of coordinating design choices of several layers in the technology stack. For example, this may be designing algorithms which can most efficiently take advantage of the acceleration properties of a given architecture, while simultaneously designing the hardware to support the structural needs of a class of comput… ▽ More Co-design is a prominent topic presently in computing, speaking to the mutual benefit of coordinating design choices of several layers in the technology stack. For example, this may be designing algorithms which can most efficiently take advantage of the acceleration properties of a given architecture, while simultaneously designing the hardware to support the structural needs of a class of computation. The implications of these design decisions are influential enough to be deemed a lottery, enabling an idea to win out over others irrespective of the individual merits. Coordination is a well studied topic in the mathematics of game theory, where in many cases without a coordination mechanism the outcome is sub-optimal. Here we consider what insights game theoretic analysis can offer for computer architecture co-design. In particular, we consider the interplay between algorithm and architecture advances in the field of neuromorphic computing. Analyzing developments of spiking neural network algorithms and neuromorphic hardware as a co-design game we use the Stag Hunt model to illustrate challenges for spiking algorithms or architectures to advance the field independently and advocate for a strategic pursuit to advance neuromorphic computing. △ Less

Submitted 11 December, 2023; originally announced December 2023.

Comments: 8 pages, 2 figures, accepted to First Workshop on Machine Learning with New Compute Paradigms at NeurIPS 2023

Report number: SAND2023-14477C

Synaptic Sampling of Neural Networks

Authors: James B. Aimone, William Severa, J. Darby Smith

Abstract: Probabilistic artificial neural networks offer intriguing prospects for enabling the uncertainty of artificial intelligence methods to be described explicitly in their function; however, the development of techniques that quantify uncertainty by well-understood methods such as Monte Carlo sampling has been limited by the high costs of stochastic sampling on deterministic computing hardware. Emergi… ▽ More Probabilistic artificial neural networks offer intriguing prospects for enabling the uncertainty of artificial intelligence methods to be described explicitly in their function; however, the development of techniques that quantify uncertainty by well-understood methods such as Monte Carlo sampling has been limited by the high costs of stochastic sampling on deterministic computing hardware. Emerging computing systems that are amenable to hardware-level probabilistic computing, such as those that leverage stochastic devices, may make probabilistic neural networks more feasible in the not-too-distant future. This paper describes the scANN technique -- \textit{sampling (by coinflips) artificial neural networks} -- which enables neural networks to be sampled directly by treating the weights as Bernoulli coin flips. This method is natively well suited for probabilistic computing techniques that focus on tunable stochastic devices, nearly matches fully deterministic performance while also describing the uncertainty of correct and incorrect neural network outputs. △ Less

Submitted 21 November, 2023; originally announced November 2023.

Comments: 9 pages, accepted to 2023 IEEE International Conference on Rebooting Computing

Report number: SAND2023-13757C

Induced Distributions from Generalized Unfair Dice

Authors: Douglas T. Pfeffer, J. Darby Smith, William Severa

Abstract: In this paper we analyze the probability distributions associated with rolling (possibly unfair) dice infinitely often. Specifically, given a $q$-sided die, if $x_i\in\{0,\ldots,q-1\}$ denotes the outcome of the $i^{\text{th}}$ toss, then the distribution function is $F(x)=\mathbb{P}[X\leq x]$, where $X = \sum_{i=1}^\infty x_i q^{-i}$. We show that $F$ is singular and establish a piecewise linear,… ▽ More In this paper we analyze the probability distributions associated with rolling (possibly unfair) dice infinitely often. Specifically, given a $q$-sided die, if $x_i\in\{0,\ldots,q-1\}$ denotes the outcome of the $i^{\text{th}}$ toss, then the distribution function is $F(x)=\mathbb{P}[X\leq x]$, where $X = \sum_{i=1}^\infty x_i q^{-i}$. We show that $F$ is singular and establish a piecewise linear, iterative construction for it. We investigate two ways of comparing $F$ to the fair distribution -- one using supremum norms and another using arclength. In the case of coin flips, we also address the case where each independent flip could come from a different distribution. In part, this work aims to address outstanding claims in the literature on Bernoulli schemes. The results herein are motivated by emerging needs, desires, and opportunities in computation to leverage physical stochasticity in microelectronic devices for random number generation. △ Less

Submitted 19 September, 2023; v1 submitted 13 September, 2023; originally announced September 2023.

Comments: 18 pages, 1 figure

MSC Class: 60E05

Stochastic Neuromorphic Circuits for Solving MAXCUT

Authors: Bradley H. Theilman, Yipu Wang, Ojas D. Parekh, William Severa, J. Darby Smith, James B. Aimone

Abstract: Finding the maximum cut of a graph (MAXCUT) is a classic optimization problem that has motivated parallel algorithm development. While approximate algorithms to MAXCUT offer attractive theoretical guarantees and demonstrate compelling empirical performance, such approximation approaches can shift the dominant computational cost to the stochastic sampling operations. Neuromorphic computing, which u… ▽ More Finding the maximum cut of a graph (MAXCUT) is a classic optimization problem that has motivated parallel algorithm development. While approximate algorithms to MAXCUT offer attractive theoretical guarantees and demonstrate compelling empirical performance, such approximation approaches can shift the dominant computational cost to the stochastic sampling operations. Neuromorphic computing, which uses the organizing principles of the nervous system to inspire new parallel computing architectures, offers a possible solution. One ubiquitous feature of natural brains is stochasticity: the individual elements of biological neural networks possess an intrinsic randomness that serves as a resource enabling their unique computational capacities. By designing circuits and algorithms that make use of randomness similarly to natural brains, we hypothesize that the intrinsic randomness in microelectronics devices could be turned into a valuable component of a neuromorphic architecture enabling more efficient computations. Here, we present neuromorphic circuits that transform the stochastic behavior of a pool of random devices into useful correlations that drive stochastic solutions to MAXCUT. We show that these circuits perform favorably in comparison to software solvers and argue that this neuromorphic hardware implementation provides a path for scaling advantages. This work demonstrates the utility of combining neuromorphic principles with intrinsic randomness as a computational resource for new computational architectures. △ Less

Submitted 5 October, 2022; originally announced October 2022.

Report number: SAND2022-13654 O

Spiking Neural Streaming Binary Arithmetic

Authors: James B. Aimone, Aaron J. Hill, William M. Severa, Craig M. Vineyard

Abstract: Boolean functions and binary arithmetic operations are central to standard computing paradigms. Accordingly, many advances in computing have focused upon how to make these operations more efficient as well as exploring what they can compute. To best leverage the advantages of novel computing paradigms it is important to consider what unique computing approaches they offer. However, for any special… ▽ More Boolean functions and binary arithmetic operations are central to standard computing paradigms. Accordingly, many advances in computing have focused upon how to make these operations more efficient as well as exploring what they can compute. To best leverage the advantages of novel computing paradigms it is important to consider what unique computing approaches they offer. However, for any special-purpose co-processor, Boolean functions and binary arithmetic operations are useful for, among other things, avoiding unnecessary I/O on-and-off the co-processor by pre- and post-processing data on-device. This is especially true for spiking neuromorphic architectures where these basic operations are not fundamental low-level operations. Instead, these functions require specific implementation. Here we discuss the implications of an advantageous streaming binary encoding method as well as a handful of circuits designed to exactly compute elementary Boolean and binary operations. △ Less

Submitted 23 March, 2022; originally announced March 2022.

Comments: Accepted and presented at the 2021 International Conference on Rebooting Computing (ICRC)

Report number: SAND2021-13472 C

Neuromorphic scaling advantages for energy-efficient random walk computation

Authors: J. Darby Smith, Aaron J. Hill, Leah E. Reeder, Brian C. Franke, Richard B. Lehoucq, Ojas Parekh, William Severa, James B. Aimone

Abstract: Computing stands to be radically improved by neuromorphic computing (NMC) approaches inspired by the brain's incredible efficiency and capabilities. Most NMC research, which aims to replicate the brain's computational structure and architecture in man-made hardware, has focused on artificial intelligence; however, less explored is whether this brain-inspired hardware can provide value beyond cogni… ▽ More Computing stands to be radically improved by neuromorphic computing (NMC) approaches inspired by the brain's incredible efficiency and capabilities. Most NMC research, which aims to replicate the brain's computational structure and architecture in man-made hardware, has focused on artificial intelligence; however, less explored is whether this brain-inspired hardware can provide value beyond cognitive tasks. We demonstrate that high-degree parallelism and configurability of spiking neuromorphic architectures makes them well-suited to implement random walks via discrete time Markov chains. Such random walks are useful in Monte Carlo methods, which represent a fundamental computational tool for solving a wide range of numerical computing tasks. Additionally, we show how the mathematical basis for a probabilistic solution involving a class of stochastic differential equations can leverage those simulations to provide solutions for a range of broadly applicable computational tasks. Despite being in an early development stage, we find that NMC platforms, at a sufficient scale, can drastically reduce the energy demands of high-performance computing (HPC) platforms. △ Less

Submitted 27 July, 2021; originally announced July 2021.

Comments: Paper, figures, supplement

Report number: SAND2021-9085 O

Journal ref: Nature Electronics 2022

Solving a steady-state PDE using spiking networks and neuromorphic hardware

Authors: J. Darby Smith, William Severa, Aaron J. Hill, Leah Reeder, Brian Franke, Richard B. Lehoucq, Ojas D. Parekh, James B. Aimone

Abstract: The widely parallel, spiking neural networks of neuromorphic processors can enable computationally powerful formulations. While recent interest has focused on primarily machine learning tasks, the space of appropriate applications is wide and continually expanding. Here, we leverage the parallel and event-driven structure to solve a steady state heat equation using a random walk method. The random… ▽ More The widely parallel, spiking neural networks of neuromorphic processors can enable computationally powerful formulations. While recent interest has focused on primarily machine learning tasks, the space of appropriate applications is wide and continually expanding. Here, we leverage the parallel and event-driven structure to solve a steady state heat equation using a random walk method. The random walk can be executed fully within a spiking neural network using stochastic neuron behavior, and we provide results from both IBM TrueNorth and Intel Loihi implementations. Additionally, we position this algorithm as a potential scalable benchmark for neuromorphic systems. △ Less

Submitted 21 May, 2020; originally announced May 2020.

Comments: Submitted to 2020 International Conference on Neuromorphic Systems (2020 ICONS)

Report number: SAND2020-5296 O

Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment

Authors: Maryam Parsa, Catherine D. Schuman, Prasanna Date, Derek C. Rose, Bill Kay, J. Parker Mitchell, Steven R. Young, Ryan Dellana, William Severa, Thomas E. Potok, Kaushik Roy

Abstract: Training neural networks for neuromorphic deployment is non-trivial. There have been a variety of approaches proposed to adapt back-propagation or back-propagation-like algorithms appropriate for training. Considering that these networks often have very different performance characteristics than traditional neural networks, it is often unclear how to set either the network topology or the hyperpar… ▽ More Training neural networks for neuromorphic deployment is non-trivial. There have been a variety of approaches proposed to adapt back-propagation or back-propagation-like algorithms appropriate for training. Considering that these networks often have very different performance characteristics than traditional neural networks, it is often unclear how to set either the network topology or the hyperparameters to achieve optimal performance. In this work, we introduce a Bayesian approach for optimizing the hyperparameters of an algorithm for training binary communication networks that can be deployed to neuromorphic hardware. We show that by optimizing the hyperparameters on this algorithm for each dataset, we can achieve improvements in accuracy over the previous state-of-the-art for this algorithm on each dataset (by up to 15 percent). This jump in performance continues to emphasize the potential when converting traditional neural networks to binary communication applicable to neuromorphic hardware. △ Less

Submitted 20 April, 2020; originally announced May 2020.

Comments: 9 pages, 3 figures, To appear in WCCI 2020

Evaluating complexity and resilience trade-offs in emerging memory inference machines

Authors: Christopher H. Bennett, Ryan Dellana, T. Patrick Xiao, Ben Feinberg, Sapan Agarwal, Suma Cardwell, Matthew J. Marinella, William Severa, Brad Aimone

Abstract: Neuromorphic-style inference only works well if limited hardware resources are maximized properly, e.g. accuracy continues to scale with parameters and complexity in the face of potential disturbance. In this work, we use realistic crossbar simulations to highlight that compact implementations of deep neural networks are unexpectedly susceptible to collapse from multiple system disturbances. Our w… ▽ More Neuromorphic-style inference only works well if limited hardware resources are maximized properly, e.g. accuracy continues to scale with parameters and complexity in the face of potential disturbance. In this work, we use realistic crossbar simulations to highlight that compact implementations of deep neural networks are unexpectedly susceptible to collapse from multiple system disturbances. Our work proposes a middle path towards high performance and strong resilience utilizing the Mosaics framework, and specifically by re-using synaptic connections in a recurrent neural network implementation that possesses a natural form of noise-immunity. △ Less

Submitted 25 February, 2020; originally announced March 2020.

Composing Neural Algorithms with Fugu

Authors: James B Aimone, William Severa, Craig M Vineyard

Abstract: Neuromorphic hardware architectures represent a growing family of potential post-Moore's Law Era platforms. Largely due to event-driving processing inspired by the human brain, these computer platforms can offer significant energy benefits compared to traditional von Neumann processors. Unfortunately there still remains considerable difficulty in successfully programming, configuring and deploying… ▽ More Neuromorphic hardware architectures represent a growing family of potential post-Moore's Law Era platforms. Largely due to event-driving processing inspired by the human brain, these computer platforms can offer significant energy benefits compared to traditional von Neumann processors. Unfortunately there still remains considerable difficulty in successfully programming, configuring and deploying neuromorphic systems. We present the Fugu framework as an answer to this need. Rather than necessitating a developer attain intricate knowledge of how to program and exploit spiking neural dynamics to utilize the potential benefits of neuromorphic computing, Fugu is designed to provide a higher level abstraction as a hardware-independent mechanism for linking a variety of scalable spiking neural algorithms from a variety of sources. Individual kernels linked together provide sophisticated processing through compositionality. Fugu is intended to be suitable for a wide-range of neuromorphic applications, including machine learning, scientific computing, and more brain-inspired neural algorithms. Ultimately, we hope the community adopts this and other open standardization attempts allowing for free exchange and easy implementations of the ever-growing list of spiking neural algorithms. △ Less

Submitted 28 May, 2019; originally announced May 2019.

Comments: Accepted to 2019 International Conference on Neuromorphic Systems (ICONS)

Making BREAD: Biomimetic strategies for Artificial Intelligence Now and in the Future

Authors: Jeffrey L. Krichmar, William Severa, Salar M. Khan, James L. Olds

Abstract: The Artificial Intelligence (AI) revolution foretold of during the 1960s is well underway in the second decade of the 21st century. Its period of phenomenal growth likely lies ahead. Still, we believe, there are crucial lessons that biology can offer that will enable a prosperous future for AI. For machines in general, and for AI's especially, operating over extended periods or in extreme environm… ▽ More The Artificial Intelligence (AI) revolution foretold of during the 1960s is well underway in the second decade of the 21st century. Its period of phenomenal growth likely lies ahead. Still, we believe, there are crucial lessons that biology can offer that will enable a prosperous future for AI. For machines in general, and for AI's especially, operating over extended periods or in extreme environments will require energy usage orders of magnitudes more efficient than exists today. In many operational environments, energy sources will be constrained. Any plans for AI devices operating in a challenging environment must begin with the question of how they are powered, where fuel is located, how energy is stored and made available to the machine, and how long the machine can operate on specific energy units. Hence, the materials and technologies that provide the needed energy represent a critical challenge towards future use-scenarios of AI and should be integrated into their design. Here we make four recommendations for stakeholders and especially decision makers to facilitate a successful trajectory for this technology. First, that scientific societies and governments coordinate Biomimetic Research for Energy-efficient, AI Designs (BREAD); a multinational initiative and a funding strategy for investments in the future integrated design of energetics into AI. Second, that biomimetic energetic solutions be central to design consideration for future AI. Third, that a pre-competitive space be organized between stakeholder partners and fourth, that a trainee pipeline be established to ensure the human capital required for success in this area. △ Less

Submitted 3 December, 2018; originally announced December 2018.

Whetstone: A Method for Training Deep Artificial Neural Networks for Binary Communication

Authors: William Severa, Craig M. Vineyard, Ryan Dellana, Stephen J. Verzi, James B. Aimone

Abstract: This paper presents a new technique for training networks for low-precision communication. Targeting minimal communication between nodes not only enables the use of emerging spiking neuromorphic platforms, but may additionally streamline processing conventionally. Low-power and embedded neuromorphic processors potentially offer dramatic performance-per-Watt improvements over traditional von Neuman… ▽ More This paper presents a new technique for training networks for low-precision communication. Targeting minimal communication between nodes not only enables the use of emerging spiking neuromorphic platforms, but may additionally streamline processing conventionally. Low-power and embedded neuromorphic processors potentially offer dramatic performance-per-Watt improvements over traditional von Neumann processors, however programming these brain-inspired platforms generally requires platform-specific expertise which limits their applicability. To date, the majority of artificial neural networks have not operated using discrete spike-like communication. We present a method for training deep spiking neural networks using an iterative modification of the backpropagation optimization algorithm. This method, which we call Whetstone, effectively and reliably configures a network for a spiking hardware target with little, if any, loss in performance. Whetstone networks use single time step binary communication and do not require a rate code or other spike-based coding scheme, thus producing networks comparable in timing and size to conventional ANNs, albeit with binarized communication. We demonstrate Whetstone on a number of image classification networks, describing how the sharpening process interacts with different training optimizers and changes the distribution of activity within the network. We further note that Whetstone is compatible with several non-classification neural network applications, such as autoencoders and semantic segmentation. Whetstone is widely extendable and currently implemented using custom activation functions within the Keras wrapper to the popular TensorFlow machine learning framework. △ Less

Submitted 26 October, 2018; originally announced October 2018.

Spiking Neural Algorithms for Markov Process Random Walk

Authors: William Severa, Rich Lehoucq, Ojas Parekh, James B. Aimone

Abstract: The random walk is a fundamental stochastic process that underlies many numerical tasks in scientific computing applications. We consider here two neural algorithms that can be used to efficiently implement random walks on spiking neuromorphic hardware. The first method tracks the positions of individual walkers independently by using a modular code inspired by the grid cell spatial representation… ▽ More The random walk is a fundamental stochastic process that underlies many numerical tasks in scientific computing applications. We consider here two neural algorithms that can be used to efficiently implement random walks on spiking neuromorphic hardware. The first method tracks the positions of individual walkers independently by using a modular code inspired by the grid cell spatial representation in the brain. The second method tracks the densities of random walkers at each spatial location directly. We analyze the scaling complexity of each of these methods and illustrate their ability to model random walkers under different probabilistic conditions. △ Less

Submitted 1 May, 2018; originally announced May 2018.

Context-modulation of hippocampal dynamics and deep convolutional networks

Authors: James B. Aimone, William M. Severa

Abstract: Complex architectures of biological neural circuits, such as parallel processing pathways, has been behaviorally implicated in many cognitive studies. However, the theoretical consequences of circuit complexity on neural computation have only been explored in limited cases. Here, we introduce a mechanism by which direct and indirect pathways from cortex to the CA3 region of the hippocampus can bal… ▽ More Complex architectures of biological neural circuits, such as parallel processing pathways, has been behaviorally implicated in many cognitive studies. However, the theoretical consequences of circuit complexity on neural computation have only been explored in limited cases. Here, we introduce a mechanism by which direct and indirect pathways from cortex to the CA3 region of the hippocampus can balance both contextual gating of memory formation and driving network activity. We implement this concept in a deep artificial neural network by enabling a context-sensitive bias. The motivation for this is to improve performance of a size-constrained network. Using direct knowledge of the superclass information in the CIFAR-100 and Fashion-MNIST datasets, we show a dramatic increase in performance without an increase in network size. △ Less

Submitted 27 November, 2017; originally announced November 2017.

Comments: 4 pages; short paper accepted to 2017 NIPS Cognitively Informed AI Workshop

Data-driven Feature Sampling for Deep Hyperspectral Classification and Segmentation

Authors: William M. Severa, Jerilyn A. Timlin, Suraj Kholwadwala, Conrad D. James, James B. Aimone

Abstract: The high dimensionality of hyperspectral imaging forces unique challenges in scope, size and processing requirements. Motivated by the potential for an in-the-field cell sorting detector, we examine a $\textit{Synechocystis sp.}$ PCC 6803 dataset wherein cells are grown alternatively in nitrogen rich or deplete cultures. We use deep learning techniques to both successfully classify cells and gener… ▽ More The high dimensionality of hyperspectral imaging forces unique challenges in scope, size and processing requirements. Motivated by the potential for an in-the-field cell sorting detector, we examine a $\textit{Synechocystis sp.}$ PCC 6803 dataset wherein cells are grown alternatively in nitrogen rich or deplete cultures. We use deep learning techniques to both successfully classify cells and generate a mask segmenting the cells/condition from the background. Further, we use the classification accuracy to guide a data-driven, iterative feature selection method, allowing the design neural networks requiring 90% fewer input features with little accuracy degradation. △ Less

Submitted 26 October, 2017; originally announced October 2017.

MSC Class: 68T01 ACM Class: I.2.1; I.4.6; J.3

Neurogenesis Deep Learning

Authors: Timothy J. Draelos, Nadine E. Miner, Christopher C. Lamb, Jonathan A. Cox, Craig M. Vineyard, Kristofor D. Carlson, William M. Severa, Conrad D. James, James B. Aimone

Abstract: Neural machine learning methods, such as deep neural networks (DNN), have achieved remarkable success in a number of complex data processing tasks. These methods have arguably had their strongest impact on tasks such as image and audio processing - data processing domains in which humans have long held clear advantages over conventional algorithms. In contrast to biological neural systems, which a… ▽ More Neural machine learning methods, such as deep neural networks (DNN), have achieved remarkable success in a number of complex data processing tasks. These methods have arguably had their strongest impact on tasks such as image and audio processing - data processing domains in which humans have long held clear advantages over conventional algorithms. In contrast to biological neural systems, which are capable of learning continuously, deep artificial networks have a limited ability for incorporating new information in an already trained network. As a result, methods for continuous learning are potentially highly impactful in enabling the application of deep networks to dynamic data sets. Here, inspired by the process of adult neurogenesis in the hippocampus, we explore the potential for adding new neurons to deep layers of artificial neural networks in order to facilitate their acquisition of novel information while preserving previously trained data representations. Our results on the MNIST handwritten digit dataset and the NIST SD 19 dataset, which includes lower and upper case letters and digits, demonstrate that neurogenesis is well suited for addressing the stability-plasticity dilemma that has long challenged adaptive machine learning algorithms. △ Less

Submitted 28 March, 2017; v1 submitted 12 December, 2016; originally announced December 2016.

Comments: 8 pages, 8 figures, Accepted to 2017 International Joint Conference on Neural Networks (IJCNN 2017)

Report number: SAND2017-2174 C

Geometric representation of the infimax S-adic family

Authors: Philip Boyland, William Severa

Abstract: We construct geometric realizations for the infimax family of substitutions by generalizing the Rauzy-Canterini-Siegel method for a single substitution to the S-adic case. The composition of each countably infinite subcollection of substitutions from the family has an asymptotic fixed sequence whose shift orbit closure is an infimax minimal set $Δ^+$. The subcollection of substitutions also genera… ▽ More We construct geometric realizations for the infimax family of substitutions by generalizing the Rauzy-Canterini-Siegel method for a single substitution to the S-adic case. The composition of each countably infinite subcollection of substitutions from the family has an asymptotic fixed sequence whose shift orbit closure is an infimax minimal set $Δ^+$. The subcollection of substitutions also generates an infinite Bratteli-Vershik diagram with prefix-suffix labeled edges. Paths in the diagram give the Dumont-Thomas expansion of sequences in $Δ^+$ which in turn gives a projection onto the asymptotic stable direction of the infinite product of the Abelianization matrices. The projections of all sequences from $Δ^+$ is the generalized Rauzy fractal which has subpieces corresponding to the images of symbolic cylinder sets. The intervals containing these subpieces are shown to be disjoint except at endpoints, and thus the induced map derived from the symbolic shift translates them. Therefore the process yields an Interval Translation Map, and the Rauzy fractal is proved to be its attractor. △ Less

Submitted 19 June, 2017; v1 submitted 31 March, 2016; originally announced March 2016.

Comments: Revised version published in Fundamenta Mathematicae

MSC Class: 37B10; 37E05