Showing 1–13 of 13 results for author: Nowotny, T

Spyx: A Library for Just-In-Time Compiled Optimization of Spiking Neural Networks

Authors: Kade M. Heckel, Thomas Nowotny

Abstract: As the role of artificial intelligence becomes increasingly pivotal in modern society, the efficient training and deployment of deep neural networks have emerged as critical areas of focus. Recent advancements in attention-based large neural architectures have spurred the development of AI accelerators, facilitating the training of extensive, multi-billion parameter models. Despite their effective… ▽ More As the role of artificial intelligence becomes increasingly pivotal in modern society, the efficient training and deployment of deep neural networks have emerged as critical areas of focus. Recent advancements in attention-based large neural architectures have spurred the development of AI accelerators, facilitating the training of extensive, multi-billion parameter models. Despite their effectiveness, these powerful networks often incur high execution costs in production environments. Neuromorphic computing, inspired by biological neural processes, offers a promising alternative. By utilizing temporally-sparse computations, Spiking Neural Networks (SNNs) offer to enhance energy efficiency through a reduced and low-power hardware footprint. However, the training of SNNs can be challenging due to their recurrent nature which cannot as easily leverage the massive parallelism of modern AI accelerators. To facilitate the investigation of SNN architectures and dynamics researchers have sought to bridge Python-based deep learning frameworks such as PyTorch or TensorFlow with custom-implemented compute kernels. This paper introduces Spyx, a new and lightweight SNN simulation and optimization library designed in JAX. By pre-staging data in the expansive vRAM of contemporary accelerators and employing extensive JIT compilation, Spyx allows for SNN optimization to be executed as a unified, low-level program on NVIDIA GPUs or Google TPUs. This approach achieves optimal hardware utilization, surpassing the performance of many existing SNN training frameworks while maintaining considerable flexibility. △ Less

Submitted 29 February, 2024; originally announced February 2024.

ACM Class: I.2.5

NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systems

Authors: Jason Yik, Korneel Van den Berghe, Douwe den Blanken, Younes Bouhadjar, Maxime Fabre, Paul Hueber, Denis Kleyko, Noah Pacik-Nelson, Pao-Sheng Vincent Sun, Guangzhi Tang, Shenqi Wang, Biyan Zhou, Soikat Hasan Ahmed, George Vathakkattil Joseph, Benedetto Leto, Aurora Micheli, Anurag Kumar Mishra, Gregor Lenz, Tao Sun, Zergham Ahmed, Mahmoud Akl, Brian Anderson, Andreas G. Andreou, Chiara Bartolozzi, Arindam Basu , et al. (73 additional authors not shown)

Abstract: Neuromorphic computing shows promise for advancing computing efficiency and capabilities of AI applications using brain-inspired principles. However, the neuromorphic research field currently lacks standardized benchmarks, making it difficult to accurately measure technological advancements, compare performance with conventional methods, and identify promising future research directions. Prior neu… ▽ More Neuromorphic computing shows promise for advancing computing efficiency and capabilities of AI applications using brain-inspired principles. However, the neuromorphic research field currently lacks standardized benchmarks, making it difficult to accurately measure technological advancements, compare performance with conventional methods, and identify promising future research directions. Prior neuromorphic computing benchmark efforts have not seen widespread adoption due to a lack of inclusive, actionable, and iterative benchmark design and guidelines. To address these shortcomings, we present NeuroBench: a benchmark framework for neuromorphic computing algorithms and systems. NeuroBench is a collaboratively-designed effort from an open community of nearly 100 co-authors across over 50 institutions in industry and academia, aiming to provide a representative structure for standardizing the evaluation of neuromorphic approaches. The NeuroBench framework introduces a common set of tools and systematic methodology for inclusive benchmark measurement, delivering an objective reference framework for quantifying neuromorphic approaches in both hardware-independent (algorithm track) and hardware-dependent (system track) settings. In this article, we present initial performance baselines across various model architectures on the algorithm track and outline the system track benchmark tasks and guidelines. NeuroBench is intended to continually expand its benchmarks and features to foster and track the progress made by the research community. △ Less

Submitted 17 January, 2024; v1 submitted 10 April, 2023; originally announced April 2023.

Comments: Updated from whitepaper to full perspective article preprint

Learning efficient backprojections across cortical hierarchies in real time

Authors: Kevin Max, Laura Kriener, Garibaldi Pineda García, Thomas Nowotny, Ismael Jaras, Walter Senn, Mihai A. Petrovici

Abstract: Models of sensory processing and learning in the cortex need to efficiently assign credit to synapses in all areas. In deep learning, a known solution is error backpropagation, which however requires biologically implausible weight transport from feed-forward to feedback paths. We introduce Phaseless Alignment Learning (PAL), a bio-plausible method to learn efficient feedback weights in layered… ▽ More Models of sensory processing and learning in the cortex need to efficiently assign credit to synapses in all areas. In deep learning, a known solution is error backpropagation, which however requires biologically implausible weight transport from feed-forward to feedback paths. We introduce Phaseless Alignment Learning (PAL), a bio-plausible method to learn efficient feedback weights in layered cortical hierarchies. This is achieved by exploiting the noise naturally found in biophysical systems as an additional carrier of information. In our dynamical system, all weights are learned simultaneously with always-on plasticity and using only information locally available to the synapses. Our method is completely phase-free (no forward and backward passes or phased learning) and allows for efficient error propagation across multi-layer cortical hierarchies, while maintaining biologically plausible signal transport and learning. Our method is applicable to a wide class of models and improves on previously known biologically plausible ways of credit assignment: compared to random synaptic feedback, it can solve complex tasks with less neurons and learn more useful latent representations. We demonstrate this on various classification tasks using a cortical microcircuit model with prospective coding. △ Less

Submitted 2 February, 2024; v1 submitted 20 December, 2022; originally announced December 2022.

Comments: Updated with streamlined main part, CIFAR-10 simulations, including DFA and minor fixes

Loss sha** enhances exact gradient learning with EventProp in Spiking Neural Networks

Authors: Thomas Nowotny, James P. Turner, James C. Knight

Abstract: Event-based machine learning promises more energy-efficient AI on future neuromorphic hardware. Here, we investigate how the recently discovered Eventprop algorithm for gradient descent on exact gradients in spiking neural networks can be scaled up to challenging keyword recognition benchmarks. We implemented Eventprop in the GPU-enhanced Neural Networks framework and used it for training recurren… ▽ More Event-based machine learning promises more energy-efficient AI on future neuromorphic hardware. Here, we investigate how the recently discovered Eventprop algorithm for gradient descent on exact gradients in spiking neural networks can be scaled up to challenging keyword recognition benchmarks. We implemented Eventprop in the GPU-enhanced Neural Networks framework and used it for training recurrent spiking neural networks on the Spiking Heidelberg Digits and Spiking Speech Commands datasets. We found that learning depended strongly on the loss function and extended Eventprop to a wider class of loss functions to enable effective training. When combined with the right additional mechanisms from the machine learning toolbox, Eventprop networks achieved state-of-the-art performance on Spiking Heidelberg Digits and good accuracy on Spiking Speech Commands. This work is a significant step towards a low-power neuromorphic alternative to current machine learning paradigms. △ Less

Submitted 2 June, 2024; v1 submitted 2 December, 2022; originally announced December 2022.

Comments: 23 pages, 7 figures, 5 tables

MSC Class: 68T05 (Primary) 68T10; 68T07 (Secondary) ACM Class: I.2; I.2.6; I.5.1

Scalability and Optimization Strategies for GPU Enhanced Neural Networks (GeNN)

Authors: Naresh Balaji, Esin Yavuz, Thomas Nowotny

Abstract: Simulation of spiking neural networks has been traditionally done on high-performance supercomputers or large-scale clusters. Utilizing the parallel nature of neural network computation algorithms, GeNN (GPU Enhanced Neural Network) provides a simulation environment that performs on General Purpose NVIDIA GPUs with a code generation based approach. GeNN allows the users to design and simulate neur… ▽ More Simulation of spiking neural networks has been traditionally done on high-performance supercomputers or large-scale clusters. Utilizing the parallel nature of neural network computation algorithms, GeNN (GPU Enhanced Neural Network) provides a simulation environment that performs on General Purpose NVIDIA GPUs with a code generation based approach. GeNN allows the users to design and simulate neural networks by specifying the populations of neurons at different stages, their synapse connection densities and the model of individual neurons. In this report we describe work on how to scale synaptic weights based on the configuration of the user-defined network to ensure sufficient spiking and subsequent effective learning. We also discuss optimization strategies particular to GPU computing: sparse representation of synapse connections and occupancy based block-size determination. △ Less

Submitted 1 December, 2014; originally announced December 2014.

ICT System Design & Implementation Using Wireless Sensors to Support Elderly In-home Assistance

Authors: Thomas J. Lampoltshammer, Thomas Nowotny, Stefan Plank

Abstract: Around the globe the number of older people in relation to the rest is constantly growing. As a result, medical and care facilities cannot handle the growing number of patients. Therefore, elderly in-home assistance gets more attention an importance. Due to issues regarding memory, physical strength and reduced self-assessment, old people face a lot of challenges in accomplishing their activities… ▽ More Around the globe the number of older people in relation to the rest is constantly growing. As a result, medical and care facilities cannot handle the growing number of patients. Therefore, elderly in-home assistance gets more attention an importance. Due to issues regarding memory, physical strength and reduced self-assessment, old people face a lot of challenges in accomplishing their activities of daily living. This thesis is meant to address these problems by analysing the required infrastructure of a home-care facility as well as the arising issues regarding used components, especially wireless sensors. After the analysis, a prototype of a home-care system is designed and implemented. Furthermore, the issue of energy consumption of the used wireless sensor node is addressed by modifying the intelligence of the used sensor. After that, the design and components of the prototype used for the energy consumption analysis is explained, together with the programming structure of the sensor nodes used in this thesis. Thereupon, the results are of the simulations are discussed and compared with the authors' expectations. Finally the overall outcomes of the thesis are analysed and summed up, followed by a short outlook of further possible improvements and developments. △ Less

Submitted 2 March, 2013; originally announced March 2013.

Emergent Properties in Structurally Dynamic Disordered Cellular Networks

Authors: Thomas Nowotny, Manfred Requardt

Abstract: We relate structurally dynamic cellular networks, a class of models we developed in fundamental space-time physics, to SDCA, introduced some time ago by Ilachinski and Halpern. We emphasize the crucial property of a non-linear interaction of network geometry with the matter degrees of freedom in order to emulate the supposedly highly erratic and strongly fluctuating space-time structure on the P… ▽ More We relate structurally dynamic cellular networks, a class of models we developed in fundamental space-time physics, to SDCA, introduced some time ago by Ilachinski and Halpern. We emphasize the crucial property of a non-linear interaction of network geometry with the matter degrees of freedom in order to emulate the supposedly highly erratic and strongly fluctuating space-time structure on the Planck scale. We then embark on a detailed numerical analysis of various large scale characteristics of several classes of models in order to understand what will happen if some sort of macroscopic or continuum limit is performed. Of particular relevance in this context is a notion of network dimension and its behavior in this limit. Furthermore, the possibility of phase transitions is discussed. △ Less

Submitted 16 November, 2006; originally announced November 2006.

Comments: 20 pages, Latex, 6 figures

Journal ref: J.Cell.Automata2:273-289,2007

Spatial representation of temporal information through spike timing dependent plasticity

Authors: Thomas Nowotny, Misha I. Rabinovich, Henry D. I. Abarbanel

Abstract: We suggest a mechanism based on spike time dependent plasticity (STDP) of synapses to store, retrieve and predict temporal sequences. The mechanism is demonstrated in a model system of simplified integrate-and-fire type neurons densely connected by STDP synapses. All synapses are modified according to the so-called normal STDP rule observed in various real biological synapses. After conditioning… ▽ More We suggest a mechanism based on spike time dependent plasticity (STDP) of synapses to store, retrieve and predict temporal sequences. The mechanism is demonstrated in a model system of simplified integrate-and-fire type neurons densely connected by STDP synapses. All synapses are modified according to the so-called normal STDP rule observed in various real biological synapses. After conditioning through repeated input of a limited number of of temporal sequences the system is able to complete the temporal sequence upon receiving the input of a fraction of them. This is an example of effective unsupervised learning in an biologically realistic system. We investigate the dependence of learning success on entrainment time, system size and presence of noise. Possible applications include learning of motor sequences, recognition and prediction of temporal sensory information in the visual as well as the auditory system and late processing in the olfactory system of insects. △ Less

Submitted 14 August, 2003; v1 submitted 4 September, 2002; originally announced September 2002.

Comments: 13 pages, 14 figures, completely revised and augmented version

Journal ref: Phys. Rev. E 68 (2003) 011908

Phase diagram of the random field Ising model on the Bethe lattice

Authors: Thomas Nowotny, Heiko Patzlaff, Ulrich Behn

Abstract: The phase diagram of the random field Ising model on the Bethe lattice with a symmetric dichotomous random field is closely investigated with respect to the transition between the ferromagnetic and paramagnetic regime. Refining arguments of Bleher, Ruiz and Zagrebnov [J. Stat. Phys. 93, 33 (1998)] an exact upper bound for the existence of a unique paramagnetic phase is found which considerably i… ▽ More The phase diagram of the random field Ising model on the Bethe lattice with a symmetric dichotomous random field is closely investigated with respect to the transition between the ferromagnetic and paramagnetic regime. Refining arguments of Bleher, Ruiz and Zagrebnov [J. Stat. Phys. 93, 33 (1998)] an exact upper bound for the existence of a unique paramagnetic phase is found which considerably improves the earlier results. Several numerical estimates of transition lines between a ferromagnetic and a paramagnetic regime are presented. The obtained results do not coincide with a lower bound for the onset of ferromagnetism proposed by Bruinsma [Phys. Rev. B 30, 289 (1984)]. If the latter one proves correct this would hint to a region of coexistence of stable ferromagnetic phases and a stable paramagnetic phase. △ Less

Submitted 24 September, 2001; v1 submitted 5 June, 2001; originally announced June 2001.

Comments: Article has been condensed and reorganized; Figs 3,5,6 merged; Fig 4 omitted; Some discussion added at end of Sec. III; 9 pages, 5 figs, RevTeX4, AMSTeX

Report number: LU-ITP 2001/014

Convolution of multifractals and the local magnetization in a random field Ising chain

Authors: Thomas Nowotny, Ulrich Behn

Abstract: The local magnetization in the one-dimensional random-field Ising model is essentially the sum of two effective fields with multifractal probability measure. The probability measure of the local magnetization is thus the convolution of two multifractals. In this paper we prove relations between the multifractal properties of two measures and the multifractal properties of their convolution. The… ▽ More The local magnetization in the one-dimensional random-field Ising model is essentially the sum of two effective fields with multifractal probability measure. The probability measure of the local magnetization is thus the convolution of two multifractals. In this paper we prove relations between the multifractal properties of two measures and the multifractal properties of their convolution. The pointwise dimension at the boundary of the support of the convolution is the sum of the pointwise dimensions at the boundary of the support of the convoluted measures and the generalized box dimensions of the convolution are bounded from above by the sum of the generalized box dimensions of the convoluted measures. The generalized box dimensions of the convolution of Cantor sets with weights can be calculated analytically for certain parameter ranges and illustrate effects we also encounter in the case of the measure of the local magnetization. Returning to the study of this measure we apply the general inequalities and present numerical approximations of the D_q-spectrum. For the first time we are able to obtain results on multifractal properties of a physical quantity in the one-dimensional random-field Ising model which in principle could be measured experimentally. The numerically generated probability densities for the local magnetization show impressively the gradual transition from a monomodal to a bimodal distribution for growing random field strength h. △ Less

Submitted 3 September, 2001; v1 submitted 19 February, 2001; originally announced February 2001.

Comments: An error in figure 1 was corrected, small additions were made to the introduction and the conclusions, some typos were corrected, 24 pages, LaTeX2e, 9 figures

Report number: NTZ 3/2001

Orbits and phase transitions in the multifractal spectrum

Authors: Thomas Nowotny, Heiko Patzlaff, Ulrich Behn

Abstract: We consider the one dimensional classical Ising model in a symmetric dichotomous random field. The problem is reduced to a random iterated function system for an effective field. The D_q-spectrum of the invariant measure of this effective field exhibits a sharp drop of all D_q with q < 0 at some critical strength of the random field. We introduce the concept of orbits which naturally group the p… ▽ More We consider the one dimensional classical Ising model in a symmetric dichotomous random field. The problem is reduced to a random iterated function system for an effective field. The D_q-spectrum of the invariant measure of this effective field exhibits a sharp drop of all D_q with q < 0 at some critical strength of the random field. We introduce the concept of orbits which naturally group the points of the support of the invariant measure. We then show that the pointwise dimension at all points of an orbit has the same value and calculate it for a class of periodic orbits and their so-called offshoots as well as for generic orbits in the non-overlap** case. The sharp drop in the D_q-spectrum is analytically explained by a drastic change of the scaling properties of the measure near the points of a certain periodic orbit at a critical strength of the random field which is explicitly given. A similar drastic change near the points of a special family of periodic orbits explains a second, hitherto unnoticed transition in the D_q-spectrum. As it turns out, a decisive role in this mechanism is played by a specific offshoot. We furthermore give rigorous upper and/or lower bounds on all D_q in a wide parameter range. In most cases the numerically obtained D_q coincide with either the upper or the lower bound. The results in this paper are relevant for the understanding of random iterated function systems in the case of moderate overlap in which periodic orbits with weak singularity can play a decisive role. △ Less

Submitted 21 November, 2000; v1 submitted 12 May, 1999; originally announced May 1999.

Comments: The article has been completely rewritten; the title has changed; a section about the typical pointwise dimension as well as several references and remarks about more general systems have been added; to appear in J. Phys. A; 25 pages, 11 figures, LaTeX2e

Journal ref: J. Phys. A 34 (2001) 1-23

Pregeometric Concepts on Graphs and Cellular Networks as Possible Models of Space-Time at the Planck-Scale

Authors: Thomas Nowotny, Manfred Requardt

Abstract: Starting from the working hypothesis that both physics and the corresponding mathematics have to be described by means of discrete concepts on the Planck-scale, one of the many problems one has to face is to find the discrete protoforms of the building blocks of continuum physics and mathematics. In the following we embark on develo** such concepts for irregular structures like (large) graphs… ▽ More Starting from the working hypothesis that both physics and the corresponding mathematics have to be described by means of discrete concepts on the Planck-scale, one of the many problems one has to face is to find the discrete protoforms of the building blocks of continuum physics and mathematics. In the following we embark on develo** such concepts for irregular structures like (large) graphs or networks which are intended to emulate (some of) the generic properties of the presumed combinatorial substratum from which continuum physics is assumed to emerge as a coarse grained and secondary model theory. We briefly indicate how various concepts of discrete (functional) analysis and geometry can be naturally constructed within this framework, leaving a larger portion of the paper to the systematic developement of dimensional concepts and their properties, which may have a possible bearing on various branches of modern physics beyond quantum gravity. △ Less

Submitted 29 January, 1998; originally announced January 1998.

Comments: 16 pages, Invited paper to appear in the special issue of the Journal of Chaos, Solitons and Fractals on: "Superstrings, M, F, S ... Theory" (M.S. El Naschie, C. Castro, Editors)

Journal ref: Chaos Solitons Fractals 10 (1999) 469

Dimension Theory of Graphs and Networks

Authors: Thomas Nowotny, Manfred Requardt

Abstract: Starting from the working hypothesis that both physics and the corresponding mathematics have to be described by means of discrete concepts on the Planck-scale, one of the many problems one has to face in this enterprise is to find the discrete protoforms of the building blocks of continuum physics and mathematics. A core concept is the notion of dimension. In the following we develop such a not… ▽ More Starting from the working hypothesis that both physics and the corresponding mathematics have to be described by means of discrete concepts on the Planck-scale, one of the many problems one has to face in this enterprise is to find the discrete protoforms of the building blocks of continuum physics and mathematics. A core concept is the notion of dimension. In the following we develop such a notion for irregular structures like (large) graphs and networks and derive a number of its properties. Among other things we show its stability under a wide class of perturbations which is important if one has 'dimensional phase transitions' in mind. Furthermore we systematically construct graphs with almost arbitrary 'fractal dimension' which may be of some use in the context of 'dimensional renormalization' or statistical mechanics on irregular sets. △ Less

Submitted 9 July, 1997; originally announced July 1997.

Comments: 20 pages, 7 figures, LaTex2e, uses amsmath, amsfonts, amssymb, latexsym, epsfig

Journal ref: J.Phys.A31:2447-2463,1998