-
Efficient Deep Reinforcement Learning with Predictive Processing Proximal Policy Optimization
Authors:
Burcu Küçükoğlu,
Walraaf Borkent,
Bodo Rueckauer,
Nasir Ahmad,
Umut Güçlü,
Marcel van Gerven
Abstract:
Advances in reinforcement learning (RL) often rely on massive compute resources and remain notoriously sample inefficient. In contrast, the human brain is able to efficiently learn effective control strategies using limited resources. This raises the question whether insights from neuroscience can be used to improve current RL methods. Predictive processing is a popular theoretical framework which…
▽ More
Advances in reinforcement learning (RL) often rely on massive compute resources and remain notoriously sample inefficient. In contrast, the human brain is able to efficiently learn effective control strategies using limited resources. This raises the question whether insights from neuroscience can be used to improve current RL methods. Predictive processing is a popular theoretical framework which maintains that the human brain is actively seeking to minimize surprise. We show that recurrent neural networks which predict their own sensory states can be leveraged to minimise surprise, yielding substantial gains in cumulative reward. Specifically, we present the Predictive Processing Proximal Policy Optimization (P4O) agent; an actor-critic reinforcement learning agent that applies predictive processing to a recurrent variant of the PPO algorithm by integrating a world model in its hidden state. Even without hyperparameter tuning, P4O significantly outperforms a baseline recurrent variant of the PPO algorithm on multiple Atari games using a single GPU. It also outperforms other state-of-the-art agents given the same wall-clock time and exceeds human gamer performance on multiple games including Seaquest, which is a particularly challenging environment in the Atari domain. Altogether, our work underscores how insights from the field of neuroscience may support the development of more capable and efficient artificial agents.
△ Less
Submitted 29 January, 2024; v1 submitted 11 November, 2022;
originally announced November 2022.
-
NxTF: An API and Compiler for Deep Spiking Neural Networks on Intel Loihi
Authors:
Bodo Rueckauer,
Connor Bybee,
Ralf Goettsche,
Yashwardhan Singh,
Joyesh Mishra,
Andreas Wild
Abstract:
Spiking Neural Networks (SNNs) are a promising paradigm for efficient event-driven processing of spatio-temporally sparse data streams. SNNs have inspired the design and can take advantage of the emerging class of neuromorphic processors like Intel Loihi. These novel hardware architectures expose a variety of constraints that affect firmware, compiler and algorithm development alike. To enable rap…
▽ More
Spiking Neural Networks (SNNs) are a promising paradigm for efficient event-driven processing of spatio-temporally sparse data streams. SNNs have inspired the design and can take advantage of the emerging class of neuromorphic processors like Intel Loihi. These novel hardware architectures expose a variety of constraints that affect firmware, compiler and algorithm development alike. To enable rapid and flexible development of SNN algorithms on Loihi, we developed NxTF: a programming interface derived from Keras and compiler optimized for map** deep convolutional SNNs to the multi-core Intel Loihi architecture. We evaluate NxTF on DNNs trained directly on spikes as well as models converted from traditional DNNs, processing both sparse event-based and dense frame-based data sets. Further, we assess the effectiveness of the compiler to distribute models across a large number of cores and to compress models by exploiting Loihi's weight sharing features. Finally, we evaluate model accuracy, energy and time to solution compared to other architectures. The compiler achieves near optimal resource utilization of 80% across 16 Loihi chips for a 28-layer, 4M parameter MobileNet model with input size 128x128. In addition, we report the lowest error rate of 8.52% for the CIFAR-10 dataset on neuromorphic hardware, using an off-the-shelf MobileNet.
△ Less
Submitted 11 January, 2021;
originally announced January 2021.
-
Closing the Accuracy Gap in an Event-Based Visual Recognition Task
Authors:
Bodo Rückauer,
Nicolas Känzig,
Shih-Chii Liu,
Tobi Delbruck,
Yulia Sandamirskaya
Abstract:
Mobile and embedded applications require neural networks-based pattern recognition systems to perform well under a tight computational budget. In contrast to commonly used synchronous, frame-based vision systems and CNNs, asynchronous, spiking neural networks driven by event-based visual input respond with low latency to sparse, salient features in the input, leading to high efficiency at run-time…
▽ More
Mobile and embedded applications require neural networks-based pattern recognition systems to perform well under a tight computational budget. In contrast to commonly used synchronous, frame-based vision systems and CNNs, asynchronous, spiking neural networks driven by event-based visual input respond with low latency to sparse, salient features in the input, leading to high efficiency at run-time. The discrete nature of the event-based data streams makes direct training of asynchronous neural networks challenging. This paper studies asynchronous spiking neural networks, obtained by conversion from a conventional CNN trained on frame-based data. As an example, we consider a CNN trained to steer a robot to follow a moving target. We identify possible pitfalls of the conversion and demonstrate how the proposed solutions bring the classification accuracy of the asynchronous network to only 3\% below the performance of the original synchronous CNN, while requiring 12x fewer computations. While being applied to a simple task, this work is an important step towards low-power, fast, and embedded neural networks-based vision solutions for robotic applications.
△ Less
Submitted 6 May, 2019;
originally announced June 2019.
-
Theory and Tools for the Conversion of Analog to Spiking Convolutional Neural Networks
Authors:
Bodo Rueckauer,
Iulia-Alexandra Lungu,
Yuhuang Hu,
Michael Pfeiffer
Abstract:
Deep convolutional neural networks (CNNs) have shown great potential for numerous real-world machine learning applications, but performing inference in large CNNs in real-time remains a challenge. We have previously demonstrated that traditional CNNs can be converted into deep spiking neural networks (SNNs), which exhibit similar accuracy while reducing both latency and computational load as a con…
▽ More
Deep convolutional neural networks (CNNs) have shown great potential for numerous real-world machine learning applications, but performing inference in large CNNs in real-time remains a challenge. We have previously demonstrated that traditional CNNs can be converted into deep spiking neural networks (SNNs), which exhibit similar accuracy while reducing both latency and computational load as a consequence of their data-driven, event-based style of computing. Here we provide a novel theory that explains why this conversion is successful, and derive from it several new tools to convert a larger and more powerful class of deep networks into SNNs. We identify the main sources of approximation errors in previous conversion methods, and propose simple mechanisms to fix these issues. Furthermore, we develop spiking implementations of common CNN operations such as max-pooling, softmax, and batch-normalization, which allow almost loss-less conversion of arbitrary CNN architectures into the spiking domain. Empirical evaluation of different network architectures on the MNIST and CIFAR10 benchmarks leads to the best SNN results reported to date.
△ Less
Submitted 13 December, 2016;
originally announced December 2016.
