-
Efficient Neural Compression with Inference-time Decoding
Authors:
C. Metz,
O. Bichler,
A. Dupret
Abstract:
This paper explores the combination of neural network quantization and entropy coding for memory footprint minimization. Edge deployment of quantized models is hampered by the harsh Pareto frontier of the accuracy-to-bitwidth tradeoff, causing dramatic accuracy loss below a certain bitwidth. This accuracy loss can be alleviated thanks to mixed precision quantization, allowing for more flexible bit…
▽ More
This paper explores the combination of neural network quantization and entropy coding for memory footprint minimization. Edge deployment of quantized models is hampered by the harsh Pareto frontier of the accuracy-to-bitwidth tradeoff, causing dramatic accuracy loss below a certain bitwidth. This accuracy loss can be alleviated thanks to mixed precision quantization, allowing for more flexible bitwidth allocation. However, standard mixed precision benefits remain limited due to the 1-bit frontier, that forces each parameter to be encoded on at least 1 bit of data. This paper introduces an approach that combines mixed precision, zero-point quantization and entropy coding to push the compression boundary of Resnets beyond the 1-bit frontier with an accuracy drop below 1% on the ImageNet benchmark. From an implementation standpoint, a compact decoder architecture features reduced latency, thus allowing for inference-compatible decoding.
△ Less
Submitted 10 June, 2024;
originally announced June 2024.
-
OvA-INN: Continual Learning with Invertible Neural Networks
Authors:
G. Hocquet,
O. Bichler,
D. Querlioz
Abstract:
In the field of Continual Learning, the objective is to learn several tasks one after the other without access to the data from previous tasks. Several solutions have been proposed to tackle this problem but they usually assume that the user knows which of the tasks to perform at test time on a particular sample, or rely on small samples from previous data and most of them suffer of a substantial…
▽ More
In the field of Continual Learning, the objective is to learn several tasks one after the other without access to the data from previous tasks. Several solutions have been proposed to tackle this problem but they usually assume that the user knows which of the tasks to perform at test time on a particular sample, or rely on small samples from previous data and most of them suffer of a substantial drop in accuracy when updated with batches of only one class at a time. In this article, we propose a new method, OvA-INN, which is able to learn one class at a time and without storing any of the previous data. To achieve this, for each class, we train a specific Invertible Neural Network to extract the relevant features to compute the likelihood on this class. At test time, we can predict the class of a sample by identifying the network which predicted the highest likelihood. With this method, we show that we can take advantage of pretrained models by stacking an Invertible Network on top of a feature extractor. This way, we are able to outperform state-of-the-art approaches that rely on features learning for the Continual Learning of MNIST and CIFAR-100 datasets. In our experiments, we reach 72% accuracy on CIFAR-100 after training our model one class at a time.
△ Less
Submitted 24 June, 2020;
originally announced June 2020.
-
SpikeGrad: An ANN-equivalent Computation Model for Implementing Backpropagation with Spikes
Authors:
Johannes Christian Thiele,
Olivier Bichler,
Antoine Dupret
Abstract:
Event-based neuromorphic systems promise to reduce the energy consumption of deep learning tasks by replacing expensive floating point operations on dense matrices by low power sparse and asynchronous operations on spike events. While these systems can be trained increasingly well using approximations of the back-propagation algorithm, these implementations usually require high precision errors fo…
▽ More
Event-based neuromorphic systems promise to reduce the energy consumption of deep learning tasks by replacing expensive floating point operations on dense matrices by low power sparse and asynchronous operations on spike events. While these systems can be trained increasingly well using approximations of the back-propagation algorithm, these implementations usually require high precision errors for training and are therefore incompatible with the typical communication infrastructure of neuromorphic circuits. In this work, we analyze how the gradient can be discretized into spike events when training a spiking neural network. To accelerate our simulation, we show that using a special implementation of the integrate-and-fire neuron allows us to describe the accumulated activations and errors of the spiking neural network in terms of an equivalent artificial neural network, allowing us to largely speed up training compared to an explicit simulation of all spike events. This way we are able to demonstrate that even for deep networks, the gradients can be discretized sufficiently well with spikes if the gradient is properly rescaled. This form of spike-based backpropagation enables us to achieve equivalent or better accuracies on the MNIST and CIFAR10 dataset than comparable state-of-the-art spiking neural networks trained with full precision gradients. The algorithm, which we call SpikeGrad, is based on accumulation and comparison operations and can naturally exploit sparsity in the gradient computation, which makes it an interesting choice for a spiking neuromorphic systems with on-chip learning capacities.
△ Less
Submitted 3 June, 2019;
originally announced June 2019.
-
A Spiking Network for Inference of Relations Trained with Neuromorphic Backpropagation
Authors:
Johannes C. Thiele,
Olivier Bichler,
Antoine Dupret,
Sergio Solinas,
Giacomo Indiveri
Abstract:
The increasing need for intelligent sensors in a wide range of everyday objects requires the existence of low power information processing systems which can operate autonomously in their environment. In particular, merging and processing the outputs of different sensors efficiently is a necessary requirement for mobile agents with cognitive abilities. In this work, we present a multi-layer spiking…
▽ More
The increasing need for intelligent sensors in a wide range of everyday objects requires the existence of low power information processing systems which can operate autonomously in their environment. In particular, merging and processing the outputs of different sensors efficiently is a necessary requirement for mobile agents with cognitive abilities. In this work, we present a multi-layer spiking neural network for inference of relations between stimuli patterns in dedicated neuromorphic systems. The system is trained with a new version of the backpropagation algorithm adapted to on-chip learning in neuromorphic hardware: Error gradients are encoded as spike signals which are propagated through symmetric synapses, using the same integrate-and-fire hardware infrastructure as used during forward propagation. We demonstrate the strength of the approach on an arithmetic relation inference task and on visual XOR on the MNIST dataset. Compared to previous, biologically-inspired implementations of networks for learning and inference of relations, our approach is able to achieve better performance with less neurons. Our architecture is the first spiking neural network architecture with on-chip learning capabilities, which is able to perform relational inference on complex visual stimuli. These features make our system interesting for sensor fusion applications and embedded learning in autonomous neuromorphic agents.
△ Less
Submitted 11 March, 2019;
originally announced March 2019.
-
Pavlov's dog associative learning demonstrated on synaptic-like organic transistors
Authors:
O. Bichler,
W. Zhao,
F. Alibart,
S. Pleutin,
S. Lenfant,
D. Vuillaume,
C. Gamrat
Abstract:
In this letter, we present an original demonstration of an associative learning neural network inspired by the famous Pavlov's dogs experiment. A single nanoparticle organic memory field effect transistor (NOMFET) is used to implement each synapse. We show how the physical properties of this dynamic memristive device can be used to perform low power write operations for the learning and implement…
▽ More
In this letter, we present an original demonstration of an associative learning neural network inspired by the famous Pavlov's dogs experiment. A single nanoparticle organic memory field effect transistor (NOMFET) is used to implement each synapse. We show how the physical properties of this dynamic memristive device can be used to perform low power write operations for the learning and implement short-term association using temporal coding and spike timing dependent plasticity based learning. An electronic circuit was built to validate the proposed learning scheme with packaged devices, with good reproducibility despite the complex synaptic-like dynamic of the NOMFET in pulse regime.
△ Less
Submitted 13 February, 2013;
originally announced February 2013.
-
A memristive nanoparticle/organic hybrid synapstor for neuro-inspired computing
Authors:
F. Alibart,
S. Pleutin,
O. Bichler,
C. Gamrat,
T. Serrano-Gotarredona,
B. Linares-Barranco,
D. Vuillaume
Abstract:
A large effort is devoted to the research of new computing paradigms associated to innovative nanotechnologies that should complement and/or propose alternative solutions to the classical Von Neumann/CMOS association. Among various propositions, Spiking Neural Network (SNN) seems a valid candidate. (i) In terms of functions, SNN using relative spike timing for information coding are deemed to be t…
▽ More
A large effort is devoted to the research of new computing paradigms associated to innovative nanotechnologies that should complement and/or propose alternative solutions to the classical Von Neumann/CMOS association. Among various propositions, Spiking Neural Network (SNN) seems a valid candidate. (i) In terms of functions, SNN using relative spike timing for information coding are deemed to be the most effective at taking inspiration from the brain to allow fast and efficient processing of information for complex tasks in recognition or classification. (ii) In terms of technology, SNN may be able to benefit the most from nanodevices, because SNN architectures are intrinsically tolerant to defective devices and performance variability. Here we demonstrate Spike-Timing-Dependent Plasticity (STDP), a basic and primordial learning function in the brain, with a new class of synapstor (synapse-transistor), called Nanoparticle Organic Memory Field Effect Transistor (NOMFET). We show that this learning function is obtained with a simple hybrid material made of the self-assembly of gold nanoparticles and organic semiconductor thin films. Beyond mimicking biological synapses, we also demonstrate how the shape of the applied spikes can tailor the STDP learning function. Moreover, the experiments and modeling show that this synapstor is a memristive device. Finally, these synapstors are successfully coupled with a CMOS platform emulating the pre- and post-synaptic neurons, and a behavioral macro-model is developed on usual device simulator.
△ Less
Submitted 14 December, 2011;
originally announced December 2011.
