-
Exploring the functional hierarchy of different pyramidal cell types in temporal processing
Authors:
Anh Duong Vo,
Elisabeth Abs,
Pau Vilimelis Aceituno,
Benjamin Friedrich Grewe,
Katharina Anna Wilmes
Abstract:
Recent research has revealed the unique functionality of cortical pyramidal cell subtypes, namely intratelencephalic neurons (IT) and pyramidal-tract neurons (PT). How these two populations interact with each other to fulfill their functional roles remains poorly understood. We propose the existence of a functional hierarchy between IT and PT due to their unidirectional connection and distinct rol…
▽ More
Recent research has revealed the unique functionality of cortical pyramidal cell subtypes, namely intratelencephalic neurons (IT) and pyramidal-tract neurons (PT). How these two populations interact with each other to fulfill their functional roles remains poorly understood. We propose the existence of a functional hierarchy between IT and PT due to their unidirectional connection and distinct roles in sensory discrimination and motor tasks. To investigate this hypothesis, we conducted a literature review of recent studies that explored the properties and functionalities of IT and PT, including causal lesion studies, population-based encoding, and calcium imaging experiments. Further, we suggest future experiments to determine the relevance of the canonical IT-PT circuit motif for temporal processing. Our work provides a novel perspective on the mechanistic role of IT and PT in temporal processing.
△ Less
Submitted 12 December, 2023;
originally announced December 2023.
-
Bio-Inspired, Task-Free Continual Learning through Activity Regularization
Authors:
Francesco Lässig,
Pau Vilimelis Aceituno,
Martino Sorbaro,
Benjamin F. Grewe
Abstract:
The ability to sequentially learn multiple tasks without forgetting is a key skill of biological brains, whereas it represents a major challenge to the field of deep learning. To avoid catastrophic forgetting, various continual learning (CL) approaches have been devised. However, these usually require discrete task boundaries. This requirement seems biologically implausible and often limits the ap…
▽ More
The ability to sequentially learn multiple tasks without forgetting is a key skill of biological brains, whereas it represents a major challenge to the field of deep learning. To avoid catastrophic forgetting, various continual learning (CL) approaches have been devised. However, these usually require discrete task boundaries. This requirement seems biologically implausible and often limits the application of CL methods in the real world where tasks are not always well defined. Here, we take inspiration from neuroscience, where sparse, non-overlap** neuronal representations have been suggested to prevent catastrophic forgetting. As in the brain, we argue that these sparse representations should be chosen on the basis of feed forward (stimulus-specific) as well as top-down (context-specific) information. To implement such selective sparsity, we use a bio-plausible form of hierarchical credit assignment known as Deep Feedback Control (DFC) and combine it with a winner-take-all sparsity mechanism. In addition to sparsity, we introduce lateral recurrent connections within each layer to further protect previously learned representations. We evaluate the new sparse-recurrent version of DFC on the split-MNIST computer vision benchmark and show that only the combination of sparsity and intra-layer recurrent connections improves CL performance with respect to standard backpropagation. Our method achieves similar performance to well-known CL methods, such as Elastic Weight Consolidation and Synaptic Intelligence, without requiring information about task boundaries. Overall, we showcase the idea of adopting computational principles from the brain to derive new, task-free learning algorithms for CL.
△ Less
Submitted 8 December, 2022;
originally announced December 2022.
-
Disentangling the Predictive Variance of Deep Ensembles through the Neural Tangent Kernel
Authors:
Sei** Kobayashi,
Pau Vilimelis Aceituno,
Johannes von Oswald
Abstract:
Identifying unfamiliar inputs, also known as out-of-distribution (OOD) detection, is a crucial property of any decision making process. A simple and empirically validated technique is based on deep ensembles where the variance of predictions over different neural networks acts as a substitute for input uncertainty. Nevertheless, a theoretical understanding of the inductive biases leading to the pe…
▽ More
Identifying unfamiliar inputs, also known as out-of-distribution (OOD) detection, is a crucial property of any decision making process. A simple and empirically validated technique is based on deep ensembles where the variance of predictions over different neural networks acts as a substitute for input uncertainty. Nevertheless, a theoretical understanding of the inductive biases leading to the performance of deep ensemble's uncertainty estimation is missing. To improve our description of their behavior, we study deep ensembles with large layer widths operating in simplified linear training regimes, in which the functions trained with gradient descent can be described by the neural tangent kernel. We identify two sources of noise, each inducing a distinct inductive bias in the predictive variance at initialization. We further show theoretically and empirically that both noise sources affect the predictive variance of non-linear deep ensembles in toy models and realistic settings after training. Finally, we propose practical ways to eliminate part of these noise sources leading to significant changes and improved OOD detection in trained deep ensembles.
△ Less
Submitted 18 October, 2022;
originally announced October 2022.
-
Credit Assignment in Neural Networks through Deep Feedback Control
Authors:
Alexander Meulemans,
Matilde Tristany Farinha,
Javier García Ordóñez,
Pau Vilimelis Aceituno,
João Sacramento,
Benjamin F. Grewe
Abstract:
The success of deep learning sparked interest in whether the brain learns by using similar techniques for assigning credit to each synaptic weight for its contribution to the network output. However, the majority of current attempts at biologically-plausible learning methods are either non-local in time, require highly specific connectivity motives, or have no clear link to any known mathematical…
▽ More
The success of deep learning sparked interest in whether the brain learns by using similar techniques for assigning credit to each synaptic weight for its contribution to the network output. However, the majority of current attempts at biologically-plausible learning methods are either non-local in time, require highly specific connectivity motives, or have no clear link to any known mathematical optimization method. Here, we introduce Deep Feedback Control (DFC), a new learning method that uses a feedback controller to drive a deep neural network to match a desired output target and whose control signal can be used for credit assignment. The resulting learning rule is fully local in space and time and approximates Gauss-Newton optimization for a wide range of feedback connectivity patterns. To further underline its biological plausibility, we relate DFC to a multi-compartment model of cortical pyramidal neurons with a local voltage-dependent synaptic plasticity rule, consistent with recent theories of dendritic processing. By combining dynamical system theory with mathematical optimization theory, we provide a strong theoretical foundation for DFC that we corroborate with detailed results on toy experiments and standard computer-vision benchmarks.
△ Less
Submitted 17 January, 2022; v1 submitted 15 June, 2021;
originally announced June 2021.
-
Minimizing costs of communication with random constant weight codes
Authors:
Pau Vilimelis Aceituno
Abstract:
We present a framework for minimizing costs in constant weight codes while maintaining a certain amount of differentiable codewords. Our calculations are based on a combinatorial view of constant weight codes and relay on simple approximations.
We present a framework for minimizing costs in constant weight codes while maintaining a certain amount of differentiable codewords. Our calculations are based on a combinatorial view of constant weight codes and relay on simple approximations.
△ Less
Submitted 6 May, 2021;
originally announced May 2021.
-
Resonances induced by Spiking Time Dependent Plasticity
Authors:
Pau Vilimelis Aceituno
Abstract:
Neural populations exposed to a certain stimulus learn to represent it better. However, the process that leads local, self-organized rules to do so is unclear. We address the question of how can a neural periodic input be learned and use the Differential Hebbian Learning framework, coupled with a homeostatic mechanism to derive two self-consistency equations that lead to increased responses to the…
▽ More
Neural populations exposed to a certain stimulus learn to represent it better. However, the process that leads local, self-organized rules to do so is unclear. We address the question of how can a neural periodic input be learned and use the Differential Hebbian Learning framework, coupled with a homeostatic mechanism to derive two self-consistency equations that lead to increased responses to the same stimulus. Although all our simulations are done with simple Leaky-Integrate and Fire neurons and standard Spiking Time Dependent Plasticity learning rules, our results can be easily interpreted in terms of rates and population codes.
△ Less
Submitted 15 June, 2020;
originally announced June 2020.
-
Synaptic Time-Dependent Plasticity Leads to Efficient Coding of Predictions
Authors:
Pau Vilimelis Aceituno,
Masud Ehsani,
Jürgen Jost
Abstract:
Latency reduction of postsynaptic spikes is a well-known effect of Synaptic Time-Dependent Plasticity. We expand this notion for long postsynaptic spike trains, showing that, for a fixed input spike train, STDP reduces the number of postsynaptic spikes and concentrates the remaining ones. Then we study the consequences of this phenomena in terms of coding, finding that this mechanism improves the…
▽ More
Latency reduction of postsynaptic spikes is a well-known effect of Synaptic Time-Dependent Plasticity. We expand this notion for long postsynaptic spike trains, showing that, for a fixed input spike train, STDP reduces the number of postsynaptic spikes and concentrates the remaining ones. Then we study the consequences of this phenomena in terms of coding, finding that this mechanism improves the neural code by increasing the signal-to-noise ratio and lowering the metabolic costs of frequent stimuli. Finally, we illustrate that the reduction of postsynaptic latencies can lead to the emergence of predictions.
△ Less
Submitted 25 July, 2019;
originally announced July 2019.
-
Universal hypotrochoidic law for random matrices with cyclic correlations
Authors:
Pau Vilimelis Aceituno,
Tim Rogers,
Henning Schomerus
Abstract:
The celebrated elliptic law describes the distribution of eigenvalues of random matrices with correlations between off-diagonal pairs of elements, having applications to a wide range of physical and biological systems. Here, we investigate the generalization of this law to random matrices exhibiting higher-order cyclic correlations between $k$-tuples of matrix entries. We show that the eigenvalue…
▽ More
The celebrated elliptic law describes the distribution of eigenvalues of random matrices with correlations between off-diagonal pairs of elements, having applications to a wide range of physical and biological systems. Here, we investigate the generalization of this law to random matrices exhibiting higher-order cyclic correlations between $k$-tuples of matrix entries. We show that the eigenvalue spectrum in this ensemble is bounded by a hypotrochoid curve with $k$-fold rotational symmetry. This hypotrochoid law applies to full matrices as well as sparse ones, and thereby holds with remarkable universality. We further extend our analysis to matrices and graphs with competing cycle motifs, which are described more generally by polytrochoid spectral boundaries.
△ Less
Submitted 2 March, 2019; v1 submitted 14 December, 2018;
originally announced December 2018.
-
Eigenvalues of random graphs with cycles
Authors:
Pau Vilimelis Aceituno
Abstract:
Networks are often studied using the eigenvalues of their adjacency matrix, a powerful mathematical tool with a wide range of applications. Since in real systems the exact graph structure is not known, researchers resort to random graphs to obtain eigenvalue properties from known structural features. However, this theory is far from intuitive and often requires training of free probability, cavity…
▽ More
Networks are often studied using the eigenvalues of their adjacency matrix, a powerful mathematical tool with a wide range of applications. Since in real systems the exact graph structure is not known, researchers resort to random graphs to obtain eigenvalue properties from known structural features. However, this theory is far from intuitive and often requires training of free probability, cavity methods or a strong familiarity with probability theory. In this note we offer a different perspective on this field by focusing on the cycles in a graph. We use the so-called method of moments to obtain relation between eigenvalues and cycle weights and then we obtain spectral properties of random graphs with cyclic motifs. We use it to explore properties of the eigenvalues of adjacency matrices of graphs with short cycles and of circulant directed graphs. Although our result is not as powerful as the some of the existing methods, they are nevertheless useful and far easier to understand.
△ Less
Submitted 29 January, 2020; v1 submitted 13 April, 2018;
originally announced April 2018.
-
Tailoring Artificial Neural Networks for Optimal Learning
Authors:
Pau Vilimelis Aceituno,
Yan Gang,
Yang-Yu Liu
Abstract:
As one of the most important paradigms of recurrent neural networks, the echo state network (ESN) has been applied to a wide range of fields, from robotics to medicine, finance, and language processing. A key feature of the ESN paradigm is its reservoir --- a directed and weighted network of neurons that projects the input time series into a high dimensional space where linear regression or classi…
▽ More
As one of the most important paradigms of recurrent neural networks, the echo state network (ESN) has been applied to a wide range of fields, from robotics to medicine, finance, and language processing. A key feature of the ESN paradigm is its reservoir --- a directed and weighted network of neurons that projects the input time series into a high dimensional space where linear regression or classification can be applied. Despite extensive studies, the impact of the reservoir network on the ESN performance remains unclear. Combining tools from physics, dynamical systems and network science, we attempt to open the black box of ESN and offer insights to understand the behavior of general artificial neural networks. Through spectral analysis of the reservoir network we reveal a key factor that largely determines the ESN memory capacity and hence affects its performance. Moreover, we find that adding short loops to the reservoir network can tailor ESN for specific tasks and optimize learning. We validate our findings by applying ESN to forecast both synthetic and real benchmark time series. Our results provide a new way to design task-specific ESN. More importantly, it demonstrates the power of combining tools from physics, dynamical systems and network science to offer new insights in understanding the mechanisms of general artificial neural networks.
△ Less
Submitted 25 February, 2020; v1 submitted 8 July, 2017;
originally announced July 2017.
