Bio-Inspired, Task-Free Continual Learning through Activity Regularization
Authors:
Francesco Lässig,
Pau Vilimelis Aceituno,
Martino Sorbaro,
Benjamin F. Grewe
Abstract:
The ability to sequentially learn multiple tasks without forgetting is a key skill of biological brains, whereas it represents a major challenge to the field of deep learning. To avoid catastrophic forgetting, various continual learning (CL) approaches have been devised. However, these usually require discrete task boundaries. This requirement seems biologically implausible and often limits the ap…
▽ More
The ability to sequentially learn multiple tasks without forgetting is a key skill of biological brains, whereas it represents a major challenge to the field of deep learning. To avoid catastrophic forgetting, various continual learning (CL) approaches have been devised. However, these usually require discrete task boundaries. This requirement seems biologically implausible and often limits the application of CL methods in the real world where tasks are not always well defined. Here, we take inspiration from neuroscience, where sparse, non-overlap** neuronal representations have been suggested to prevent catastrophic forgetting. As in the brain, we argue that these sparse representations should be chosen on the basis of feed forward (stimulus-specific) as well as top-down (context-specific) information. To implement such selective sparsity, we use a bio-plausible form of hierarchical credit assignment known as Deep Feedback Control (DFC) and combine it with a winner-take-all sparsity mechanism. In addition to sparsity, we introduce lateral recurrent connections within each layer to further protect previously learned representations. We evaluate the new sparse-recurrent version of DFC on the split-MNIST computer vision benchmark and show that only the combination of sparsity and intra-layer recurrent connections improves CL performance with respect to standard backpropagation. Our method achieves similar performance to well-known CL methods, such as Elastic Weight Consolidation and Synaptic Intelligence, without requiring information about task boundaries. Overall, we showcase the idea of adopting computational principles from the brain to derive new, task-free learning algorithms for CL.
△ Less
Submitted 8 December, 2022;
originally announced December 2022.
Darts: User-Friendly Modern Machine Learning for Time Series
Authors:
Julien Herzen,
Francesco Lässig,
Samuele Giuliano Piazzetta,
Thomas Neuer,
Léo Tafti,
Guillaume Raille,
Tomas Van Pottelbergh,
Marek Pasieka,
Andrzej Skrodzki,
Nicolas Huguenin,
Maxime Dumonal,
Jan Kościsz,
Dennis Bader,
Frédérick Gusset,
Mounir Benheddi,
Camila Williamson,
Michal Kosinski,
Matej Petrik,
Gaël Grosch
Abstract:
We present Darts, a Python machine learning library for time series, with a focus on forecasting. Darts offers a variety of models, from classics such as ARIMA to state-of-the-art deep neural networks. The emphasis of the library is on offering modern machine learning functionalities, such as supporting multidimensional series, meta-learning on multiple series, training on large datasets, incorpor…
▽ More
We present Darts, a Python machine learning library for time series, with a focus on forecasting. Darts offers a variety of models, from classics such as ARIMA to state-of-the-art deep neural networks. The emphasis of the library is on offering modern machine learning functionalities, such as supporting multidimensional series, meta-learning on multiple series, training on large datasets, incorporating external data, ensembling models, and providing a rich support for probabilistic forecasting. At the same time, great care goes into the API design to make it user-friendly and easy to use. For instance, all models can be used using fit()/predict(), similar to scikit-learn.
△ Less
Submitted 19 May, 2022; v1 submitted 7 October, 2021;
originally announced October 2021.
