-
Sparse Distributed Memory is a Continual Learner
Abstract: Continual learning is a problem for artificial neural networks that their biological counterparts are adept at solving. Building on work using Sparse Distributed Memory (SDM) to connect a core neural circuit with the powerful Transformer model, we create a modified Multi-Layered Perceptron (MLP) that is a strong continual learner. We find that every component of our MLP variant translated from bio… ▽ More
Submitted 20 March, 2023; originally announced March 2023.
Comments: 9 Pages. ICLR Acceptance
Journal ref: ICLR 2023
-
Attention Approximates Sparse Distributed Memory
Abstract: While Attention has come to be an important mechanism in deep learning, there remains limited intuition for why it works so well. Here, we show that Transformer Attention can be closely related under certain data conditions to Kanerva's Sparse Distributed Memory (SDM), a biologically plausible associative memory model. We confirm that these conditions are satisfied in pre-trained GPT2 Transformer… ▽ More
Submitted 17 January, 2022; v1 submitted 9 November, 2021; originally announced November 2021.
Journal ref: 35th Conference on Neural Information Processing Systems (NeurIPS 2021)