The Relational Bottleneck as an Inductive Bias for Efficient Abstraction
Authors:
Taylor W. Webb,
Steven M. Frankland,
Awni Altabaa,
Simon Segert,
Kamesh Krishnamurthy,
Declan Campbell,
Jacob Russin,
Tyler Giallanza,
Zack Dulberg,
Randall O'Reilly,
John Lafferty,
Jonathan D. Cohen
Abstract:
A central challenge for cognitive science is to explain how abstract concepts are acquired from limited experience. This has often been framed in terms of a dichotomy between connectionist and symbolic cognitive models. Here, we highlight a recently emerging line of work that suggests a novel reconciliation of these approaches, by exploiting an inductive bias that we term the relational bottleneck…
▽ More
A central challenge for cognitive science is to explain how abstract concepts are acquired from limited experience. This has often been framed in terms of a dichotomy between connectionist and symbolic cognitive models. Here, we highlight a recently emerging line of work that suggests a novel reconciliation of these approaches, by exploiting an inductive bias that we term the relational bottleneck. In that approach, neural networks are constrained via their architecture to focus on relations between perceptual inputs, rather than the attributes of individual inputs. We review a family of models that employ this approach to induce abstractions in a data-efficient manner, emphasizing their potential as candidate models for the acquisition of abstract concepts in the human mind and brain.
△ Less
Submitted 1 May, 2024; v1 submitted 12 September, 2023;
originally announced September 2023.
Learning Representations that Support Extrapolation
Authors:
Taylor W. Webb,
Zachary Dulberg,
Steven M. Frankland,
Alexander A. Petrov,
Randall C. O'Reilly,
Jonathan D. Cohen
Abstract:
Extrapolation -- the ability to make inferences that go beyond the scope of one's experiences -- is a hallmark of human intelligence. By contrast, the generalization exhibited by contemporary neural network algorithms is largely limited to interpolation between data points in their training corpora. In this paper, we consider the challenge of learning representations that support extrapolation. We…
▽ More
Extrapolation -- the ability to make inferences that go beyond the scope of one's experiences -- is a hallmark of human intelligence. By contrast, the generalization exhibited by contemporary neural network algorithms is largely limited to interpolation between data points in their training corpora. In this paper, we consider the challenge of learning representations that support extrapolation. We introduce a novel visual analogy benchmark that allows the graded evaluation of extrapolation as a function of distance from the convex domain defined by the training data. We also introduce a simple technique, temporal context normalization, that encourages representations that emphasize the relations between objects. We find that this technique enables a significant improvement in the ability to extrapolate, considerably outperforming a number of competitive techniques.
△ Less
Submitted 6 September, 2023; v1 submitted 9 July, 2020;
originally announced July 2020.
