Human-Timescale Adaptation in an Open-Ended Task Space
Authors:
Adaptive Agent Team,
Jakob Bauer,
Kate Baumli,
Satinder Baveja,
Feryal Behbahani,
Avishkar Bhoopchand,
Nathalie Bradley-Schmieg,
Michael Chang,
Natalie Clay,
Adrian Collister,
Vibhavari Dasagi,
Lucy Gonzalez,
Karol Gregor,
Edward Hughes,
Sheleem Kashem,
Maria Loks-Thompson,
Hannah Openshaw,
Jack Parker-Holder,
Shreya Pathak,
Nicolas Perez-Nieves,
Nemanja Rakicevic,
Tim Rocktäschel,
Yannick Schroecker,
Jakub Sygnowski,
Karl Tuyls
, et al. (3 additional authors not shown)
Abstract:
Foundation models have shown impressive adaptation and scalability in supervised and self-supervised learning problems, but so far these successes have not fully translated to reinforcement learning (RL). In this work, we demonstrate that training an RL agent at scale leads to a general in-context learning algorithm that can adapt to open-ended novel embodied 3D problems as quickly as humans. In a…
▽ More
Foundation models have shown impressive adaptation and scalability in supervised and self-supervised learning problems, but so far these successes have not fully translated to reinforcement learning (RL). In this work, we demonstrate that training an RL agent at scale leads to a general in-context learning algorithm that can adapt to open-ended novel embodied 3D problems as quickly as humans. In a vast space of held-out environment dynamics, our adaptive agent (AdA) displays on-the-fly hypothesis-driven exploration, efficient exploitation of acquired knowledge, and can successfully be prompted with first-person demonstrations. Adaptation emerges from three ingredients: (1) meta-reinforcement learning across a vast, smooth and diverse task distribution, (2) a policy parameterised as a large-scale attention-based memory architecture, and (3) an effective automated curriculum that prioritises tasks at the frontier of an agent's capabilities. We demonstrate characteristic scaling laws with respect to network size, memory length, and richness of the training task distribution. We believe our results lay the foundation for increasingly general and adaptive RL agents that perform well across ever-larger open-ended domains.
△ Less
Submitted 18 January, 2023;
originally announced January 2023.
Learning Robust Real-Time Cultural Transmission without Human Data
Authors:
Cultural General Intelligence Team,
Avishkar Bhoopchand,
Bethanie Brownfield,
Adrian Collister,
Agustin Dal Lago,
Ashley Edwards,
Richard Everett,
Alexandre Frechette,
Yanko Gitahy Oliveira,
Edward Hughes,
Kory W. Mathewson,
Piermaria Mendolicchio,
Julia Pawar,
Miruna Pislar,
Alex Platonov,
Evan Senter,
Sukhdeep Singh,
Alexander Zacherl,
Lei M. Zhang
Abstract:
Cultural transmission is the domain-general social skill that allows agents to acquire and use information from each other in real-time with high fidelity and recall. In humans, it is the inheritance process that powers cumulative cultural evolution, expanding our skills, tools and knowledge across generations. We provide a method for generating zero-shot, high recall cultural transmission in arti…
▽ More
Cultural transmission is the domain-general social skill that allows agents to acquire and use information from each other in real-time with high fidelity and recall. In humans, it is the inheritance process that powers cumulative cultural evolution, expanding our skills, tools and knowledge across generations. We provide a method for generating zero-shot, high recall cultural transmission in artificially intelligent agents. Our agents succeed at real-time cultural transmission from humans in novel contexts without using any pre-collected human data. We identify a surprisingly simple set of ingredients sufficient for generating cultural transmission and develop an evaluation methodology for rigorously assessing it. This paves the way for cultural evolution as an algorithm for develo** artificial general intelligence.
△ Less
Submitted 1 March, 2022;
originally announced March 2022.
