AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning
Authors:
Michaël Mathieu,
Sherjil Ozair,
Srivatsan Srinivasan,
Caglar Gulcehre,
Shangtong Zhang,
Ray Jiang,
Tom Le Paine,
Richard Powell,
Konrad Żołna,
Julian Schrittwieser,
David Choi,
Petko Georgiev,
Daniel Toyama,
Aja Huang,
Roman Ring,
Igor Babuschkin,
Timo Ewalds,
Mahyar Bordbar,
Sarah Henderson,
Sergio Gómez Colmenarejo,
Aäron van den Oord,
Wojciech Marian Czarnecki,
Nando de Freitas,
Oriol Vinyals
Abstract:
StarCraft II is one of the most challenging simulated reinforcement learning environments; it is partially observable, stochastic, multi-agent, and mastering StarCraft II requires strategic planning over long time horizons with real-time low-level execution. It also has an active professional competitive scene. StarCraft II is uniquely suited for advancing offline RL algorithms, both because of it…
▽ More
StarCraft II is one of the most challenging simulated reinforcement learning environments; it is partially observable, stochastic, multi-agent, and mastering StarCraft II requires strategic planning over long time horizons with real-time low-level execution. It also has an active professional competitive scene. StarCraft II is uniquely suited for advancing offline RL algorithms, both because of its challenging nature and because Blizzard has released a massive dataset of millions of StarCraft II games played by human players. This paper leverages that and establishes a benchmark, called AlphaStar Unplugged, introducing unprecedented challenges for offline reinforcement learning. We define a dataset (a subset of Blizzard's release), tools standardizing an API for machine learning methods, and an evaluation protocol. We also present baseline agents, including behavior cloning, offline variants of actor-critic and MuZero. We improve the state of the art of agents using only offline data, and we achieve 90% win rate against previously published AlphaStar behavior cloning agent.
△ Less
Submitted 7 August, 2023;
originally announced August 2023.
A Generalist Agent
Authors:
Scott Reed,
Konrad Zolna,
Emilio Parisotto,
Sergio Gomez Colmenarejo,
Alexander Novikov,
Gabriel Barth-Maron,
Mai Gimenez,
Yury Sulsky,
Jackie Kay,
Jost Tobias Springenberg,
Tom Eccles,
Jake Bruce,
Ali Razavi,
Ashley Edwards,
Nicolas Heess,
Yutian Chen,
Raia Hadsell,
Oriol Vinyals,
Mahyar Bordbar,
Nando de Freitas
Abstract:
Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, dec…
▽ More
Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato.
△ Less
Submitted 11 November, 2022; v1 submitted 12 May, 2022;
originally announced May 2022.
