-
Modelling Behavioural Diversity for Learning in Open-Ended Games
Authors:
Nicolas Perez Nieves,
Yaodong Yang,
Oliver Slumbers,
David Henry Mguni,
Ying Wen,
Jun Wang
Abstract:
Promoting behavioural diversity is critical for solving games with non-transitive dynamics where strategic cycles exist, and there is no consistent winner (e.g., Rock-Paper-Scissors). Yet, there is a lack of rigorous treatment for defining diversity and constructing diversity-aware learning dynamics. In this work, we offer a geometric interpretation of behavioural diversity in games and introduce…
▽ More
Promoting behavioural diversity is critical for solving games with non-transitive dynamics where strategic cycles exist, and there is no consistent winner (e.g., Rock-Paper-Scissors). Yet, there is a lack of rigorous treatment for defining diversity and constructing diversity-aware learning dynamics. In this work, we offer a geometric interpretation of behavioural diversity in games and introduce a novel diversity metric based on determinantal point processes (DPP). By incorporating the diversity metric into best-response dynamics, we develop diverse fictitious play and diverse policy-space response oracle for solving normal-form games and open-ended games. We prove the uniqueness of the diverse best response and the convergence of our algorithms on two-player games. Importantly, we show that maximising the DPP-based diversity metric guarantees to enlarge the gamescape -- convex polytopes spanned by agents' mixtures of strategies. To validate our diversity-aware solvers, we test on tens of games that show strong non-transitivity. Results suggest that our methods achieve at least the same, and in most games, lower exploitability than PSRO solvers by finding effective and diverse strategies.
△ Less
Submitted 10 June, 2021; v1 submitted 14 March, 2021;
originally announced March 2021.
-
Online Double Oracle
Authors:
Le Cong Dinh,
Yaodong Yang,
Stephen McAleer,
Zheng Tian,
Nicolas Perez Nieves,
Oliver Slumbers,
David Henry Mguni,
Haitham Bou Ammar,
Jun Wang
Abstract:
Solving strategic games with huge action space is a critical yet under-explored topic in economics, operations research and artificial intelligence. This paper proposes new learning algorithms for solving two-player zero-sum normal-form games where the number of pure strategies is prohibitively large. Specifically, we combine no-regret analysis from online learning with Double Oracle (DO) methods…
▽ More
Solving strategic games with huge action space is a critical yet under-explored topic in economics, operations research and artificial intelligence. This paper proposes new learning algorithms for solving two-player zero-sum normal-form games where the number of pure strategies is prohibitively large. Specifically, we combine no-regret analysis from online learning with Double Oracle (DO) methods from game theory. Our method -- \emph{Online Double Oracle (ODO)} -- is provably convergent to a Nash equilibrium (NE). Most importantly, unlike normal DO methods, ODO is \emph{rationale} in the sense that each agent in ODO can exploit strategic adversary with a regret bound of $\mathcal{O}(\sqrt{T k \log(k)})$ where $k$ is not the total number of pure strategies, but rather the size of \emph{effective strategy set} that is linearly dependent on the support size of the NE. On tens of different real-world games, ODO outperforms DO, PSRO methods, and no-regret algorithms such as Multiplicative Weight Update by a significant margin, both in terms of convergence rate to a NE and average payoff against strategic adversaries.
△ Less
Submitted 15 February, 2023; v1 submitted 13 March, 2021;
originally announced March 2021.
-
SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
Authors:
Ming Zhou,
Jun Luo,
Julian Villella,
Yaodong Yang,
David Rusu,
Jiayu Miao,
Weinan Zhang,
Montgomery Alban,
Iman Fadakar,
Zheng Chen,
Aurora Chongxi Huang,
Ying Wen,
Kimia Hassanzadeh,
Daniel Graves,
Dong Chen,
Zhengbang Zhu,
Nhat Nguyen,
Mohamed Elsayed,
Kun Shao,
Sanjeevan Ahilan,
Baokuan Zhang,
Jiannan Wu,
Zhengang Fu,
Kasra Rezaee,
Peyman Yadmellat
, et al. (12 additional authors not shown)
Abstract:
Multi-agent interaction is a fundamental aspect of autonomous driving in the real world. Despite more than a decade of research and development, the problem of how to competently interact with diverse road users in diverse scenarios remains largely unsolved. Learning methods have much to offer towards solving this problem. But they require a realistic multi-agent simulator that generates diverse a…
▽ More
Multi-agent interaction is a fundamental aspect of autonomous driving in the real world. Despite more than a decade of research and development, the problem of how to competently interact with diverse road users in diverse scenarios remains largely unsolved. Learning methods have much to offer towards solving this problem. But they require a realistic multi-agent simulator that generates diverse and competent driving interactions. To meet this need, we develop a dedicated simulation platform called SMARTS (Scalable Multi-Agent RL Training School). SMARTS supports the training, accumulation, and use of diverse behavior models of road users. These are in turn used to create increasingly more realistic and diverse interactions that enable deeper and broader research on multi-agent interaction. In this paper, we describe the design goals of SMARTS, explain its basic architecture and its key features, and illustrate its use through concrete multi-agent experiments on interactive scenarios. We open-source the SMARTS platform and the associated benchmark tasks and evaluation metrics to encourage and empower research on multi-agent learning for autonomous driving. Our code is available at https://github.com/huawei-noah/SMARTS.
△ Less
Submitted 31 October, 2020; v1 submitted 19 October, 2020;
originally announced October 2020.
