-
A social path to human-like artificial intelligence
Authors:
Edgar A. Duéñez-Guzmán,
Suzanne Sadedin,
Jane X. Wang,
Kevin R. McKee,
Joel Z. Leibo
Abstract:
Traditionally, cognitive and computer scientists have viewed intelligence solipsistically, as a property of unitary agents devoid of social context. Given the success of contemporary learning algorithms, we argue that the bottleneck in artificial intelligence (AI) progress is shifting from data assimilation to novel data generation. We bring together evidence showing that natural intelligence emer…
▽ More
Traditionally, cognitive and computer scientists have viewed intelligence solipsistically, as a property of unitary agents devoid of social context. Given the success of contemporary learning algorithms, we argue that the bottleneck in artificial intelligence (AI) progress is shifting from data assimilation to novel data generation. We bring together evidence showing that natural intelligence emerges at multiple scales in networks of interacting agents via collective living, social relationships and major evolutionary transitions, which contribute to novel data generation through mechanisms such as population pressures, arms races, Machiavellian selection, social learning and cumulative culture. Many breakthroughs in AI exploit some of these processes, from multi-agent structures enabling algorithms to master complex games like Capture-The-Flag and StarCraft II, to strategic communication in Diplomacy and the sha** of AI data streams by other AIs. Moving beyond a solipsistic view of agency to integrate these mechanisms suggests a path to human-like compounding innovation through ongoing novel data generation.
△ Less
Submitted 22 May, 2024;
originally announced May 2024.
-
Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia
Authors:
Alexander Sasha Vezhnevets,
John P. Agapiou,
Avia Aharon,
Ron Ziv,
Jayd Matyas,
Edgar A. Duéñez-Guzmán,
William A. Cunningham,
Simon Osindero,
Danny Karmon,
Joel Z. Leibo
Abstract:
Agent-based modeling has been around for decades, and applied widely across the social and natural sciences. The scope of this research method is now poised to grow dramatically as it absorbs the new affordances provided by Large Language Models (LLM)s. Generative Agent-Based Models (GABM) are not just classic Agent-Based Models (ABM)s where the agents talk to one another. Rather, GABMs are constr…
▽ More
Agent-based modeling has been around for decades, and applied widely across the social and natural sciences. The scope of this research method is now poised to grow dramatically as it absorbs the new affordances provided by Large Language Models (LLM)s. Generative Agent-Based Models (GABM) are not just classic Agent-Based Models (ABM)s where the agents talk to one another. Rather, GABMs are constructed using an LLM to apply common sense to situations, act "reasonably", recall common semantic knowledge, produce API calls to control digital technologies like apps, and communicate both within the simulation and to researchers viewing it from the outside. Here we present Concordia, a library to facilitate constructing and working with GABMs. Concordia makes it easy to construct language-mediated simulations of physically- or digitally-grounded environments. Concordia agents produce their behavior using a flexible component system which mediates between two fundamental operations: LLM calls and associative memory retrieval. A special agent called the Game Master (GM), which was inspired by tabletop role-playing games, is responsible for simulating the environment where the agents interact. Agents take actions by describing what they want to do in natural language. The GM then translates their actions into appropriate implementations. In a simulated physical world, the GM checks the physical plausibility of agent actions and describes their effects. In digital environments simulating technologies such as apps and services, the GM may handle API calls to integrate with external tools such as general AI assistants (e.g., Bard, ChatGPT), and digital apps (e.g., Calendar, Email, Search, etc.). Concordia was designed to support a wide array of applications both in scientific research and for evaluating performance of real digital services by simulating users and/or generating synthetic data.
△ Less
Submitted 13 December, 2023; v1 submitted 6 December, 2023;
originally announced December 2023.
-
Doing the right thing for the right reason: Evaluating artificial moral cognition by probing cost insensitivity
Authors:
Yiran Mao,
Madeline G. Reinecke,
Markus Kunesch,
Edgar A. Duéñez-Guzmán,
Ramona Comanescu,
Julia Haas,
Joel Z. Leibo
Abstract:
Is it possible to evaluate the moral cognition of complex artificial agents? In this work, we take a look at one aspect of morality: `doing the right thing for the right reasons.' We propose a behavior-based analysis of artificial moral cognition which could also be applied to humans to facilitate like-for-like comparison. Morally-motivated behavior should persist despite mounting cost; by measuri…
▽ More
Is it possible to evaluate the moral cognition of complex artificial agents? In this work, we take a look at one aspect of morality: `doing the right thing for the right reasons.' We propose a behavior-based analysis of artificial moral cognition which could also be applied to humans to facilitate like-for-like comparison. Morally-motivated behavior should persist despite mounting cost; by measuring an agent's sensitivity to this cost, we gain deeper insight into underlying motivations. We apply this evaluation to a particular set of deep reinforcement learning agents, trained by memory-based meta-reinforcement learning. Our results indicate that agents trained with a reward function that includes other-regarding preferences perform hel** behavior in a way that is less sensitive to increasing cost than agents trained with more self-interested preferences.
△ Less
Submitted 29 May, 2023;
originally announced May 2023.
-
Heterogeneous Social Value Orientation Leads to Meaningful Diversity in Sequential Social Dilemmas
Authors:
Udari Madhushani,
Kevin R. McKee,
John P. Agapiou,
Joel Z. Leibo,
Richard Everett,
Thomas Anthony,
Edward Hughes,
Karl Tuyls,
Edgar A. Duéñez-Guzmán
Abstract:
In social psychology, Social Value Orientation (SVO) describes an individual's propensity to allocate resources between themself and others. In reinforcement learning, SVO has been instantiated as an intrinsic motivation that remaps an agent's rewards based on particular target distributions of group reward. Prior studies show that groups of agents endowed with heterogeneous SVO learn diverse poli…
▽ More
In social psychology, Social Value Orientation (SVO) describes an individual's propensity to allocate resources between themself and others. In reinforcement learning, SVO has been instantiated as an intrinsic motivation that remaps an agent's rewards based on particular target distributions of group reward. Prior studies show that groups of agents endowed with heterogeneous SVO learn diverse policies in settings that resemble the incentive structure of Prisoner's dilemma. Our work extends this body of results and demonstrates that (1) heterogeneous SVO leads to meaningfully diverse policies across a range of incentive structures in sequential social dilemmas, as measured by task-specific diversity metrics; and (2) learning a best response to such policy diversity leads to better zero-shot generalization in some situations. We show that these best-response agents learn policies that are conditioned on their co-players, which we posit is the reason for improved zero-shot generalization results.
△ Less
Submitted 1 May, 2023;
originally announced May 2023.
-
Melting Pot 2.0
Authors:
John P. Agapiou,
Alexander Sasha Vezhnevets,
Edgar A. Duéñez-Guzmán,
Jayd Matyas,
Yiran Mao,
Peter Sunehag,
Raphael Köster,
Udari Madhushani,
Kavya Kopparapu,
Ramona Comanescu,
DJ Strouse,
Michael B. Johanson,
Sukhdeep Singh,
Julia Haas,
Igor Mordatch,
Dean Mobbs,
Joel Z. Leibo
Abstract:
Multi-agent artificial intelligence research promises a path to develop intelligent technologies that are more human-like and more human-compatible than those produced by "solipsistic" approaches, which do not consider interactions between agents. Melting Pot is a research tool developed to facilitate work on multi-agent artificial intelligence, and provides an evaluation protocol that measures ge…
▽ More
Multi-agent artificial intelligence research promises a path to develop intelligent technologies that are more human-like and more human-compatible than those produced by "solipsistic" approaches, which do not consider interactions between agents. Melting Pot is a research tool developed to facilitate work on multi-agent artificial intelligence, and provides an evaluation protocol that measures generalization to novel social partners in a set of canonical test scenarios. Each scenario pairs a physical environment (a "substrate") with a reference set of co-players (a "background population"), to create a social situation with substantial interdependence between the individuals involved. For instance, some scenarios were inspired by institutional-economics-based accounts of natural resource management and public-good-provision dilemmas. Others were inspired by considerations from evolutionary biology, game theory, and artificial life. Melting Pot aims to cover a maximally diverse set of interdependencies and incentives. It includes the commonly-studied extreme cases of perfectly-competitive (zero-sum) motivations and perfectly-cooperative (shared-reward) motivations, but does not stop with them. As in real-life, a clear majority of scenarios in Melting Pot have mixed incentives. They are neither purely competitive nor purely cooperative and thus demand successful agents be able to navigate the resulting ambiguity. Here we describe Melting Pot 2.0, which revises and expands on Melting Pot. We also introduce support for scenarios with asymmetric roles, and explain how to integrate them into the evaluation protocol. This report also contains: (1) details of all substrates and scenarios; (2) a complete description of all baseline algorithms and results. Our intention is for it to serve as a reference for researchers using Melting Pot 2.0.
△ Less
Submitted 30 October, 2023; v1 submitted 24 November, 2022;
originally announced November 2022.
-
Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibria
Authors:
Kavya Kopparapu,
Edgar A. Duéñez-Guzmán,
Jayd Matyas,
Alexander Sasha Vezhnevets,
John P. Agapiou,
Kevin R. McKee,
Richard Everett,
Janusz Marecki,
Joel Z. Leibo,
Thore Graepel
Abstract:
A key challenge in the study of multiagent cooperation is the need for individual agents not only to cooperate effectively, but to decide with whom to cooperate. This is particularly critical in situations when other agents have hidden, possibly misaligned motivations and goals. Social deduction games offer an avenue to study how individuals might learn to synthesize potentially unreliable informa…
▽ More
A key challenge in the study of multiagent cooperation is the need for individual agents not only to cooperate effectively, but to decide with whom to cooperate. This is particularly critical in situations when other agents have hidden, possibly misaligned motivations and goals. Social deduction games offer an avenue to study how individuals might learn to synthesize potentially unreliable information about others, and elucidate their true motivations. In this work, we present Hidden Agenda, a two-team social deduction game that provides a 2D environment for studying learning agents in scenarios of unknown team alignment. The environment admits a rich set of strategies for both teams. Reinforcement learning agents trained in Hidden Agenda show that agents can learn a variety of behaviors, including partnering and voting without need for communication in natural language.
△ Less
Submitted 5 January, 2022;
originally announced January 2022.
-
Statistical discrimination in learning agents
Authors:
Edgar A. Duéñez-Guzmán,
Kevin R. McKee,
Yiran Mao,
Ben Coppin,
Silvia Chiappa,
Alexander Sasha Vezhnevets,
Michiel A. Bakker,
Yoram Bachrach,
Suzanne Sadedin,
William Isaac,
Karl Tuyls,
Joel Z. Leibo
Abstract:
Undesired bias afflicts both human and algorithmic decision making, and may be especially prevalent when information processing trade-offs incentivize the use of heuristics. One primary example is \textit{statistical discrimination} -- selecting social partners based not on their underlying attributes, but on readily perceptible characteristics that covary with their suitability for the task at ha…
▽ More
Undesired bias afflicts both human and algorithmic decision making, and may be especially prevalent when information processing trade-offs incentivize the use of heuristics. One primary example is \textit{statistical discrimination} -- selecting social partners based not on their underlying attributes, but on readily perceptible characteristics that covary with their suitability for the task at hand. We present a theoretical model to examine how information processing influences statistical discrimination and test its predictions using multi-agent reinforcement learning with various agent architectures in a partner choice-based social dilemma. As predicted, statistical discrimination emerges in agent policies as a function of both the bias in the training population and of agent architecture. All agents showed substantial statistical discrimination, defaulting to using the readily available correlates instead of the outcome relevant features. We show that less discrimination emerges with agents that use recurrent neural networks, and when their training environment has less bias. However, all agent algorithms we tried still exhibited substantial bias after learning in biased training populations.
△ Less
Submitted 21 October, 2021;
originally announced October 2021.
-
DeepMind Lab2D
Authors:
Charles Beattie,
Thomas Köppe,
Edgar A. Duéñez-Guzmán,
Joel Z. Leibo
Abstract:
We present DeepMind Lab2D, a scalable environment simulator for artificial intelligence research that facilitates researcher-led experimentation with environment design. DeepMind Lab2D was built with the specific needs of multi-agent deep reinforcement learning researchers in mind, but it may also be useful beyond that particular subfield.
We present DeepMind Lab2D, a scalable environment simulator for artificial intelligence research that facilitates researcher-led experimentation with environment design. DeepMind Lab2D was built with the specific needs of multi-agent deep reinforcement learning researchers in mind, but it may also be useful beyond that particular subfield.
△ Less
Submitted 12 December, 2020; v1 submitted 13 November, 2020;
originally announced November 2020.
-
Model-free conventions in multi-agent reinforcement learning with heterogeneous preferences
Authors:
Raphael Köster,
Kevin R. McKee,
Richard Everett,
Laura Weidinger,
William S. Isaac,
Edward Hughes,
Edgar A. Duéñez-Guzmán,
Thore Graepel,
Matthew Botvinick,
Joel Z. Leibo
Abstract:
Game theoretic views of convention generally rest on notions of common knowledge and hyper-rational models of individual behavior. However, decades of work in behavioral economics have questioned the validity of both foundations. Meanwhile, computational neuroscience has contributed a modernized 'dual process' account of decision-making where model-free (MF) reinforcement learning trades off with…
▽ More
Game theoretic views of convention generally rest on notions of common knowledge and hyper-rational models of individual behavior. However, decades of work in behavioral economics have questioned the validity of both foundations. Meanwhile, computational neuroscience has contributed a modernized 'dual process' account of decision-making where model-free (MF) reinforcement learning trades off with model-based (MB) reinforcement learning. The former captures habitual and procedural learning while the latter captures choices taken via explicit planning and deduction. Some conventions (e.g. international treaties) are likely supported by cognition that resonates with the game theoretic and MB accounts. However, convention formation may also occur via MF mechanisms like habit learning; though this possibility has been understudied. Here, we demonstrate that complex, large-scale conventions can emerge from MF learning mechanisms. This suggests that some conventions may be supported by habit-like cognition rather than explicit reasoning. We apply MF multi-agent reinforcement learning to a temporo-spatially extended game with incomplete information. In this game, large parts of the state space are reachable only by collective action. However, heterogeneity of tastes makes such coordinated action difficult: multiple equilibria are desirable for all players, but subgroups prefer a particular equilibrium over all others. This creates a coordination problem that can be solved by establishing a convention. We investigate start-up and free rider subproblems as well as the effects of group size, intensity of intrinsic preference, and salience on the emergence dynamics of coordination conventions. Results of our simulations show agents establish and switch between conventions, even working against their own preferred outcome when doing so is necessary for effective coordination.
△ Less
Submitted 14 December, 2020; v1 submitted 18 October, 2020;
originally announced October 2020.
-
D3C: Reducing the Price of Anarchy in Multi-Agent Learning
Authors:
Ian Gemp,
Kevin R. McKee,
Richard Everett,
Edgar A. Duéñez-Guzmán,
Yoram Bachrach,
David Balduzzi,
Andrea Tacchetti
Abstract:
In multiagent systems, the complex interaction of fixed incentives can lead agents to outcomes that are poor (inefficient) not only for the group, but also for each individual. Price of anarchy is a technical, game-theoretic definition that quantifies the inefficiency arising in these scenarios -- it compares the welfare that can be achieved through perfect coordination against that achieved by se…
▽ More
In multiagent systems, the complex interaction of fixed incentives can lead agents to outcomes that are poor (inefficient) not only for the group, but also for each individual. Price of anarchy is a technical, game-theoretic definition that quantifies the inefficiency arising in these scenarios -- it compares the welfare that can be achieved through perfect coordination against that achieved by self-interested agents at a Nash equilibrium. We derive a differentiable, upper bound on a price of anarchy that agents can cheaply estimate during learning. Equipped with this estimator, agents can adjust their incentives in a way that improves the efficiency incurred at a Nash equilibrium. Agents do so by learning to mix their reward (equiv. negative loss) with that of other agents by following the gradient of our derived upper bound. We refer to this approach as D3C. In the case where agent incentives are differentiable, D3C resembles the celebrated Win-Stay, Lose-Shift strategy from behavioral game theory, thereby establishing a connection between the global goal of maximum welfare and an established agent-centric learning rule. In the non-differentiable setting, as is common in multiagent reinforcement learning, we show the upper bound can be reduced via evolutionary strategies, until a compromise is reached in a distributed fashion. We demonstrate that D3C improves outcomes for each agent and the group as a whole on several social dilemmas including a traffic network exhibiting Braess's paradox, a prisoner's dilemma, and several multiagent domains.
△ Less
Submitted 20 February, 2022; v1 submitted 1 October, 2020;
originally announced October 2020.
-
Social diversity and social preferences in mixed-motive reinforcement learning
Authors:
Kevin R. McKee,
Ian Gemp,
Brian McWilliams,
Edgar A. Duéñez-Guzmán,
Edward Hughes,
Joel Z. Leibo
Abstract:
Recent research on reinforcement learning in pure-conflict and pure-common interest games has emphasized the importance of population heterogeneity. In contrast, studies of reinforcement learning in mixed-motive games have primarily leveraged homogeneous approaches. Given the defining characteristic of mixed-motive games--the imperfect correlation of incentives between group members--we study the…
▽ More
Recent research on reinforcement learning in pure-conflict and pure-common interest games has emphasized the importance of population heterogeneity. In contrast, studies of reinforcement learning in mixed-motive games have primarily leveraged homogeneous approaches. Given the defining characteristic of mixed-motive games--the imperfect correlation of incentives between group members--we study the effect of population heterogeneity on mixed-motive reinforcement learning. We draw on interdependence theory from social psychology and imbue reinforcement learning agents with Social Value Orientation (SVO), a flexible formalization of preferences over group outcome distributions. We subsequently explore the effects of diversity in SVO on populations of reinforcement learning agents in two mixed-motive Markov games. We demonstrate that heterogeneity in SVO generates meaningful and complex behavioral variation among agents similar to that suggested by interdependence theory. Empirical results in these mixed-motive dilemmas suggest agents trained in heterogeneous populations develop particularly generalized, high-performing policies relative to those trained in homogeneous populations.
△ Less
Submitted 12 February, 2020; v1 submitted 6 February, 2020;
originally announced February 2020.
-
Evolving intrinsic motivations for altruistic behavior
Authors:
Jane X. Wang,
Edward Hughes,
Chrisantha Fernando,
Wojciech M. Czarnecki,
Edgar A. Duenez-Guzman,
Joel Z. Leibo
Abstract:
Multi-agent cooperation is an important feature of the natural world. Many tasks involve individual incentives that are misaligned with the common good, yet a wide range of organisms from bacteria to insects and humans are able to overcome their differences and collaborate. Therefore, the emergence of cooperative behavior amongst self-interested individuals is an important question for the fields…
▽ More
Multi-agent cooperation is an important feature of the natural world. Many tasks involve individual incentives that are misaligned with the common good, yet a wide range of organisms from bacteria to insects and humans are able to overcome their differences and collaborate. Therefore, the emergence of cooperative behavior amongst self-interested individuals is an important question for the fields of multi-agent reinforcement learning (MARL) and evolutionary theory. Here, we study a particular class of multi-agent problems called intertemporal social dilemmas (ISDs), where the conflict between the individual and the group is particularly sharp. By combining MARL with appropriately structured natural selection, we demonstrate that individual inductive biases for cooperation can be learned in a model-free way. To achieve this, we introduce an innovative modular architecture for deep reinforcement learning agents which supports multi-level selection. We present results in two challenging environments, and interpret these in the context of cultural and ecological evolution.
△ Less
Submitted 11 March, 2019; v1 submitted 14 November, 2018;
originally announced November 2018.
-
Inequity aversion improves cooperation in intertemporal social dilemmas
Authors:
Edward Hughes,
Joel Z. Leibo,
Matthew G. Phillips,
Karl Tuyls,
Edgar A. Duéñez-Guzmán,
Antonio García Castañeda,
Iain Dunning,
Tina Zhu,
Kevin R. McKee,
Raphael Koster,
Heather Roff,
Thore Graepel
Abstract:
Groups of humans are often able to find ways to cooperate with one another in complex, temporally extended social dilemmas. Models based on behavioral economics are only able to explain this phenomenon for unrealistic stateless matrix games. Recently, multi-agent reinforcement learning has been applied to generalize social dilemma problems to temporally and spatially extended Markov games. However…
▽ More
Groups of humans are often able to find ways to cooperate with one another in complex, temporally extended social dilemmas. Models based on behavioral economics are only able to explain this phenomenon for unrealistic stateless matrix games. Recently, multi-agent reinforcement learning has been applied to generalize social dilemma problems to temporally and spatially extended Markov games. However, this has not yet generated an agent that learns to cooperate in social dilemmas as humans do. A key insight is that many, but not all, human individuals have inequity averse social preferences. This promotes a particular resolution of the matrix game social dilemma wherein inequity-averse individuals are personally pro-social and punish defectors. Here we extend this idea to Markov games and show that it promotes cooperation in several types of sequential social dilemma, via a profitable interaction with policy learnability. In particular, we find that inequity aversion improves temporal credit assignment for the important class of intertemporal social dilemmas. These results help explain how large-scale cooperation may emerge and persist.
△ Less
Submitted 27 September, 2018; v1 submitted 23 March, 2018;
originally announced March 2018.
-
Dynamics of alliance formation and the egalitarian revolution
Authors:
Sergey Gavrilets,
Edgar A. Duenez-Guzman,
Michael D. Vose
Abstract:
Arguably the most influential force in human history is the formation of social coalitions and alliances (i.e., long-lasting coalitions) and their impact on individual power. In most great ape species, coalitions occur at individual and group levels and among both kin and non-kin. Nonetheless, ape societies remain essentially hierarchical, and coalitions rarely weaken social inequality. In contr…
▽ More
Arguably the most influential force in human history is the formation of social coalitions and alliances (i.e., long-lasting coalitions) and their impact on individual power. In most great ape species, coalitions occur at individual and group levels and among both kin and non-kin. Nonetheless, ape societies remain essentially hierarchical, and coalitions rarely weaken social inequality. In contrast, human hunter-gatherers show a remarkable tendency to egalitarianism, and human coalitions and alliances occur not only among individuals and groups, but also among groups of groups. Here, we develop a stochastic model describing the emergence of networks of allies resulting from within-group competition for status or mates between individuals utilizing dyadic information. The model shows that alliances often emerge in a phase transition-like fashion if the group size, awareness, aggressiveness, and persuasiveness of individuals are large and the decay rate of individual affinities is small. With cultural inheritance of social networks, a single leveling alliance including all group members can emerge in several generations. Our results suggest that a rapid transition from a hierarchical society of great apes to an egalitarian society of hunter-gatherers (often referred to as "egalitarian revolution") could indeed follow an increase in human cognitive abilities. The establishment of stable group-wide egalitarian alliances creates conditions promoting the origin of cultural norms favoring the group interests over those of individuals.
△ Less
Submitted 1 September, 2008;
originally announced September 2008.
