Towards Principled Superhuman AI for Multiplayer Symmetric Games

For any set $\mathcal{A}$, its cardinality is represented by $|\mathcal{A}|$, and $\Delta(\mathcal{A})$ denotes a probability distribution over $\mathcal{A}$. We employ $\mathcal{A}^{\otimes n}$ to denote the Cartesian product of $n$ instances of $\mathcal{A}$. Given a distribution $x$ over $\mathcal{A}$, $x^{\otimes n}$ represents the joint distribution of $n$ independent copies of $x$, forming a distribution over $\mathcal{A}^{\otimes n}$. For a function $f:\mathcal{A}\rightarrow\mathbb{R}$, we denote $\|f\|_{\infty}:=\max_{a\in\mathcal{A}}|f(a)|$. We use $[n]$ to denote the set $\{1,\ldots,n\}$. In this paper, we use $C$ to denote universal constants, which may vary from line to line.

An $n$-player normal-form game consists of a finite set of $n$ players, where each player has an action space $\mathcal{A}_{i}$ and a corresponding payoff function $U_{i}:\mathcal{A}_{1}\times\cdots\times\mathcal{A}_{n}\to[-1,1]$ with $U_{i}(a_{1},\ldots,a_{n})$ denotes the payoff received by the $i$-th player if $n$ players are taking joint actions $(a_{1},\ldots,a_{n})$. This paper focuses on symmetric and zero-sum normal-form games.⁴⁴4Our work trivially extends to symmetric and constant-sum normal-form games, since one can convert constant-sum games to zero-sum games by offsetting all players’ payoff functions by a constant.

Briefly speaking, in a symmetric game, the payoffs for employing a specific action are determined solely by the actions used by others, agnostic of the identities of the players using them. Thus, the payoff function of the first player, denoted as $U_{1}$, is sufficient to encapsulate the entire game. For $i\in[n]$, $x_{i}\in\Delta(\mathcal{A})$ denotes a mixed strategy of the $i$-th player and $x_{-i}\in\Delta(\mathcal{A}^{\otimes n-1})$ denotes a mixed strategy of the other players. Correspondingly, $a_{i}\in\mathcal{A}$ denotes the action of the $i$-th player and $a_{-i}\in\mathcal{A}^{\otimes n-1}$ denotes the action of the other players. For any $a_{i}\in\mathcal{A}$, we denote $U_{i}(a_{i},x_{-i}):=\mathbb{E}_{a_{-i}\sim x_{-i}}\left[U_{i}(a_{i},a_{-i})\right].$

Given a mixed strategy $x_{-i}$ of the other $n-1$ players, the best response set ${\rm BR}_{i}(x_{-i})$ of the $i$-th player is defined as ${\rm BR}_{i}(x_{-i}):={\arg\max}_{a_{i}\in\mathcal{A}_{i}}U_{i}(a_{i},x_{-i})$.

Nash equilibrium is the most commonly-used solution concept for games: a mixed strategy $x\in\Delta(\mathcal{A}_{1}\times\cdots\times\mathcal{A}_{n})$ of all players is said to be Nash Equilibrium (NE) if $x$ is a product distribution, and no player could gain by deviating her own strategy while holding all other players’ strategies fixed. That is, for all $i\in[n]$ and $a^{\prime}_{i}\in\mathcal{A}_{i}$, $\mathbb{E}_{a\sim x}[U_{i}(a_{i},a_{-i})]\geq\mathbb{E}_{a\sim x}[U_{i}(a^{\prime}_{i},a_{-i})]$.

There are also two equilibrium notions relaxing the notion of Nash equilibrium by no longer requiring $x$ to be a product distribution. It allows general joint distribution $x$ which describes correlated policies among players. In particular, (1) $x$ is a Correlated Equilibrium (CE) if for all $i\in[n]$ and $a^{\prime}_{i}\in\mathcal{A}_{i}$, $\mathbb{E}_{a\sim x}[U_{i}(a_{i},a_{-i})\mid a_{i}]\geq\mathbb{E}_{a\sim x}[U_{i}(a^{\prime}_{i},a_{-i})\mid a_{i}]$, and (2) $x$ is Coarse Correlated Equilibrium (CCE) if for all $i\in[n]$ and $a^{\prime}_{i}\in\mathcal{A}_{i}$: $\mathbb{E}_{a\sim x}[U_{i}(a_{i},a_{-i})]\geq\mathbb{E}_{a\sim x}[U_{i}(a^{\prime}_{i},a_{-i})]$. The major difference between those two notions is in the cases when the agent deviates from her current strategy, whether she is still allowed to observe the randomness in drawing actions from the correlated policy. The relationship among various equilibrium concepts is encapsulated by ${\rm NE}\subset{\rm CE}\subset{\rm CCE}$.

We note in two-player zero-sum games, a Nash equilibrium is non-exploitable. In math, if $(\mu^{\star},\nu^{\star})$ is the Nash equilibrium, we have $\min_{\nu}U_{1}(\mu^{\star},\nu)=\max_{\mu}\min_{\nu}U_{1}(\mu,\nu)=0$.

No-regret learning is a commonly adopted strategy in game theory to find equilibrium solutions. We consider a $T$-step learning procedure, where for each round $t\in[T]$: (1) the agent picks a mixed strategy $\mu^{t}$ over $\mathcal{A}$, (2) the environment picks an adversarial loss $\ell_{t}\in[0,1]^{|\mathcal{A}|}$. The expected utility for $t$-th round is defined as $-\langle\mu^{t},\ell_{t}\rangle$. To measure the performance of a particular algorithm, a common approach is to consider the regret, where the algorithm’s performance is compared against the single best action in hindsight. Specifically, for policy sequence $(\mu^{1},\ldots,\mu^{T})$ taken by an algorithm, we define

$${\text{Reg}(T)}:=\sum_{t=1}^{T}\langle\mu^{t},\ell_{t}\rangle-\min_{a\in\mathcal{A}}\sum^{T}_{t=1}\ell_{t}(a),$$

We say that the algorithm is a no-regret algorithm if ${\text{Reg}(T)}=o(T)$. One of such no-regret learning algorithms is Hedge algorithm, which performs the following exponential weight updates:

$$\mu^{t+1}(a)\propto\mu^{t}(a)e^{-\eta_{t}\ell_{t}(a)},\quad\text{for}\quad\forall a\in\mathcal{A}.$$

(1)

where $\eta_{t}$ is the learning rate.

It is known that in the normal-form game setup, if each player independently plays a no-regret learning algorithm treating all other players as adversarial environments, and we denote $x^{t}=x_{1}^{t}\times\cdots\times x_{n}^{t}$ as the joint strategy by all players at round $t$, then we have the average policy $(1/T)\sum_{t=1}^{T}x^{t}$ converges to CCEs when $T$ is large.

In this setup, both strategies $(0,0,0)$ and $(1,1,1)$ constitute NE. However, the existence of multiple NEs creates a predicament for the AI agent. It must choose which equilibrium to follow, yet there’s always the risk that the other two players may coordinate on the opposing NE, leading to a negative payoff for the AI agent. In other words, adhering to a single NE does not reliably ensure an average payoff when multiple equilibria exist. Since ${\rm NE}\subset{\rm CE}\subset{\rm CCE}$, we know the same limitation also holds for CE and CCE.

Self-play is a training method in which an AI agent improves its performance by repeatedly playing against copies of itself without human supervision. Typical self-play is carried out by optimizing one agent’s policy (denoted as the main agent) while fixing all other agents’ policies to be the current policy or the earlier versions of the main agent. When no-regret learning algorithms such as Hedge are adopted as the optimizer, the self-play algorithm is guaranteed to converge to a CCE. Moreover, it may converge to a certain “good” subclass of CCEs depending on the choice of specific optimizer and initialization.

Here, we argue that self-play from scratch (Brown and Sandholm,, 2019) fails to always achieve non-negative expected payoff in multiplayer symmetric zero-sum games. The example is again the three-player majority vote game. Imagine two parallel worlds, with two other players always playing 1 in one world, and 0 in the other. The learner must play the exact same action as the other two players to receive non-negative expected payoff. Noting that self-play from scratch does not check other players’ actual policy, the learner has no clue which world it is, and thus loses in at least one of the two worlds.

We note that several recent systems (Li et al.,, 2020; Jacob et al.,, 2022) combine self-play with human policy modeling which avoids the drawback of self-play from scratch. However, our experiments demonstrate that many of these solutions remain ineffective even in some simple normal-form games (See Section 6).

We start by considering the general cases where opponents can possibly deploy different strategies.

Here, we define “no collusion” by requiring the policies of other players to be statistically independent without any shared randomness.

This observation underscores a crucial aspect of the multiplayer symmetric game dynamic where diverse strategies by opponents can drastically affect the possibility of achieving an equal share.

Claim 4.1 leaves out no choice but to consider the setting where opponents must deploy identical strategy. While it may seem limiting to assume that opponents employ identical strategies, we argue that this assumption is actually already implicit in the majority of practical systems (Li et al.,, 2020; Brown and Sandholm,, 2019; Bakhtin et al.,, 2022; † et al., 2022, FAIR). These systems utilize a self-play framework, equating the strategies of all opponents with those of the learner. Identical strategy assumption is also naturally satisfied in modern multiplayer gaming platforms with a large player base (See Section 4.1).

After making the strategies of opponents equal, one may wonder whether we can find a fixed non-exploitable policy similar to the case of two-player zero-sum games.

This observation again makes multiplayer games significantly different from two-player zero-sum games. This leads to the following conclusion:

To reliably secure an equal share in symmetric games, an agent must be in a scenario where: (a) opponents employ the identical strategy, and (b) the AI agent adapts its strategy to the opponents’ strategies, which involves opponent modeling.

It is not difficult to find solution concepts—such as (1) behavior cloning for the strategy by opponents; and (2) best response to the identical strategy by opponents—that satisfy both criteria, and are guaranteed to secure an equal share.

Finally, we rigorously summarize our discussion using mathematical language, detailing the relationships between various minimax quantities.

We first consider the standard majority vote game (Example 1) with $3$ players. A payoff of 0.5 is evenly distributed among the majority, and a payoff of -0.5 is evenly distributed among the minority. Note that everyone receives 0 if choosing the same action. Here the mixed strategy of the human population is assumed to be $\pi_{{\text{human}}}=[0.49,0.51]$. It can be seen that [1, 0], [0, 1], and [1/2, 1/2] are all NE.

In each round, players simultaneously choose an action from actions $A$, $B$, or $C$. Let $N$ be the total number of players and $n_{A}$ be the number of agents choosing action $A$, We define the game rule as:

$$\begin{cases}B\succ A\succ C&\text{if }n_{A}>0.2N,\\ C\succ B\succ A&\text{otherwise },\end{cases}$$

where the rule $i\succ j\succ k$ intuitively means that action $i$ dominates both $j$ and $k$, and action $j$ dominates $k$. SDG is designed so that $C$ is a dominated action when there is a reasonable number of players taking action $A$, but a dominating action otherwise. Concretely, for $i\succ j\succ k$, we use the payoff defined as:

	$\displaystyle r_{i}$	$\displaystyle=\mathbb{I}[n_{j}+n_{k}>0],$
	$\displaystyle r_{j}$	$\displaystyle=\mathbb{I}[n_{k}>0]-\mathbb{I}[n_{j}+n_{k}>0]\frac{n_{i}}{n_{j}+n_{k}},$
	$\displaystyle r_{k}$	$\displaystyle=-\mathbb{I}[n_{j}+n_{k}>0]\frac{n_{i}}{n_{j}+n_{k}}-\mathbb{I}[n_{k}>0]\frac{n_{j}}{n_{k}}.$

This payoff design guarantees that SDG is a zero-sum game. Throughout our experiments, we choose $N=30$ and the human meta-strategy $\pi_{{\text{human}}}=[0.399,0.6,0.001]$ (in the order of $A$, $B$, and $C$). Note that while this game has an NE policy [0, 0, 1], its utility is negative against all other players playing $\pi_{{\text{human}}}$.