Networked Communication for Decentralised Agents in Mean-Field Games

Patrick Benjamin
Department of Computer Science
University of Oxford
Oxford, United Kingdom
[email protected]
&Alessandro Abate
Department of Computer Science
University of Oxford
Oxford, United Kingdom
[email protected]

Abstract

We introduce networked communication to the mean-field game framework, in particular to oracle-free settings where $N$ decentralised agents learn along a single, non-episodic run of the empirical system. We prove that our architecture, with only a few reasonable assumptions about network structure, has sample guarantees bounded between those of the centralised- and independent-learning cases. We discuss how the sample guarantees of the three theoretical algorithms do not actually result in practical convergence. We therefore show that in practical settings where the theoretical parameters are not observed (leading to poor estimation of the Q-function), our communication scheme significantly accelerates convergence over the independent case (and often even the centralised case), without relying on the assumption of a centralised learner. We contribute further practical enhancements to all three theoretical algorithms, allowing us to present their first empirical demonstrations. Our experiments confirm that we can remove several of the theoretical assumptions of the algorithms, and display the empirical convergence benefits brought by our new networked communication. We additionally show that the networked approach has significant advantages, over both the centralised and independent alternatives, in terms of robustness to unexpected learning failures and to changes in population size.

1 Introduction

The mean-field game (MFG) framework [1, 2] seeks to tackle the difficulty faced by multi-agent reinforcement learning (MARL) regarding computational scalability as the number of agents increases. It models a representative agent as interacting not with the other individuals in the population on a per-agent basis, but instead with a distribution of other agents, known as the mean field. The MFG framework analyses the limiting case when the population consists of an infinite number of symmetric and anonymous agents, that is, they have identical reward and transition functions which depend on the mean-field distribution rather than on the actions of specific other players. In this work we focus on MFGs with stationary population distributions (‘stationary MFGs’) [3, 4, 5, 6], for which the solution concept is the MFG-Nash equilibrium (MFG-NE), which reflects the situation when each agent responds optimally to the population distribution that arises when all other agents follow that same optimal behaviour. The MFG-NE can be used as an approximation for the Nash equilibrium (NE) in a finite-agent game, with the error in the solution reducing as the number of agents N tends to infinity [4, 7, 8, 9, 10]. MFGs have been applied to a wide variety of real world problems [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28]; see Appx. F for further details.

For large, complex many-agent systems in the real world (e.g. swarm robotics or autonomous vehicle traffic), it may not be feasible to find MFG-NEs analytically or via oracles/simulations of an infinite population (as they have been traditionally), such that learning must instead be conducted directly by the finite population in its deployed environment. In such settings, in contrast to many previous approaches, desirable qualities for MFG algorithms include: learning from the empirical distribution of $N$ agents (without generation or manipulation of this distribution by the algorithm itself or by an external oracle); learning from a single continued system run that is not arbitrarily reset as in episodic learning; model-free learning; decentralisation; fast practical convergence; and robustness to unexpected failures of decentralised learners or changes in the size of the population [29].

Conversely, MFG frameworks have traditionally been largely theoretical, and methods for finding equilibria have often relied on assumptions that are too strong for real-world applications [2, 4, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41]; see Appx. F for an extended discussion of this related work. In particular, almost all prior work relies on a centralised controller to orchestrate the learning of all agents [3, 4, 5, 32, 42]. However, outside of MFGs, the multi-agent systems community has recognised that the existence of a central controller is a very strong assumption, as well as one that can both restrict scalability by constituting a bottleneck for computation and communication, and reveal a single point of failure for the whole system [43, 44, 45, 46, 47, 48]. For example, if the single server coordinating all the autonomous vehicles in a smart city were to crash, the entire road network would cease to operate. As an alternative, recent work has explored MFG algorithms for independent learning [6, 49, 50, 51, 52, 53, 54, 55]. However, prior works focus on theoretical sample guarantees rather than on practical convergence speed, and have largely not considered robustness in the senses we address, despite fault-tolerance being one of the original motivations behind many-agent systems.

We address all of these desiderata by novelly introducing a communication network to the MFG setting. Communication networks have had success in other multi-agent settings, removing the reliance on inflexible, centralised structures [43, 45, 46, 47, 56, 57, 58, 59, 60]. We focus on ‘coordination games’, i.e. where agents can increase their individual rewards by following the same strategy as others and therefore have an incentive to communicate policies, even if the MFG setting itself is technically non-cooperative. Thus our work can be applied to real-world problems in e.g. traffic signal control, formation control in swarm robotics, and consensus and synchronisation e.g. for sensor networks.

In this work, we show that when the agents’ state-action value functions (Q-functions) can be only roughly estimated due to fewer samples/updates, possibly leading to high variance in policy updates, then propagating policies that are estimated to be better through the population via the communication network leads to faster convergence than that achieved by agents learning entirely independently. This is crucial in large complex environments that may be encountered in real applications, where the idealised hyperparameter choices (such as learning rates and numbers of iterations) required in previous works for theoretical convergence guarantees will be infeasible in practice.¹¹1This work therefore also serves as a bridge with our ongoing work extending our algorithm with neural networks, for settings requiring (non-linear) function approximation. We compare our networked architecture with modified versions of earlier theoretical algorithms for the centralised and independent settings; we extend the original algorithms with experience replay buffers, without which we found them unable to converge in practical time. While the use of buffers means that the original theoretical sample guarantees no longer apply, we argue that this is preferable since these guarantees were in any case impractical. On this basis, we conduct numerical comparisons of the three architectures, demonstrating the benefits of communication for both convergence speed and system robustness. For further discussion of how networked communication can benefit robustness in large multi-agent systems, see Appx. D. In summary, our key contributions include the following:

•

We prove that a theoretical version of our new networked algorithm (Alg. 1) has sample guarantees bounded between those of the centralised and independent settings for learning with a single, non-episodic run of the empirical system (Sec. 3.3).
•

All three theoretical algorithms do not permit convergence in practical time; we show that in practical settings our communication scheme can significantly benefit convergence speed over the independent case, and often even the centralised case (Sec. 3.4.1).
•

We novelly modify all three theoretical algorithms (Alg. 2) to make their practical convergence feasible, most notably by including an experience replay buffer, allowing us to contribute the first empirical demonstrations of all three algorithms (Sec. 3.4.2).
•

Our experiments demonstrate the convergence benefits brought by our networked communication, and show we can remove several of the algorithms’ theoretical assumptions (a goal shared by other work on the practicality of MFG algorithms [61]) (Sec. 4.1).
•

We further demonstrate that our decentralised communication architecture brings significant benefits over both the centralised and independent alternatives in terms of robustness to unexpected learning failures and changes in population size (Sec. 4.1).

The main paper is structured as follows: notation and preliminaries are given in Sec. 2; the theoretical algorithms and results are given in Sections 3.1-3.3; practical enhancements to the algorithms are given in Sec. 3.4; experiments and discussion in Sec. 4. Limitations are found in Appx. G.

2 Preliminaries

We use the following notation. $N$ is the number of agents in a population, with $\mathcal{S}$ and $\mathcal{A}$ representing the finite state and common action spaces, respectively. The sets $\mathcal{S}$ and $\mathcal{A}$ are equipped with the discrete metric $d(x,y)=\mathbb{1}_{x\neq y}$ . The set of probability measures on a finite set $\mathcal{X}$ is denoted $\Delta_{\mathcal{X}}$ , and $\mathbf{e}_{x}\in\Delta_{\mathcal{X}}$ for $x\in\mathcal{X}$ is a one-hot vector with only the entry corresponding to $x$ set to 1, and all others set to 0. For time $t\geq 0$ , $\hat{\mu}_{t}$ = $\frac{1}{N}\sum^{N}_{i=1}\sum_{s\in\mathcal{S}}$ $\mathbb{1}_{s^{i}_{t}=s}\mathbf{e}_{s}$ $\in$ $\Delta_{\mathcal{S}}$ is a vector denoting the empirical state distribution of the $N$ agents at time $t$ . The set of policies is $\Pi$ = { $\pi$ : $\mathcal{S}\rightarrow\Delta_{\mathcal{A}}$ }, and the set of Q-functions is denoted $\mathcal{Q}=\{q:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}\}$ . For $\pi,\pi^{\prime}\in\Pi$ and $q,q^{\prime}\in\mathcal{Q}$ , we have the norms $||\pi-\pi^{\prime}||_{1}$ := $\sup_{s\in\mathcal{S}}||\pi(s)-\pi^{\prime}(s)||_{1}$ and $||q-q^{\prime}||_{\infty}$ := $\sup_{s\in\mathcal{S},a\in\mathcal{A}}|q(s,a)-q^{\prime}(s,a)|$ .

Function $h:\Delta_{\mathcal{A}}\rightarrow\mathbb{R}_{\geq 0}$ denotes a strongly concave function, which we implement as the scaled entropy regulariser $\lambda h_{ent}(u)=-\lambda\sum_{a}u(a)\log u(a)$ , for $a\in\mathcal{A}$ , $u\in\Delta_{\mathcal{A}}$ and $\lambda>0$ . As in some earlier works [4, 6, 40, 62, 63, 64], regularisation is theoretically required to ensure the contractivity of operators and continued exploration, and hence algorithmic convergence. However, it has been recognised that modifying the RL objective in this way can bias the NE [6, 10, 30, 65]. We show in our experiments that we are able to reduce $\lambda$ to 0 with no detriment to convergence.

Definition 1 (N-player symmetric anonymous games).

An N-player stochastic game with symmetric, anonymous agents is given by the tuple $\langle$ $N$ , $\mathcal{S}$ , $\mathcal{A}$ , $P$ , $R$ , $\gamma$ $\rangle$ , where $\mathcal{A}$ is the action space, identical for each agent; $\mathcal{S}$ is the identical state space of each agent, such that their initial states are { $s^{i}_{0}$ } ${}_{i=1}^{N}\in\mathcal{S}^{N}$ and their policies are { $\pi^{i}$ } ${}_{i=1}^{N}\in\Pi^{N}$ . $P$ : $\mathcal{S}$ $\times$ $\mathcal{A}$ $\times$ $\Delta_{\mathcal{S}}$ $\rightarrow$ $\Delta_{\mathcal{S}}$ is the transition function and $R$ : $\mathcal{S}$ $\times$ $\mathcal{A}$ $\times$ $\Delta_{\mathcal{S}}$ $\rightarrow$ [0,1] is the reward function, which map each agent’s local state and action and the population’s empirical distribution to transition probabilities and bounded rewards, respectively, i.e.

s^{i}_{t+1}\sim P(\cdot|s^{i}_{t},a^{i}_{t},\hat{\mu}_{t}),\;\;\;r^{i}_{t}=R(s% ^{i}_{t},a^{i}_{t},\hat{\mu}_{t}),\;\;\;\forall i=1,\dots,N.\qed

The policy of an agent is given by $a^{i}_{t}\sim\pi^{i}(s^{i}_{t})$ , that is, each agent only observes its own state, and not the joint state or empirical distribution of the population.

Definition 2 (N-player discounted regularised return).

With joint policies $\boldsymbol{\pi}$ := ( $\pi^{1},\dots,\pi^{N}$ ) $\in\Pi^{N}$ , initial states sampled from a distribution $\upsilon_{0}\in\Delta_{\mathcal{S}}$ and $\gamma$ $\in$ [0,1) as a discount factor, the expected discounted regularised returns of each agent $i$ in the symmetric anonymous game are given by

$\Theta^{i}_{h}(\boldsymbol{\pi},\upsilon_{0})=\mathbb{E}\left[\sum^{\infty}_{t% =0}\gamma^{t}(R(s^{i}_{t},a^{i}_{t},\hat{\mu}_{t})+h(\pi^{i}(s^{i}_{t})))\bigg% {|}\begin{subarray}{c}s^{j}_{0}\sim\upsilon_{0}\\ a^{j}_{t}\sim\pi^{j}(s^{j}_{t})\\ s^{j}_{t+1}\sim P(\cdot|s^{j}_{t},a^{j}_{t},\hat{\mu}_{t})\end{subarray},% \forall t\geq 0,j\in\{1,\dots,N\}\right].\qed$

Definition 3 ( $\delta$ -NE).

Say $\delta>0$ and $(\pi,\boldsymbol{\pi}^{-i})$ := $(\pi^{1},\dots\pi^{i-1},\pi,\pi^{i+1},\dots,\pi^{N})\in\Pi^{N}$ . An initial distribution $\upsilon_{0}\in\Delta_{\mathcal{S}}$ and an $N$ -tuple of policies $\boldsymbol{\pi}$ := ( $\pi^{1},\dots,\pi^{N}$ ) $\in\Pi^{N}$ form a $\delta$ -NE ( $\boldsymbol{\pi}$ , $\upsilon_{0}$ ), if $\forall i$ = 1, $\dots,N$

\Theta^{i}_{h}(\boldsymbol{\pi},\upsilon_{0})\geq\max_{\pi\in\Pi}\Theta^{i}_{h% }((\pi,\boldsymbol{\pi}^{-i}),\upsilon_{0})-\delta.\qed

At the limit as $N\rightarrow\infty$ , the population of infinitely many agents can be characterised as a limit distribution $\mu\in\Delta_{\mathcal{S}}$ . We denote the expected discounted return of the representative agent in the infinite-agent game - termed an MFG - as $V$ , rather than $\Theta$ as in the finite $N$ -agent case.

Definition 4 (Mean-field discounted regularised return).

For a policy-population pair ( $\pi,\mu$ ) $\in\Pi\times\Delta_{\mathcal{S}}$ ,

V_{h}(\pi,\mu)=\mathbb{E}\left[\sum^{\infty}_{t=0}\gamma^{t}(R(s_{t},a_{t},\mu% )+h(\pi(s_{t})))\bigg{|}\begin{subarray}{c}s_{0}\sim\mu\\ a_{t}\sim\pi(s_{t})\\ s_{t+1}\sim P(\cdot|s_{t},a_{t},{\mu})\end{subarray}\right].\qed

A stationary MFG is one that has a unique population distribution that is stable with respect to a given policy, and the agents’ policies are not time-dependent.

Definition 5 (NE of stationary MFG).

For a policy $\pi^{*}\in\Pi$ and a population distribution $\mu^{*}\in\Delta_{\mathcal{S}}$ , the pair ( $\pi^{*},\mu^{*}$ ) is a stationary MFG-NE if the following optimality and stability conditions hold:

	$\displaystyle\textrm{optimality:}\quad V_{h}(\pi^{},\mu^{})=\max_{\pi}V_{h}(% \pi,\mu^{*}),$
	$\displaystyle\textrm{stability:}\quad\mu^{}(s)=\sum_{s^{\prime},a^{\prime}}% \mu^{}(s^{\prime})\pi^{}(a^{\prime}\|s^{\prime})P(s\|s^{\prime},a^{\prime},\mu% ^{}).$

If the optimality condition is only satisfied with $V_{h}(\pi^{*}_{\delta},\mu^{*}_{\delta})\geq\max_{\pi}V_{h}(\pi,\mu^{*}_{% \delta})-\delta$ , then ( $\pi^{*}_{\delta},\mu^{*}_{\delta}$ ) is a $\delta$ -NE of the MFG, where $\mu^{*}_{\delta}$ is obtained from the stability equation and $\pi^{*}_{\delta}$ . ∎

The MFG-NE is an approximate NE of the $N$ -player game, which is difficult to solve in itself [6, 30]:

Proposition 1 (N-player NE and MFG-NE (Thm. 1, [4])).

If ( $\pi^{*},\mu^{*}$ ) is a MFG-NE, then, under certain Lipschitz conditions [4], for any $\delta>0$ , there exists $N(\delta)\in\mathbb{N}_{>0}$ such that, for all $N\geq N(\delta)$ , the joint policy $\boldsymbol{\pi}=\{\pi^{*},\pi^{*},\dots,\pi^{*}\}\in\Pi^{N}$ is a $\delta$ -NE of the $N$ -player game. ∎

Remark 1.

It can be shown that $\delta$ can be characterised further in terms of $N$ , with ( $\pi^{*},\mu^{*}$ ) being an $\mathcal{O}$ ( $\frac{1}{\sqrt{N}}$ )-NE of the $N$ -player symmetric anonymous game [6].∎

For our new, networked learning algorithm, we also introduce the concept of a time-varying communication network, where the links between agents that make up the network may change at each time step $t$ . Most commonly we might think of such a network as depending on the spatial locations of decentralised agents, such as physical robots, which can communicate with neighbours that fall within a given broadcast radius. When the agents move in the environment, their neighbours and therefore communication links may change. However, the dynamic network can also depend on other factors that may or may not depend on each agent’s state $s^{i}_{t}$ . For example, even a network of fixed-location agents can change depending on which agents are active and broadcasting at a given time $t$ , or if their broadcast radius changes, perhaps in relation to signal or battery strength.

Definition 6 (Time-varying communication network).

The time-varying communication network { $\mathcal{G}_{t}$ }_t≥0 is given by $\mathcal{G}_{t}$ = ( $\mathcal{N},\mathcal{E}_{t}$ ), where $\mathcal{N}$ is the set of vertices each representing an agent $i=1,\dots,N$ , and the edge set $\mathcal{E}_{t}$ $\subseteq$ {(i,j) : i,j $\in$ $\mathcal{N}$ , i $\neq$ j} is the set of undirected communication links by which information can be shared at time t.∎

We say a network is connected if there is a sequence of distinct edges that form a path between each distinct pair of vertices. The union of a collection of graphs { $\mathcal{G}_{t},\mathcal{G}_{t+1},\cdots,\mathcal{G}_{t+\omega}\}$ (for $\omega\in\mathbb{N}$ ) is the graph with vertices and edge set equalling the union of the vertices and edge sets of the graphs in the collection [66]. The collection is jointly connected if the union of its members is connected.

3 Learning with networked, decentralised agents

Summary We first discuss theoretical versions of our operators and algorithm (Sections 3.1, 3.2) to show that our networked framework has sample guarantees bounded between those of the centralised- and independent-learning cases (Sec. 3.3). We then show that our novel incorporation of an experience replay buffer (Sec. 3.4.2), along with networked communication, means that empirically we can remove many of the theoretical assumptions and practically infeasible hyperparameter choices that are required by the sample guarantees of the theoretical algorithms, in which cases we demonstrate that our networked algorithm can significantly outperform the independent algorithm, and sometimes even the centralised one (Sec. 4).

3.1 Learning with $N$ agents from a single run

We begin by outlining the basic procedure for solving the MFG using the $N$ -agent empirical distribution and a single, continuous system run. The two underlying operators are the same for the centralised, independent and networked architectures; in the latter two cases all agents apply the operators individually, while in the centralised setting a single central agent estimates the Q-function and computes an updated policy that is pushed to all the other agents.

We define, for $h_{max}>0$ and $h:\Delta_{\mathcal{A}}\rightarrow[0,h_{max}]$ , $u_{max}\in\Delta_{\mathcal{A}}$ such that $h(u_{max})=h_{max}$ . We further define $q_{max}:=\frac{1+h_{max}}{1-\gamma}$ , and set $\pi_{max}\in\Pi$ such that $\pi_{max}(s)=u_{max},\forall s\in\mathcal{S}$ . For any $\Delta h\in\mathbb{R}_{>0}$ , we also define the convex set $\mathcal{U}_{\Delta h}$ := { $u\in\Delta_{\mathcal{A}}:h(u)\geq h_{max}-\Delta h\}$ .

Learning agents use the stochastic temporal difference (TD)-learning operator to repeatedly update an estimate of the Q-function of their current policy with respect to the current empirical distribution, i.e. to approximate the operator $\Gamma_{q}$ (Def. 12, Appx. A):

Definition 7 (Stochastic TD-learning operator, simplified from Definition 4.1 in Yardim et al. [6]).

We define $\mathcal{Z}:=\mathcal{S}\times\mathcal{A}\times[0,1]\times\mathcal{S}\times% \mathcal{A}$ , and say that $\zeta^{i}_{t}$ is the transition observed by agent $i$ at time $t$ , given by $\zeta^{i}_{t}=(s^{i}_{t},a^{i}_{t},r^{i}_{t},s^{i}_{t+1},a^{i}_{t+1})$ . The TD-learning operator $\tilde{F}^{\pi}_{\beta}:\mathcal{Q}\times\mathcal{Z}\rightarrow\mathcal{Q}$ is defined, for any $Q\in\mathcal{Q},\zeta_{t}\in\mathcal{Z},\beta\in\mathbb{R}$ , as

\tilde{F}^{\pi}_{\beta}(Q,\zeta_{t})=Q(s_{t},a_{t})-\beta\left(Q(s_{t},a_{t})-% r_{t}-h(\pi(s_{t}))-\gamma\left(Q(s_{t+1},a_{t+1})\right)\right).\qed

Having estimated the Q-function of their current policy, agents update this policy by selecting, for each state, a probability distribution over their actions that maximises the combination of three terms (Def. 8): 1. the value of the given state with respect to the estimated Q-function; 2. a regulariser over the action probability distribution (in practice, we maximise the scaled entropy of the distribution); 3. a metric of similarity between the new action probabilities for the given state and those of the previous policy, given by the squared two-norm of the difference between the two distributions. We can alter the importance of the similarity metric relative to the other two terms by varying a parameter $\eta$ , which is equivalent to changing the learning rate of the policy update. The three terms in the maximisation function can be seen in the policy mirror ascent (PMA) operator:

Definition 8 (Policy mirror ascent operator (Definition 3.5, [6])).

For a learning rate $\eta>0$ and $L_{h}:=L_{a}+\gamma\frac{L_{s}K_{a}}{2-\gamma K_{s}}$ (where these constants are defined in Assumption 1 in Appx. A), the PMA update operator $\Gamma^{md}_{\eta}:\mathcal{Q}\times\Pi\rightarrow\Pi$ is defined as

$\Gamma^{md}_{\eta}(Q,\pi)(s):=\underset{u\in\mathcal{U}_{L_{h}}}{\arg\max}% \left(\langle u,q(s,\cdot)\rangle+h(u)-\frac{1}{2\eta}||u-\pi(s)||^{2}_{2}% \right),\forall s\in\mathcal{S},\forall Q\in\mathcal{Q},\forall\pi\in\Pi.\qed$

The theoretical learning algorithm has three nested loops (see Lines 4, 6 and 7 of Alg. 1). The policy update is applied $K$ times. Before the policy update in each of the $K$ loops, agents update their estimate of the Q-function by applying the stochastic TD-learning operator $M_{pg}$ times. Prior to the TD update in each of the $M_{pg}$ loops, agents take $M_{td}$ steps in the environment without updating. The $M_{td}$ loops exist to create a delay between each TD update to reduce bias when using the empirical distribution to approximate the mean field [67]. However, we find in our experiments that we are able to essentially remove the inner $M_{td}$ loops (Sec. 4.1).

3.2 Decentralised communication between agents

In our novel algorithm Alg. 1, agents compute policy updates in a decentralised manner as in the independent case (Lines 5-12, before broadcasting their updated policy $\pi^{i}_{k+1}$ to their neighbours (Line 15). Agents also broadcast a value $\sigma^{i}_{k+1}$ that they generate associated with the policy. Agents have a certain broadcast radius, which defines the structure of the possibly time-varying communication network. Of the policies and associated values received by a given agent (including its own) (Line 16), the agent selects a $\sigma^{i}_{k+1}$ with a probability defined by a softmax function over the received values, and adopts the policy associated with this $\sigma^{i}_{k+1}$ (Lines 17, 18). This process continues for $C$ communication rounds, before the the Q-function estimation steps begin again. After each round, the agents take a step in the environment (Line 19), such that if the communication network is affected by the agents’ states, then agents that are unconnected from any others in a given communication round might become connected in the next. (In our experiments we set $C$ as 1 to show the benefits to convergence speed brought by even a single communication round.) We assume that the softmax function is subject to a possibly time-varying temperature parameter $\tau_{k}$ . We discuss the effects of the values of $C$ and $\tau_{k}$ , as well as the mechanism for generating $\sigma^{i}_{k+1}$ , in subsequent sections.

Remark 2.

Our networked architecture is in effect a generalisation of both the centralised and independent settings (Algorithms 2 and 3, Yardim et al. [6]). The independent setting is the special case where there are no communication rounds, i.e. $C=0$ . The centralised setting is the special case when the method for generating $\sigma^{i}_{k+1}$ involves a unique ID for each agent, with the central learner agent being assumed to generate the highest value by default. In this case we assume $\tau_{k}\rightarrow 0$ (such that the softmax becomes a max function), and that the communication network becomes jointly connected repeatedly, such that the central learner’s policy is always adopted by the entire population, assuming $C$ is large enough that the number of jointly connected collections of graphs occurring within $C$ is equal to the largest diameter of the union of any collection [68, 69]. ∎

Remark 3.

In practice, when referring in the following to a centralised version of the networked Alg. 1, for simplicity we assume that there is no communication and instead that the updated policy $\pi^{1}_{k+1}$ of the central learner $i=1$ is pushed to all other agents after Line 12, as in Alg. 2 of [6]. ∎

Algorithm 1 Networked learning with single system run

1:loop parameters

K,M_{pg},M_{td},C

, learning parameters

\eta,\{\beta_{m}\}_{m\in\{0,\dots,M_{pg}-1\}}

\lambda,\gamma

\{\tau_{k}\}_{k\in\{0,\dots,K-1\}}

2:initial states {

s^{i}_{0}

}_i,

i=1,\ldots,N

3:Set

\pi^{i}_{0}=\pi_{max},\forall i

and

t\leftarrow 0

4:for

k=0,\dots,K-1

\forall s,a,i:\hat{Q}^{i}_{0}(s,a)=Q_{max}

6: for

m=0,\dots,M_{pg}-1

7: for

M_{td}

iterations do

8: Take step

\forall i:a^{i}_{t}\sim\pi^{i}_{k}(\cdot|s^{i}_{t}),r^{i}_{t}=R(s^{i}_{t},a^{i% }_{t},\hat{\mu}_{t}),s^{i}_{t+1}\sim P(\cdot|s^{i}_{t},a^{i}_{t},\hat{\mu}_{t})

;

t\leftarrow t+1

9: end for

10: Compute TD update (

\forall i

\hat{Q}^{i}_{m+1}=\tilde{F}^{\pi^{i}_{k}}_{\beta_{m}}(\hat{Q}^{i}_{m},\zeta^{i% }_{t-2})

(see Def. 7)

11: end for

12: PMA step

\forall i:\pi^{i}_{k+1}=\Gamma^{md}_{\eta}(\hat{Q}^{i}_{M_{pg}},\pi^{i}_{k})

(see Def. 8)

13:

\forall i:

Generate

\sigma^{i}_{k+1}

associated with

\pi^{i}_{k+1}

14: for

C

rounds do

15:

\forall i:

Broadcast

\sigma^{i}_{k+1},\pi^{i}_{k+1}

16:

\forall i:J^{i}_{t}=i\cup\{j\in\mathcal{N}:(i,j)\in\mathcal{E}_{t}

}

17:

\forall i:

Select

\sigma^{adopted}_{k+1}\sim

\left(\sigma^{adopted}_{k+1}=\sigma^{j}_{k+1}\right)

\frac{\exp{(\sigma^{j}_{k+1}}/\tau_{k})}{\sum^{[J^{i}_{t}]}_{x=1}\exp{(\sigma^% {x}_{k+1}}/\tau_{k})}

\forall j\in J^{i}_{t}

18:

\forall i:\sigma^{i}_{k+1}=\sigma^{adopted}_{k+1},\pi^{i}_{k+1}=\pi^{adopted}_% {k+1}

19: Take step

\forall i:a^{i}_{t}\sim\pi^{i}_{k+1}(\cdot|s^{i}_{t}),r^{i}_{t}=R(s^{i}_{t},a^% {i}_{t},\hat{\mu}_{t}),s^{i}_{t+1}\sim P(\cdot|s^{i}_{t},a^{i}_{t},\hat{\mu}_{% t})

;

t\leftarrow t+1

20: end for

21:end for

22:Return policies {

\pi^{i}_{K}

}_i,

i=1,\ldots,N

3.3 Properties of policy adoption

We give two results comparing the sample guarantees of our networked case with those of the other settings; each result depends on how networked agents select which communicated policies to adopt.

Theorem 1 (Networked learning with random adoption).

Full version in Appx. B.2. Say that $\pi^{*}$ is the unique MFG-NE policy, and let $\varepsilon>0$ be an arbitrary value that reduces as $K$ increases. If we assume that $C>0$ , with $\tau_{k}\rightarrow\infty$ , then under certain technical conditions the random output { $\pi^{i}_{K}$ }_i of Alg. 1 preserves the sample guarantees of the independent-learning case given in Lemma 3, i.e. the output satisfies, for all agents $i=1,\dots,N$ , $\mathbb{E}\left[||\pi^{i}_{K}-\pi^{*}||_{1}\right]\;\leq\;\varepsilon+\mathcal% {O}\left(\frac{1}{\sqrt{N}}\right).$ ∎

Proof sketch. Random exchange of policies learnt in a decentralised manner does not change the expectation of the random output of the purely independent-learning setting i.e. where this exchange does not occur. Full proof in Appx. B.3.

Moreover we can show that if $\sigma^{i}_{k+1}$ is generated arbitrarily and uniquely for each $i$ , then for $\tau_{k}\in\mathbb{R}_{>0}$ , the sample complexity of the networked learning algorithm is bounded between that of the centralised and independent algorithms:

Theorem 2 (Networked learning with non-random adoption).

Full version in Appx. B.5. Assume that Alg. 1 is run as in Thm. 1, except now $\tau_{k}\in\mathbb{R}_{>0}$ . Assume also that $\sigma^{i}_{k+1}$ is generated uniquely for each $i$ , in a manner independent of any metric related to $\pi^{i}_{k+1}$ , e.g. $\sigma^{i}_{k+1}$ is random or related only to the index $i$ (so as not to bias the spread of any particular policy). Let the random output of this Algorithm be denoted as { $\pi^{i,net}_{K}$ }_i. Also consider an independent-learning version of the algorithm (i.e. with the same parameters except $C=0$ ) and denote its random output { $\pi^{j,ind}_{K}$ }_j; and a centralised version of the algorithm with the same parameters (see Rem. 3) and denote its random output as $\pi^{cent}_{K}$ . Then under certain technical conditions, for all agents $i=1,\dots,N$ and $j=1,\dots,N$ , the random outputs { $\pi^{i,net}_{K}$ }_i, { $\pi^{j,ind}_{K}$ }_j and $\pi^{cent}_{K}$ satisfy

\mathbb{E}\left[||\pi^{cent}_{K}-\pi^{*}||_{1}\right]\;\leq\;\mathbb{E}\left[|% |\pi^{i,net}_{K}-\pi^{*}||_{1}\right]\leq\;\mathbb{E}\left[||\pi^{j,ind}_{K}-% \pi^{*}||_{1}\right]\;\leq\;\varepsilon+\mathcal{O}\left(\frac{1}{\sqrt{N}}% \right).\qed

Proof sketch. Fewer distinct policies in the population means there is less bias in each learner’s estimation of the Q function. So methods for reducing the number of policies in the population (such as non-random adoption in the networked case) lead to faster convergence. Full proof in Appx. B.6.

Remark 4.

The expected sample complexity of the centralised algorithm is known to be significantly better than that of the independent case [6]. Where the expected sample complexity of the networked case lies on the spectrum between the centralised and independent cases depends on how much closer the divergence between the networked agents’ policies is to 0, as in the centralised case, than it is to the divergence in the independent case (Appx. B.6). This in turn depends on the interaction between the parameters $\tau_{k}$ and $C$ , and also on the structure of the time-varying network, i.e. how frequently it becomes jointly connected. See Rem. 6 in Appx. B.7 for further details. ∎

In Thm. 2, we assume $\sigma^{i}_{k+1}$ is generated independently of any metric related to $\pi^{i}_{k+1}$ ; e.g. $\sigma^{i}_{k+1}$ is random or related only to the index $i$ . Next we show that an appropriate generation of $\sigma^{i}_{k+1}$ dependent on $\pi^{i}_{k+1}$ can advantageously bias the spread of particular policies in practical scenarios.

3.4 Practical running of algorithms

3.4.1 Generation of $\sigma^{i}_{k+1}$

The theoretical analysis in Sec. 3.3 requires algorithmic hyperparameters that render convergence impractically slow in all of the centralised, independent and networked cases (see Rem. 7, Appx. B.8). For practical convergence of the algorithms, we seek to drastically increase $\{\beta_{m}\}$ and reduce $M_{td}$ and $M_{pg}$ , though this will naturally break the theoretical guarantees and give a poorer estimation of the Q-function $\hat{Q}^{i}_{M_{pg}}$ , and hence a greater variance in the quality of the updated policies $\pi^{i}_{k+1}$ . However, in such cases we found empirically that an appropriate method for generating $\sigma^{i}_{k+1}$ dependent on $\pi^{i}_{k+1}$ allows our networked algorithm to significantly outperform the independent setting, and sometimes even the centralised setting.

We do so by setting $\sigma^{i}_{k+1}$ to a finite approximation $\widehat{\Theta^{i}_{h,k+1}}(\boldsymbol{\pi}_{k+1},\upsilon_{0})$ of $\Theta^{i}_{h,k+1}(\boldsymbol{\pi}_{k+1},\upsilon_{0})$ where $\boldsymbol{\pi}_{k+1}$ := ( $\pi^{1}_{k+1},\dots,\pi^{N}_{k+1}$ ), by tracking the discounted return for $E$ evaluation steps. This is given by

\widehat{\Theta^{i}_{h,k+1}}(\boldsymbol{\pi}_{k+1},\upsilon_{0})=\left[\sum^{% E}_{e=0}\gamma^{e}(R(s^{i}_{t},a^{i}_{t},\hat{\mu}_{t})\right.+h(\pi^{i}(s^{i}% _{t})))\left|\begin{subarray}{c}t=t+e\\ a^{j}_{t}\sim\pi^{j}_{k+1}(s^{j}_{t})\\ s^{j}_{t+1}\sim P(\cdot|s^{j}_{t},a^{j}_{t},\hat{\mu}_{t})\end{subarray},% \forall j\in\{1,\dots,N\}\right].

Generating $\sigma^{i}_{k+1}$ in this way means that the policies that are more likely to be adopted and spread through the communication network are those which are estimated to receive a higher return in reality, despite being generated from poorly estimated Q-functions. This explains why our networked method can in practice outperform even the centralised case, where the updated policy of the agent with arbitrary index $i=1$ gets pushed to all other agents regardless of its quality. Naturally the quality of the finite approximation depends on the number of evaluation steps $E$ , but we found empirically that $E$ can be much smaller than $M_{pg}$ and still give significant convergence benefits.

3.4.2 Algorithm acceleration by use of experience-replay buffer

Even with networked communication, the empirical convergence of our original algorithm is too slow for practical demonstration, as also in the centralised and independent cases. We therefore offer a further technical contribution allowing the first practical demonstrations of all three architectures for learning from a single continued system run.

The modifications made to our Alg. 1 are shown in blue in Alg. 2, Appx. C. Instead of using a transition $\zeta^{i}_{t-2}$ to compute the TD update within each $M_{pg}$ iteration and then discarding the transition, we store the transition in a buffer (Line 11) until after the $M_{pg}$ loops. Replay buffers are a common (MA)RL tool used especially with deep learning, precisely to improve data efficiency and reduce autocorrelation [70, 71, 72]. When learning does take place in our modified algorithm (Lines 13-18), it involves cycling through the buffer for $L$ iterations - randomly shuffling the buffer between each - and thus conducting the TD update on each stored transition $L$ times. This allows us to reduce the number of $M_{pg}$ loops, as well as not requiring as small a learning rate $\{\beta_{m}\}$ , allowing much faster learning in practice. Moreover, by shuffling the buffer before each cycle we reduce bias resulting from the dependency of samples along the single path, which may justify being able to achieve adequate stable learning even when reducing the number of $M_{td}$ waiting steps within each $M_{pg}$ loop. Naturally the experience replay buffer means that theoretical guarantees given in Sec. 3.3 no longer apply, but we trade this off for practical convergence times. See Appx. C for further discussion.

Refer to caption — Figure 1: ‘Target agreement’ game. Even with only a single round of communication, our networked case outperforms the independent case in terms of exploitability, and significantly outperforms in terms of return. The fact that the lowest broadcast radius (0.2) ends with similar exploitability to the independent case yet much higher return shows that our networked algorithm can help agents find more ‘preferable’ equilibria. Moreover, the scenarios with the largest two broadcast radii outperform even the centralised algorithm. CPU time for 5 trials = 128,228 secs.

4 Experiments

Our technical contribution of the experience replay buffer to MFG algorithms for learning from continuous system runs allows us also to contribute the first empirical demonstrations of these algorithms, not just in the networked case but also in the centralised and independent cases. The latter two serve as baselines to demonstrate the advantages of the networked architecture.We follow prior works on stationary MFGs in the types of game used in our demonstrations [5, 30, 40, 73, 74]. We focus on grid-world environments where agents can move in one of the four cardinal directions or remain in place. We present results from two tasks defined by the reward functions of the agents; see Appx. E.1 for full technical description of our task settings.

Cluster. Agents are rewarded for gathering together. The agents are given no indication where they should cluster, agreeing this themselves over time.

Target agreement. The agents are rewarded for visiting any of a given number of targets, but their reward is proportional to the number of other agents co-located at that target. The agents must therefore coordinate on which single target they will all meet at to maximise their individual rewards.

As well as the standard scenario for these tasks, we conduct robustness tests in two settings, reflecting those elaborated in Appx. D. The first illustrates robustness to learning failures: at every iteration $k$ each learner (whether centralised or decentralised) fails to update its policy (i.e. Line 12 of Alg. 1 is not executed such that $\pi^{i}_{k+1}=\pi^{i}_{k}$ ) with a 50% probability. The second test illustrates robustness to increases in population size. Instead of having 250 agents throughout, the population begins with 50 agents learning normally, and a further 200 agents are added to the population at the marked point.

Experiments are evaluated via three metrics (see Appx. E.2 for a full discussion): an approximation of the exploitability of the joint policy $\boldsymbol{\pi}_{k}$ ; the average discounted return of the agents’ policies $\pi_{k}^{i}$ ; and the population’s policy divergence. Hyperparameters are discussed in Appx. E.3.

4.1 Discussion

We give here three example figures illustrating the benefits of the networked architecture; in each the decimals refer to each agent’s broadcast radius as a fraction of the maximum possible distance in the grid (i.e. the diagonal). See figure captions for details, and Appx. E.4 for further experiments and discussion. As well as allowing convergence in a practical number of iterations, even with only a single communication round, the combination of the buffer and the networked architecture allows us to remove in our experiments a number of the assumptions required for the theoretical algorithms:

•

We significantly reduce $M_{pg}$ while still converging within a reasonable $K$ . With smaller values for the parameters $M_{pg}$ (the number of samples in the buffer) and $L$ (the number of loops through the buffer for updating the Q-function), and hence with worse estimation of the Q-function, the networked architecture outperforms the independent case to an even greater extent. This underlines its advantages in allowing faster convergence in practical settings.
•

We can reduce the $M_{td}$ parameter (theoretically required for the learner to wait between collecting samples when learning from a single system run) to 1, effectively removing the innermost loop of the nested learning algorithm (see Line 7 of Alg. 1).
•

We can reduce the scaling parameter $\lambda$ of the entropy regulariser to 0, i.e. we converge even without regularisation, allowing us to leave the NE unbiased, and also removing Assumption 3 (Appx. A). In general an unregularised MFG-NE is not unique [6]; the ability of the agents to coordinate on one of the multiple solutions in the centralised and networked cases may explain why they outperform the independent-learning case.
•

For the PMA operator (Def. 8), we conduct the optimisation over the set $u\in\Delta_{\mathcal{A}}$ instead of $u\in\mathcal{U}_{L_{h}}$ , i.e. we can choose from all possible probability distributions over actions instead of needing to identify the Lipschitz constants given in Assumption 1 (Appx. A).

For limitations and ongoing work, see Appx. G; for impact statement, see Appx. H.

References

Lasry and Lions [2007] Jean-Michel Lasry and Pierre-Louis Lions. Mean Field Games. Japanese Journal of Mathematics, 2(1):229–260, 2007.
Huang et al. [2006] Minyi Huang, Roland P. Malhamé, and Peter E. Caines. Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle. Communications in Information & Systems, 6(3):221 – 252, 2006.
Xie et al. [2021] Qiaomin Xie, Zhuoran Yang, Zhaoran Wang, and Andreea Minca. Learning While Playing in Mean-Field Games: Convergence and Optimality. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11436–11447. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/xie21g.html.
Anahtarci et al. [2023] Berkay Anahtarci, Can Deha Kariksiz, and Naci Saldi. Q-learning in regularized mean-field games. Dynamic Games and Applications, 13(1):89–117, 2023.
uz Zaman et al. [2023] Muhammad Aneeq uz Zaman, Alec Koppel, Sujay Bhatt, and Tamer Başar. Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Path, 2023.
Yardim et al. [2023] Batuhan Yardim, Semih Cayci, Matthieu Geist, and Niao He. Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Games. In International Conference on Machine Learning, pages 39722–39754. PMLR, 2023.
Saldi et al. [2018] Naci Saldi, Tamer Başar, and Maxim Raginsky. Markov–Nash Equilibria in Mean-Field Games with Discounted Cost. SIAM Journal on Control and Optimization, 56(6):4256–4287, 2018. doi: 10.1137/17M1112583. URL https://doi.org/10.1137/17M1112583.
Yardim et al. [2024] Batuhan Yardim, Artur Goldman, and Niao He. When is Mean-Field Reinforcement Learning Tractable and Relevant? arXiv preprint arXiv:2402.05757, 2024.
Toumi et al. [2024] Noureddine Toumi, Roland Malhame, and Jerome Le Ny. A mean field game approach for a class of linear quadratic discrete choice problems with congestion avoidance. Automatica, 160:111420, 2024. ISSN 0005-1098. doi: https://doi.org/10.1016/j.automatica.2023.111420. URL https://www.sciencedirect.com/science/article/pii/S0005109823005873.
Hu and Zhang [2024] Anran Hu and Junzi Zhang. MF-OML: Online Mean-Field Reinforcement Learning with Occupation Measures for Large Population Games, 2024.
Trimborn et al. [2018] Torsten Trimborn, Martin Frank, and Stephan Martin. Mean field limit of a behavioral financial market model. Physica A: Statistical Mechanics and its Applications, 505:613–631, 2018. ISSN 0378-4371. doi: https://doi.org/10.1016/j.physa.2018.03.079. URL https://www.sciencedirect.com/science/article/pii/S0378437118303984.
Li et al. [2022] Zongxi Li, A. Max Reppen, and Ronnie Sircar. A Mean Field Games Model for Cryptocurrency Mining, 2022.
Aggarwal et al. [2024] Shubham Aggarwal, Muhammad Aneeq uz Zaman, Melih Bastopcu, Sennur Ulukus, and Tamer Başar. A Mean Field Game Model for Timely Computation in Edge Computing Systems, 2024.
Shen et al. [2024] Shigen Shen, Chenpeng Cai, Yizhou Shen, ** Wu, Wenlong Ke, and Shui Yu. MFGD3QN: Enhancing Edge Intelligence Defense against DDoS with Mean-Field Games and Dueling Double Deep Q-network. IEEE Internet of Things Journal, pages 1–1, 2024. doi: 10.1109/JIOT.2024.3387090.
Miao et al. [2024] Li Miao, Shuai Li, Xiangjuan Wu, and Bingjie Liu. Mean-Field Stackelberg Game-Based Security Defense and Resource Optimization in Edge Computing. Applied Sciences, 14(9), 2024. ISSN 2076-3417. doi: 10.3390/app14093538. URL https://www.mdpi.com/2076-3417/14/9/3538.
Huang et al. [2020] Kuang Huang, Xuan Di, Qiang Du, and Xi Chen. A game-theoretic framework for autonomous vehicles velocity control: Bridging microscopic differential games and macroscopic mean field games. Discrete and Continuous Dynamical Systems - B, 25(12):4869–4903, 2020. ISSN 1531-3492. doi: 10.3934/dcdsb.2020131.
Hu et al. [2023] Tianfeng Hu, Zhiqun hu, Zhaoming Lu, and Xiangming Wen. Dynamic traffic signal control using mean field multi-agent reinforcement learning in large scale road-networks. IET Intelligent Transport Systems, 04 2023. doi: 10.1049/itr2.12364.
Mao et al. [2022] Weichao Mao, Haoran Qiu, Chen Wang, Hubertus Franke, Zbigniew T. Kalbarczyk, Ravishankar K. Iyer, and Tamer Başar. A mean-field game approach to cloud resource management with function approximation. In Proceedings of the 36th Conference on Advances in Neural Information Processing Systems (NIPS 2022), volume 36, pages 1–12, New Orleans, LA, USA, 2022. Curran Associates, Inc.
Bauso and Tembine [2016] Dario Bauso and Hamidou Tembine. Crowd-Averse Cyber-Physical Systems: The Paradigm of Robust Mean-Field Games. IEEE Transactions on Automatic Control, 61(8):2312–2317, 2016. doi: 10.1109/TAC.2015.2492038.
Mishra et al. [2023] Rajesh Mishra, Sriram Vishwanath, and Deepanshu Vasal. Model-free Reinforcement Learning for Mean Field Games. IEEE Transactions on Control of Network Systems, pages 1–11, 2023. doi: 10.1109/TCNS.2023.3264934.
Yoshioka et al. [2024] Hidekazu Yoshioka, Motoh Tsujimura, and Yumi Yoshioka. Numerical analysis of an extended mean field game for harvesting common fishery resource. Computers & Mathematics with Applications, 165:88–105, 2024. ISSN 0898-1221. doi: https://doi.org/10.1016/j.camwa.2024.04.003. URL https://www.sciencedirect.com/science/article/pii/S0898122124001615.
Wang et al. [2024] Yao Wang, Chungang Yang, Tong Li, Xinru Mi, Lixin Li, and Zhu Han. A Survey On Mean-Field Game for Dynamic Management and Control in Space-Air-Ground Network. IEEE Communications Surveys & Tutorials, pages 1–1, 2024. doi: 10.1109/COMST.2024.3393369.
Le Ménec [0] Stéphane Le Ménec. Swarm Guidance Based on Mean Field Game Concepts. International Game Theory Review, 0(0):2440008, 0. doi: 10.1142/S0219198924400085. URL https://doi.org/10.1142/S0219198924400085.
Emami et al. [2024] Yousef Emami, Hao Gao, Kai Li, Luis Almeida, Eduardo Tovar, and Zhu Han. Age of Information Minimization using Multi-agent UAVs based on AI-Enhanced Mean Field Resource Allocation. IEEE Transactions on Vehicular Technology, pages 1–14, 2024. doi: 10.1109/TVT.2024.3394235.
Yang et al. [2023] Yaoqi Yang, Bangning Zhang, Daoxing Guo, Renhui Xu, Neeraj Kumar, and Weizheng Wang. Mean Field Game and Broadcast Encryption-Based Joint Data Freshness Optimization and Privacy Preservation for Mobile Crowdsensing. IEEE Transactions on Vehicular Technology, 72(11):14860–14874, 2023. doi: 10.1109/TVT.2023.3282694.
Benamor et al. [2022] Amani Benamor, Oussama Habachi, Inès Kammoun, and Jean-Pierre Cances. NOMA-based Power Control for Machine-Type Communications: A Mean Field Game Approach. In 2022 IEEE International Performance, Computing, and Communications Conference (IPCCC), pages 338–343, 2022. doi: 10.1109/IPCCC55026.2022.9894296.
Dey and Xu [2023] Shawon Dey and Hao Xu. Intelligent Distributed Charging Control for Large Scale Electric Vehicles: A Multi-Cluster Mean Field Game Approach. In Proceedings of Cyber-Physical Systems and Internet of Things Week 2023, CPS-IoT Week ’23, page 146–151, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400700491. doi: 10.1145/3576914.3587709. URL https://doi.org/10.1145/3576914.3587709.
Wang et al. [2020a] Ximing Wang, Yuhua Xu, ** Chen, Chunguo Li, Xin Liu, Dianxiong Liu, and Yifan Xu. Mean Field Reinforcement Learning Based Anti-Jamming Communications for Ultra-Dense Internet of Things in 6G. In 2020 International Conference on Wireless Communications and Signal Processing (WCSP), pages 195–200, 2020a. doi: 10.1109/WCSP49889.2020.9299742.
Korecki et al. [2023] Marcin Korecki, Damian Dailisan, and Dirk Helbing. How Well Do Reinforcement Learning Approaches Cope With Disruptions? The Case of Traffic Signal Control. IEEE Access, 11:36504–36515, 2023. doi: 10.1109/ACCESS.2023.3266644.
Lauriere et al. [2022] Mathieu Lauriere, Sarah Perrin, Sertan Girgin, Paul Muller, Ayush Jain, Theophile Cabannes, Georgios Piliouras, Julien Perolat, Romuald Elie, Olivier Pietquin, and Matthieu Geist. Scalable Deep Reinforcement Learning Algorithms for Mean Field Games. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 12078–12095. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/lauriere22a.html.
Perrin et al. [2020] Sarah Perrin, Julien Pérolat, Mathieu Laurière, Matthieu Geist, Romuald Elie, and Olivier Pietquin. Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Laurière et al. [2022] Mathieu Laurière, Sarah Perrin, Matthieu Geist, and Olivier Pietquin. Learning Mean Field Games: A Survey, 2022. URL https://arxiv.longhoe.net/abs/2205.12944.
Guo et al. [2019a] Xin Guo, Anran Hu, Renyuan Xu, and Junzi Zhang. Learning Mean-Field Games. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019a. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/030e65da2b1c944090548d36b244b28d-Paper.pdf.
Perrin et al. [2021] Sarah Perrin, Mathieu Laurière, Julien Pérolat, Matthieu Geist, Romuald Élie, and Olivier Pietquin. Mean field games flock! the reinforcement learning way. In IJCAI, 2021.
Elie et al. [2020] Romuald Elie, Julien Pérolat, Mathieu Laurière, Matthieu Geist, and Olivier Pietquin. On the Convergence of Model Free Learning in Mean Field Games. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7143–7150, Apr. 2020. doi: 10.1609/aaai.v34i05.6203. URL https://ojs.aaai.org/index.php/AAAI/article/view/6203.
Carmona and Laurière [2021] René Carmona and Mathieu Laurière. Deep Learning for Mean Field Games and Mean Field Control with Applications to Finance, 2021.
Cao et al. [2021] Haoyang Cao, Xin Guo, and Mathieu Laurière. Connecting GANs, MFGs, and OT, 2021.
Germain et al. [2022] Maximilien Germain, Joseph Mikael, and Xavier Warin. Numerical resolution of McKean-Vlasov FBSDEs using neural networks, 2022.
Fouque and Zhang [2020] Jean-Pierre Fouque and Zhaoyu Zhang. Deep Learning Methods for Mean Field Control Problems With Delay. Frontiers in Applied Mathematics and Statistics, 6, 2020. ISSN 2297-4687. doi: 10.3389/fams.2020.00011. URL https://www.frontiersin.org/articles/10.3389/fams.2020.00011.
Algumaei et al. [2023] Talal Algumaei, Ruben Solozabal, Reda Alami, Hakim Hacid, Merouane Debbah, and Martin Takac. Regularization of the policy updates for stabilizing Mean Field Games, 2023.
Angiuli et al. [2023] Andrea Angiuli, Jean-Pierre Fouque, Mathieu Laurière, and Mengrui Zhang. Convergence of Multi-Scale Reinforcement Q-Learning Algorithms for Mean Field Game and Control Problems. arXiv preprint arXiv:2312.06659, 2023.
Guo et al. [2019b] Xin Guo, Anran Hu, Renyuan Xu, and Junzi Zhang. Learning Mean-Field Games, 2019b. URL https://arxiv.longhoe.net/abs/1901.09585.
Zhang et al. [2021] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. “Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms", pages 321–384. Springer International Publishing, Cham, 2021. ISBN 978-3-030-60990-0. doi: 10.1007/978-3-030-60990-0_12. URL https://doi.org/10.1007/978-3-030-60990-0_12.
Zhang et al. [2018] Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5872–5881. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/zhang18n.html.
Wai et al. [2018] Hoi-To Wai, Zhuoran Yang, Zhaoran Wang, and Mingyi Hong. Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 9672–9683, Red Hook, NY, USA, 2018. Curran Associates Inc.
Zhang et al. [2019] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Decentralized Multi-Agent Reinforcement Learning with Networked Agents: Recent Advances, 2019. URL https://arxiv.longhoe.net/abs/1912.03821.
Chen et al. [2021] Mingzhe Chen, Deniz Gündüz, Kaibin Huang, Walid Saad, Mehdi Bennis, Aneta Vulgarakis Feljan, and H. Vincent Poor. Distributed Learning in Wireless Networks: Recent Progress and Future Challenges, 2021. URL https://arxiv.longhoe.net/abs/2104.02151.
Jiang et al. [2024] Jiechuan Jiang, Kefan Su, and Zongqing Lu. Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Survey. arXiv preprint arXiv:2401.04934, 2024.
Mguni et al. [2018] David Mguni, Joel Jennings, and Enrique Munoz de Cote. Decentralised Learning in Systems With Many, Many Strategic Agents. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/aaai.v32i1.11586. URL https://ojs.aaai.org/index.php/AAAI/article/view/11586.
Yongacoglu et al. [2022a] Bora Yongacoglu, Gürdal Arslan, and Serdar Yüksel. Independent Learning in Mean-Field Games: Satisficing Paths and Convergence to Subjective Equilibria, 2022a. URL https://arxiv.longhoe.net/abs/2209.05703.
Yongacoglu et al. [2022b] Bora Yongacoglu, Gürdal Arslan, and Serdar Yüksel. Independent Learning and Subjectivity in Mean-Field Games. In 2022 IEEE 61st Conference on Decision and Control (CDC), pages 2845–2850, 2022b. doi: 10.1109/CDC51059.2022.9992399.
Grammatico et al. [2015a] Sergio Grammatico, Basilio Gentile, Francesca Parise, and John Lygeros. A Mean Field control approach for demand side management of large populations of Thermostatically Controlled Loads. In 2015 European Control Conference (ECC), pages 3548–3553, 2015a. doi: 10.1109/ECC.2015.7331083.
Grammatico et al. [2015b] Sergio Grammatico, Francesca Parise, and John Lygeros. Constrained linear quadratic deterministic mean field control: Decentralized convergence to Nash equilibria in large populations of heterogeneous agents. In 2015 54th IEEE Conference on Decision and Control (CDC), pages 4412–4417, 2015b. doi: 10.1109/CDC.2015.7402908.
Parise et al. [2015] Francesca Parise, Sergio Grammatico, Basilio Gentile, and John Lygeros. Network Aggregative Games and Distributed Mean Field Control via Consensus Theory, 2015.
Grammatico et al. [2016] Sergio Grammatico, Francesca Parise, Marcello Colombino, and John Lygeros. Decentralized Convergence to Nash Equilibria in Constrained Deterministic Mean Field Control. IEEE Transactions on Automatic Control, 61(11):3315–3329, 2016. doi: 10.1109/TAC.2015.2513368.
Doan et al. [2019] Thinh T. Doan, Siva Theja Maguluri, and Justin Romberg. Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation for Multi-Agent Reinforcement Learning, 2019. URL https://arxiv.longhoe.net/abs/1902.07393.
Lin et al. [2019] Yixuan Lin, Kaiqing Zhang, Zhuoran Yang, Zhaoran Wang, Tamer Başar, Romeil Sandhu, and Ji Liu. A Communication-Efficient Multi-Agent Actor-Critic Algorithm for Distributed Reinforcement Learning. In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 5562–5567, 2019. doi: 10.1109/CDC40024.2019.9029257.
Heredia et al. [2020] Paulo Heredia, Hasan Ghadialy, and Shaoshuai Mou. Finite-Sample Analysis of Distributed Q-learning for Multi-Agent Networks. In 2020 American Control Conference (ACC), pages 3511–3516, 2020. doi: 10.23919/ACC45564.2020.9147428.
Kar et al. [2013] Soummya Kar, José M. F. Moura, and H. Vincent Poor. ${{\cal Q}{\cal D}}$ -Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through ${\rm Consensus}+{\rm Innovations}$ . IEEE Transactions on Signal Processing, 61(7):1848–1862, 2013. doi: 10.1109/TSP.2013.2241057.
Suttle et al. [2019] Wesley Suttle, Zhuoran Yang, Kaiqing Zhang, Zhaoran Wang, Tamer Basar, and Ji Liu. A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning, 2019. URL https://arxiv.longhoe.net/abs/1903.06372.
Cui et al. [2023a] Kai Cui, Gökçe Dayanıklı, Mathieu Laurière, Matthieu Geist, Olivier Pietquin, and Heinz Koeppl. Learning Discrete-Time Major-Minor Mean Field Games. arXiv preprint arXiv:2312.10787, 2023a.
Cui and Koeppl [2021] Kai Cui and Heinz Koeppl. Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning, 2021. URL https://arxiv.longhoe.net/abs/2102.01585.
Guo et al. [2022] Xin Guo, Renyuan Xu, and Thaleia Zariphopoulou. Entropy Regularization for Mean Field Games with Learning. Math. Oper. Res., 47(4):3239–3260, nov 2022. ISSN 0364-765X. doi: 10.1287/moor.2021.1238. URL https://doi.org/10.1287/moor.2021.1238.
Yu and Yuan [2023] Xiang Yu and Fengyi Yuan. Time-inconsistent mean-field stop** problems: A regularized equilibrium approach. arXiv preprint arXiv:2311.00381, 2023.
Su and Lu [2022] Kefan Su and Zongqing Lu. Divergence-Regularized Multi-Agent Actor-Critic. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 20580–20603. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/su22b.html.
Jadbabaie et al. [2003] A. Jadbabaie, Jie Lin, and A.S. Morse. Coordination of groups of mobile autonomous agents using nearest neighbor rules. IEEE Transactions on Automatic Control, 48(6):988–1001, 2003. doi: 10.1109/TAC.2003.812781.
Kotsalis et al. [2022] Georgios Kotsalis, Guanghui Lan, and Tianjiao Li. Simple and Optimal Methods for Stochastic Variational Inequalities, II: Markovian Noise and Policy Evaluation in Reinforcement Learning. SIAM Journal on Optimization, 32(2):1120–1155, 2022. doi: 10.1137/20M1381691. URL https://doi.org/10.1137/20M1381691.
Rajagopalan and Shah [2010] Shreevatsa Rajagopalan and Devavrat Shah. Distributed Averaging in Dynamic Networks. In Proceedings of the ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS ’10, page 369–370, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 9781450300384. doi: 10.1145/1811039.1811091. URL https://doi.org/10.1145/1811039.1811091.
Zhang et al. [2020] Kaiqing Zhang, Yang Liu, Ji Liu, Mingyan Liu, and Tamer Basar. Distributed learning of average belief over networks using sequential observations. Automatica, 115:108857, 2020. ISSN 0005-1098. doi: https://doi.org/10.1016/j.automatica.2020.108857. URL https://www.sciencedirect.com/science/article/pii/S0005109820300558.
Lin [1992] Long-Ji Lin. Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching. Mach. Learn., 8(3–4):293–321, may 1992. ISSN 0885-6125. doi: 10.1007/BF00992699. URL https://doi.org/10.1007/BF00992699.
Fedus et al. [2020] William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting Fundamentals of Experience Replay. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020.
Xu et al. [2024] Linjie Xu, Zichuan Liu, Alexander Dockhorn, Diego Perez-Liebana, **yu Wang, Lei Song, and Jiang Bian. Higher Replay Ratio Empowers Sample-Efficient Multi-Agent Reinforcement Learning, 2024.
Laurière [2021] Mathieu Laurière. Numerical Methods for Mean Field Games and Mean Field Type Control, 2021.
Cui et al. [2023b] Kai Cui, Christian Fabian, and Heinz Koeppl. Multi-Agent Reinforcement Learning via Mean Field Control: Common Noise, Major Agents and Approximation Properties, 2023b.
Eck et al. [2023] Adam Eck, Leen-Kiat Soh, and Prashant Doshi. Decision making in open agent systems. AI Mag., 44(4):508–523, dec 2023. ISSN 0738-4602. doi: 10.1002/aaai.12131. URL https://doi.org/10.1002/aaai.12131.
Gao et al. [2024] Yuzhao Gao, Yiming Nie, and Hongliang Wang. A Scalable Multi-agent Reinforcement Learning Approach Based on¬†Value Function Decomposition. In Yi Qu, Mancang Gu, Yifeng Niu, and Wenxing Fu, editors, Proceedings of 3rd 2023 International Conference on Autonomous Unmanned Systems (3rd ICAUS 2023), pages 88–96, Singapore, 2024. Springer Nature Singapore. ISBN 978-981-97-1087-4.
Dawood et al. [2023] Murad Dawood, Sicong Pan, Nils Dengler, Siqi Zhou, Angela P Schoellig, and Maren Bennewitz. Safe Multi-Agent Reinforcement Learning for Formation Control without Individual Reference Targets. arXiv preprint arXiv:2312.12861, 2023.
Wu et al. [2024a] Zida Wu, Mathieu Lauriere, Samuel Jia Cong Chua, Matthieu Geist, Olivier Pietquin, and Ankur Mehta. Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learning. arXiv preprint arXiv:2403.03552, 2024a.
Pérolat et al. [2022] Julien Pérolat, Sarah Perrin, Romuald Elie, Mathieu Laurière, Georgios Piliouras, Matthieu Geist, Karl Tuyls, and Olivier Pietquin. Scaling Mean Field Games by Online Mirror Descent. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’22, page 1028–1037, Richland, SC, 2022. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450392136.
Guo et al. [2020] Xin Guo, Anran Hu, Renyuan Xu, and Junzi Zhang. A General Framework for Learning Mean-Field Games, 2020. URL https://arxiv.longhoe.net/abs/2003.06069.
Subramanian and Mahajan [2019] Jayakumar Subramanian and Aditya Mahajan. Reinforcement Learning in Stationary Mean-Field Games. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’19, page 251–259, Richland, SC, 2019. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450363099.
Yang et al. [2018a] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean Field Multi-Agent Reinforcement Learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5571–5580. PMLR, 10–15 Jul 2018a. URL https://proceedings.mlr.press/v80/yang18d.html.
Subramanian et al. [2020] Sriram Ganapathi Subramanian, Matthew E. Taylor, Mark Crowley, and Pascal Poupart. Partially Observable Mean Field Reinforcement Learning, 2020. URL https://arxiv.longhoe.net/abs/2012.15791.
Subramanian et al. [2022] Sriram Ganapathi Subramanian, Pascal Poupart, Matthew E. Taylor, and Nidhi Hegde. Multi Type Mean Field Reinforcement Learning, 2022.
Subramanian et al. [2021] Sriram Ganapathi Subramanian, Matthew E. Taylor, Mark Crowley, and Pascal Poupart. Decentralized Mean Field Games, 2021. URL https://arxiv.longhoe.net/abs/2112.09099.
Busoniu et al. [2008] Lucian Busoniu, Robert Babuska, and Bart De Schutter. A Comprehensive Survey of Multiagent Reinforcement Learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156–172, 2008. doi: 10.1109/TSMCC.2007.913919.
Sutton and Barto [2018] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
Leottau et al. [2018] David L. Leottau, Javier Ruiz del Solar, and Robert Babuka. Decentralized Reinforcement Learning of Robot Behaviors. Artificial Intelligence, 256:130–159, 2018. ISSN 0004-3702. doi: https://doi.org/10.1016/j.artint.2017.12.001. URL https://www.sciencedirect.com/science/article/pii/S0004370217301674.
Lv et al. [2023] Zefang Lv, Liang Xiao, Yousong Du, Guohang Niu, Chengwen Xing, and Wenyuan Xu. Multi-Agent Reinforcement Learning based UAV Swarm Communications Against Jamming. IEEE Transactions on Wireless Communications, pages 1–1, 2023. doi: 10.1109/TWC.2023.3268082.
Orr and Dutta [2023] James Orr and Ayan Dutta. Multi-Agent Deep Reinforcement Learning for Multi-Robot Applications: A Survey. Sensors, 23(7), 2023. ISSN 1424-8220. doi: 10.3390/s23073625. URL https://www.mdpi.com/1424-8220/23/7/3625.
Guan et al. [2024] Yue Guan, Sai Zou, Haixia Peng, Wei Ni, Yanglong Sun, and Hongfeng Gao. Cooperative UAV Trajectory Design for Disaster Area Emergency Communications: A Multiagent PPO Method. IEEE Internet of Things Journal, 11(5):8848–8859, 2024. doi: 10.1109/JIOT.2023.3320796.
Ali et al. [2023] Adeeba Ali, Rashid Ali, and M.F. Baig. Distributed Multi-Agent Deep Reinforcement Learning based Navigation and Control of UAV Swarm for Wildfire Monitoring. In 2023 IEEE 4th Annual Flagship India Council International Subsections Conference (INDISCON), pages 1–8, 2023. doi: 10.1109/INDISCON58499.2023.10270198.
Shalev-Shwartz et al. [2016] Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295, 2016.
Mannion et al. [2016] Patrick Mannion, Jim Duggan, and Enda Howley. An Experimental Review of Reinforcement Learning Algorithms for Adaptive Traffic Signal Control, pages 47–66. Springer International Publishing, Cham, 2016. ISBN 978-3-319-25808-9. doi: 10.1007/978-3-319-25808-9_4. URL https://doi.org/10.1007/978-3-319-25808-9_4.
Samvelyan et al. [2019] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’19, page 2186–2188, Richland, SC, 2019. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450363099.
Vinyals et al. [2019a] Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojtek Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, Timo Ewalds, Dan Horgan, Manuel Kroiss, Ivo Danihelka, John Agapiou, Junhyuk Oh, Valentin Dalibard, David Choi, Laurent Sifre, Yury Sulsky, Sasha Vezhnevets, James Molloy, Trevor Cai, David Budden, Tom Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Toby Pohlen, Dani Yogatama, Julia Cohen, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Chris Apps, Koray Kavukcuoglu, Demis Hassabis, and David Silver. AlphaStar: Mastering the Real-Time Strategy Game StarCraft II. https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/, 2019a.
Berner et al. [2019] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d. O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with Large Scale Deep Reinforcement Learning, 2019.
Rashedi et al. [2016] Navid Rashedi, Mohammad Amin Tajeddini, and Hamed Kebriaei. Markov game approach for multi-agent competitive bidding strategies in electricity market. IET Generation, Transmission & Distribution, 10:3756–3763(7), November 2016. ISSN 1751-8687. URL https://digital-library.theiet.org/content/journals/10.1049/iet-gtd.2016.0075.
Shavandi and Khedmati [2022] Ali Shavandi and Majid Khedmati. A multi-agent deep reinforcement learning framework for algorithmic trading in financial markets. Expert Systems with Applications, 208:118124, 2022. ISSN 0957-4174. doi: https://doi.org/10.1016/j.eswa.2022.118124. URL https://www.sciencedirect.com/science/article/pii/S0957417422013082.
Leibo et al. [2017] Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi-Agent Reinforcement Learning in Sequential Social Dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’17, page 464–473, Richland, SC, 2017. International Foundation for Autonomous Agents and Multiagent Systems.
Cao et al. [2018] Kris Cao, Angeliki Lazaridou, Marc Lanctot, Joel Z. Leibo, Karl Tuyls, and Stephen Clark. Emergent Communication through Negotiation. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=Hk6WhagRW.
Jaques et al. [2019] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A. Ortega, DJ Strouse, Joel Z. Leibo, and Nando de Freitas. Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning, 2019.
McKee et al. [2020] Kevin R. McKee, Ian Gemp, Brian McWilliams, Edgar A. Duéñez-Guzmán, Edward Hughes, and Joel Z. Leibo. Social diversity and social preferences in mixed-motive reinforcement learning, 2020.
Daskalakis et al. [2006] Constantinos Daskalakis, Paul W. Goldberg, and Christos H. Papadimitriou. The Complexity of Computing a Nash Equilibrium. In Proceedings of the Thirty-Eighth Annual ACM Symposium on Theory of Computing, STOC ’06, page 71–78, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595931341. doi: 10.1145/1132516.1132527. URL https://doi.org/10.1145/1132516.1132527.
Vinyals et al. [2019b] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, L. Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Caglar Gulcehre, Ziyun Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, pages 1–5, 2019b.
Mcaleer et al. [2020] Stephen Mcaleer, JB Lanier, Roy Fox, and Pierre Baldi. Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20238–20248. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/e9bcd1b063077573285ae1a41025f5dc-Paper.pdf.
Wang et al. [2020b] Lingxiao Wang, Zhuoran Yang, and Zhaoran Wang. Breaking the Curse of Many Agents: Provable Mean Embedding Q-Iteration for Mean-Field Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020b.
Zheng et al. [2018] Lianmin Zheng, Jiacheng Yang, Han Cai, Ming Zhou, Weinan Zhang, Jun Wang, and Yong Yu. MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Cui et al. [2022] Kai Cui, Anam Tahir, Gizem Ekinci, Ahmed Elshamanhory, Yannick Eich, Mengguang Li, and Heinz Koeppl. A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2209.03859, 2022.
Ornia et al. [2022] Daniel Jarne Ornia, Pedro J. Zufiria, and Manuel Mazo Jr. Mean Field Behavior of Collaborative Multiagent Foragers. IEEE Transactions on Robotics, 38(4):2151–2165, 2022. doi: 10.1109/TRO.2022.3152691.
Shiri et al. [2019] Hamid Shiri, Jihong Park, and Mehdi Bennis. Massive Autonomous UAV Path Planning: A Neural Network Based Mean-Field Game Theoretic Approach, 2019.
Andréen et al. [2016] David Andréen, Petra Jenning, Nils Napp, and Kirstin Petersen. Emergent structures assembled by large swarms of simple robots. In Acadia, pages 54–61, 2016.
Chang et al. [2023] Lu Chang, Liang Shan, Weilong Zhang, and Yuewei Dai. Hierarchical multi-robot navigation and formation in unknown environments via deep reinforcement learning and distributed optimization. Robotics and Computer-Integrated Manufacturing, 83:102570, 2023. ISSN 0736-5845. doi: https://doi.org/10.1016/j.rcim.2023.102570. URL https://www.sciencedirect.com/science/article/pii/S0736584523000467.
Meigs et al. [2020] Emily Meigs, Francesca Parise, Asuman E. Ozdaglar, and Daron Acemoglu. Optimal dynamic information provision in traffic routing. CoRR, abs/2001.03232, 2020. URL https://arxiv.longhoe.net/abs/2001.03232.
Achdou and Capuzzo-Dolcetta [2010] Yves Achdou and Italo Capuzzo-Dolcetta. Mean Field Games: Numerical Methods. SIAM Journal on Numerical Analysis, 48(3):1136–1162, 2010. doi: 10.1137/090758477. URL https://doi.org/10.1137/090758477.
Carlini and Silva [2014] E. Carlini and F. J. Silva. A Fully Discrete Semi-Lagrangian Scheme for a First Order Mean Field Game Problem. SIAM Journal on Numerical Analysis, 52(1):45–67, 2014. doi: 10.1137/120902987. URL https://doi.org/10.1137/120902987.
Briceño-Arias et al. [2018] Luis Briceño-Arias, Dante Kalise, and Francisco Silva. Proximal methods for stationary Mean Field Games with local couplings. SIAM Journal on Control and Optimization, 56:801–, 03 2018.
Achdou and Laurière [2020] Yves Achdou and Mathieu Laurière. Mean Field Games and Applications: Numerical Aspects, 2020.
Huang et al. [2024a] Jiawei Huang, Batuhan Yardim, and Niao He. On the Statistical Efficiency of Mean-Field Reinforcement Learning with General Function Approximation . In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 289–297. PMLR, 02–04 May 2024a. URL https://proceedings.mlr.press/v238/huang24a.html.
Huang et al. [2024b] Jiawei Huang, Niao He, and Andreas Krause. Model-Based RL for Mean-Field Games is not Statistically Harder than Single-Agent RL, 2024b.
Angiuli et al. [2021] Andrea Angiuli, Jean-Pierre Fouque, and Mathieu Laurière. Unified Reinforcement Q-Learning for Mean Field Game and Control Problems, 2021.
Mishra et al. [2020] Rajesh K Mishra, Deepanshu Vasal, and Sriram Vishwanath. Model-free Reinforcement Learning for Non-stationary Mean Field Games. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 1032–1037, 2020. doi: 10.1109/CDC42340.2020.9304340.
Cacace, Simone et al. [2021] Cacace, Simone, Camilli, Fabio, and Goffi, Alessandro. A policy iteration method for mean field games. ESAIM: COCV, 27:85, 2021. doi: 10.1051/cocv/2021081. URL https://doi.org/10.1051/cocv/2021081.
Pérolat et al. [2021] Julien Pérolat, Sarah Perrin, Romuald Elie, Mathieu Laurière, Georgios Piliouras, Matthieu Geist, Karl Tuyls, and Olivier Pietquin. Scaling up Mean Field Games with Online Mirror Descent, 2021.
Lee et al. [2021] Kiyeob Lee, Desik Rengarajan, Dileep Kalathil, and Srinivas Shakkottai. Reinforcement Learning for Mean Field Games with Strategic Complementarities . In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 2458–2466. PMLR, 13–15 Apr 2021. URL https://proceedings.mlr.press/v130/lee21b.html.
Anahtarci et al. [2019] Berkay Anahtarci, Can Deha Karıksız, and Naci Saldi. Fitted Q-Learning in Mean-field Games. ArXiv, abs/1912.13309, 2019.
Fu et al. [2019] Zuyue Fu, Zhuoran Yang, Yongxin Chen, and Zhaoran Wang. Actor-critic provably finds Nash equilibria of linear-quadratic mean-field games. arXiv preprint arXiv:1910.07498, 2019.
Tembine et al. [2012] Hamidou Tembine, Raul Tempone, and Pedro Vilanova. Mean-Field Learning: a Survey, 2012.
Cardaliaguet, Pierre and Hadikhanloo, Saeed [2017] Cardaliaguet, Pierre and Hadikhanloo, Saeed. Learning in mean field games: The fictitious play. ESAIM: COCV, 23(2):569–591, 2017. doi: 10.1051/cocv/2016004. URL https://doi.org/10.1051/cocv/2016004.
Geist et al. [2022] Matthieu Geist, Julien Pérolat, Mathieu Laurière, Romuald Elie, Sarah Perrin, Olivier Bachem, Rémi Munos, and Olivier Pietquin. Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint, 2022.
Bonnans et al. [2021] J Frédéric Bonnans, Pierre Lavigne, and Laurent Pfeiffer. Generalized conditional gradient and learning in potential mean field games, 2021.
Yu et al. [2024] Hanfei Yu, Jian Li, Yang Hua, Xu Yuan, and Hao Wang. Cheaper and Faster: Distributed Deep Reinforcement Learning with Serverless Computing. 2024.
Wiggins et al. [2023] Samuel Wiggins, Yuan Meng, Rajgopal Kannan, and Viktor Prasanna. Characterizing Speed Performance of Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2309.07108, 2023.
Wu et al. [2024b] Peiliang Wu, Liqiang Tian, Qian Zhang, Bingyi Mao, and Wenbai Chen. MARRGM: Learning Framework for Multi-agent Reinforcement Learning via Reinforcement Recommendation and Group Modification. IEEE Robotics and Automation Letters, pages 1–8, 2024b. doi: 10.1109/LRA.2024.3389813.
Patel et al. [2024] Bhrij Patel, Wesley A. Suttle, Alec Koppel, Vaneet Aggarwal, Brian M. Sadler, Amrit Singh Bedi, and Dinesh Manocha. Global Optimality without Mixing Time Oracles in Average-reward RL via Multi-level Actor-Critic, 2024.
Huang and Lai [2024] Han Huang and Rongjie Lai. Unsupervised Solution Operator Learning for Mean-Field Games via Sampling-Invariant Parametrizations. arXiv preprint arXiv:2401.15482, 2024.
Bauso et al. [2012] Dario Bauso, Hamidou Tembine, and Tamer Başar. Robust Mean Field Games with Application to Production of an Exhaustible Resource. IFAC Proceedings Volumes, 45(13):454–459, 2012. ISSN 1474-6670. doi: https://doi.org/10.3182/20120620-3-DK-2025.00135. URL https://www.sciencedirect.com/science/article/pii/S1474667015377302. 7th IFAC Symposium on Robust Control Design.
Bauso et al. [2016] Dario Bauso, Hamidou Tembine, and Tamer Başar. Robust mean field games. Dynamic games and applications, 6(3):277–303, 2016.
Moon and Başar [2017] Jun Moon and Tamer Başar. Linear Quadratic Risk-Sensitive and Robust Mean Field Games. IEEE Transactions on Automatic Control, 62(3):1062–1077, 2017. doi: 10.1109/TAC.2016.2579264.
Huang and Huang [2017] Jianhui Huang and Minyi Huang. Robust Mean Field Linear-Quadratic-Gaussian Games with Unknown $L^{2}$ -Disturbance. SIAM Journal on Control and Optimization, 55(5):2811–2840, 2017. doi: 10.1137/15M1014437. URL https://doi.org/10.1137/15M1014437.
Yang et al. [2018b] Chungang Yang, Haoxiang Dai, Jiandong Li, Yue Zhang, and Zhu Han. Distributed Interference-Aware Power Control in Ultra-Dense Small Cell Networks: A Robust Mean Field Game. IEEE Access, 6:12608–12619, 2018b. doi: 10.1109/ACCESS.2018.2799138.
Tirumalai and Baras [2022] Amoolya Tirumalai and John S. Baras. A Robust Mean-field Game of Boltzmann-Vlasov-like Traffic Flow. In 2022 American Control Conference (ACC), pages 556–561, 2022. doi: 10.23919/ACC53348.2022.9867331.
Aydın and Saldi [2023] Uğur Aydın and Naci Saldi. Robustness and Approximation of Discrete-time Mean-field Games under Discounted Cost Criterion. arXiv preprint arXiv:2310.10828, 2023.
Eiben and Smith [2015] A. E. Eiben and J. E. Smith. What Is an Evolutionary Algorithm?, pages 25–48. Springer Berlin Heidelberg, Berlin, Heidelberg, 2015. ISBN 978-3-662-44874-8. doi: 10.1007/978-3-662-44874-8_3. URL https://doi.org/10.1007/978-3-662-44874-8_3.
Haasdijk et al. [2014] Evert Haasdijk, Nicolas Bredeche, and Agoston E Eiben. Combining environment-driven adaptation and task-driven optimisation in evolutionary robotics. PloS one, 9(6):e98466, 2014.
Trueba et al. [2015] Pedro Trueba, Abraham Prieto, Francisco Bellas, and Richard J. Duro. Embodied Evolution for Collective Indoor Surveillance and Location. In Proceedings of the Companion Publication of the 2015 Annual Conference on Genetic and Evolutionary Computation, GECCO Companion ’15, page 1241–1242, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450334884. doi: 10.1145/2739482.2768490. URL https://doi.org/10.1145/2739482.2768490.
Hart et al. [2015] Emma Hart, Andreas Steyven, and Ben Paechter. Improving Survivability in Environment-Driven Distributed Evolutionary Algorithms through Explicit Relative Fitness and Fitness Proportionate Communication. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, GECCO ’15, page 169–176, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450334723. doi: 10.1145/2739480.2754688. URL https://doi.org/10.1145/2739480.2754688.
Fernández Pérez et al. [2018] Iñaki Fernández Pérez, Amine Boumaza, and François Charpillet. Maintaining Diversity in Robot Swarms with Distributed Embodied Evolution. In Marco Dorigo, Mauro Birattari, Christian Blum, Anders L. Christensen, Andreagiovanni Reina, and Vito Trianni, editors, Swarm Intelligence, pages 395–402, Cham, 2018. Springer International Publishing. ISBN 978-3-030-00533-7.
Fernández Pérez and Sanchez [2019] Iñaki Fernández Pérez and Stéphane Sanchez. Influence of Local Selection and Robot Swarm Density on the Distributed Evolution of GRNs. In Paul Kaufmann and Pedro A. Castillo, editors, Applications of Evolutionary Computation, pages 567–582, Cham, 2019. Springer International Publishing. ISBN 978-3-030-16692-2.
Gomes and Christensen [2013] Jorge Gomes and Anders L. Christensen. Generic Behaviour Similarity Measures for Evolutionary Swarm Robotics. In Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, GECCO ’13, page 199–206, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450319638. doi: 10.1145/2463372.2463398. URL https://doi.org/10.1145/2463372.2463398.
Prieto et al. [2016] Abraham Prieto, Francisco Bellas, Pedro Trueba, and Richard J Duro. Real-time optimization of dynamic problems through distributed embodied evolution. Integrated Computer-Aided Engineering, 23(3):237–253, 2016.
Cambier et al. [2020] Nicolas Cambier, Roman Miletitch, Vincent Fremont, Marco Dorigo, Eliseo Ferrante, and Vito Trianni. Language Evolution in Swarm Robotics: A Perspective. Frontiers in Robotics and AI, 7, 2020. ISSN 2296-9144. doi: 10.3389/frobt.2020.00012. URL https://www.frontiersin.org/articles/10.3389/frobt.2020.00012.
Cambier et al. [2018] Nicolas Cambier, Vincent Frémont, Vito Trianni, and Eliseo Ferrante. Embodied evolution of self-organised aggregation by cultural propagation. In Marco Dorigo, Mauro Birattari, Christian Blum, Anders L. Christensen, Andreagiovanni Reina, and Vito Trianni, editors, Swarm Intelligence, pages 351–359, Cham, 2018. Springer International Publishing.
Cambier et al. [2021] Nicolas Cambier, Dario Albani, Vincent Fremont, Vito Trianni, and Eliseo Ferrante. Cultural evolution of probabilistic aggregation in synthetic swarms. Applied Soft Computing, 113:108010, 2021. ISSN 1568-4946. doi: https://doi.org/10.1016/j.asoc.2021.108010. URL https://www.sciencedirect.com/science/article/pii/S1568494621009327.
Berner et al. [2023] Rico Berner, Thilo Gross, Christian Kuehn, Jürgen Kurths, and Serhiy Yanchuk. Adaptive Dynamical Networks, 2023.
Ma et al. [2024] Zhuangzhuang Ma, Lei Shi, Kai Chen, **liang Shao, and Yuhua Cheng. Multi-Agent Bipartite Flocking Control over Cooperation-Competition Networks with Asynchronous Communications. IEEE Transactions on Signal and Information Processing over Networks, pages 1–12, 2024. doi: 10.1109/TSIPN.2024.3384817.
Piazza et al. [2024] Nancirose Piazza, Vahid Behzadan, and Stefan Sarkadi. The Power in Communication: Power Regularization of Communication for Autonomy in Cooperative Multi-Agent Reinforcement Learning, 2024.
Tang et al. [2024] Huaze Tang, Yuanquan Hu, Fanfan Zhao, Junji Yan, Ting Dong, and Wenbo Ding. M3ARL: Moment-Embedded Mean-Field Multi-Agent Reinforcement Learning for Continuous Action Space. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7250–7254, 2024. doi: 10.1109/ICASSP48485.2024.10448058.
Vieillard et al. [2020] Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Munchausen Reinforcement Learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4235–4246. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/2c6a0bae0f071cbbf0bb3d5b11d90a82-Paper.pdf.
Wu et al. [2024c] Zida Wu, Mathieu Lauriere, Samuel Jia Cong Chua, Matthieu Geist, Olivier Pietquin, and Ankur Mehta. Population-aware Online Mirror Descent for Mean-Field Games by Deep Reinforcement Learning, 2024c.
Perrin et al. [2022] Sarah Perrin, Mathieu Laurière, Julien Pérolat, Romuald Élie, Matthieu Geist, and Olivier Pietquin. Generalization in mean field games by learning master policies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 9413–9421, 2022.
Carmona et al. [2021] René Carmona, Mathieu Laurière, and Zongjun Tan. Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning, 2021.
Zhang and Sutton [2017] Shangtong Zhang and Richard S Sutton. A deeper look at experience replay. arXiv preprint arXiv:1712.01275, 2017.

Technical appendices to article
“Networked Communication for Decentralised Agents in Mean-Field Games”

Appendix A Further definitions and assumptions for theorems in Sec. 3.3

Assumption 1 (Lipschitz continuity of $P$ and $R$ , from Assumption 1, Yardim et al. [6]).

There exist constants $K_{\mu},K_{s},K_{a},L_{\mu},L_{s},L_{a}\in\mathbb{R}_{\geq 0}$ such that $\forall s,s^{\prime}\in\mathcal{S},\forall a,a^{\prime}\in\mathcal{A},\forall% \mu,\mu^{\prime}\in\Delta_{\mathcal{S}}$ ,

||P(\cdot|s,a,\mu)-P(\cdot|s^{\prime},a^{\prime},\mu^{\prime})||_{1}\leq K_{% \mu}||\mu-\mu^{\prime}||_{1}+K_{s}d(s,s^{\prime})+K_{a}d(a,a^{\prime}),

|R(s,a,\mu)-R(s^{\prime},a^{\prime},\mu^{\prime})|\leq L_{\mu}||\mu-\mu^{% \prime}||_{1}+L_{s}d(s,s^{\prime})+L_{a}d(a,a^{\prime}).\qed

Definition 9 (Population update operator, from Definition 3.1, Yardim et al. [6]).

The single-step population update operator $\Gamma_{pop}:\Delta_{\mathcal{S}}\times\Pi\rightarrow\Delta_{\mathcal{S}}$ is defined as, $\forall s\in\mathcal{S}$ :

\Gamma_{pop}(\mu,\pi)(s):=\sum_{s^{\prime}\in\mathcal{S}}\sum_{a^{\prime}\in% \mathcal{A}}\mu(s^{\prime})\pi(a^{\prime}|s^{\prime})P(s|s^{\prime},a^{\prime}% ,\mu).

Let us use the short hand notation $\Gamma^{n}_{pop}(\mu,\pi):=\underbrace{\Gamma_{pop}(\dots\Gamma_{pop}(\Gamma_{% pop}(\mu,\pi),\pi),\dots,\pi)}_{n\text{ times}}$ . ∎

We recall that $\Gamma_{pop}$ is known to be Lipschitz:

Lemma 1 (Lipschitz population updates, from Lemma 3.2, Yardim et al. [6]).

$\Gamma_{pop}$ is Lipschitz with

||\Gamma_{pop}(\mu,\pi)-\Gamma_{pop}(\mu^{\prime},\pi^{\prime})||_{1}\leq L_{% pop,\mu}||\mu-\mu^{\prime}||_{1}+\frac{K_{a}}{2}||\pi-\pi^{\prime}||_{1},

where $L_{pop,\mu}$ := $\left(\frac{K_{s}}{2}+\frac{K_{a}}{2}+K_{\mu}\right)$ , $\forall\pi\in\Pi,\mu\in\Delta_{\mathcal{S}}$ . ∎

For stationary MFGs the population distribution must be stable with respect to a policy, requiring that $\Gamma_{pop}(\cdot,\pi)$ is contractive $\forall\pi\in\Pi$ :

Assumption 2 (Stable population, from Assumption 2, Yardim et al. [6]).

Population updates are stable, i.e. $L_{pop,\mu}<1$ . ∎

Definition 10 (Stable population operator $\Gamma_{pop}^{\infty}$ , from Definition 3.3, Yardim et al. [6]).

Given Assumption 2, the operator $\Gamma_{pop}^{\infty}:\Pi\rightarrow\Delta_{\mathcal{S}}$ maps a given policy to its unique stable population distribution such that $\Gamma_{pop}(\Gamma_{pop}^{\infty}(\pi),\pi)=\Gamma_{pop}^{\infty}(\pi)$ , i.e. the unique fixed point of $\Gamma_{pop}(\cdot,\pi):\Delta_{\mathcal{S}}\rightarrow\Delta_{\mathcal{S}}.$ ∎

Definition 11 ( $Q_{h}$ and $q_{h}$ functions).

We define, for any pair $(s,a)\in\mathcal{S}\times\mathcal{A}$ :

Q_{h}(s,a|\pi,\mu):=\mathbb{E}\left[\sum^{\infty}_{t=0}\gamma^{t}(R(s_{t},a_{t% },\mu)+h(\pi(s_{t})))\left|\begin{subarray}{c}s_{0}=s,\\ a_{0}=a\end{subarray},\begin{subarray}{c}s_{t+1}\sim P(\cdot|s_{t},a_{t},\mu),% \\ a_{t+1}\sim\pi(\cdot|s_{t+1})\end{subarray},\forall t\geq 0\vphantom{\sum^{% \infty}_{t=0}\gamma^{t}(R(s_{t},a_{t},\mu)+}\right]\right.

and

q_{h}(s,a|\pi,\mu):=R(s,a,\mu)+\gamma\sum_{s^{\prime},a^{\prime}}P(s^{\prime}|% s,a,\mu)\pi(a^{\prime}|s^{\prime})Q_{h}(s^{\prime},a^{\prime}|\pi,\mu).\qed

Definition 12 ( $\Gamma_{q}$ operator).

The operator $\Gamma_{q}:\Pi\times\Delta_{\mathcal{S}}\rightarrow\mathcal{Q}$ map** population-policy pairs to Q-functions is defined as $\Gamma_{q}(\pi,\mu):=q_{h}(\cdot,\cdot|\pi,\mu)\in\mathcal{Q}$ $\forall\pi\in\Pi,\mu\in\Delta_{\mathcal{S}}$ .∎

We also assume that the regulariser $h$ ensures that all actions at all states are explored with non-zero probability:

Assumption 3 (Persistence of excitation, from Assumption 3, Yardim et al. [6]).

We assume there exists $p_{inf}>0$ such that:

1.

$\pi_{max}(a|s)\geq p_{inf}$ $\forall s\in\mathcal{S},a\in\mathcal{A}$ ,
2.

For any $\pi\in\Pi$ and $q\in\mathcal{Q}$ that satisfy, $\forall(s,a)\in\mathcal{S}\times\mathcal{A}$ , $\pi(a|s)\geq p_{inf}$ and $0\leq q(s,a)\leq Q_{max}$ , it holds that $\Gamma^{md}_{\eta}(q,\pi)(a|s)\geq p_{inf}$ , $\forall(s,a)\in\mathcal{S}\times\mathcal{A}$ . ∎

Assumption 4 (Sufficient mixing, from Assumption 4, Yardim et al. [6]).

For any $\pi\in\Pi$ satisfying $\pi(a|s)\geq p_{inf}>0$ $\forall s\in\mathcal{S},a\in\mathcal{A}$ , and any initial states { $s^{i}_{0}$ } ${}_{i}\in\mathcal{S}^{N}$ , there exist $T_{mix}>0,\delta_{mix}>0$ such that $\mathbb{P}(s^{j}_{T_{mix}}=s^{\prime}|$ { $s^{i}_{0}$ } ${}_{i})$ $\geq\delta_{mix}$ , $\forall s^{\prime}\in\mathcal{S},j\in[N]$ .∎

Definition 13 (Nested learning operator).

For a learning rate $\eta>0$ , $\Gamma_{\eta}:\Pi\rightarrow\Pi$ is defined as

\Gamma_{\eta}(\pi):=\Gamma_{\eta}^{md}(\Gamma_{q}(\pi,\Gamma_{pop}^{\infty}(% \pi)),\pi).\qed

Lemma 2 (Lipschitz continuity of $\Gamma_{\eta}$ , from Lemma 3.7, [6])).

For any $\eta>0$ , the operator $\Gamma_{\eta}:\Pi\rightarrow\Pi$ is Lipschitz with constant $L_{\Gamma_{\eta}}$ on ( $\Pi,||\cdot||_{1}$ ).∎

Appendix B Full theorems and proofs

B.1 Sample guarantees of independent-learning case

Lemma 3 (Independent learning, from Theorem 4.5, Yardim et al. [6]).

Define $t_{0}:=\frac{16(1+\gamma)^{2}}{((1-\gamma)\delta_{mix}p_{inf})^{2}}$ . Assume that Assumptions 1, 2, 3 and 4 hold, that $\eta>0$ satisfies $L_{\Gamma_{\eta}}<1$ , and that $\pi^{*}$ is the unique MFG-NE. The learning rates are $\beta_{m}=\frac{2}{(1-\gamma)(t_{0}+m-1)}$ $\forall m\geq 0$ , and let $\varepsilon>0$ be arbitrary. There exists a problem-dependent constant $a\in[0,\infty)$ such that if $K=\frac{\log 8\varepsilon^{-1}}{\log L^{-1}_{\Gamma_{\eta}}}$ , $M_{pg}>\mathcal{O}(\varepsilon^{-2-a})$ and $M_{td}>\mathcal{O}(log^{2}\varepsilon^{-1})$ , then the random output { $\pi^{i}_{K}$ }_i of Alg. 1 when run with $C=0$ (such that there is no communication) satisfies for all agents $i=1,\dots,N$ ,

\mathbb{E}\left[||\pi^{i}_{K}-\pi^{*}||_{1}\right]\;\leq\;\varepsilon+\mathcal% {O}\left(\frac{1}{\sqrt{N}}\right).\qed

B.2 Full version of Theorem 1

Theorem 1 (Networked learning with random adoption).

For $p_{inf}$ and $\delta_{mix}$ defined in Assumptions 3 and 4 respectively, define $t_{0}:=\frac{16(1+\gamma)^{2}}{((1-\gamma)\delta_{mix}p_{inf})^{2}}$ . Assume that Assumptions 1, 2, 3 and 4 hold, and that $\pi^{*}$ is the unique MFG-NE policy. For $L_{\Gamma_{\eta}}$ defined in Lemma 2, we assume $\eta>0$ satisfies $L_{\Gamma_{\eta}}<1$ . The learning rates are $\beta_{m}=\frac{2}{(1-\gamma)(t_{0}+m-1)}$ $\forall m\geq 0$ , and let $\varepsilon>0$ be arbitrary. Assume also that $C>0$ , with $\tau_{k}\rightarrow\infty$ . There exists a problem-dependent constant $a\in[0,\infty)$ such that if $K=\frac{\log 8\varepsilon^{-1}}{\log L^{-1}_{\Gamma_{\eta}}}$ , $M_{pg}>\mathcal{O}(\varepsilon^{-2-a})$ and $M_{td}>\mathcal{O}(log^{2}\varepsilon^{-1})$ , then the random output { $\pi^{i}_{K}$ }_i of Alg. 1 preserves the sample guarantees of the independent-learning case given in Lemma 3, i.e. the output satisfies, for all agents $i=1,\dots,N$ ,

\mathbb{E}\left[||\pi^{i}_{K}-\pi^{*}||_{1}\right]\;\leq\;\varepsilon+\mathcal% {O}\left(\frac{1}{\sqrt{N}}\right).

(Proof in Appx. B.3.)∎

B.3 Proof of Theorem 1

Proof.

If $\tau_{k}\rightarrow\infty$ , the softmax function that defines the probability of a received policy being adopted in Line 17 of Alg. 1 gives a uniform distribution. Policies are thus exchanged at random between communicating agents an arbitrary $C>0$ times, which does not affect the random output of the algorithm, such that the random output satisfies the same expectation as if $C=0$ . ∎

B.4 Conditional TD learning from a single continuous run of the empirical distribution of $N$ agents

Lemma 4 (Conditional TD learning from a single continuous run of the empirical distribution of $N$ agents, from Theorem 4.2, Yardim et al. [6]).

Define $t_{0}:=\frac{16(1+\gamma)^{2}}{((1-\gamma)\delta_{mix}p_{inf})^{2}}$ . Assume Assumption 4 holds and let policies { $\pi^{i}$ }_i be given such that $\pi^{i}(a|s)\geq p_{inf}$ $\forall i$ . Assume Lines 5-11 of Alg. 1 are run with policies { $\pi^{i}$ }_i, arbitrary initial agents states { $s^{i}_{0}$ }_i, learning rates $\beta_{m}=\frac{2}{(1-\gamma)(t_{0}+m-1)}$ , $\forall m\geq 0$ and $M_{pg}>\mathcal{O}(\varepsilon^{-2})$ , $M_{td}>\mathcal{O}(\log\varepsilon^{-1})$ . If $\bar{\pi}\in\Pi$ is an arbitrary policy, $\Delta$ := $\sum_{i=1}^{N}||\pi^{i}-\bar{\pi}||_{1}$ and $Q^{*}$ := $Q_{h}(\cdot,\cdot|\bar{\pi},\mu_{\bar{\pi}})$ , then the random output $\hat{Q}^{i}_{M_{pg}}$ of Lines 5-11 satisfies

\mathbb{E}\left[||\hat{Q}^{i}_{M_{pg}}-\;Q^{*}||_{\infty}\right]\leq% \varepsilon+\mathcal{O}\left(\frac{1}{\sqrt{N}}+\frac{1}{N}\Delta+||\pi^{i}-% \bar{\pi}||_{1}\right).

∎

B.5 Full version of Theorem 2

Theorem 2 (Networked learning with non-random adoption).

Assume that Assumptions 1, 2, 3 and 4 hold, and that Alg. 1 is run with learning rates and constants as defined in Thm. 1, except now $\tau_{k}\in\mathbb{R}_{>0}$ . Assume that $\sigma^{i}_{k+1}$ is generated uniquely for each $i$ , in a manner independent of any metric related to $\pi^{i}_{k+1}$ , e.g. $\sigma^{i}_{k+1}$ is random or related only to the index $i$ (so as not to bias the spread of any particular policy). Let the random output of this Algorithm be denoted as { $\pi^{i,net}_{K}$ }_i. Also consider an independent-learning version of the algorithm (i.e. with the same parameters except $C=0$ ) and denote its random output { $\pi^{j,ind}_{K}$ }_j; and a centralised version of the algorithm with the same parameters (see Rem. 3) and denote its random output as $\pi^{cent}_{K}$ . Then for all agents $i=1,\dots,N$ and $j=1,\dots,N$ , the random outputs { $\pi^{i,net}_{K}$ }_i, { $\pi^{j,ind}_{K}$ }_j and $\pi^{cent}_{K}$ satisfy

\mathbb{E}\left[||\pi^{cent}_{K}-\pi^{*}||_{1}\right]\;\leq\;\mathbb{E}\left[|% |\pi^{i,net}_{K}-\pi^{*}||_{1}\right]\leq\;\mathbb{E}\left[||\pi^{j,ind}_{K}-% \pi^{*}||_{1}\right]\;\leq\;\varepsilon+\mathcal{O}\left(\frac{1}{\sqrt{N}}% \right).

(Proof in Appx. B.6.)∎

B.6 Proof of Theorem 2

Proof.

We build off the proof of our Lemma 3, given in Theorem D.9 of Yardim et al. [6], where the sample guarantees of the independent case are worse than those of the centralised algorithm as a result of the divergence between the decentralised policies due to the stochasticity of the PMA updates. For an arbitrary policy $\bar{\pi}_{k}\in\Pi$ , for all $k=0,1,\dots,K$ define the policy divergence as the random variable $\Delta_{k}$ := $\sum_{i=1}^{N}||\pi^{i}_{k}-\bar{\pi}_{k}||_{1}$ . We can say that $\Delta_{k,cent}=0$ $\forall k$ is the divergence in the centralised case, while in the networked case the policy divergence is $\Delta_{k+1,c}$ after communication round $c\in 1,\dots,C$ . The independent case is equivalent to the scenario when $C=0$ , such that its policy divergence can be written $\Delta_{k+1,0}$ .

For $\tau_{k}\in\mathbb{R}_{>0}$ , the adoption probability Pr $\left(\sigma^{adopted}_{k+1}=\sigma^{j}_{k+1}\right)$ = $\frac{\exp{(\sigma^{j}_{k+1}}/\tau_{k})}{\sum^{[J^{i}_{t}]}_{x=1}\exp{(\sigma^% {x}_{k+1}}/\tau_{k})}$ (as in Line 17 of Alg. 1) is higher for some $j\in J^{i}_{t}$ than for others. This means that for $c>0$ for which there are communication links in the population, in expectation the number of unique policies in the population will decrease, as it will likely become that $\pi^{i}_{k+1}=\pi^{j}_{k+1}$ for some $i,j\in\{1,\dots,N\}$ . As such, $\Delta_{k+1,cent}\leq\mathbb{E}\left[\Delta_{k+1,c}\right]\leq\mathbb{E}\left[% \Delta_{k+1,0}\right]$ , i.e. the policy divergence in the independent-learning case is expected to be greater than or equal to that of the networked case.

The proof of Lemma 3 given in Theorem D.9 of Yardim et al. [6] ends with, for constants $\chi$ and $\xi$ ,

\mathbb{E}\left[||\pi^{i}_{K}-\pi^{*}||_{1}\right]\leq 2L_{\Gamma_{\eta}}^{K}+% \frac{\chi}{1-L_{\Gamma_{\eta}}}+\xi\sum^{K-1}_{k=1}L_{\Gamma_{\eta}}^{K-k-1}% \mathbb{E}\left[\Delta_{k}\right],

where in our context the policy divergence in the independent case $\mathbb{E}\left[\Delta_{k+1}\right]$ is equivalent to $\mathbb{E}\left[\Delta_{k+1,c}\right]$ when $C=0$ , i.e. $\mathbb{E}\left[\Delta_{k+1,0}\right]$ .

Since $\Delta_{k+1,cent}\leq\mathbb{E}\left[\Delta_{k+1,c}\right]\leq\mathbb{E}\left[% \Delta_{k+1,0}\right]$ , we obtain our result, i.e. for all agents $i=1,\dots,N$ and $j=1,\dots,N$ , the random outputs { $\pi^{i,net}_{K}$ }_i, { $\pi^{j,ind}_{K}$ }_j and $\pi^{cent}_{K}$ satisfy

\mathbb{E}\left[||\pi^{cent}_{K}-\pi^{*}||_{1}\right]\;\leq\;\mathbb{E}\left[|% |\pi^{i,net}_{K}-\pi^{*}||_{1}\right]\leq\;\mathbb{E}\left[||\pi^{j,ind}_{K}-% \pi^{*}||_{1}\right]\;\leq\;\varepsilon+\mathcal{O}\left(\frac{1}{\sqrt{N}}% \right).

∎

Remark 5.

It may help to see that our result is a consequence of the following. Denote $\hat{Q}^{i,net}_{M_{pg}}$ , $\hat{Q}^{i,ind}_{M_{pg}}$ and $\hat{Q}^{cent}_{M_{pg}}$ as the random outputs of Lines 5-11 of Alg. 1 in the networked, independent and centralised cases respectively. In Lemma 4, we can see that policy divergence gives bias terms in the estimation of the Q-value. Therefore, given $\Delta_{k+1,cent}\leq\mathbb{E}\left[\Delta_{k+1,c}\right]\leq\mathbb{E}\left[% \Delta_{k+1,0}\right]$ , we can also say

\mathbb{E}\left[||\hat{Q}^{cent}_{M_{pg}}-Q^{*}||_{\infty}\right]\;\leq\;% \mathbb{E}\left[||\hat{Q}^{i,net}_{M_{pg}}-Q^{*}||_{\infty}\right]\\ \leq\;\mathbb{E}\left[||\hat{Q}^{i,ind}_{M_{pg}}-Q^{*}||_{\infty}\right].

In other words, the networked case will require the same or fewer outer iterations $K$ to reduce the variance caused by this bias than the independent case requires (where the bias is non-vanishing), and the same or more iterations than the centralised case requires. ∎

B.7 Continuation of Rem. 4

Remark 6.

For an arbitrary policy $\bar{\pi}_{k}\in\Pi$ , for all $k=0,1,\dots,K$ define the policy divergence as the random variable $\Delta_{k}$ := $\sum_{i=1}^{N}||\pi^{i}_{k}-\bar{\pi}_{k}||_{1}$ . We can say that $\Delta_{k,cent}=0$ $\forall k$ is the divergence in the centralised case, while in the networked case the policy divergence is $\Delta_{k+1,c}$ after communication round $c\in C$ . As detailed in Appx. B.6 and as per Rem. 2, if $\tau_{k}\rightarrow 0$ and $C$ is large enough that the number of jointly connected collections of graphs occurring within $C$ is equal to the largest diameter of the union of any collection, then $\Delta_{k,C}=\Delta_{k,cent}=0$ $\forall k\in K$ . (If it is always $\sigma^{1}_{k+1}$ and $\pi^{1}_{k+1}$ that is adopted by the whole population then this is the same as the centralised case; if the $\sigma^{i}_{k+1}$ and $\pi^{i}_{k+1}$ that gets adopted has different $i$ for each $k$ then this is akin to a version of the centralised setting where the central learning agent may differ for each $k$ .) If instead $\tau_{k}\in\mathbb{R}_{>0}$ , we have $\lim_{C\rightarrow\infty}\mathbb{E}\left[\Delta_{k+1,C}\right]\rightarrow 0$ , assuming that the communication network becomes jointly connected infinitely often. ∎

B.8 Remark on theoretical hyperparameters when used in practical settings

Remark 7.

The theoretical analysis in Sec. 3.3 and Appx. B requires algorithmic hyperparameters (Thms. 1 and 2) that render convergence impractically slow in all of the centralised, independent and networked cases. (Indeed Yardim et al. [6] do not provide empirical demonstrations of their algorithms for the centralised and independent cases.) In particular, the values of $\delta_{mix}$ and $p_{inf}$ give rise to very large $t_{0}$ , causing very small learning rates $\{\beta_{m}\}_{m\in\{0,\dots,M_{pg}-1\}}$ , and necessitating very large values for $M_{td}$ and $M_{pg}$ .

Appendix C Algorithm acceleration by use of experience-replay buffer (further details)

The intuition behind the better learning efficiency resulting from our experience replay buffer in Alg. 2 is as follows. The value of a state-action pair $p$ is dependent on the values of subsequent states reached, but the value of $p$ is only updated when the TD update is conducted on $p$ , rather than every time a subsequent pair is updated. By learning from each stored transition multiple times, we not only make repeated use of the reward and transition information in each costly experience, but also repeatedly update each state-action pair in light of its likewise updated subsequent states.

We leave $\beta_{m}$ fixed across all iterations, as we found empirically that this yields sufficient learning. We have not experimented with decreasing $\beta$ as $l$ increases, though this may benefit learning.

The transitions in the buffer are discarded after the replay cycles and a new buffer is initialised for the next iteration $k$ , as in Line 6. As such the space complexity of the buffer only grows linearly with the number of $M_{pg}$ iterations within each outer loop $k$ , rather than with the number of $K$ loops.

Algorithm 2 Networked learning with experience replay

1:loop parameters

K,M_{pg},M_{td},C

L

, E, learning parameters

\eta

\beta

\lambda,\gamma

\{\tau_{k}\}_{k\in\{0,\dots,K-1\}}

2:initial states {

s^{i}_{0}

}_i,

i=1,\ldots,N

3:Set

\pi^{i}_{0}=\pi_{max},\forall i

and

t\leftarrow 0

4:for

k=0,\dots,K-1

\forall s,a,i:\hat{Q}^{i}_{0}(s,a)=Q_{max}

\forall i

: Empty

i

’s buffer

7: for

m=0,\dots,M_{pg}-1

8: for

M_{td}

iterations do

9: Take step

\forall i:a^{i}_{t}\sim\pi^{i}_{k}(\cdot|s^{i}_{t}),r^{i}_{t}=R(s^{i}_{t},a^{i% }_{t},\hat{\mu}_{t}),s^{i}_{t+1}\sim P(\cdot|s^{i}_{t},a^{i}_{t},\hat{\mu}_{t})

;

t\leftarrow t+1

10: end for

11:

\forall i

: Add

\zeta^{i}_{t-2}

i

’s buffer

12: end for

13: for

l=0,\dots,L-1

14:

\forall i:

Shuffle buffer

15: for transition

\zeta^{i}_{b}

i

’s buffer do (

\forall i

)

16: Compute TD update (

\forall i

\hat{Q}^{i}_{m+1}=\tilde{F}^{\pi^{i}_{k}}_{\beta}(\hat{Q}^{i}_{m},\zeta^{i}_{t% -2})

(see Def. 7)

17: end for

18: end for

19: PMA step

\forall i:\pi^{i}_{k+1}=\Gamma^{md}_{\eta}(\hat{Q}^{i}_{M_{pg}},\pi^{i}_{k})

(see Def. 8)

20:

\forall i:

\sigma^{i}_{k+1}=0

21: for

e=0,\dots,E-1

evaluation steps do

22: Take step

\forall i:a^{i}_{t}\sim\pi^{i}_{k+1}(\cdot|s^{i}_{t}),r^{i}_{t}=R(s^{i}_{t},a^% {i}_{t},\hat{\mu}_{t}),s^{i}_{t+1}\sim P(\cdot|s^{i}_{t},a^{i}_{t},\hat{\mu}_{% t})

23:

\forall i:

\sigma^{i}_{k+1}=\sigma^{i}_{k+1}+\gamma^{e}(r^{i}_{t}+h(\pi^{i}_{k+1}(s^{i}_{% t})))

24:

t\leftarrow t+1

25: end for

26: for

C

rounds do

27:

\forall i:

Broadcast

\sigma^{i}_{k+1},\pi^{i}_{k+1}

28:

\forall i:J^{i}_{t}=i\cup\{j\in\mathcal{N}:(i,j)\in\mathcal{E}_{t}

}

29:

\forall i:

Select

\sigma^{adopted}_{k+1}\sim

\left(\sigma^{adopted}_{k+1}=\sigma^{j}_{k+1}\right)

\frac{\exp{(\sigma^{j}_{k+1}}/\tau_{k})}{\sum^{[J^{i}_{t}]}_{x=1}\exp{(\sigma^% {x}_{k+1}}/\tau_{k})}

\forall j\in J^{i}_{t}

30:

\forall i:\sigma^{i}_{k+1}=\sigma^{adopted}_{k+1},\pi^{i}_{k+1}=\pi^{adopted}_% {k+1}

31: Take step

\forall i:a^{i}_{t}\sim\pi^{i}_{k+1}(\cdot|s^{i}_{t}),r^{i}_{t}=R(s^{i}_{t},a^% {i}_{t},\hat{\mu}_{t}),s^{i}_{t+1}\sim P(\cdot|s^{i}_{t},a^{i}_{t},\hat{\mu}_{% t})

;

t\leftarrow t+1

32: end for

33:end for

34:Return policies {

\pi^{i}_{K}

}_i,

i=1,\ldots,N

Appendix D Extended discussion on robustness of communication networks in MFGs and related experimental settings

We consider two scenarios to which we desire real-world many-agent systems (e.g. robotic swarms or autonomous vehicle traffic) to be robust; these scenarios form the basis of our experiments on robustness (see Sec. 4 and Figs. 3, 3, 5 and 6). The networked setup affords population fault-tolerance and online scalability, which are motivating qualities of many-agent systems.

Firstly, we consider a scenario in which the learning/updating procedure of agents fails with a certain probability within each iteration, in which cases $\pi^{i}_{k+1}=\pi^{i}_{k}$ (see Figs. 3 and 5 for our experimental results and discussion of this scenario). In real-life decentralised settings, this might be particularly liable to occur since the updating process might only be synchronised between agents by internal clock ticks, such that some agents may not complete their update in the allotted time but will nevertheless be required to take the next step in the environment. Such failures slow the improvement of the population in the independent case, and in the centralised setting it means no improvement occurs at all in any iteration in which failure occurs, as there is a single point of failure. Networked communication instead provides redundancy in case of failures, with the updated policies of any agents that have managed to learn spreading through the population to those that have not. This feature thus ensures that improvement can continue for potentially the whole population even if a high number of agents do not manage to learn at a given iteration.

Secondly, we may want to arbitrarily increase the size of a population of agents that are already learning or operating in the environment (we can imagine extra fleets of autonomous cars or drones being deployed) - see Appx. F for comparison with other works considering this type of robustness [75, 76, 77, 78]. A purely independent setting would require all the new agents to learn a policy individually given the existing distribution, and the process of their following and improving policies from scratch may itself disturb the NE that has already been achieved by the original population. With a communication network, however, the policies that have been learnt so far can quickly be shared with the new agents in a decentralised way, hopefully before their unoptimised policies can destabilise the current NE. This would provide, for example, a way to bootstrap a large population from a smaller pre-trained group, if training were considered expensive in a given setting. See Figs. 3 and 6 for our experimental results and discussion of this scenario.

Appendix E Experiments

Experiments were conducted on a MacBook Pro, Apple M1 Max chip, 32 GB, 10 cores. We use scipy.optimize.minimize (employing Sequential Least Squares Programming) to conduct the optimisation step in Def. 8, and the JAX framework to accelerate and vectorise some elements of our code. We report the CPU time for each experiment in its caption; the total CPU time for the experiments presented in this work was 1,747,001 secs. ( $\sim$ 485 hours), though the full research project required more compute time due to testing other hyperparameter values (see Table 1) for which we do not give plots here.

For reproducibility, the code to run our experiments is provided with our Supplementary Material, and will be made publicly available upon publication.

E.1 Games

We conduct numerical tests with two games (defined by the agents’ objectives), chosen for being particularly amenable to intuitive and visualisable understanding of whether the agents are learning behaviours that are appropriate and explainable for the respective objective functions. In all cases, rewards are normalised in [0,1] after they are computed.

Cluster.

This is the inverse of the ‘exploration’ game in [30], where in our case agents are encouraged to gather together by the reward function $R(s^{i}_{t},a^{i}_{t},\hat{\mu}_{t})=$ log $(\hat{\mu}_{t}(s^{i}_{t}))$ . That is, agent $i$ receives a reward that is logarithmically proportional to the fraction of the population that is co-located with it at time $t$ . We give the population no indication where they should cluster, agreeing this themselves over time.

Agree on a single target.

Unlike in the above ‘cluster’ game, the agents are given options of locations at which to gather, and they must reach consensus among themselves. If the agents are co-located with one of a number of specified targets $\phi\in\Phi$ (in our experiments we place one target in each of the four corners of the grid), and other agents are also at that target, they get a reward proportional to the fraction of the population found there; otherwise they receive a penalty of -1. In other words, the agents must coordinate on which of a number of mutually beneficial points will be their single gathering place. The reward function is given by $R(s^{i}_{t},a^{i}_{t},\hat{\mu}_{t})=r_{targ}(r_{collab}(\hat{\mu}_{t}(s^{i}_{% t})))$ , where

r_{targ}(x)=\begin{cases}x\quad&\text{if}\,\exists\phi\in\Phi\text{ s.t. dist}% (s^{i}_{t},\phi)=0\\ -1\quad&\text{otherwise,}\\ \end{cases}

r_{collab}(x)=\begin{cases}x\quad&\text{if}\,\hat{\mu}_{t}(s^{i}_{t})>1/N\\ -1\quad&\text{otherwise.}\\ \end{cases}

E.2 Experimental metrics

To give as informative results as possible about both performance and proximity to the NE, we provide several metrics for each experiment. All metrics are plotted with 2-sigma error bars (2 $\times$ standard deviation), computed over the five trials (each with a random seed) of the system evolution in each setting. This is computed based on a call to numpy.std for each metric over each run.

E.2.1 Exploitability

Works on MFGs frequently use the exploitability metric to evaluate how close a given policy $\pi$ is to a NE policy $\pi^{*}$ [30, 31, 32, 40, 79]. The metric quantifies how much an agent can benefit by deviating from the policy pursued by the rest of the population, by measuring the difference between the return given by a policy that maximises the expected discounted regularised (via $h$ ) reward $V_{h}$ for a given population distribution, and the return given by the policy that gives rise to this distribution. If $\pi$ has a large exploitability then an agent can significantly improve its return by deviating from $\pi$ , meaning that $\pi$ is far from $\pi^{*}$ , whereas an exploitability of 0 implies that $\pi=\pi^{*}$ . Denote by $\mu^{\pi}$ the distribution generated when $\pi$ is the policy followed by all of the population aside from the deviating agent; then the exploitability of policy $\pi$ is defined as:

\mathcal{E}(\pi)=\max_{\pi^{\prime}}V_{h}(\pi^{\prime},\mu^{\pi})-V_{h}(\pi,% \mu^{\pi}).

Since we do not have access to the exact best response policy $\arg\max_{\pi^{\prime}}V_{h}(\pi^{\prime},\mu^{\pi})$ , we instead approximate the exploitability metric, similarly to [34], as follows. We freeze the policy of all agents apart from a deviating agent, for which we store its current policy and then conduct 40 $k$ loops of policy improvement (we found that 40 iterations was enough to converge to a policy that maximised $V_{h}$ for the given population distribution). To approximate the expectations, we take the best return of the deviating agent across the 40 $k$ loops, as well as the mean of all the other agents’ returns across these same loops. We then revert the agent back to its stored policy, before learning continues for all agents. As such, the quality of our approximation is limited by the number of policy improvement rounds, which must be restricted for the sake of running speed of the experiments. Due to the expensive computations required for this metric, we evaluate it on alternate $k$ iterations.

Since prior works conducting empirical testing have generally focused on the centralised setting, evaluations have not had to consider the exploitability metric when not all agents are following a single policy $\pi_{k}$ , as may occur in the the independent or networked settings. The method described above for approximating exploitability involves calculating the mean return of all non-deviating agents’ policies. While this is $\pi_{k}$ in the centralised case, if the non-deviating agents do not share a single policy, then this method is in fact approximating the exploitability of their joint policy $\boldsymbol{\pi}^{-d}_{k}$ , where $d$ is the deviating agent.

E.2.2 Average discounted return

We record the average discounted return of the agents’ policies $\pi_{k}^{i}$ during the $M_{pg}$ steps - this allows us to observe that settings that converge to similar exploitability values may not have similar average agent returns, suggesting that some algorithms are better than others at finding not just NE, but preferable NE. See for example Fig. 10, where the networked agents converge to similar exploitability as the independent agents, but receive higher average reward.

E.2.3 Policy divergence

We record the population’s average policy divergence $\frac{1}{N}\Delta_{k}$ := $\frac{1}{N}\sum_{i=1}^{N}||\pi_{k}^{i}-{\pi_{k}^{1}}||_{1}$ for the arbitrary policy $\bar{\pi}={\pi^{1}}$ . This allows us to demonstrate that populations approaching the NE (i.e. with joint exploitability approaching zero) do not necessarily actually share a single policy $\pi^{*}$ as suggested by the theoretical sample guarantees in Sec. 3.3. Our experimental plots show that this is particularly often the case in the independent setting. The greater divergence in the independent case also indicates why convergence is slower here (see Rem. 5).

E.3 Hyperparameters

See Table 1 for our hyperparameter choices. In general, we seek to show that our networked algorithm is robust to ‘poor’ choices of hyperparameters e.g. low numbers of iterations, as may be required when aiming for practical convergence times in complex real-world problems. By contrast, the convergence speed of the independent-learning algorithm (and sometimes also the centralised algorithm) suffers much more significantly without idealised hyperparameter choices. As such, our experimental demonstrations in the plots generally involve hyperparameter choices at the low end of the values we tested during our research.

We can group our hyperparameters into those controlling the size of the experiment, those controlling the number of iterations of each loop in the algorithm and those affecting the learning/policy updates or policy adoption ( $\beta,\eta,\lambda,\tau,\gamma$ ).

\tablefirsthead

Hyper-parameter Value Comment
\tablehead Hyper-parameter Value Comment

\tabletail

Continued on next page
\tablelasttail

Table 1: Hyperparameters

Hyper-parameter	Value	Comment
Gridsize	8x8 / 16x16	Most experiments are run on the smaller grid, while Figures 7 and 8 showcase learning in a larger state space.
Trials	5	We run 5 trials with different random seeds for each experiment. We plot the mean and 2-sigma error bars for each metric across the trials.
Population	250	We tested $N$ in {25,50,100,200,250,500}, with the networked architecture generally performing equally well with all population sizes $\geq$ 50. We chose 250 for our demonstrations, to show that our algorithm can handle large populations, indeed often larger than those demonstrated in other mean-field works, especially for grid-world environments [50, 62, 74, 80, 81, 82, 83, 84, 85], while also being feasible to simulate wrt. time and computation constraints. In experiments testing robustness to population increase, the population instead begins at 50 agents and has 200 added at the marked point.
$K$	200	$K$ is chosen to be large enough to see exploitability reducing, and converging where possible.
$M_{pg}$	500 / 1000	We wish to illustrate the benefits of our networked architecture and replay buffer in reducing the number of loops required for convergence, i.e. we wish to select a low value that still permits learning. We tested $M_{pg}$ in {300,500,600,800,1000,1200,1300, 1400,1500,1800,2000,2500,3000}, and chose 500 for demonstrations on the 8x8 grids, and 1000 for the 16x16 grids. It may be possible to optimise these values further in combination with other hyperparameters.
$M_{td}$	1	We tested $M_{td}$ in {1,2,10,100}, and found that we were still able to achieve convergence with $M_{td}=1$ . This is much lower than the requirements of the theoretical algorithms, essentially allowing us to remove the innermost nested learning loop.
$C$	1	We tested $C$ in {1,20,50,300}. We choose 1 to show the convergence benefits brought by even a single communication round, even in networks that may have limited connectivity; higher $C$ has even better performance.
$L$	100	As with $M_{pg}$ , we wish to select a low value that still permits learning. We tested $L$ in {50,100,200,300,400,500}. In combination with our other hyperparameters, we found $L\leq 50$ led to less good results, but it may be possible to optimise this hyperparameter further.
$E$	100	We tested $E$ in {100,300,1000}, and choose the lowest value to show the benefit to convergence even from very few evaluation steps. It may be possible to reduce this value further and still achieve similar results.
$\gamma$	0.9	Standard choice across RL literature.
$\beta$	0.1	We tested $\beta$ in {0.01,0.1} and found 0.1 to be small enough for sufficient learning at an acceptable speed. Further optimising this hyperparameter (including by having it decay with increasing $l\in 0,\dots,L-1$ , rather than leaving it fixed) may lead to better results.
$\eta$	0.01	We tested $\eta$ in {0.001,0.01,0.1,1,10} and found that 0.01 gave stable learning that progressed sufficiently quickly.
$\lambda$	0	We tested $\lambda$ in {0,0.0001,0.001,0.01,0.1,1}. Since we are able to reduce $\lambda$ to 0 with no detriment to empirical convergence, we do so in order not to bias the NE.
$\tau_{k}$	cf. comment	For fixed $\tau_{k}$ $\forall k$ , we tested {1,10,100,1000}. In our experiments for fixed $\tau_{k}$ the value is 100 (see Figs. 9 and 10); this yields learning, but does not perform as well as if we anneal $\tau_{k}$ as follows. We begin with $\tau_{0}=10000/(10**\lceil(K-1)/10\rceil$ ), and multiply $\tau_{k}$ by 10 whenever $k\bmod 10=1$ i.e. every 10 iterations. Further optimising the annealing process may lead to better results.

E.4 Additional experiments and discussion

In this section we showcase results with our standard hyperparameter choices continuing from those shown in Sec. 4.1 (Figs. 4, 5 and 6), and we also vary several hyperparameters to show their effects on convergence (Figs. 7 - 10).

Remark 8.

Note that the reward structure of our games is such that exploitability sometimes increases from its initial value before it decreases down to 0. This is because agents are rewarded proportionally to how many other agents are co-located with them: when agents are evenly dispersed at the beginning of the run, it is difficult for even a deviating, best-responding agent to significantly increase its reward. However, once some agents start to aggregate, a best-responding agent can take advantage of this to substantially increase its reward (giving higher exploitability), before all the other agents catch up and aggregate at a single point, reducing the exploitability down to 0. Due to this arc, in some of our plots the independent case may have lower exploitability at certain points than the other architectures, but this is not necessarily a sign of good performance. In fact, we can see in some such cases that the independent case is not learning at all, with the independent agents’ average return not increasing and the exploitability staying level rather than ultimately decreasing (see, for example, Figs. 3, 4, 6) and 7.∎

In our additional experiments, where the results are discussed fully in each figure’s caption, the factors we vary to show the effects on convergence are as follows:

•

Grid size. Figs. 7 and 8 show the result of learning on a grid of size 16x16 instead of 8x8 as in all other experiments. There is greater differentiation in this setting than in the 8x8 grid between the performances of the different broadcast radii of the networked architecture (as is to be expected in a less densely populated environment). The networked architecture continues to significantly outperform the independent case for most broadcast radii, and sometimes even the centralised case.
•

Choice of softmax temperature. Figs. 9 and 10 illustrate the effect of fixed $\{\tau_{k}\}_{k\in\{0,\dots,K-1\}}=100$ , where the networked architecture does not perform as well as if we use the stepped annealing scheme employed in all the other experiments and detailed in Table 1. The intuition behind the better performance achieved with the annealing scheme is as follows. If we begin with $\tau_{k}\rightarrow 0$ (such that the softmax approaches being a max function), we heavily favour the adoption of the highest rewarded policies to speed up progress in the early stages of learning. Subsequently we increase $\tau_{k}$ in steps, promoting greater randomness in adoption, so that as the agents come closer to equilibrium, poorer policy updates that nevertheless receive a high return (due to randomness) do not introduce too much instability to learning and prevent convergence.

Appendix F Extended related work

Multi-agent reinforcement learning (MARL) [43, 86] is a generalisation of reinforcement learning [87] that has recently seen empirical success in a variety of domains, underpinned by breakthroughs in deep learning, including robotics [88, 89, 90, 91, 92], smart autonomy and infrastructures [93, 94], complex games [95, 96, 97], economics [98, 99], social science and cooperative AI [100, 101, 102, 103]. However, it has been computationally difficult to scale MARL algorithms beyond configurations with agents numbering in the low tens, as the joint state and action spaces grow exponentially with the number of agents [3, 30, 31, 99, 104, 105, 106]. Nevertheless, the value of reasoning about interactions among very large populations of agents has been recognised, and an informal distinction is sometimes drawn between multi- and many-agent systems [107, 108, 109]. The latter situation can be more useful (as in cases where better solutions arise from the presence of more agents [90, 110, 111, 75]), more parallelisable [112], more fault tolerant [113], or otherwise more reflective of certain real-world systems involving large numbers of decision makers [75, 98, 99, 114]. Indeed, MFGs have been applied to a wide variety of real world problems, including financial markets [11]; cryptocurrency mining [12]; autonomous vehicles [16]; traffic signal control [17]; resource management in fisheries [21]; crowdsensing [25]; electric vehicle charging [27]; communication networks [22, 28]; swarms [23]; data collection by UAVs [24]; edge computing [13, 14, 15]; cloud resource management [18]; smart grids, and other large-scale cyber-physical systems [19, 20, 26].

Our networked communication framework possesses all of the following desirable qualities for mean-field algorithms when applied to large, complex real-world many-agent systems: learning from the empirical distribution of $N$ agents without generation or manipulation of this distribution by the algorithm itself or by an external oracle; learning from a single continued system run that is not arbitrarily reset as in episodic learning (also referred to in other works as a single sample path/trajectory [5, 6]); model-free learning; decentralisation; fast practical convergence; and robustness to unexpected failures of decentralised learners or changes in the size of the population.

Conversely, as we emphasise in Sec. 1, the MFG framework was originally mainly theoretical [1, 2]. The MFG-NE is traditionally found by solving a coupled system of dynamical equations: a forward evolution equation for the mean-field distribution, and a backwards equation for the representative agent’s optimal response to the mean-field distribution, as in Def. 5 [21]; crucially, these methods relied on the assumption of an infinite population [32]. Early work solved the coupled equations using numerical methods that did not scale well for more complex state and action spaces [115, 116, 117, 118]; or, even if they could handle higher-dimensional problems, the methods were based on known models of the environment’s dynamics (i.e. they were model-based) [4, 33, 36, 37, 38, 39, 119, 120], and/or computed a best-response to the mean-field distribution [2, 30, 31, 32, 33, 34, 35, 40]. The latter approach is both computationally inefficient in non-trivial settings [6, 32], and in many cases is not convergent (as in general it does not induce a contractive operator) [30, 62]. Subsequent work, including our own, has therefore moved towards model-free and/or policy-improvement scenarios [20, 32, 80, 81, 121, 122, 123, 124, 125], possibly with learning taking place by observing N-agent empirical population distributions [6, 10, 50].

Most prior works, including algorithms designed to solve the MFG using an N-agent empirical distribution, have also assumed an oracle that can generate samples of the game dynamics (for any distribution) to be provided to the learning agent [4, 33, 80, 126, 127], or otherwise that the algorithm has direct control over the population distribution at each time step, such as in cases when the agents’ policies and distribution are updated on different timescales [41], with the fictitious play method being particularly popular [3, 5, 18, 30, 31, 34, 49, 61, 73, 81, 121, 128, 129, 130, 131]. In practice, many-agent problems may not admit such arbitrary generation or manipulation (for example, in the context of robotics or controlling vehicle traffic), and so a desirable quality of learning algorithms is that they update only the agents’ policies, rather than being able to arbitrarily reset their states. Learning may thus also need to leverage continuing, rather than episodic, tasks [87]. Yardim et al. [6], Yongacoglu et al. [50] and our own work therefore present algorithms that seek the MFG-NE using only a single run of the empirical population. Almost all prior work relies on a centralised controller to conduct learning on behalf of all the agents [3, 4, 5, 32, 42]. More recent work, including our own, has explored MFG algorithms for decentralised learning with N agents [6, 49, 50, 51, 52, 53, 54, 55].

Naturally, inter-agent communication is most applicable in settings where learning takes place along a continuing system run, rather than the distribution being manipulated by an oracle or arbitrarily reset for new episodes, since these imply a level of external control over the population that results in centralised learning. Equally, it is in situations of learning from finite numbers of real, deployed agents (rather than simulated settings) that we are most likely to be concerned with fault tolerance. As such, our work is most closely related to Yardim et al. [6] and Yongacoglu et al. [50], which provide algorithms for centralised and independent learning with empirical distributions along continued system runs: we contribute a networked learning algorithm in this setting. Yongacoglu et al. [50] empirically demonstrates an independent learning algorithm when agents observe compressed information about the mean-field distribution as well as their local state, but they do not compare this to any other algorithms or baselines. Yardim et al. [6] compares algorithms for centralised and independent learning theoretically, but does not provide empirical demonstrations. In contrast, in addition to providing theoretical guarantees, we empirically demonstrate our networked learning algorithm, where agents observe only their local state, in comparison to both centralised and independent baselines, as well as concerning ourselves with the speed of practical convergence and robustness, unlike these works.

Improving the training speed and sample efficiency of (deep) (multi-agent) RL is gaining increasing attention [132, 133, 134, 135], though our own work is one of the only on MFGs to be concerned with this. Huang and Lai [136] trains on a distribution of MFG configurations to speed up inference on unseen problems, but does not learn online in a decentralised manner as in our own work. Similarly, while some attention has been given to the robustness of multi-agent systems to varying numbers of agents, where it is sometimes referred to as ‘ad-hoc teaming’, ‘open-agent systems’, ‘scalability’ or ‘generalisation’ [75], it has more commonly been addressed in MARL [76, 77] than in MFGs [78]. Wu et al. [78] presents an MFG approach that allows new agents to join the population during execution, but training itself takes place offline in a centralised, episodic manner. Our networked communication framework presented in the current work, on the other hand, allows decentralised agents to join the population during online learning and to have minimal impact on the learning process by adopting policies from existing members of the population through communication.

An existing area of work called ‘robust mean-field games’ studies the robustness of these games to uncertainty in the transition and reward functions [19, 137, 138, 139, 140, 141, 142, 143], but does not consider fault-tolerance, despite this being one of the original motivations behind many-agent systems. On the other hand, we focus on robustness to failures and changes in the agent population itself.

We note a similarity between 1. our method for deciding which policies to propagate through the population (described in Sec. 3.4.1) and 2. the computation of evaluation/fitness functions within evolutionary algorithms to indicate which solutions are desirable to keep in the population for the next generation [144]. Moreover, the research avenue broadly referred to as ‘distributed embodied evolution’ involves swarms of agents independently running evolutionary algorithms while operating within a physical/simulated environment and communicating behaviour parameters to neighbours [145, 146], and is therefore even more similar to our setting, where decentralised RL updates are computed locally and then shared with neighbours. In distributed embodied evolution, the computed fitness of solutions helps determine both which are preserved by agents during local updates, and also which are chosen for broadcast or adoption between neighbours [147, 148, 149]. Indeed, some works on distributed embodied evolution specifically consider features or rewards relating to the joint behaviour of the whole population [150, 151], similar to MFGs. The adjacent research area of cultural/language evolution for swarm robotics [152, 153, 154] has similarly demonstrated the combination of evolutionary approaches and multi-agent communication networks for self-organised behaviours in swarms. However, unlike our own work, none of these areas employ reinforcement learning in the update of policies or the computation of the fitness functions.

We preempt objections that communication with neighbours might violate the anonymity that is characteristic of the mean-field paradigm, by emphasising that the communication in our algorithm takes place outside of the ongoing learning-and-updating parts of each iteration. Thus the core learning assumptions of the mean-field paradigm are unaffected, as they essentially apply at a different level of abstraction (a convenient approximation) to the reality we face of $N$ agents that interact within the same environment. Indeed, prior works have combined networks with mean-field theory, such as using a mean field to describe adaptive dynamical networks [155].

Appendix G Limitations and ongoing/future work

Our algorithm for the networked case (Alg. 1), as well as prior work on the centralised and independent cases [6], all have multiple nested loops. This is a potential limitation for real-world implementation, since the decentralised agents might be sensitive to failures in synchronising these loops. However, in practice, we show that our networked architecture provides redundancy and robustness (which the independent-learning algorithm lacks) in case of learning failures that may result from the necessities of synchronisation (see Appx. D). We have also shown that networked communication in combination with the replay buffer allows us to reduce the hyperparameter $M_{td}$ to 1, essentially removing the inner ‘waiting’ loop. Nevertheless, our algorithm still features multiple loops, and future work lies in simplifying the algorithms further to aid practical implementation, possibly by techniques such as asynchronous communication [156].

Since the MFG setting is technically non-cooperative, we have preempted objections that agents would not have incentive to communicate their policies by focusing on coordination games, i.e. where agents seek to maximise only their individual returns, but receive higher rewards when they follow the same strategy as other agents. In this case they stand to benefit by exchanging their policies with others. Nevertheless, in real-world settings, the communication network could be vulnerable to malfunctioning agents or adversarial actors poisoning the equilibrium by broadcasting untrue policy information. It is outside the scope of this paper to analyse how much false information would have to be broadcast by how many agents to affect the equilibrium, but real-world applications may need to compute this and prevent it. Future research to mitigate this risk might build on work such as Piazza et al. [157], where ‘power regularisation’ of information flow is proposed to limit the adverse effects of communication by misaligned agents.

While our MFG algorithms are designed to handle arbitrarily large numbers of agents (and theoretically perform better as $N\rightarrow\infty$ ), the code for our experiments naturally still suffers from a bottleneck of computational speed when simulating agents that in the real world would be acting and learning in parallel, since the GPU can only process JAX-vectorised elements in batches of a certain size.

Our experiments are based on relatively small toy examples that clearly demonstrate the advantages of our new approach, but which lack the complexity of the real-world applications to which we wish to address the approach. It is feasible that in more complex problems, it may not be possible to remove theoretical assumptions and reduce hyperparameter values to the same extent we have demonstrated in our experimental examples.

Moreover, real-world examples would likely require handling larger and continuous state/action spaces (the latter perhaps building on related work such as Tang et al. [158]), which in turn may require (non-linear) function approximation. Our ongoing work therefore involves incorporating neural networks into our networked communication architecture for oracle-free, non-episodic MFG settings. Extending our algorithms in this way, which depends on modifying the PMA step [159, 160], allows us to introduce communication networks to MFGs with non-stationary equilibria, in addition to those with larger state/action spaces. Our method for the latter variety of game is to have agents’ policies depend both on their local state and also on the population distribution [32, 122, 161, 162], but such a high-dimensional observation object requires moving beyond tabular settings to those of function approximation. The present work demonstrates the benefits of the networked architecture when the Q-function is poorly estimated and introduces experience relay buffers to the setting of learning from a continuous run of the empirical system. Both elements are an important bridge to employing (non-linear) function approximation in this setting, where the problems of data efficiency and imprecise value estimation can be even more acute, and where we also employ experience replay buffers to provide uncorrelated data to train the neural networks [163]. When the policy functions are approximated rather than tabular, our agents communicate the functions’ parameters instead of the whole policy as now.

In our future work with non-stationary equilibria, where agents’ policies will also depend on the population distribution, it may be a strong assumption to suppose that decentralised agents with local state observations and limited communication radius would be able to observe the entire population distribution. We will therefore explore a framework of networked agents estimating the empirical distribution from only their local neighbourhood as in [83], and possibly also improving this estimation by communicating with neighbours [50], such that this useful information spreads through the network along with policy parameters.

Appendix H Negative impacts and mitigation

As with many advances in machine learning, and those relating to multi-agent systems in particular, in the long term our research on large populations of coordinating agents could have negative social outcomes if pursued by malicious actors, including surveillance and military uses. However, our work is primarily foundational and far from deployments, and it also has a large range of potential beneficial applications (such as smart grids and disaster response). Moreover, better understanding the dynamics of large multi-agent systems (as we seek to do in this paper) can contribute to ensuring safety by reducing the risks of unintended failures or outcomes.

We hope to help mitigate potential harmful consequences of this research by fostering transparency through submitting our code in the Supplementary Material, which we commit to publishing online under license upon acceptance of the paper. Details of how to install the dependencies and run the code are provided in the README file, and our hyperparameter choices are provided in Table 1. Details of the computational resources required are given in E.

Networked Communication for Decentralised Agents in Mean-Field Games

Abstract

1 Introduction

2 Preliminaries

Definition 1 (N-player symmetric anonymous games).

Definition 2 (N-player discounted regularised return).

Definition 3 (δ𝛿\deltaitalic_δ-NE).

Definition 4 (Mean-field discounted regularised return).

Definition 5 (NE of stationary MFG).

Proposition 1 (N-player NE and MFG-NE (Thm. 1, [4])).

Remark 1.

Definition 6 (Time-varying communication network).

3 Learning with networked, decentralised agents

3.1 Learning with N𝑁Nitalic_N agents from a single run

Definition 7 (Stochastic TD-learning operator, simplified from Definition 4.1 in Yardim et al. [6]).

Definition 8 (Policy mirror ascent operator (Definition 3.5, [6])).

3.2 Decentralised communication between agents

Remark 2.

Remark 3.

3.3 Properties of policy adoption

Theorem 1 (Networked learning with random adoption).

Theorem 2 (Networked learning with non-random adoption).

Remark 4.

3.4 Practical running of algorithms

3.4.1 Generation of σk+1isubscriptsuperscript𝜎𝑖𝑘1\sigma^{i}_{k+1}italic_σ start_POSTSUPERSCRIPT italic_i end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_k + 1 end_POSTSUBSCRIPT

3.4.2 Algorithm acceleration by use of experience-replay buffer

4 Experiments

4.1 Discussion

References

Technical appendices to article “Networked Communication for Decentralised Agents in Mean-Field Games”

Appendix A Further definitions and assumptions for theorems in Sec. 3.3

Assumption 1 (Lipschitz continuity of P𝑃Pitalic_P and R𝑅Ritalic_R, from Assumption 1, Yardim et al. [6]).

Definition 9 (Population update operator, from Definition 3.1, Yardim et al. [6]).

Lemma 1 (Lipschitz population updates, from Lemma 3.2, Yardim et al. [6]).

Assumption 2 (Stable population, from Assumption 2, Yardim et al. [6]).

Definition 10 (Stable population operator Γp⁢o⁢p∞superscriptsubscriptΓ𝑝𝑜𝑝\Gamma_{pop}^{\infty}roman_Γ start_POSTSUBSCRIPT italic_p italic_o italic_p end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∞ end_POSTSUPERSCRIPT, from Definition 3.3, Yardim et al. [6]).

Definition 11 (Qhsubscript𝑄ℎQ_{h}italic_Q start_POSTSUBSCRIPT italic_h end_POSTSUBSCRIPT and qhsubscript𝑞ℎq_{h}italic_q start_POSTSUBSCRIPT italic_h end_POSTSUBSCRIPT functions).

Definition 12 (ΓqsubscriptΓ𝑞\Gamma_{q}roman_Γ start_POSTSUBSCRIPT italic_q end_POSTSUBSCRIPT operator).

Assumption 3 (Persistence of excitation, from Assumption 3, Yardim et al. [6]).

Assumption 4 (Sufficient mixing, from Assumption 4, Yardim et al. [6]).

Definition 13 (Nested learning operator).

Lemma 2 (Lipschitz continuity of ΓηsubscriptΓ𝜂\Gamma_{\eta}roman_Γ start_POSTSUBSCRIPT italic_η end_POSTSUBSCRIPT, from Lemma 3.7, [6])).

Appendix B Full theorems and proofs

B.1 Sample guarantees of independent-learning case

Lemma 3 (Independent learning, from Theorem 4.5, Yardim et al. [6]).

B.2 Full version of Theorem 1

Theorem 1 (Networked learning with random adoption).

B.3 Proof of Theorem 1

Proof.

B.4 Conditional TD learning from a single continuous run of the empirical distribution of N𝑁Nitalic_N agents

Lemma 4 (Conditional TD learning from a single continuous run of the empirical distribution of N𝑁Nitalic_N agents, from Theorem 4.2, Yardim et al. [6]).

B.5 Full version of Theorem 2

Theorem 2 (Networked learning with non-random adoption).

B.6 Proof of Theorem 2

Proof.

Remark 5.

B.7 Continuation of Rem. 4

Remark 6.

B.8 Remark on theoretical hyperparameters when used in practical settings

Remark 7.

Appendix C Algorithm acceleration by use of experience-replay buffer (further details)

Appendix D Extended discussion on robustness of communication networks in MFGs and related experimental settings

Appendix E Experiments

E.1 Games

Cluster.

Agree on a single target.

E.2 Experimental metrics

E.2.1 Exploitability

E.2.2 Average discounted return

E.2.3 Policy divergence

E.3 Hyperparameters

E.4 Additional experiments and discussion

Remark 8.

Appendix F Extended related work

Appendix G Limitations and ongoing/future work

Appendix H Negative impacts and mitigation

Definition 3 ( $\delta$ -NE).

3.1 Learning with $N$ agents from a single run

3.4.1 Generation of $\sigma^{i}_{k+1}$

Technical appendices to article
“Networked Communication for Decentralised Agents in Mean-Field Games”

Assumption 1 (Lipschitz continuity of $P$ and $R$ , from Assumption 1, Yardim et al. [6]).

Definition 10 (Stable population operator $\Gamma_{pop}^{\infty}$ , from Definition 3.3, Yardim et al. [6]).

Definition 11 ( $Q_{h}$ and $q_{h}$ functions).

Definition 12 ( $\Gamma_{q}$ operator).

Lemma 2 (Lipschitz continuity of $\Gamma_{\eta}$ , from Lemma 3.7, [6])).

B.4 Conditional TD learning from a single continuous run of the empirical distribution of $N$ agents

Lemma 4 (Conditional TD learning from a single continuous run of the empirical distribution of $N$ agents, from Theorem 4.2, Yardim et al. [6]).