-
Strategizing against Q-learners: A Control-theoretical Approach
Authors:
Yuksel Arslantas,
Ege Yuceel,
Muhammed O. Sayin
Abstract:
In this paper, we explore the susceptibility of the independent Q-learning algorithms (a classical and widely used multi-agent reinforcement learning method) to strategic manipulation of sophisticated opponents in normal-form games played repeatedly. We quantify how much strategically sophisticated agents can exploit naive Q-learners if they know the opponents' Q-learning algorithm. To this end, w…
▽ More
In this paper, we explore the susceptibility of the independent Q-learning algorithms (a classical and widely used multi-agent reinforcement learning method) to strategic manipulation of sophisticated opponents in normal-form games played repeatedly. We quantify how much strategically sophisticated agents can exploit naive Q-learners if they know the opponents' Q-learning algorithm. To this end, we formulate the strategic actors' interactions as a stochastic game (whose state encompasses Q-function estimates of the Q-learners) as if the Q-learning algorithms are the underlying dynamical system. We also present a quantization-based approximation scheme to tackle the continuum state space and analyze its performance for two competing strategic actors and a single strategic actor both analytically and numerically.
△ Less
Submitted 25 May, 2024; v1 submitted 13 March, 2024;
originally announced March 2024.
-
Team Collaboration vs Competition: New Fictitious Play Dynamics for Multi-team Zero-Sum Games
Authors:
Ahmed Said Donmez,
Yuksel Arslantas,
Muhammed O. Sayin
Abstract:
This paper presents a new variant of fictitious play (FP) called team-fictitious-play (Team-FP) that can reach equilibrium in multi-team competition, different from the other variants of FP. We specifically focus on zero-sum potential team games with network separable interactions (ZSPTGs), unifying potential games (if there is a single team) and zero-sum polymatrix games (if each team has a singl…
▽ More
This paper presents a new variant of fictitious play (FP) called team-fictitious-play (Team-FP) that can reach equilibrium in multi-team competition, different from the other variants of FP. We specifically focus on zero-sum potential team games with network separable interactions (ZSPTGs), unifying potential games (if there is a single team) and zero-sum polymatrix games (if each team has a single member) due to their wide range of applications from robotics to financial markets beyond two-team games. Similar to the FP dynamics, in Team-FP, agents follow a simple behavioral rule where they respond (with some inertia and exploration in the update of actions) to the last actions of the neighboring team members and the beliefs formed about the other neighbors' strategies as if the opponents are playing according to some stationary strategy. We show the almost sure convergence of the empirical averages of teams' action profiles to near team-Nash equilibrium in ZSPTGs under standard assumptions on the step sizes used. We formulate a bound on the approximation error, decaying with the exploration in the agents' responses. We further examine the performance of the Team-FP dynamics numerically.
△ Less
Submitted 3 February, 2024;
originally announced February 2024.
-
Convergence of Heterogeneous Learning Dynamics in Zero-sum Stochastic Games
Authors:
Yuksel Arslantas,
Ege Yuceel,
Yigit Yalin,
Muhammed O. Sayin
Abstract:
This paper presents new families of algorithms for the repeated play of two-agent (near) zero-sum games and two-agent zero-sum stochastic games. For example, the family includes fictitious play and its variants as members. Commonly, the algorithms in this family are all uncoupled, rational, and convergent even in heterogeneous cases, e.g., where the dynamics may differ in terms of learning rates,…
▽ More
This paper presents new families of algorithms for the repeated play of two-agent (near) zero-sum games and two-agent zero-sum stochastic games. For example, the family includes fictitious play and its variants as members. Commonly, the algorithms in this family are all uncoupled, rational, and convergent even in heterogeneous cases, e.g., where the dynamics may differ in terms of learning rates, full, none or temporal access to opponent actions, and model-based vs model-free learning. The convergence of heterogeneous dynamics is of practical interest especially in competitive environments since agents may have no means or interests in following the same dynamic with the same parameters. We prove that any mixture of such asymmetries does not impact the algorithms' convergence to equilibrium (or near equilibrium if there is experimentation) in zero-sum games with repeated play and in zero-sum (irreducible) stochastic games with sufficiently small discount factors.
△ Less
Submitted 1 November, 2023;
originally announced November 2023.
