Showing 1–14 of 14 results for author: Kovarik, V

Extinction Risks from AI: Invisible to Science?

Authors: Vojtech Kovarik, Christian van Merwijk, Ida Mattsson

Abstract: In an effort to inform the discussion surrounding existential risks from AI, we formulate Extinction-level Goodhart's Law as "Virtually any goal specification, pursued to the extreme, will result in the extinction of humanity", and we aim to understand which formal models are suitable for investigating this hypothesis. Note that we remain agnostic as to whether Extinction-level Goodhart's Law hold… ▽ More In an effort to inform the discussion surrounding existential risks from AI, we formulate Extinction-level Goodhart's Law as "Virtually any goal specification, pursued to the extreme, will result in the extinction of humanity", and we aim to understand which formal models are suitable for investigating this hypothesis. Note that we remain agnostic as to whether Extinction-level Goodhart's Law holds or not. As our key contribution, we identify a set of conditions that are necessary for a model that aims to be informative for evaluating specific arguments for Extinction-level Goodhart's Law. Since each of the conditions seems to significantly contribute to the complexity of the resulting model, formally evaluating the hypothesis might be exceedingly difficult. This raises the possibility that whether the risk of extinction from artificial intelligence is real or not, the underlying dynamics might be invisible to current scientific methods. △ Less

Submitted 2 February, 2024; originally announced March 2024.

Recursive Joint Simulation in Games

Authors: Vojtech Kovarik, Caspar Oesterheld, Vincent Conitzer

Abstract: Game-theoretic dynamics between AI agents could differ from traditional human-human interactions in various ways. One such difference is that it may be possible to accurately simulate an AI agent, for example because its source code is known. Our aim is to explore ways of leveraging this possibility to achieve more cooperative outcomes in strategic settings. In this paper, we study an interaction… ▽ More Game-theoretic dynamics between AI agents could differ from traditional human-human interactions in various ways. One such difference is that it may be possible to accurately simulate an AI agent, for example because its source code is known. Our aim is to explore ways of leveraging this possibility to achieve more cooperative outcomes in strategic settings. In this paper, we study an interaction between AI agents where the agents run a recursive joint simulation. That is, the agents first jointly observe a simulation of the situation they face. This simulation in turn recursively includes additional simulations (with a small chance of failure, to avoid infinite recursion), and the results of all these nested simulations are observed before an action is chosen. We show that the resulting interaction is strategically equivalent to an infinitely repeated version of the original game, allowing a direct transfer of existing results such as the various folk theorems. △ Less

Submitted 1 March, 2024; v1 submitted 12 February, 2024; originally announced February 2024.

Game Theory with Simulation of Other Players

Authors: Vojtech Kovarik, Caspar Oesterheld, Vincent Conitzer

Abstract: Game-theoretic interactions with AI agents could differ from traditional human-human interactions in various ways. One such difference is that it may be possible to simulate an AI agent (for example because its source code is known), which allows others to accurately predict the agent's actions. This could lower the bar for trust and cooperation. In this paper, we formalize games in which one play… ▽ More Game-theoretic interactions with AI agents could differ from traditional human-human interactions in various ways. One such difference is that it may be possible to simulate an AI agent (for example because its source code is known), which allows others to accurately predict the agent's actions. This could lower the bar for trust and cooperation. In this paper, we formalize games in which one player can simulate another at a cost. We first derive some basic properties of such games and then prove a number of results for them, including: (1) introducing simulation into generic-payoff normal-form games makes them easier to solve; (2) if the only obstacle to cooperation is a lack of trust in the possibly-simulated agent, simulation enables equilibria that improve the outcome for both agents; and however (3) there are settings where introducing simulation results in strictly worse outcomes for both players. △ Less

Submitted 19 March, 2024; v1 submitted 18 May, 2023; originally announced May 2023.

Comments: The latest version fixes some typos in the proof of Theorem 5

Revisiting Game Representations: The Hidden Costs of Efficiency in Sequential Decision-making Algorithms

Authors: Vojtěch Kovařík, David Milec, Michal Šustr, Dominik Seitz, Viliam Lisý

Abstract: Recent advancements in algorithms for sequential decision-making under imperfect information have shown remarkable success in large games such as limit- and no-limit poker. These algorithms traditionally formalize the games using the extensive-form game formalism, which, as we show, while theoretically sound, is memory-inefficient and computationally intensive in practice. To mitigate these challe… ▽ More Recent advancements in algorithms for sequential decision-making under imperfect information have shown remarkable success in large games such as limit- and no-limit poker. These algorithms traditionally formalize the games using the extensive-form game formalism, which, as we show, while theoretically sound, is memory-inefficient and computationally intensive in practice. To mitigate these challenges, a popular workaround involves using a specialized representation based on player specific information-state trees. However, as we show, this alternative significantly narrows the set of games that can be represented efficiently. In this study, we identify the set of large games on which modern algorithms have been benchmarked as being naturally represented by Sequential Bayesian Games. We elucidate the critical differences between extensive-form game and sequential Bayesian game representations, both theoretically and empirically. We further argue that the impressive experimental results often cited in the literature may be skewed, as they frequently stem from testing these algorithms only on this restricted class of games. By understanding these nuances, we aim to guide future research in develo** more universally applicable and efficient algorithms for sequential decision-making under imperfect information. △ Less

Submitted 5 December, 2023; v1 submitted 20 December, 2021; originally announced December 2021.

Solving Zero-Sum One-Sided Partially Observable Stochastic Games

Authors: Karel Horák, Branislav Bošanský, Vojtěch Kovařík, Christopher Kiekintveld

Abstract: Many security and other real-world situations are dynamic in nature and can be modelled as strictly competitive (or zero-sum) dynamic games. In these domains, agents perform actions to affect the environment and receive observations -- possibly imperfect -- about the situation and the effects of the opponent's actions. Moreover, there is no limitation on the total number of actions an agent can pe… ▽ More Many security and other real-world situations are dynamic in nature and can be modelled as strictly competitive (or zero-sum) dynamic games. In these domains, agents perform actions to affect the environment and receive observations -- possibly imperfect -- about the situation and the effects of the opponent's actions. Moreover, there is no limitation on the total number of actions an agent can perform -- that is, there is no fixed horizon. These settings can be modelled as partially observable stochastic games (POSGs). However, solving general POSGs is computationally intractable, so we focus on a broad subclass of POSGs called one-sided POSGs. In these games, only one agent has imperfect information while their opponent has full knowledge of the current situation. We provide a full picture for solving one-sided POSGs: we (1) give a theoretical analysis of one-sided POSGs and their value functions, (2) show that a variant of a value-iteration algorithm converges in this setting, (3) adapt the heuristic search value-iteration algorithm for solving one-sided POSGs, (4) describe how to use approximate value functions to derive strategies in the game, and (5) demonstrate that our algorithm can solve one-sided POSGs of non-trivial sizes and analyze the scalability of our algorithm in three different domains: pursuit-evasion, patrolling, and search games. △ Less

Submitted 21 October, 2020; originally announced October 2020.

(When) Is Truth-telling Favored in AI Debate?

Authors: Vojtěch Kovařík, Ryan Carey

Abstract: For some problems, humans may not be able to accurately judge the goodness of AI-proposed solutions. Irving et al. (2018) propose that in such cases, we may use a debate between two AI systems to amplify the problem-solving capabilities of a human judge. We introduce a mathematical framework that can model debates of this type and propose that the quality of debate designs should be measured by th… ▽ More For some problems, humans may not be able to accurately judge the goodness of AI-proposed solutions. Irving et al. (2018) propose that in such cases, we may use a debate between two AI systems to amplify the problem-solving capabilities of a human judge. We introduce a mathematical framework that can model debates of this type and propose that the quality of debate designs should be measured by the accuracy of the most persuasive answer. We describe a simple instance of the debate framework called feature debate and analyze the degree to which such debates track the truth. We argue that despite being very simple, feature debates nonetheless capture many aspects of practical debates such as the incentives to confuse the judge or stall to prevent losing. We then outline how these models should be generalized to analyze a wider range of debate phenomena. △ Less

Submitted 16 March, 2021; v1 submitted 11 November, 2019; originally announced November 2019.

Comments: In SafeAI Workshop at AAAI, 2019

Rethinking Formal Models of Partially Observable Multiagent Decision Making

Authors: Vojtěch Kovařík, Martin Schmid, Neil Burch, Michael Bowling, Viliam Lisý

Abstract: Multiagent decision-making in partially observable environments is usually modelled as either an extensive-form game (EFG) in game theory or a partially observable stochastic game (POSG) in multiagent reinforcement learning (MARL). One issue with the current situation is that while most practical problems can be modelled in both formalisms, the relationship of the two models is unclear, which hind… ▽ More Multiagent decision-making in partially observable environments is usually modelled as either an extensive-form game (EFG) in game theory or a partially observable stochastic game (POSG) in multiagent reinforcement learning (MARL). One issue with the current situation is that while most practical problems can be modelled in both formalisms, the relationship of the two models is unclear, which hinders the transfer of ideas between the two communities. A second issue is that while EFGs have recently seen significant algorithmic progress, their classical formalization is unsuitable for efficient presentation of the underlying ideas, such as those around decomposition. To solve the first issue, we introduce factored-observation stochastic games (FOSGs), a minor modification of the POSG formalism which distinguishes between private and public observation and thereby greatly simplifies decomposition. To remedy the second issue, we show that FOSGs and POSGs are naturally connected to EFGs: by "unrolling" a FOSG into its tree form, we obtain an EFG. Conversely, any perfect-recall timeable EFG corresponds to some underlying FOSG in this manner. Moreover, this relationship justifies several minor modifications to the classical EFG formalization that recently appeared as an implicit response to the model's issues with decomposition. Finally, we illustrate the transfer of ideas between EFGs and MARL by presenting three key EFG techniques -- counterfactual regret minimization, sequence form, and decomposition -- in the FOSG framework. △ Less

Submitted 28 September, 2021; v1 submitted 26 June, 2019; originally announced June 2019.

Comments: A 2020 update of the original 2019 version of the paper. (Rewrote the main text and clarified the relationship between FOSGs/POSGs and EFGs. Some of the technical results are now presented in the appendix.)

Value Functions for Depth-Limited Solving in Zero-Sum Imperfect-Information Games

Authors: Vojtěch Kovařík, Dominik Seitz, Viliam Lisý, Jan Rudolf, Shuo Sun, Karel Ha

Abstract: We provide a formal definition of depth-limited games together with an accessible and rigorous explanation of the underlying concepts, both of which were previously missing in imperfect-information games. The definition works for an arbitrary extensive-form game and is not tied to any specific game-solving algorithm. Moreover, this framework unifies and significantly extends three approaches to de… ▽ More We provide a formal definition of depth-limited games together with an accessible and rigorous explanation of the underlying concepts, both of which were previously missing in imperfect-information games. The definition works for an arbitrary extensive-form game and is not tied to any specific game-solving algorithm. Moreover, this framework unifies and significantly extends three approaches to depth-limited solving that previously existed in extensive-form games and multiagent reinforcement learning but were not known to be compatible. A key ingredient of these depth-limited games are value functions. Focusing on two-player zero-sum imperfect-information games, we show how to obtain optimal value functions and prove that public information provides both necessary and sufficient context for computing them. We provide a domain-independent encoding of the domains that allows for approximating value functions even by simple feed-forward neural networks, which are then able to generalize to unseen parts of the game. We use the resulting value network to implement a depth-limited version of counterfactual regret minimization. In three distinct domains, we show that the algorithm's exploitability is roughly linearly dependent on the value network's quality and that it is not difficult to train a value network with which depth-limited CFR's performance is as good as that of CFR with access to the full game. △ Less

Submitted 24 March, 2022; v1 submitted 31 May, 2019; originally announced June 2019.

Comments: The first two authors contributed equally

Problems with the EFG formalism: a solution attempt using observations

Authors: Vojtěch Kovařík, Viliam Lisý

Abstract: We argue that the extensive-form game (EFG) model isn't powerful enough to express all important aspects of imperfect information games, such as those related to decomposition and online game solving. We present a principled attempt to fix the formalism by considering information partitions that correspond to observations. We show that EFGs cannot be "fixed" without additional knowledge about the… ▽ More We argue that the extensive-form game (EFG) model isn't powerful enough to express all important aspects of imperfect information games, such as those related to decomposition and online game solving. We present a principled attempt to fix the formalism by considering information partitions that correspond to observations. We show that EFGs cannot be "fixed" without additional knowledge about the original task, and show how to extend the EFG model under this assumption. However, during our investigation, we ultimately concluded that a better solution is to abandon EFGs entirely and instead adopt a model based on partially observable stochastic games. The primary contribution of the text thus lies in exposing the problems with EFGs and presenting a detailed study on introducing imperfect information by adding observations into an underlying perfect-information model. △ Less

Submitted 14 June, 2019; originally announced June 2019.

Approximation capability of neural networks on spaces of probability measures and tree-structured domains

Authors: Tomas Pevny, Vojtech Kovarik

Abstract: This paper extends the proof of density of neural networks in the space of continuous (or even measurable) functions on Euclidean spaces to functions on compact sets of probability measures. By doing so the work parallels a more then a decade old results on mean-map embedding of probability measures in reproducing kernel Hilbert spaces. The work has wide practical consequences for multi-instance l… ▽ More This paper extends the proof of density of neural networks in the space of continuous (or even measurable) functions on Euclidean spaces to functions on compact sets of probability measures. By doing so the work parallels a more then a decade old results on mean-map embedding of probability measures in reproducing kernel Hilbert spaces. The work has wide practical consequences for multi-instance learning, where it theoretically justifies some recently proposed constructions. The result is then extended to Cartesian products, yielding universal approximation theorem for tree-structured domains, which naturally occur in data-exchange formats like JSON, XML, YAML, AVRO, and ProtoBuffer. This has important practical implications, as it enables to automatically create an architecture of neural networks for processing structured data (AutoML paradigms), as demonstrated by an accompanied library for JSON format. △ Less

Submitted 3 June, 2019; originally announced June 2019.

Monte Carlo Continual Resolving for Online Strategy Computation in Imperfect Information Games

Authors: Michal Sustr, Vojtech Kovarik, Viliam Lisy

Abstract: Online game playing algorithms produce high-quality strategies with a fraction of memory and computation required by their offline alternatives. Continual Resolving (CR) is a recent theoretically sound approach to online game playing that has been used to outperform human professionals in poker. However, parts of the algorithm were specific to poker, which enjoys many properties not shared by othe… ▽ More Online game playing algorithms produce high-quality strategies with a fraction of memory and computation required by their offline alternatives. Continual Resolving (CR) is a recent theoretically sound approach to online game playing that has been used to outperform human professionals in poker. However, parts of the algorithm were specific to poker, which enjoys many properties not shared by other imperfect information games. We present a domain-independent formulation of CR applicable to any two-player zero-sum extensive-form games that works with an abstract resolving algorithm. We further describe and implement its Monte Carlo variant (MCCR) which uses Monte Carlo Counterfactual Regret Minimization (MCCFR) as a resolver. We prove the correctness of CR and show an $O(T^{-1/2})$-dependence of MCCR's exploitability on the computation time. Furthermore, we present an empirical comparison of MCCR with incremental tree building to Online Outcome Sampling and Information-set MCTS on several domains. △ Less

Submitted 8 March, 2019; v1 submitted 18 December, 2018; originally announced December 2018.

Analysis of Hannan Consistent Selection for Monte Carlo Tree Search in Simultaneous Move Games

Authors: Vojtěch Kovařík, Viliam Lisý

Abstract: Hannan consistency, or no external regret, is a~key concept for learning in games. An action selection algorithm is Hannan consistent (HC) if its performance is eventually as good as selecting the~best fixed action in hindsight. If both players in a~zero-sum normal form game use a~Hannan consistent algorithm, their average behavior converges to a~Nash equilibrium (NE) of the~game. A similar result… ▽ More Hannan consistency, or no external regret, is a~key concept for learning in games. An action selection algorithm is Hannan consistent (HC) if its performance is eventually as good as selecting the~best fixed action in hindsight. If both players in a~zero-sum normal form game use a~Hannan consistent algorithm, their average behavior converges to a~Nash equilibrium (NE) of the~game. A similar result is known about extensive form games, but the~played strategies need to be Hannan consistent with respect to the~counterfactual values, which are often difficult to obtain. We study zero-sum extensive form games with simultaneous moves, but otherwise perfect information. These games generalize normal form games and they are a special case of extensive form games. We study whether applying HC algorithms in each decision point of these games directly to the~observed payoffs leads to convergence to a~Nash equilibrium. This learning process corresponds to a~class of Monte Carlo Tree Search algorithms, which are popular for playing simultaneous-move games but do not have any known performance guarantees. We show that using HC algorithms directly on the~observed payoffs is not sufficient to guarantee the~convergence. With an~additional averaging over joint actions, the~convergence is guaranteed, but empirically slower. We further define an~additional property of HC algorithms, which is sufficient to guarantee the~convergence without the~averaging and we empirically show that commonly used HC algorithms have this property. △ Less

Submitted 7 July, 2019; v1 submitted 23 April, 2018; originally announced April 2018.

Comments: arXiv admin note: substantial text overlap with arXiv:1509.00149

Analysis of Hannan Consistent Selection for Monte Carlo Tree Search in Simultaneous Move Games

Authors: Vojtěch Kovařík, Viliam Lisý

Abstract: Monte Carlo Tree Search (MCTS) has recently been successfully used to create strategies for playing imperfect-information games. Despite its popularity, there are no theoretic results that guarantee its convergence to a well-defined solution, such as Nash equilibrium, in these games. We partially fill this gap by analysing MCTS in the class of zero-sum extensive-form games with simultaneous moves… ▽ More Monte Carlo Tree Search (MCTS) has recently been successfully used to create strategies for playing imperfect-information games. Despite its popularity, there are no theoretic results that guarantee its convergence to a well-defined solution, such as Nash equilibrium, in these games. We partially fill this gap by analysing MCTS in the class of zero-sum extensive-form games with simultaneous moves but otherwise perfect information. The lack of information about the opponent's concurrent moves already causes that optimal strategies may require randomization. We present theoretic as well as empirical investigation of the speed and quality of convergence of these algorithms to the Nash equilibria. Primarily, we show that after minor technical modifications, MCTS based on any (approximately) Hannan consistent selection function always converges to an (approximate) subgame perfect Nash equilibrium. Without these modifications, Hannan consistency is not sufficient to ensure such convergence and the selection function must satisfy additional properties, which empirically hold for the most common Hannan consistent algorithms. △ Less

Submitted 1 September, 2015; originally announced September 2015.

Convergence of Monte Carlo Tree Search in Simultaneous Move Games

Authors: Viliam Lisý, Vojtěch Kovařík, Marc Lanctot, Branislav Bošanský

Abstract: We study Monte Carlo tree search (MCTS) in zero-sum extensive-form games with perfect information and simultaneous moves. We present a general template of MCTS algorithms for these games, which can be instantiated by various selection methods. We formally prove that if a selection method is $ε$-Hannan consistent in a matrix game and satisfies additional requirements on exploration, then the MCTS a… ▽ More We study Monte Carlo tree search (MCTS) in zero-sum extensive-form games with perfect information and simultaneous moves. We present a general template of MCTS algorithms for these games, which can be instantiated by various selection methods. We formally prove that if a selection method is $ε$-Hannan consistent in a matrix game and satisfies additional requirements on exploration, then the MCTS algorithm eventually converges to an approximate Nash equilibrium (NE) of the extensive-form game. We empirically evaluate this claim using regret matching and Exp3 as the selection methods on randomly generated games and empirically selected worst case games. We confirm the formal result and show that additional MCTS variants also converge to approximate NE on the evaluated games. △ Less

Submitted 5 November, 2013; v1 submitted 31 October, 2013; originally announced October 2013.

Comments: NIPS 2013 paper including appendix

Journal ref: Advances in Neural Information Processing Systems 26, pp 2112-2120, 2013