-
Learning to Play No-Press Diplomacy with Best Response Policy Iteration
Authors:
Thomas Anthony,
Tom Eccles,
Andrea Tacchetti,
János Kramár,
Ian Gemp,
Thomas C. Hudson,
Nicolas Porcel,
Marc Lanctot,
Julien Pérolat,
Richard Everett,
Roman Werpachowski,
Satinder Singh,
Thore Graepel,
Yoram Bachrach
Abstract:
Recent advances in deep reinforcement learning (RL) have led to considerable progress in many 2-player zero-sum games, such as Go, Poker and Starcraft. The purely adversarial nature of such games allows for conceptually simple and principled application of RL methods. However real-world settings are many-agent, and agent interactions are complex mixtures of common-interest and competitive aspects.…
▽ More
Recent advances in deep reinforcement learning (RL) have led to considerable progress in many 2-player zero-sum games, such as Go, Poker and Starcraft. The purely adversarial nature of such games allows for conceptually simple and principled application of RL methods. However real-world settings are many-agent, and agent interactions are complex mixtures of common-interest and competitive aspects. We consider Diplomacy, a 7-player board game designed to accentuate dilemmas resulting from many-agent interactions. It also features a large combinatorial action space and simultaneous moves, which are challenging for RL algorithms. We propose a simple yet effective approximate best response operator, designed to handle large combinatorial action spaces and simultaneous moves. We also introduce a family of policy iteration methods that approximate fictitious play. With these methods, we successfully apply RL to Diplomacy: we show that our agents convincingly outperform the previous state-of-the-art, and game theoretic equilibrium analysis shows that the new process yields consistent improvements.
△ Less
Submitted 4 January, 2022; v1 submitted 8 June, 2020;
originally announced June 2020.
-
Detecting Overfitting via Adversarial Examples
Authors:
Roman Werpachowski,
András György,
Csaba Szepesvári
Abstract:
The repeated community-wide reuse of test sets in popular benchmark problems raises doubts about the credibility of reported test-error rates. Verifying whether a learned model is overfitted to a test set is challenging as independent test sets drawn from the same data distribution are usually unavailable, while other test sets may introduce a distribution shift. We propose a new hypothesis test t…
▽ More
The repeated community-wide reuse of test sets in popular benchmark problems raises doubts about the credibility of reported test-error rates. Verifying whether a learned model is overfitted to a test set is challenging as independent test sets drawn from the same data distribution are usually unavailable, while other test sets may introduce a distribution shift. We propose a new hypothesis test that uses only the original test data to detect overfitting. It utilizes a new unbiased error estimate that is based on adversarial examples generated from the test data and importance weighting. Overfitting is detected if this error estimate is sufficiently different from the original test error rate. We develop a specialized variant of our test for multiclass image classification, and apply it to testing overfitting of recent models to the popular ImageNet benchmark. Our method correctly indicates overfitting of the trained model to the training set, but is not able to detect any overfitting to the test set, in line with other recent work on this topic.
△ Less
Submitted 14 November, 2019; v1 submitted 6 March, 2019;
originally announced March 2019.
-
Microsimulations of demographic changes in England and Wales under different EU referendum scenarios
Authors:
Agnieszka Werpachowska,
Roman Werpachowski
Abstract:
We perform stochastic microsimulations of the dynamics of England and Wales population after the British referendum on EU membership, considering different possible outcomes. Employing available survey data, we model the demographics of the region over the next generation, as shaped by births, deaths and international migration. The migration patterns between England and Wales and the remaining EU…
▽ More
We perform stochastic microsimulations of the dynamics of England and Wales population after the British referendum on EU membership, considering different possible outcomes. Employing available survey data, we model the demographics of the region over the next generation, as shaped by births, deaths and international migration. The migration patterns between England and Wales and the remaining EU countries are modified according to the possible scenarios of their future relations. We find that Brexit will accelerate the overall population ageing and the deepening imbalance between workers and retirees but reduce the population growth and the fraction of women of reproductive age. In the alternative scenarios of remaining in the EU these effects will be partially forestalled by the influx of immigrants from current and prospective EU countries and their children. In all considered scenarios the native British population declines. Our study demonstrates that microsimulations can be a useful tool for designing and evaluating the country's policies in the advent of fundamental transformations.
△ Less
Submitted 8 September, 2017; v1 submitted 15 June, 2016;
originally announced June 2016.
-
Cross-sectional Markov model for trend analysis of observed discrete distributions of population characteristics
Authors:
Agnieszka Werpachowska,
Roman Werpachowski
Abstract:
We present a stochastic model of population dynamics exploiting cross-sectional data in trend analysis and forecasts for groups and cohorts of a population. While sharing the convenient features of classic Markov models, it alleviates the practical problems experienced in longitudinal studies. Based on statistical and information-theoretical analysis, we adopt maximum likelihood estimation to dete…
▽ More
We present a stochastic model of population dynamics exploiting cross-sectional data in trend analysis and forecasts for groups and cohorts of a population. While sharing the convenient features of classic Markov models, it alleviates the practical problems experienced in longitudinal studies. Based on statistical and information-theoretical analysis, we adopt maximum likelihood estimation to determine model parameters, facilitating the use of a range of model selection methods. Their application to several synthetic and empirical datasets shows that the proposed approach is robust, stable and superior to a regression-based one. We extend the basic framework to simulate ageing cohorts, processes with finite memory, distinguishing their short and long-term trends, introduce regularisation to avoid the ecological fallacy, and generalise it to mixtures of cross-sectional and (possibly incomplete) longitudinal data. The presented model illustrations yield new and interesting results, such as an implied common driving factor in obesity for all generations of the English population and "yo-yo" dieting in the U.S. data.
△ Less
Submitted 17 June, 2017; v1 submitted 22 October, 2015;
originally announced October 2015.
-
On the solutions of generalized discrete Poisson equation
Authors:
Roman Werpachowski
Abstract:
The set of common numerical and analytical problems is introduced in the form of the generalized multidimensional discrete Poisson equation. It is shown that its solutions with square-summable discrete derivatives are unique up to a constant. The proof uses the Fourier transform as the main tool. The necessary condition for the existence of the solution is provided.
The set of common numerical and analytical problems is introduced in the form of the generalized multidimensional discrete Poisson equation. It is shown that its solutions with square-summable discrete derivatives are unique up to a constant. The proof uses the Fourier transform as the main tool. The necessary condition for the existence of the solution is provided.
△ Less
Submitted 19 June, 2007;
originally announced June 2007.
-
On the approximation of real powers of sparse, infinite, bounded and Hermitian matrices
Authors:
Roman Werpachowski
Abstract:
We describe a way to approximate the matrix elements of a real power $α$ of a positive (for $α\ge 0$) or non-negative (for $α\in \mathbb{R}$), infinite, bounded, sparse and Hermitian matrix $W$. The approximation uses only a finite part of the matrix $W$.
We describe a way to approximate the matrix elements of a real power $α$ of a positive (for $α\ge 0$) or non-negative (for $α\in \mathbb{R}$), infinite, bounded, sparse and Hermitian matrix $W$. The approximation uses only a finite part of the matrix $W$.
△ Less
Submitted 26 April, 2007; v1 submitted 11 October, 2006;
originally announced October 2006.
-
A simple renormalization group approximation of the groundstate properties of interacting bosonic systems
Authors:
Roman Werpachowski,
Jerzy Kijowski
Abstract:
We present a new, simple renormalization group method of investigating groundstate properties of interacting bosonic systems. Our method reduces the number of particles in a system, which makes numerical calculations possible for large systems. It is conceptually simple and easy to implement, and allows to investigate the properties unavailable through mean field approximations, such as one- and…
▽ More
We present a new, simple renormalization group method of investigating groundstate properties of interacting bosonic systems. Our method reduces the number of particles in a system, which makes numerical calculations possible for large systems. It is conceptually simple and easy to implement, and allows to investigate the properties unavailable through mean field approximations, such as one- and two-particle reduced density matrices of the groundstate. As an example, we model a weakly interacting 1D Bose gas in a harmonic trap. Compared to the mean-field Gross-Pitaevskii approximation, our method provides a more accurate description of the groundstate one-particle density matrix. We have also obtained the Hall-Post lower bounds for the groundstate energy of the gas. All results have been obtained by the straightforward numerical diagonalization of the Hamiltonian matrix.
△ Less
Submitted 20 April, 2007; v1 submitted 17 October, 2005;
originally announced October 2005.
-
Comment on the energy spectrum of Tonks-Girardeau gas
Authors:
Roman Werpachowski
Abstract:
Withdrawn due to major errors.
Withdrawn due to major errors.
△ Less
Submitted 7 April, 2005; v1 submitted 15 March, 2005;
originally announced March 2005.
-
Universality of affine formulation in General Relativity theory
Authors:
Jerzy Kijowski,
Roman Werpachowski
Abstract:
Affine variational principle for General Relativity, proposed in 1978 by one of us (J.K.), is a good remedy for the non-universal properties of the standard, metric formulation, arising when the matter Lagrangian depends upon the metric derivatives. Affine version of the theory cures the standard drawback of the metric version, where the leading (second order) term of the field equations depends…
▽ More
Affine variational principle for General Relativity, proposed in 1978 by one of us (J.K.), is a good remedy for the non-universal properties of the standard, metric formulation, arising when the matter Lagrangian depends upon the metric derivatives. Affine version of the theory cures the standard drawback of the metric version, where the leading (second order) term of the field equations depends upon matter fields and its causal structure violates the light cone structure of the metric. Choosing the affine connection (and not the metric one) as the gravitational configuration, simplifies considerably the canonical structure of the theory and is more suitable for purposes of its quantization along the lines of Ashtekar and Lewandowski (see http://www.arxiv.longhoe.net/gr-qc/0404018). We show how the affine formulation provides a simple method to handle boundary integrals in general relativity theory.
△ Less
Submitted 29 January, 2007; v1 submitted 22 June, 2004;
originally announced June 2004.
