-
The Wisdom of the Crowd and Higher-Order Beliefs
Authors:
Yi-Chun Chen,
Manuel Mueller-Frank,
Mallesh M Pai
Abstract:
The classic wisdom-of-the-crowd problem asks how a principal can "aggregate" information about the unknown state of the world from agents without understanding the information structure among them. We propose a new simple procedure called Population-Mean-Based Aggregation to achieve this goal. The procedure only requires eliciting agents' beliefs about the state, and also eliciting some agents' ex…
▽ More
The classic wisdom-of-the-crowd problem asks how a principal can "aggregate" information about the unknown state of the world from agents without understanding the information structure among them. We propose a new simple procedure called Population-Mean-Based Aggregation to achieve this goal. The procedure only requires eliciting agents' beliefs about the state, and also eliciting some agents' expectations of the average belief in the population. We show that this procedure fully aggregates information: in an infinite population, it always infers the true state of the world. The procedure can accommodate correlations in agents' information, misspecified beliefs, any finite number of possible states of the world, and only requires very weak assumptions on the information structure.
△ Less
Submitted 11 November, 2021; v1 submitted 4 February, 2021;
originally announced February 2021.
-
Online Multivalid Learning: Means, Moments, and Prediction Intervals
Authors:
Varun Gupta,
Christopher Jung,
Georgy Noarov,
Mallesh M. Pai,
Aaron Roth
Abstract:
We present a general, efficient technique for providing contextual predictions that are "multivalid" in various senses, against an online sequence of adversarially chosen examples $(x,y)$. This means that the resulting estimates correctly predict various statistics of the labels $y$ not just marginally -- as averaged over the sequence of examples -- but also conditionally on $x \in G$ for any $G$…
▽ More
We present a general, efficient technique for providing contextual predictions that are "multivalid" in various senses, against an online sequence of adversarially chosen examples $(x,y)$. This means that the resulting estimates correctly predict various statistics of the labels $y$ not just marginally -- as averaged over the sequence of examples -- but also conditionally on $x \in G$ for any $G$ belonging to an arbitrary intersecting collection of groups $\mathcal{G}$.
We provide three instantiations of this framework. The first is mean prediction, which corresponds to an online algorithm satisfying the notion of multicalibration from Hebert-Johnson et al. The second is variance and higher moment prediction, which corresponds to an online algorithm satisfying the notion of mean-conditioned moment multicalibration from Jung et al. Finally, we define a new notion of prediction interval multivalidity, and give an algorithm for finding prediction intervals which satisfy it. Because our algorithms handle adversarially chosen examples, they can equally well be used to predict statistics of the residuals of arbitrary point prediction methods, giving rise to very general techniques for quantifying the uncertainty of predictions of black box algorithms, even in an online adversarial setting. When instantiated for prediction intervals, this solves a similar problem as conformal prediction, but in an adversarial environment and with multivalidity guarantees stronger than simple marginal coverage guarantees.
△ Less
Submitted 5 January, 2021;
originally announced January 2021.
-
Moment Multicalibration for Uncertainty Estimation
Authors:
Christopher Jung,
Changhwa Lee,
Mallesh M. Pai,
Aaron Roth,
Rakesh Vohra
Abstract:
We show how to achieve the notion of "multicalibration" from Hébert-Johnson et al. [2018] not just for means, but also for variances and other higher moments. Informally, it means that we can find regression functions which, given a data point, can make point predictions not just for the expectation of its label, but for higher moments of its label distribution as well-and those predictions match…
▽ More
We show how to achieve the notion of "multicalibration" from Hébert-Johnson et al. [2018] not just for means, but also for variances and other higher moments. Informally, it means that we can find regression functions which, given a data point, can make point predictions not just for the expectation of its label, but for higher moments of its label distribution as well-and those predictions match the true distribution quantities when averaged not just over the population as a whole, but also when averaged over an enormous number of finely defined subgroups. It yields a principled way to estimate the uncertainty of predictions on many different subgroups-and to diagnose potential sources of unfairness in the predictive power of features across subgroups. As an application, we show that our moment estimates can be used to derive marginal prediction intervals that are simultaneously valid as averaged over all of the (sufficiently large) subgroups for which moment multicalibration has been obtained.
△ Less
Submitted 18 August, 2020;
originally announced August 2020.
-
Fair Prediction with Endogenous Behavior
Authors:
Christopher Jung,
Sampath Kannan,
Changhwa Lee,
Mallesh M. Pai,
Aaron Roth,
Rakesh Vohra
Abstract:
There is increasing regulatory interest in whether machine learning algorithms deployed in consequential domains (e.g. in criminal justice) treat different demographic groups "fairly." However, there are several proposed notions of fairness, typically mutually incompatible. Using criminal justice as an example, we study a model in which society chooses an incarceration rule. Agents of different de…
▽ More
There is increasing regulatory interest in whether machine learning algorithms deployed in consequential domains (e.g. in criminal justice) treat different demographic groups "fairly." However, there are several proposed notions of fairness, typically mutually incompatible. Using criminal justice as an example, we study a model in which society chooses an incarceration rule. Agents of different demographic groups differ in their outside options (e.g. opportunity for legal employment) and decide whether to commit crimes. We show that equalizing type I and type II errors across groups is consistent with the goal of minimizing the overall crime rate; other popular notions of fairness are not.
△ Less
Submitted 18 February, 2020;
originally announced February 2020.
-
Competing Models
Authors:
Jose Luis Montiel Olea,
Pietro Ortoleva,
Mallesh M Pai,
Andrea Prat
Abstract:
Different agents need to make a prediction. They observe identical data, but have different models: they predict using different explanatory variables. We study which agent believes they have the best predictive ability -- as measured by the smallest subjective posterior mean squared prediction error -- and show how it depends on the sample size. With small samples, we present results suggesting i…
▽ More
Different agents need to make a prediction. They observe identical data, but have different models: they predict using different explanatory variables. We study which agent believes they have the best predictive ability -- as measured by the smallest subjective posterior mean squared prediction error -- and show how it depends on the sample size. With small samples, we present results suggesting it is an agent using a low-dimensional model. With large samples, it is generally an agent with a high-dimensional model, possibly including irrelevant variables, but never excluding relevant ones. We apply our results to characterize the winning model in an auction of productive assets, to argue that entrepreneurs and investors with simple models will be over-represented in new sectors, and to understand the proliferation of "factors" that explain the cross-sectional variation of expected stock returns in the asset-pricing literature.
△ Less
Submitted 11 November, 2021; v1 submitted 8 July, 2019;
originally announced July 2019.
-
Robust Mediators in Large Games
Authors:
Michael Kearns,
Mallesh M. Pai,
Ryan Rogers,
Aaron Roth,
Jonathan Ullman
Abstract:
A mediator is a mechanism that can only suggest actions to players, as a function of all agents' reported types, in a given game of incomplete information. We study what is achievable by two kinds of mediators, "strong" and "weak." Players can choose to opt-out of using a strong mediator but cannot misrepresent their type if they opt-in. Such a mediator is "strong" because we can view it as having…
▽ More
A mediator is a mechanism that can only suggest actions to players, as a function of all agents' reported types, in a given game of incomplete information. We study what is achievable by two kinds of mediators, "strong" and "weak." Players can choose to opt-out of using a strong mediator but cannot misrepresent their type if they opt-in. Such a mediator is "strong" because we can view it as having the ability to verify player types. Weak mediators lack this ability--- players are free to misrepresent their type to a weak mediator. We show a striking result---in a prior-free setting, assuming only that the game is large and players have private types, strong mediators can implement approximate equilibria of the complete-information game. If the game is a congestion game, then the same result holds using only weak mediators. Our result follows from a novel application of differential privacy, in particular, a variant we propose called joint differential privacy.
△ Less
Submitted 10 December, 2015; v1 submitted 8 December, 2015;
originally announced December 2015.
-
The Strange Case of Privacy in Equilibrium Models
Authors:
Rachel Cummings,
Katrina Ligett,
Mallesh M. Pai,
Aaron Roth
Abstract:
We study how privacy technologies affect user and advertiser behavior in a simple economic model of targeted advertising. In our model, a consumer first decides whether or not to buy a good, and then an advertiser chooses an advertisement to show the consumer. The consumer's value for the good is correlated with her type, which determines which ad the advertiser would prefer to show to her---and h…
▽ More
We study how privacy technologies affect user and advertiser behavior in a simple economic model of targeted advertising. In our model, a consumer first decides whether or not to buy a good, and then an advertiser chooses an advertisement to show the consumer. The consumer's value for the good is correlated with her type, which determines which ad the advertiser would prefer to show to her---and hence, the advertiser would like to use information about the consumer's purchase decision to target the ad that he shows.
In our model, the advertiser is given only a differentially private signal about the consumer's behavior---which can range from no signal at all to a perfect signal, as we vary the differential privacy parameter. This allows us to study equilibrium behavior as a function of the level of privacy provided to the consumer. We show that this behavior can be highly counter-intuitive, and that the effect of adding privacy in equilibrium can be completely different from what we would expect if we ignored equilibrium incentives. Specifically, we show that increasing the level of privacy can actually increase the amount of information about the consumer's type contained in the signal the advertiser receives, lead to decreased utility for the consumer, and increased profit for the advertiser, and that generally these quantities can be non-monotonic and even discontinuous in the privacy level of the signal.
△ Less
Submitted 12 August, 2015;
originally announced August 2015.
-
An Anti-Folk Theorem for Large Repeated Games with Imperfect Monitoring
Authors:
Mallesh M. Pai,
Aaron Roth,
Jonathan Ullman
Abstract:
We study infinitely repeated games in settings of imperfect monitoring. We first prove a family of theorems that show that when the signals observed by the players satisfy a condition known as $(ε, γ)$-differential privacy, that the folk theorem has little bite: for values of $ε$ and $γ$ sufficiently small, for a fixed discount factor, any equilibrium of the repeated game involve players playing a…
▽ More
We study infinitely repeated games in settings of imperfect monitoring. We first prove a family of theorems that show that when the signals observed by the players satisfy a condition known as $(ε, γ)$-differential privacy, that the folk theorem has little bite: for values of $ε$ and $γ$ sufficiently small, for a fixed discount factor, any equilibrium of the repeated game involve players playing approximate equilibria of the stage game in every period. Next, we argue that in large games ($n$ player games in which unilateral deviations by single players have only a small impact on the utility of other players), many monitoring settings naturally lead to signals that satisfy $(ε,γ)$-differential privacy, for $ε$ and $γ$ tending to zero as the number of players $n$ grows large. We conclude that in such settings, the set of equilibria of the repeated game collapse to the set of equilibria of the stage game.
△ Less
Submitted 8 October, 2014; v1 submitted 12 February, 2014;
originally announced February 2014.
-
Mechanism Design in Large Games: Incentives and Privacy
Authors:
Michael Kearns,
Mallesh M. Pai,
Aaron Roth,
Jonathan Ullman
Abstract:
We study the problem of implementing equilibria of complete information games in settings of incomplete information, and address this problem using "recommender mechanisms." A recommender mechanism is one that does not have the power to enforce outcomes or to force participation, rather it only has the power to suggestion outcomes on the basis of voluntary participation. We show that despite these…
▽ More
We study the problem of implementing equilibria of complete information games in settings of incomplete information, and address this problem using "recommender mechanisms." A recommender mechanism is one that does not have the power to enforce outcomes or to force participation, rather it only has the power to suggestion outcomes on the basis of voluntary participation. We show that despite these restrictions, recommender mechanisms can implement equilibria of complete information games in settings of incomplete information under the condition that the game is large---i.e. that there are a large number of players, and any player's action affects any other's payoff by at most a small amount.
Our result follows from a novel application of differential privacy. We show that any algorithm that computes a correlated equilibrium of a complete information game while satisfying a variant of differential privacy---which we call joint differential privacy---can be used as a recommender mechanism while satisfying our desired incentive properties. Our main technical result is an algorithm for computing a correlated equilibrium of a large game while satisfying joint differential privacy.
Although our recommender mechanisms are designed to satisfy game-theoretic properties, our solution ends up satisfying a strong privacy property as well. No group of players can learn "much" about the type of any player outside the group from the recommendations of the mechanism, even if these players collude in an arbitrary way. As such, our algorithm is able to implement equilibria of complete information games, without revealing information about the realized types.
△ Less
Submitted 10 December, 2015; v1 submitted 17 July, 2012;
originally announced July 2012.
