-
Interference Among First-Price Pacing Equilibria: A Bias and Variance Analysis
Authors:
Luofeng Liao,
Christian Kroer,
Sergei Leonenkov,
Okke Schrijvers,
Liang Shi,
Nicolas Stier-Moses,
Congshan Zhang
Abstract:
Online A/B testing is widely used in the internet industry to inform decisions on new feature roll-outs. For online marketplaces (such as advertising markets), standard approaches to A/B testing may lead to biased results when buyers operate under a budget constraint, as budget consumption in one arm of the experiment impacts performance of the other arm. To counteract this interference, one can u…
▽ More
Online A/B testing is widely used in the internet industry to inform decisions on new feature roll-outs. For online marketplaces (such as advertising markets), standard approaches to A/B testing may lead to biased results when buyers operate under a budget constraint, as budget consumption in one arm of the experiment impacts performance of the other arm. To counteract this interference, one can use a budget-split design where the budget constraint operates on a per-arm basis and each arm receives an equal fraction of the budget, leading to ``budget-controlled A/B testing.'' Despite clear advantages of budget-controlled A/B testing, performance degrades when budget are split too small, limiting the overall throughput of such systems. In this paper, we propose a parallel budget-controlled A/B testing design where we use market segmentation to identify submarkets in the larger market, and we run parallel experiments on each submarket.
Our contributions are as follows: First, we introduce and demonstrate the effectiveness of the parallel budget-controlled A/B test design with submarkets in a large online marketplace environment. Second, we formally define market interference in first-price auction markets using the first price pacing equilibrium (FPPE) framework. Third, we propose a debiased surrogate that eliminates the first-order bias of FPPE, drawing upon the principles of sensitivity analysis in mathematical programs. Fourth, we derive a plug-in estimator for the surrogate and establish its asymptotic normality. Fifth, we provide an estimation procedure for submarket parallel budget-controlled A/B tests. Finally, we present numerical examples on semi-synthetic data, confirming that the debiasing technique achieves the desired coverage properties.
△ Less
Submitted 11 February, 2024;
originally announced February 2024.
-
Bandits for Online Calibration: An Application to Content Moderation on Social Media Platforms
Authors:
Vashist Avadhanula,
Omar Abdul Baki,
Hamsa Bastani,
Osbert Bastani,
Caner Gocmen,
Daniel Haimovich,
Darren Hwang,
Dima Karamshuk,
Thomas Leeper,
Jiayuan Ma,
Gregory Macnamara,
Jake Mullett,
Christopher Palow,
Sung Park,
Varun S Rajagopal,
Kevin Schaeffer,
Parikshit Shah,
Deeksha Sinha,
Nicolas Stier-Moses,
Peng Xu
Abstract:
We describe the current content moderation strategy employed by Meta to remove policy-violating content from its platforms. Meta relies on both handcrafted and learned risk models to flag potentially violating content for human review. Our approach aggregates these risk models into a single ranking score, calibrating them to prioritize more reliable risk models. A key challenge is that violation t…
▽ More
We describe the current content moderation strategy employed by Meta to remove policy-violating content from its platforms. Meta relies on both handcrafted and learned risk models to flag potentially violating content for human review. Our approach aggregates these risk models into a single ranking score, calibrating them to prioritize more reliable risk models. A key challenge is that violation trends change over time, affecting which risk models are most reliable. Our system additionally handles production challenges such as changing risk models and novel risk models. We use a contextual bandit to update the calibration in response to such trends. Our approach increases Meta's top-line metric for measuring the effectiveness of its content moderation strategy by 13%.
△ Less
Submitted 11 November, 2022;
originally announced November 2022.
-
QUEST: Queue Simulation for Content Moderation at Scale
Authors:
Rahul Makhijani,
Parikshit Shah,
Vashist Avadhanula,
Caner Gocmen,
Nicolás E. Stier-Moses,
Julián Mestre
Abstract:
Moderating content in social media platforms is a formidable challenge due to the unprecedented scale of such systems, which typically handle billions of posts per day. Some of the largest platforms such as Facebook blend machine learning with manual review of platform content by thousands of reviewers. Operating a large-scale human review system poses interesting and challenging methodological qu…
▽ More
Moderating content in social media platforms is a formidable challenge due to the unprecedented scale of such systems, which typically handle billions of posts per day. Some of the largest platforms such as Facebook blend machine learning with manual review of platform content by thousands of reviewers. Operating a large-scale human review system poses interesting and challenging methodological questions that can be addressed with operations research techniques. We investigate the problem of optimally operating such a review system at scale using ideas from queueing theory and simulation.
△ Less
Submitted 31 March, 2021;
originally announced March 2021.
-
Approximation and Convergence of Large Atomic Congestion Games
Authors:
Roberto Cominetti,
Marco Scarsini,
Marc Schröder,
Nicolás Stier-Moses
Abstract:
We consider the question of whether, and in what sense, Wardrop equilibria provide a good approximation for Nash equilibria in atomic unsplittable congestion games with a large number of small players. We examine two different definitions of small players. In the first setting, we consider games where each player's weight is small. We prove that when the number of players goes to infinity and thei…
▽ More
We consider the question of whether, and in what sense, Wardrop equilibria provide a good approximation for Nash equilibria in atomic unsplittable congestion games with a large number of small players. We examine two different definitions of small players. In the first setting, we consider games where each player's weight is small. We prove that when the number of players goes to infinity and their weights to zero, the random flows in all (mixed) Nash equilibria for the finite games converge in distribution to the set of Wardrop equilibria of the corresponding nonatomic limit game. In the second setting, we consider an increasing number of players with a unit weight that participate in the game with a decreasingly small probability. In this case, the Nash equilibrium flows converge in total variation towards Poisson random variables whose expected values are Wardrop equilibria of a different nonatomic game with suitably-defined costs. The latter can be viewed as symmetric equilibria in a Poisson game in the sense of Myerson, establishing a plausible connection between the Wardrop model for routing games and the stochastic fluctuations observed in real traffic. In both settings we provide explicit approximation bounds, and we study the convergence of the price of anarchy. Beyond the case of congestion games, we prove a general result on the convergence of large games with random players towards Poisson games.
△ Less
Submitted 30 April, 2022; v1 submitted 8 January, 2020;
originally announced January 2020.
-
Ordinary and prophet planning under uncertainty in Bernoulli congestion games
Authors:
Roberto Cominetti,
Marco Scarsini,
Marc Schröder,
Nicolás Stier-Moses
Abstract:
We consider an atomic congestion game in which each player $i$ either participates in the game with an exogenous and known probability $p_{i}\in(0,1]$, independently of everybody else, or stays out and incurs no cost. We compute the parameterized price of anarchy to characterize the impact of demand uncertainty on the efficiency of selfish behavior, considering two different notions of a social pl…
▽ More
We consider an atomic congestion game in which each player $i$ either participates in the game with an exogenous and known probability $p_{i}\in(0,1]$, independently of everybody else, or stays out and incurs no cost. We compute the parameterized price of anarchy to characterize the impact of demand uncertainty on the efficiency of selfish behavior, considering two different notions of a social planner. A prophet planner knows the realization of the random participation in the game; the ordinary planner does not. As a consequence, a prophet planner can compute an adaptive social optimum that selects different solutions depending on the players that turn out to be active, whereas an ordinary planner faces the same uncertainty as the players and can only compute social optima with respect to the player participation distribution. For both planners, we derive the precise price of anarchy, which arises from an optimization problem parameterized by the maximum participation probability $q=\max_{i} p_{i}$. In the case of affine costs, we provide an analytic expression for the ordinary and prophet price of anarchy, parameterized as a function of $q$.
△ Less
Submitted 2 February, 2024; v1 submitted 8 March, 2019;
originally announced March 2019.
-
Computing large market equilibria using abstractions
Authors:
Christian Kroer,
Alexander Peysakhovich,
Eric Sodomka,
Nicolas E. Stier-Moses
Abstract:
Computing market equilibria is an important practical problem for market design, for example in fair division of items. However, computing equilibria requires large amounts of information (typically the valuation of every buyer for every item) and computing power. We consider ameliorating these issues by applying a method used for solving complex games: constructing a coarsened abstraction of a gi…
▽ More
Computing market equilibria is an important practical problem for market design, for example in fair division of items. However, computing equilibria requires large amounts of information (typically the valuation of every buyer for every item) and computing power. We consider ameliorating these issues by applying a method used for solving complex games: constructing a coarsened abstraction of a given market, solving for the equilibrium in the abstraction, and lifting the prices and allocations back to the original market. We show how to bound important quantities such as regret, envy, Nash social welfare, Pareto optimality, and maximin share/proportionality when the abstracted prices and allocations are used in place of the real equilibrium. We then study two abstraction methods of interest for practitioners: (1) filling in unknown valuations using techniques from matrix completion, (2) reducing the problem size by aggregating groups of buyers/items into smaller numbers of representative buyers/items and solving for equilibrium in this coarsened market. We find that in real data allocations/prices that are relatively close to equilibria can be computed from even very coarse abstractions.
△ Less
Submitted 3 September, 2021; v1 submitted 18 January, 2019;
originally announced January 2019.
-
Pacing Equilibrium in First-Price Auction Markets
Authors:
Vincent Conitzer,
Christian Kroer,
Debmalya Panigrahi,
Okke Schrijvers,
Eric Sodomka,
Nicolas E. Stier-Moses,
Chris Wilkens
Abstract:
Mature internet advertising platforms offer high-level campaign management tools to help advertisers run their campaigns, often abstracting away the intricacies of how each ad is placed and focusing on aggregate metrics of interest to advertisers. On such platforms, advertisers often participate in auctions through a proxy bidder, so the standard incentive analyses that are common in the literatur…
▽ More
Mature internet advertising platforms offer high-level campaign management tools to help advertisers run their campaigns, often abstracting away the intricacies of how each ad is placed and focusing on aggregate metrics of interest to advertisers. On such platforms, advertisers often participate in auctions through a proxy bidder, so the standard incentive analyses that are common in the literature do not apply directly. In this paper, we take the perspective of a budget management system that surfaces aggregated incentives -- instead of individual auctions -- and compare first and second price auctions. We show that theory offers surprising endorsement for using a first price auction to sell individual impressions. In particular, first price auctions guarantee uniqueness of the steady-state equilibrium of the budget management system, monotonicity, and other desirable properties, as well as efficient computation through the solution to the well-studied Eisenberg-Gale convex program. Contrary to what one can expect from first price auctions, we show that incentives issues are not a barrier that undermines the system. Using realistic instances generated from data collected at real-world auction platforms, we show that bidders have small regret with respect to their optimal ex-post strategy, and they do not have a big incentive to misreport when they can influence equilibria directly by giving inputs strategically. Finally, budget-constrained bidders, who have significant prevalence in real-world platforms, tend to have smaller regrets. Our computations indicate that bidder budgets, pacing multipliers and regrets all have a positive association in statistical terms.
△ Less
Submitted 3 September, 2021; v1 submitted 17 November, 2018;
originally announced November 2018.
-
Multiplicative Pacing Equilibria in Auction Markets
Authors:
Vincent Conitzer,
Christian Kroer,
Eric Sodomka,
Nicolas E. Stier-Moses
Abstract:
Budgets play a significant role in real-world sequential auction markets such as those implemented by internet companies. To maximize the value provided to auction participants, spending is smoothed across auctions so budgets are used for the best opportunities. Motivated by a mechanism used in practice by several companies, this paper considers a smoothing procedure that relies on {\em pacing mul…
▽ More
Budgets play a significant role in real-world sequential auction markets such as those implemented by internet companies. To maximize the value provided to auction participants, spending is smoothed across auctions so budgets are used for the best opportunities. Motivated by a mechanism used in practice by several companies, this paper considers a smoothing procedure that relies on {\em pacing multipliers}: on behalf of each buyer, the auction market applies a factor between 0 and 1 that uniformly scales the bids across all auctions. Reinterpreting this process as a game between buyers, we introduce the notion of {\em pacing equilibrium}, and prove that they are always guaranteed to exist. We demonstrate through examples that a market can have multiple pacing equilibria with large variations in several natural objectives. We show that pacing equilibria refine another popular solution concept, competitive equilibria, and show further connections between the two solution concepts. Although we show that computing either a social-welfare-maximizing or a revenue-maximizing pacing equilibrium is NP-hard, we present a mixed-integer program (MIP) that can be used to find equilibria optimizing several relevant objectives. We use the MIP to provide evidence that: (1) equilibrium multiplicity occurs very rarely across several families of random instances, (2) static MIP solutions can be used to improve the outcomes achieved by a dynamic pacing algorithm with instances based on a real-world auction market, and (3) for the instances we study, buyers do not have an incentive to misreport bids or budgets provided there are enough participants in the auction.
△ Less
Submitted 8 May, 2021; v1 submitted 21 June, 2017;
originally announced June 2017.
-
End-to-end Planning of Fixed Millimeter-Wave Networks
Authors:
Tim Danford,
Onur Filiz,
**g Huang,
Brian Karrer,
Manohar Paluri,
Guan Pang,
Vish Ponnampalam,
Nicolas Stier-Moses,
Birce Tezel
Abstract:
This article discusses a framework to support the design and end-to-end planning of fixed millimeter-wave networks. Compared to traditional techniques, the framework allows an organization to quickly plan a deployment in a cost-effective way. We start by using LiDAR data---basically, a 3D point cloud captured from a city---to estimate potential sites to deploy antennas and whether there is line-of…
▽ More
This article discusses a framework to support the design and end-to-end planning of fixed millimeter-wave networks. Compared to traditional techniques, the framework allows an organization to quickly plan a deployment in a cost-effective way. We start by using LiDAR data---basically, a 3D point cloud captured from a city---to estimate potential sites to deploy antennas and whether there is line-of-sight between them. With that data on hand, we use combinatorial optimization techniques to determine the optimal set of locations and how they should communicate with each other, to satisfy engineering (e.g., latency, polarity), design (e.g., reliability) and financial (e.g., total cost of operation) constraints. The primary goal is to connect as many people as possible to the network. Our methodology can be used for strategic planning when an organization is in the process of deciding whether to adopt a millimeter-wave technology or choosing between locations, or for operational planning when conducting a detailed design of the actual network to be deployed in a selected location.
△ Less
Submitted 19 May, 2017;
originally announced May 2017.
-
Asymptotically tight bounds for inefficiency in risk-averse selfish routing
Authors:
Thanasis Lianeas,
Evdokia Nikolova,
Nicolas E. Stier-Moses
Abstract:
We consider a nonatomic selfish routing model with independent stochastic travel times, represented by mean and variance latency functions for each edge that depend on their flows. In an effort to decouple the effect of risk-averse player preferences from selfish behavior on the degradation of system performance, Nikolova and Stier- Moses [16] defined the concept of the price of risk aversion as t…
▽ More
We consider a nonatomic selfish routing model with independent stochastic travel times, represented by mean and variance latency functions for each edge that depend on their flows. In an effort to decouple the effect of risk-averse player preferences from selfish behavior on the degradation of system performance, Nikolova and Stier- Moses [16] defined the concept of the price of risk aversion as the worst-case ratio of the cost of an equilibrium with risk-averse players and that of an equilibrium with risk-neutral users. For risk-averse users who seek to minimize the mean plus variance of travel time on a path, they proved an upper bound on the price of risk aversion, which is independent of the latency functions, and grows linearly with the size of the graph and players' risk-aversion. In this follow-up paper, we provide a matching lower bound for graphs with number of vertices equal to powers of two, via the construction of a graph family inductively generated from the Braess graph. We also provide conceptually different bounds, which we call functional, that depend on the class of mean latency functions and provide characterizations that are independent of the network topology (first derived, in a more complicated way, by Meir and Parkes [10] in a different context with different techniques). We also supplement the upper bound with a new asymptotically-tight lower bound. Our third contribution is a tight bound on the price of risk aversion for a family of graphs that generalize series-parallel graphs which applies to users minimizing the mean plus standard deviation of a path, a much more complex model of risk-aversion due to the cost of a path being non-additive over edge costs. This is a refinement of previous results in [16] that characterized the price of risk-aversion for series-parallel graphs and for the Braess graph.
△ Less
Submitted 16 October, 2015; v1 submitted 7 October, 2015;
originally announced October 2015.
-
The Burden of Risk Aversion in Mean-Risk Selfish Routing
Authors:
E. Nikolova,
N. Stier-Moses
Abstract:
Considering congestion games with uncertain delays, we compute the inefficiency introduced in network routing by risk-averse agents. At equilibrium, agents may select paths that do not minimize the expected latency so as to obtain lower variability. A social planner, who is likely to be more risk neutral than agents because it operates at a longer time-scale, quantifies social cost with the total…
▽ More
Considering congestion games with uncertain delays, we compute the inefficiency introduced in network routing by risk-averse agents. At equilibrium, agents may select paths that do not minimize the expected latency so as to obtain lower variability. A social planner, who is likely to be more risk neutral than agents because it operates at a longer time-scale, quantifies social cost with the total expected delay along routes. From that perspective, agents may make suboptimal decisions that degrade long-term quality. We define the {\em price of risk aversion} (PRA) as the worst-case ratio of the social cost at a risk-averse Wardrop equilibrium to that where agents are risk-neutral. For networks with general delay functions and a single source-sink pair, we show that the PRA depends linearly on the agents' risk tolerance and on the degree of variability present in the network. In contrast to the {\em price of anarchy}, in general the PRA increases when the network gets larger but it does not depend on the shape of the delay functions. To get this result we rely on a combinatorial proof that employs alternating paths that are reminiscent of those used in max-flow algorithms. For {\em series-parallel} (SP) graphs, the PRA becomes independent of the network topology and its size. As a result of independent interest, we prove that for SP networks with deterministic delays, Wardrop equilibria {\em maximize} the shortest-path objective among all feasible flows.
△ Less
Submitted 4 November, 2014; v1 submitted 31 October, 2014;
originally announced November 2014.
