-
Data-Driven Switchback Experiments: Theoretical Tradeoffs and Empirical Bayes Designs
Authors:
Ruoxuan Xiong,
Alex Chin,
Sean J. Taylor
Abstract:
We study the design and analysis of switchback experiments conducted on a single aggregate unit. The design problem is to partition the continuous time space into intervals and switch treatments between intervals, in order to minimize the estimation error of the treatment effect. We show that the estimation error depends on four factors: carryover effects, periodicity, serially correlated outcomes…
▽ More
We study the design and analysis of switchback experiments conducted on a single aggregate unit. The design problem is to partition the continuous time space into intervals and switch treatments between intervals, in order to minimize the estimation error of the treatment effect. We show that the estimation error depends on four factors: carryover effects, periodicity, serially correlated outcomes, and impacts from simultaneous experiments. We derive a rigorous bias-variance decomposition and show the tradeoffs of the estimation error from these factors. The decomposition provides three new insights in choosing a design: First, balancing the periodicity between treated and control intervals reduces the variance; second, switching less frequently reduces the bias from carryover effects while increasing the variance from correlated outcomes, and vice versa; third, randomizing interval start and end points reduces both bias and variance from simultaneous experiments. Combining these insights, we propose a new empirical Bayes design approach. This approach uses prior data and experiments for designing future experiments. We illustrate this approach using real data from a ride-sharing platform, yielding a design that reduces MSE by 33% compared to the status quo design used on the platform.
△ Less
Submitted 10 June, 2024;
originally announced June 2024.
-
Displaying Things in Common to Encourage Friendship Formation: A Large Randomized Field Experiment
Authors:
Tianshu Sun,
Sean J. Taylor
Abstract:
Friendship formation is important to online social network sites and to society, but can suffer from informational friction. In this study, we demonstrate that social networks may effectively use an IT-facilitated intervention -- displaying things in common (TIC) between users (mutual hometown, interest, education, work, city) -- to encourage friendship formation. Displaying TIC updates an individ…
▽ More
Friendship formation is important to online social network sites and to society, but can suffer from informational friction. In this study, we demonstrate that social networks may effectively use an IT-facilitated intervention -- displaying things in common (TIC) between users (mutual hometown, interest, education, work, city) -- to encourage friendship formation. Displaying TIC updates an individual's belief about the shared similarity with another and reduces information friction that may be hard to overcome in offline communication. In collaboration with an online social network, we conduct a randomized field experiment that randomly varies the prominence of different things in common when a user is browsing a non-friend's profile. The dyad-level exogenous variation, orthogonal to any (un)observed structural factors in viewer-profile's network, allows us to cleanly isolate the role of preferences for TIC in driving network formation and homophily. We find that displaying TICs to viewers can significantly increase their probability of sending a friend requests and forming friendships, and is especially effective for pairs of people who have little in common. We also find that displaying TIC can improve friendship formation for a wide range of viewers with different demographics and is more effective when the TICs are more surprising to the viewer.
△ Less
Submitted 7 May, 2019;
originally announced May 2019.
-
Randomized experiments to detect and estimate social influence in networks
Authors:
Sean J. Taylor,
Dean Eckles
Abstract:
Estimation of social influence in networks can be substantially biased in observational studies due to homophily and network correlation in exposure to exogenous events. Randomized experiments, in which the researcher intervenes in the social system and uses randomization to determine how to do so, provide a methodology for credibly estimating of causal effects of social behaviors. In addition to…
▽ More
Estimation of social influence in networks can be substantially biased in observational studies due to homophily and network correlation in exposure to exogenous events. Randomized experiments, in which the researcher intervenes in the social system and uses randomization to determine how to do so, provide a methodology for credibly estimating of causal effects of social behaviors. In addition to addressing questions central to the social sciences, these estimates can form the basis for effective marketing and public policy.
In this review, we discuss the design space of experiments to measure social influence through combinations of interventions and randomizations. We define an experiment as combination of (1) a target population of individuals connected by an observed interaction network, (2) a set of treatments whereby the researcher will intervene in the social system, (3) a randomization strategy which maps individuals or edges to treatments, and (4) a measurement of an outcome of interest after treatment has been assigned. We review experiments that demonstrate potential experimental designs and we evaluate their advantages and tradeoffs for answering different types of causal questions about social influence. We show how randomization also provides a basis for statistical inference when analyzing these experiments.
△ Less
Submitted 27 September, 2017;
originally announced September 2017.
-
Discussion quality diffuses in the digital public square
Authors:
George Berry,
Sean J. Taylor
Abstract:
Studies of online social influence have demonstrated that friends have important effects on many types of behavior in a wide variety of settings. However, we know much less about how influence works among relative strangers in digital public squares, despite important conversations happening in such spaces. We present the results of a study on large public Facebook pages where we randomly used two…
▽ More
Studies of online social influence have demonstrated that friends have important effects on many types of behavior in a wide variety of settings. However, we know much less about how influence works among relative strangers in digital public squares, despite important conversations happening in such spaces. We present the results of a study on large public Facebook pages where we randomly used two different methods--most recent and social feedback--to order comments on posts. We find that the social feedback condition results in higher quality viewed comments and response comments. After measuring the average quality of comments written by users before the study, we find that social feedback has a positive effect on response quality for both low and high quality commenters. We draw on a theoretical framework of social norms to explain this empirical result. In order to examine the influence mechanism further, we measure the similarity between comments viewed and written during the study, finding that similarity increases for the highest quality contributors under the social feedback condition. This suggests that, in addition to norms, some individuals may respond with increased relevance to high-quality comments.
△ Less
Submitted 21 February, 2017;
originally announced February 2017.
-
Selection Effects in Online Sharing: Consequences for Peer Adoption
Authors:
Sean J. Taylor,
Eytan Bakshy,
Sinan Aral
Abstract:
Most models of social contagion take peer exposure to be a corollary of adoption, yet in many settings, the visibility of one's adoption behavior happens through a separate decision process. In online systems, product designers can define how peer exposure mechanisms work: adoption behaviors can be shared in a passive, automatic fashion, or occur through explicit, active sharing. The consequences…
▽ More
Most models of social contagion take peer exposure to be a corollary of adoption, yet in many settings, the visibility of one's adoption behavior happens through a separate decision process. In online systems, product designers can define how peer exposure mechanisms work: adoption behaviors can be shared in a passive, automatic fashion, or occur through explicit, active sharing. The consequences of these mechanisms are of substantial practical and theoretical interest: passive sharing may increase total peer exposure but active sharing may expose higher quality products to peers who are more likely to adopt.
We examine selection effects in online sharing through a large-scale field experiment on Facebook that randomizes whether or not adopters share Offers (coupons) in a passive manner. We derive and estimate a joint discrete choice model of adopters' sharing decisions and their peers' adoption decisions. Our results show that active sharing enables a selection effect that exposes peers who are more likely to adopt than the population exposed under passive sharing.
We decompose the selection effect into two distinct mechanisms: active sharers expose peers to higher quality products, and the peers they share with are more likely to adopt independently of product quality. Simulation results show that the user-level mechanism comprises the bulk of the selection effect. The study's findings are among the first to address downstream peer effects induced by online sharing mechanisms, and can inform design in settings where a surplus of sharing could be viewed as costly.
△ Less
Submitted 12 November, 2013;
originally announced November 2013.
