-
Boomerang: Rebounding the Consequences of Reputation Feedback on Crowdsourcing Platforms
Authors:
Snehalkumar,
S. Gaikwad,
Durim Morina,
Adam Ginzberg,
Catherine Mullings,
Shirish Goyal,
Dilrukshi Gamage,
Christopher Diemert,
Mathias Burton,
Sharon Zhou,
Mark Whiting,
Karolina Ziulkoski,
Alipta Ballav,
Aaron Gilbee,
Senadhipathige S. Niranga,
Vibhor Sehgal,
Jasmine Lin,
Leonardy Kristianto,
Angela Richmond-Fuller,
Jeff Regino,
Nalin Chhibber,
Dinesh Majeti,
Sachin Sharma,
Kamila Mananova,
Dinesh Dhakal
, et al. (13 additional authors not shown)
Abstract:
Paid crowdsourcing platforms suffer from low-quality work and unfair rejections, but paradoxically, most workers and requesters have high reputation scores. These inflated scores, which make high-quality work and workers difficult to find, stem from social pressure to avoid giving negative feedback. We introduce Boomerang, a reputation system for crowdsourcing that elicits more accurate feedback b…
▽ More
Paid crowdsourcing platforms suffer from low-quality work and unfair rejections, but paradoxically, most workers and requesters have high reputation scores. These inflated scores, which make high-quality work and workers difficult to find, stem from social pressure to avoid giving negative feedback. We introduce Boomerang, a reputation system for crowdsourcing that elicits more accurate feedback by rebounding the consequences of feedback directly back onto the person who gave it. With Boomerang, requesters find that their highly-rated workers gain earliest access to their future tasks, and workers find tasks from their highly-rated requesters at the top of their task feed. Field experiments verify that Boomerang causes both workers and requesters to provide feedback that is more closely aligned with their private opinions. Inspired by a game-theoretic notion of incentive-compatibility, Boomerang opens opportunities for interaction design to incentivize honest reporting over strategic dishonesty.
△ Less
Submitted 14 April, 2019;
originally announced April 2019.
-
Prototype Tasks: Improving Crowdsourcing Results through Rapid, Iterative Task Design
Authors:
Snehalkumar "Neil" S. Gaikwad,
Nalin Chhibber,
Vibhor Sehgal,
Alipta Ballav,
Catherine Mullings,
Ahmed Nasser,
Angela Richmond-Fuller,
Aaron Gilbee,
Dilrukshi Gamage,
Mark Whiting,
Sharon Zhou,
Sekandar Matin,
Senadhipathige Niranga,
Shirish Goyal,
Dinesh Majeti,
Preethi Srinivas,
Adam Ginzberg,
Kamila Mananova,
Karolina Ziulkoski,
Jeff Regino,
Tejas Sarma,
Akshansh Sinha,
Abhratanu Paul,
Christopher Diemert,
Mahesh Murag
, et al. (4 additional authors not shown)
Abstract:
Low-quality results have been a long-standing problem on microtask crowdsourcing platforms, driving away requesters and justifying low wages for workers. To date, workers have been blamed for low-quality results: they are said to make as little effort as possible, do not pay attention to detail, and lack expertise. In this paper, we hypothesize that requesters may also be responsible for low-quali…
▽ More
Low-quality results have been a long-standing problem on microtask crowdsourcing platforms, driving away requesters and justifying low wages for workers. To date, workers have been blamed for low-quality results: they are said to make as little effort as possible, do not pay attention to detail, and lack expertise. In this paper, we hypothesize that requesters may also be responsible for low-quality work: they launch unclear task designs that confuse even earnest workers, under-specify edge cases, and neglect to include examples. We introduce prototype tasks, a crowdsourcing strategy requiring all new task designs to launch a small number of sample tasks. Workers attempt these tasks and leave feedback, enabling the re- quester to iterate on the design before publishing it. We report a field experiment in which tasks that underwent prototype task iteration produced higher-quality work results than the original task designs. With this research, we suggest that a simple and rapid iteration cycle can improve crowd work, and we provide empirical evidence that requester "quality" directly impacts result quality.
△ Less
Submitted 18 July, 2017;
originally announced July 2017.
-
Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms
Authors:
Mark E. Whiting,
Dilrukshi Gamage,
Snehalkumar S. Gaikwad,
Aaron Gilbee,
Shirish Goyal,
Alipta Ballav,
Dinesh Majeti,
Nalin Chhibber,
Angela Richmond-Fuller,
Freddie Vargus,
Tejas Seshadri Sarma,
Varshine Chandrakanthan,
Teogenes Moura,
Mohamed Hashim Salih,
Gabriel Bayomi Tinoco Kalejaiye,
Adam Ginzberg,
Catherine A. Mullings,
Yoni Dayan,
Kristy Milland,
Henrique Orefice,
Jeff Regino,
Sayna Parsi,
Kunz Mainali,
Vibhor Sehgal,
Sekandar Matin
, et al. (3 additional authors not shown)
Abstract:
Crowd workers are distributed and decentralized. While decentralization is designed to utilize independent judgment to promote high-quality results, it paradoxically undercuts behaviors and institutions that are critical to high-quality work. Reputation is one central example: crowdsourcing systems depend on reputation scores from decentralized workers and requesters, but these scores are notoriou…
▽ More
Crowd workers are distributed and decentralized. While decentralization is designed to utilize independent judgment to promote high-quality results, it paradoxically undercuts behaviors and institutions that are critical to high-quality work. Reputation is one central example: crowdsourcing systems depend on reputation scores from decentralized workers and requesters, but these scores are notoriously inflated and uninformative. In this paper, we draw inspiration from historical worker guilds (e.g., in the silk trade) to design and implement crowd guilds: centralized groups of crowd workers who collectively certify each other's quality through double-blind peer assessment. A two-week field experiment compared crowd guilds to a traditional decentralized crowd work model. Crowd guilds produced reputation signals more strongly correlated with ground-truth worker quality than signals available on current crowd working platforms, and more accurate than in the traditional model.
△ Less
Submitted 28 February, 2017; v1 submitted 4 November, 2016;
originally announced November 2016.
