CrowdHub: Extending crowdsourcing platforms for the controlled evaluation of tasks designs
Authors:
Jorge RamÃrez,
Simone Degiacomi,
Davide Zanella,
Marcos Baez,
Fabio Casati,
Boualem Benatallah
Abstract:
We present CrowdHub, a tool for running systematic evaluations of task designs on top of crowdsourcing platforms. The goal is to support the evaluation process, avoiding potential experimental biases that, according to our empirical studies, can amount to 38% loss in the utility of the collected dataset in uncontrolled settings. Using CrowdHub, researchers can map their experimental design and aut…
▽ More
We present CrowdHub, a tool for running systematic evaluations of task designs on top of crowdsourcing platforms. The goal is to support the evaluation process, avoiding potential experimental biases that, according to our empirical studies, can amount to 38% loss in the utility of the collected dataset in uncontrolled settings. Using CrowdHub, researchers can map their experimental design and automate the complex process of managing task execution over time while controlling for returning workers and crowd demographics, thus reducing bias, increasing utility of collected data, and making more efficient use of a limited pool of subjects.
△ Less
Submitted 10 September, 2019; v1 submitted 6 September, 2019;
originally announced September 2019.
