On Optimizing Human-Machine Task Assignments
Authors:
Andreas Veit,
Michael Wilber,
Rajan Vaish,
Serge Belongie,
James Davis,
Vishal Anand,
Anshu Aviral,
Prithvijit Chakrabarty,
Yash Chandak,
Sidharth Chaturvedi,
Chinmaya Devaraj,
Ankit Dhall,
Utkarsh Dwivedi,
Sanket Gupte,
Sharath N. Sridhar,
Karthik Paga,
Anuj Pahuja,
Aditya Raisinghani,
Ayush Sharma,
Shweta Sharma,
Darpana Sinha,
Nisarg Thakkar,
K. Bala Vignesh,
Utkarsh Verma,
Kanniganti Abhishek
, et al. (26 additional authors not shown)
Abstract:
When crowdsourcing systems are used in combination with machine inference systems in the real world, they benefit the most when the machine system is deeply integrated with the crowd workers. However, if researchers wish to integrate the crowd with "off-the-shelf" machine classifiers, this deep integration is not always possible. This work explores two strategies to increase accuracy and decrease…
▽ More
When crowdsourcing systems are used in combination with machine inference systems in the real world, they benefit the most when the machine system is deeply integrated with the crowd workers. However, if researchers wish to integrate the crowd with "off-the-shelf" machine classifiers, this deep integration is not always possible. This work explores two strategies to increase accuracy and decrease cost under this setting. First, we show that reordering tasks presented to the human can create a significant accuracy improvement. Further, we show that greedily choosing parameters to maximize machine accuracy is sub-optimal, and joint optimization of the combined system improves performance.
△ Less
Submitted 24 September, 2015;
originally announced September 2015.
