Showing 1–2 of 2 results for author: Smirnova, I

Crowdsourcing subjective annotations using pairwise comparisons reduces bias and error compared to the majority-vote method

Authors: Hasti Narimanzadeh, Arash Badie-Modiri, Iuliia Smirnova, Ted Hsuan Yun Chen

Abstract: How to better reduce measurement variability and bias introduced by subjectivity in crowdsourced labelling remains an open question. We introduce a theoretical framework for understanding how random error and measurement bias enter into crowdsourced annotations of subjective constructs. We then propose a pipeline that combines pairwise comparison labelling with Elo scoring, and demonstrate that it… ▽ More How to better reduce measurement variability and bias introduced by subjectivity in crowdsourced labelling remains an open question. We introduce a theoretical framework for understanding how random error and measurement bias enter into crowdsourced annotations of subjective constructs. We then propose a pipeline that combines pairwise comparison labelling with Elo scoring, and demonstrate that it outperforms the ubiquitous majority-voting method in reducing both types of measurement error. To assess the performance of the labelling approaches, we constructed an agent-based model of crowdsourced labelling that lets us introduce different types of subjectivity into the tasks. We find that under most conditions with task subjectivity, the comparison approach produced higher $f_1$ scores. Further, the comparison approach is less susceptible to inflating bias, which majority voting tends to do. To facilitate applications, we show with simulated and real-world data that the number of required random comparisons for the same classification accuracy scales log-linearly $O(N \log N)$ with the number of labelled items. We also implemented the Elo system as an open-source Python package. △ Less

Submitted 1 June, 2023; v1 submitted 31 May, 2023; originally announced May 2023.

Comments: Accepted for publication at ACM CSCW 2023

Journal ref: Proceedings of the ACM on Human-Computer Interaction, CSCW2 (2023) 1-29

A Systematic Approach to Setting Up Distributed Global Collaborations for Software-based Products in the Automotive Domain

Authors: Inna Smirnova

Abstract: There is an increasing need for organizations to collaborate with internal and external partners on a global scale for creating software-based products and services. Many aspects and risks need to be addressed when setting up such global collaborations. Different types of collaborations such as engineering collaborations or innovation-focused collaborations need to be considered. Further aspects s… ▽ More There is an increasing need for organizations to collaborate with internal and external partners on a global scale for creating software-based products and services. Many aspects and risks need to be addressed when setting up such global collaborations. Different types of collaborations such as engineering collaborations or innovation-focused collaborations need to be considered. Further aspects such as cultural and social aspects, coordination, infrastructure, organizational change process, and communication issues need to be examined. Although there are already experiences available with respect to setting up global collaborations, they are mainly focusing on certain specific areas. An overall holistic approach that guides companies in systematically setting up global collaborations for software-based products is widely missing. The goal of this thesis is to analyze existing literature and related information and to extract topics that need be taken into account while establishing global software development collaborations - to identify solutions, risks, success factors, strategies, good experiences as well as good examples. This information is structured in a way so that it can be used by companies as a well-grounded holistic approach to guide companies effectively in setting up long-term global collaborations in the domain software development. The presented approach is based on scientific findings reported in literature, driven by industry needs, and confirmed by industry experts. △ Less

Submitted 3 October, 2014; originally announced October 2014.

Comments: Master's thesis, 72 pages