-
LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems
Authors:
Chu Li,
Zhihan Zhang,
Michael Saugstad,
Esteban Safranchik,
Minchu Kulkarni,
Xiaoyu Huang,
Shwetak Patel,
Vikram Iyer,
Tim Althoff,
Jon E. Froehlich
Abstract:
Crowdsourcing platforms have transformed distributed problem-solving, yet quality control remains a persistent challenge. Traditional quality control measures, such as prescreening workers and refining instructions, often focus solely on optimizing economic output. This paper explores just-in-time AI interventions to enhance both labeling quality and domain-specific knowledge among crowdworkers. W…
▽ More
Crowdsourcing platforms have transformed distributed problem-solving, yet quality control remains a persistent challenge. Traditional quality control measures, such as prescreening workers and refining instructions, often focus solely on optimizing economic output. This paper explores just-in-time AI interventions to enhance both labeling quality and domain-specific knowledge among crowdworkers. We introduce LabelAId, an advanced inference model combining Programmatic Weak Supervision (PWS) with FT-Transformers to infer label correctness based on user behavior and domain knowledge. Our technical evaluation shows that our LabelAId pipeline consistently outperforms state-of-the-art ML baselines, improving mistake inference accuracy by 36.7% with 50 downstream samples. We then implemented LabelAId into Project Sidewalk, an open-source crowdsourcing platform for urban accessibility. A between-subjects study with 34 participants demonstrates that LabelAId significantly enhances label precision without compromising efficiency while also increasing labeler confidence. We discuss LabelAId's success factors, limitations, and its generalizability to other crowdsourced science domains.
△ Less
Submitted 14 March, 2024;
originally announced March 2024.
-
A Pilot Study of Sidewalk Equity in Seattle Using Crowdsourced Sidewalk Assessment Data
Authors:
Chu Li,
Lisa Orii,
Mikey Saugstad,
Stephen J. Mooney,
Yochai Eisenberg,
Delphine Labbé,
Joy Hammel,
Jon E. Froehlich
Abstract:
We examine the potential of using large-scale open crowdsourced sidewalk data from Project Sidewalk to study the distribution and condition of sidewalks in Seattle, WA. While potentially noisier than professionally gathered sidewalk datasets, crowdsourced data enables large, cross-regional studies that would be otherwise expensive and difficult to manage. As an initial case study, we examine spati…
▽ More
We examine the potential of using large-scale open crowdsourced sidewalk data from Project Sidewalk to study the distribution and condition of sidewalks in Seattle, WA. While potentially noisier than professionally gathered sidewalk datasets, crowdsourced data enables large, cross-regional studies that would be otherwise expensive and difficult to manage. As an initial case study, we examine spatial patterns of sidewalk quality in Seattle and their relationship to racial diversity, income level, built density, and transit modes. We close with a reflection on our approach, key limitations, and opportunities for future work.
△ Less
Submitted 5 October, 2022;
originally announced November 2022.
-
Towards Map** and Assessing Sidewalk Accessibility Across Sociocultural and Geographic Contexts
Authors:
Jon E. Froehlich,
Michael Saugstad,
Manaswi Saha,
Matthew Johnson
Abstract:
Despite the important role of sidewalks in supporting mobility, accessibility, and public health, there is a lack of high-quality datasets and corresponding analyses on sidewalk existence and condition. Our work explores a twofold vision: first, to develop scalable mechanisms to locate and assess sidewalks in cities across the world, and second, to use this data to support new urban analyses and m…
▽ More
Despite the important role of sidewalks in supporting mobility, accessibility, and public health, there is a lack of high-quality datasets and corresponding analyses on sidewalk existence and condition. Our work explores a twofold vision: first, to develop scalable mechanisms to locate and assess sidewalks in cities across the world, and second, to use this data to support new urban analyses and mobility tools. We report on two preliminary urban science explorations enabled by our approach: exploring geo-spatial patterns and key correlates of sidewalk accessibility and examining differences in sidewalk infrastructure across regions.
△ Less
Submitted 27 July, 2022;
originally announced July 2022.
-
Towards Global-Scale Crowd+AI Techniques to Map and Assess Sidewalks for People with Disabilities
Authors:
Maryam Hosseini,
Mikey Saugstad,
Fabio Miranda,
Andres Sevtsuk,
Claudio T. Silva,
Jon E. Froehlich
Abstract:
There is a lack of data on the location, condition, and accessibility of sidewalks across the world, which not only impacts where and how people travel but also fundamentally limits interactive map** tools and urban analytics. In this paper, we describe initial work in semi-automatically building a sidewalk network topology from satellite imagery using hierarchical multi-scale attention models,…
▽ More
There is a lack of data on the location, condition, and accessibility of sidewalks across the world, which not only impacts where and how people travel but also fundamentally limits interactive map** tools and urban analytics. In this paper, we describe initial work in semi-automatically building a sidewalk network topology from satellite imagery using hierarchical multi-scale attention models, inferring surface materials from street-level images using active learning-based semantic segmentation, and assessing sidewalk condition and accessibility features using Crowd+AI. We close with a call to create a database of labeled satellite and streetscape scenes for sidewalks and sidewalk accessibility issues along with standardized benchmarks.
△ Less
Submitted 18 August, 2022; v1 submitted 27 June, 2022;
originally announced June 2022.
