-
AltGeoViz: Facilitating Accessible Geovisualization
Authors:
Chu Li,
Rock Yuren Pang,
Ather Sharif,
Arnavi Chheda-Kothary,
Jeffrey Heer,
Jon E. Froehlich
Abstract:
Geovisualizations are powerful tools for exploratory spatial analysis, enabling sighted users to discern patterns, trends, and relationships within geographic data. However, these visual tools have remained largely inaccessible to screen-reader users. We present AltGeoViz, a new system we designed to facilitate geovisualization exploration for these users. AltGeoViz dynamically generates alt-text…
▽ More
Geovisualizations are powerful tools for exploratory spatial analysis, enabling sighted users to discern patterns, trends, and relationships within geographic data. However, these visual tools have remained largely inaccessible to screen-reader users. We present AltGeoViz, a new system we designed to facilitate geovisualization exploration for these users. AltGeoViz dynamically generates alt-text descriptions based on the user's current map view, providing summaries of spatial patterns and descriptive statistics. In a study of five screen-reader users, we found that AltGeoViz enabled them to interact with geovisualizations in previously infeasible ways. Participants demonstrated a clear understanding of data summaries and their location context, and they could synthesize spatial understandings of their explorations. Moreover, we identified key areas for improvement, such as the addition of intuitive spatial navigation controls and comparative analysis features.
△ Less
Submitted 21 June, 2024; v1 submitted 19 June, 2024;
originally announced June 2024.
-
SonifyAR: Context-Aware Sound Generation in Augmented Reality
Authors:
Xia Su,
Jon E. Froehlich,
Eunyee Koh,
Chang Xiao
Abstract:
Sound plays a crucial role in enhancing user experience and immersiveness in Augmented Reality (AR). However, current platforms lack support for AR sound authoring due to limited interaction types, challenges in collecting and specifying context information, and difficulty in acquiring matching sound assets. We present SonifyAR, an LLM-based AR sound authoring system that generates context-aware s…
▽ More
Sound plays a crucial role in enhancing user experience and immersiveness in Augmented Reality (AR). However, current platforms lack support for AR sound authoring due to limited interaction types, challenges in collecting and specifying context information, and difficulty in acquiring matching sound assets. We present SonifyAR, an LLM-based AR sound authoring system that generates context-aware sound effects for AR experiences. SonifyAR expands the current design space of AR sound and implements a Programming by Demonstration (PbD) pipeline to automatically collect contextual information of AR events, including virtual content semantics and real world context. This context information is then processed by a large language model to acquire sound effects with Recommendation, Retrieval, Generation, and Transfer methods. To evaluate the usability and performance of our system, we conducted a user study with eight participants and created five example applications, including an AR-based science experiment, an improving case for AR headset safety, and an assisting example for low vision AR users.
△ Less
Submitted 15 May, 2024; v1 submitted 11 May, 2024;
originally announced May 2024.
-
GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality
Authors:
Jaewook Lee,
Jun Wang,
Elizabeth Brown,
Liam Chu,
Sebastian S. Rodriguez,
Jon E. Froehlich
Abstract:
Voice assistants (VAs) like Siri and Alexa are transforming human-computer interaction; however, they lack awareness of users' spatiotemporal context, resulting in limited performance and unnatural dialogue. We introduce GazePointAR, a fully-functional context-aware VA for wearable augmented reality that leverages eye gaze, pointing gestures, and conversation history to disambiguate speech queries…
▽ More
Voice assistants (VAs) like Siri and Alexa are transforming human-computer interaction; however, they lack awareness of users' spatiotemporal context, resulting in limited performance and unnatural dialogue. We introduce GazePointAR, a fully-functional context-aware VA for wearable augmented reality that leverages eye gaze, pointing gestures, and conversation history to disambiguate speech queries. With GazePointAR, users can ask "what's over there?" or "how do I solve this math problem?" simply by looking and/or pointing. We evaluated GazePointAR in a three-part lab study (N=12): (1) comparing GazePointAR to two commercial systems; (2) examining GazePointAR's pronoun disambiguation across three tasks; (3) and an open-ended phase where participants could suggest and try their own context-sensitive queries. Participants appreciated the naturalness and human-like nature of pronoun-driven queries, although sometimes pronoun use was counter-intuitive. We then iterated on GazePointAR and conducted a first-person diary study examining how GazePointAR performs in-the-wild. We conclude by enumerating limitations and design considerations for future context-aware VAs.
△ Less
Submitted 11 April, 2024;
originally announced April 2024.
-
LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems
Authors:
Chu Li,
Zhihan Zhang,
Michael Saugstad,
Esteban Safranchik,
Minchu Kulkarni,
Xiaoyu Huang,
Shwetak Patel,
Vikram Iyer,
Tim Althoff,
Jon E. Froehlich
Abstract:
Crowdsourcing platforms have transformed distributed problem-solving, yet quality control remains a persistent challenge. Traditional quality control measures, such as prescreening workers and refining instructions, often focus solely on optimizing economic output. This paper explores just-in-time AI interventions to enhance both labeling quality and domain-specific knowledge among crowdworkers. W…
▽ More
Crowdsourcing platforms have transformed distributed problem-solving, yet quality control remains a persistent challenge. Traditional quality control measures, such as prescreening workers and refining instructions, often focus solely on optimizing economic output. This paper explores just-in-time AI interventions to enhance both labeling quality and domain-specific knowledge among crowdworkers. We introduce LabelAId, an advanced inference model combining Programmatic Weak Supervision (PWS) with FT-Transformers to infer label correctness based on user behavior and domain knowledge. Our technical evaluation shows that our LabelAId pipeline consistently outperforms state-of-the-art ML baselines, improving mistake inference accuracy by 36.7% with 50 downstream samples. We then implemented LabelAId into Project Sidewalk, an open-source crowdsourcing platform for urban accessibility. A between-subjects study with 34 participants demonstrates that LabelAId significantly enhances label precision without compromising efficiency while also increasing labeler confidence. We discuss LabelAId's success factors, limitations, and its generalizability to other crowdsourced science domains.
△ Less
Submitted 14 March, 2024;
originally announced March 2024.
-
HandSight: DeCAF & Improved Fisher Vectors to Classify Clothing Color and Texture with a Finger-Mounted Camera
Authors:
Alexander J. Medeiros,
Lee Stearns,
Jon E. Froehlich
Abstract:
We demonstrate the use of DeCAF and Improved Fisher Vector image features to classify clothing texture. The issue of choosing clothes is a problem for the blind every day. This work attempts to solve the issue with a finger-mounted camera and state-of-the-art classification algorithms. To evaluate our solution, we collected 520 close-up images across 29 pieces of clothing. We contribute (1) the HC…
▽ More
We demonstrate the use of DeCAF and Improved Fisher Vector image features to classify clothing texture. The issue of choosing clothes is a problem for the blind every day. This work attempts to solve the issue with a finger-mounted camera and state-of-the-art classification algorithms. To evaluate our solution, we collected 520 close-up images across 29 pieces of clothing. We contribute (1) the HCTD, an image dataset taken with a NanEyeGS camera, a camera small enough to be mounted on the finger, and (2) evaluations of state-of-the-art recognition algorithms applied to our dataset - achieving an accuracy >95%. Throughout the paper, we will discuss previous work, evaluate the current work, and finally, suggest the project's future direction.
△ Less
Submitted 20 November, 2023;
originally announced November 2023.
-
A Pilot Study of Sidewalk Equity in Seattle Using Crowdsourced Sidewalk Assessment Data
Authors:
Chu Li,
Lisa Orii,
Mikey Saugstad,
Stephen J. Mooney,
Yochai Eisenberg,
Delphine Labbé,
Joy Hammel,
Jon E. Froehlich
Abstract:
We examine the potential of using large-scale open crowdsourced sidewalk data from Project Sidewalk to study the distribution and condition of sidewalks in Seattle, WA. While potentially noisier than professionally gathered sidewalk datasets, crowdsourced data enables large, cross-regional studies that would be otherwise expensive and difficult to manage. As an initial case study, we examine spati…
▽ More
We examine the potential of using large-scale open crowdsourced sidewalk data from Project Sidewalk to study the distribution and condition of sidewalks in Seattle, WA. While potentially noisier than professionally gathered sidewalk datasets, crowdsourced data enables large, cross-regional studies that would be otherwise expensive and difficult to manage. As an initial case study, we examine spatial patterns of sidewalk quality in Seattle and their relationship to racial diversity, income level, built density, and transit modes. We close with a reflection on our approach, key limitations, and opportunities for future work.
△ Less
Submitted 5 October, 2022;
originally announced November 2022.
-
Towards Semi-automatic Detection and Localization of Indoor Accessibility Issues using Mobile Depth Scanning and Computer Vision
Authors:
Xia Su,
Kaiming Cheng,
Han Zhang,
Jaewook Lee,
Jon E. Froehlich
Abstract:
To help improve the safety and accessibility of indoor spaces, researchers and health professionals have created assessment instruments that enable homeowners and trained experts to audit and improve homes. With advances in computer vision, augmented reality (AR), and mobile sensors, new approaches are now possible. We introduce RASSAR (Room Accessibility and Safety Scanning in Augmented Reality),…
▽ More
To help improve the safety and accessibility of indoor spaces, researchers and health professionals have created assessment instruments that enable homeowners and trained experts to audit and improve homes. With advances in computer vision, augmented reality (AR), and mobile sensors, new approaches are now possible. We introduce RASSAR (Room Accessibility and Safety Scanning in Augmented Reality), a new proof-of-concept prototype for semi-automatically identifying, categorizing, and localizing indoor accessibility and safety issues using LiDAR + camera data, machine learning, and AR. We present an overview of the current RASSAR prototype and a preliminary evaluation in a single home.
△ Less
Submitted 5 October, 2022;
originally announced October 2022.
-
Towards Map** and Assessing Sidewalk Accessibility Across Sociocultural and Geographic Contexts
Authors:
Jon E. Froehlich,
Michael Saugstad,
Manaswi Saha,
Matthew Johnson
Abstract:
Despite the important role of sidewalks in supporting mobility, accessibility, and public health, there is a lack of high-quality datasets and corresponding analyses on sidewalk existence and condition. Our work explores a twofold vision: first, to develop scalable mechanisms to locate and assess sidewalks in cities across the world, and second, to use this data to support new urban analyses and m…
▽ More
Despite the important role of sidewalks in supporting mobility, accessibility, and public health, there is a lack of high-quality datasets and corresponding analyses on sidewalk existence and condition. Our work explores a twofold vision: first, to develop scalable mechanisms to locate and assess sidewalks in cities across the world, and second, to use this data to support new urban analyses and mobility tools. We report on two preliminary urban science explorations enabled by our approach: exploring geo-spatial patterns and key correlates of sidewalk accessibility and examining differences in sidewalk infrastructure across regions.
△ Less
Submitted 27 July, 2022;
originally announced July 2022.
-
Towards Global-Scale Crowd+AI Techniques to Map and Assess Sidewalks for People with Disabilities
Authors:
Maryam Hosseini,
Mikey Saugstad,
Fabio Miranda,
Andres Sevtsuk,
Claudio T. Silva,
Jon E. Froehlich
Abstract:
There is a lack of data on the location, condition, and accessibility of sidewalks across the world, which not only impacts where and how people travel but also fundamentally limits interactive map** tools and urban analytics. In this paper, we describe initial work in semi-automatically building a sidewalk network topology from satellite imagery using hierarchical multi-scale attention models,…
▽ More
There is a lack of data on the location, condition, and accessibility of sidewalks across the world, which not only impacts where and how people travel but also fundamentally limits interactive map** tools and urban analytics. In this paper, we describe initial work in semi-automatically building a sidewalk network topology from satellite imagery using hierarchical multi-scale attention models, inferring surface materials from street-level images using active learning-based semantic segmentation, and assessing sidewalk condition and accessibility features using Crowd+AI. We close with a call to create a database of labeled satellite and streetscape scenes for sidewalks and sidewalk accessibility issues along with standardized benchmarks.
△ Less
Submitted 18 August, 2022; v1 submitted 27 June, 2022;
originally announced June 2022.
-
ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users
Authors:
Dhruv Jain,
Khoa Huynh Anh Nguyen,
Steven Goodman,
Rachel Grossman-Kahn,
Hung Ngo,
Aditya Kusupati,
Ruofei Du,
Alex Olwal,
Leah Findlater,
Jon E. Froehlich
Abstract:
Recent advances have enabled automatic sound recognition systems for deaf and hard of hearing (DHH) users on mobile devices. However, these tools use pre-trained, generic sound recognition models, which do not meet the diverse needs of DHH users. We introduce ProtoSound, an interactive system for customizing sound recognition models by recording a few examples, thereby enabling personalized and fi…
▽ More
Recent advances have enabled automatic sound recognition systems for deaf and hard of hearing (DHH) users on mobile devices. However, these tools use pre-trained, generic sound recognition models, which do not meet the diverse needs of DHH users. We introduce ProtoSound, an interactive system for customizing sound recognition models by recording a few examples, thereby enabling personalized and fine-grained categories. ProtoSound is motivated by prior work examining sound awareness needs of DHH people and by a survey we conducted with 472 DHH participants. To evaluate ProtoSound, we characterized performance on two real-world sound datasets, showing significant improvement over state-of-the-art (e.g., +9.7% accuracy on the first dataset). We then deployed ProtoSound's end-user training and real-time recognition through a mobile application and recruited 19 hearing participants who listened to the real-world sounds and rated the accuracy across 56 locations (e.g., homes, restaurants, parks). Results show that ProtoSound personalized the model on-device in real-time and accurately learned sounds across diverse acoustic contexts. We close by discussing open challenges in personalizable sound recognition, including the need for better recording interfaces and algorithmic improvements.
△ Less
Submitted 22 February, 2022;
originally announced February 2022.
-
Social, Environmental, and Technical: Factors at Play in the Current Use and Future Design of Small-Group Captioning
Authors:
Emma J. McDonnell,
** Liu,
Steven M. Goodman,
Raja Kushalnagar,
Jon E. Froehlich,
Leah Findlater
Abstract:
Real-time captioning is a critical accessibility tool for many d/Deaf and hard of hearing (DHH) people. While the vast majority of captioning work has focused on formal settings and technical innovations, in contrast, we investigate captioning for informal, interactive small-group conversations, which have a high degree of spontaneity and foster dynamic social interactions. This paper reports on s…
▽ More
Real-time captioning is a critical accessibility tool for many d/Deaf and hard of hearing (DHH) people. While the vast majority of captioning work has focused on formal settings and technical innovations, in contrast, we investigate captioning for informal, interactive small-group conversations, which have a high degree of spontaneity and foster dynamic social interactions. This paper reports on semi-structured interviews and design probe activities we conducted with 15 DHH participants to understand their use of existing real-time captioning services and future design preferences for both in-person and remote small-group communication. We found that our participants' experiences of captioned small-group conversations are shaped by social, environmental, and technical considerations (e.g., interlocutors' pre-established relationships, the type of captioning displays available, and how far captions lag behind speech). When considering future captioning tools, participants were interested in greater feedback on non-speech elements of conversation (e.g., speaker identity, speech rate, volume) both for their personal use and to guide hearing interlocutors toward more accessible communication. We contribute a qualitative account of DHH people's real-time captioning experiences during small-group conversation and future design considerations to better support the groups being captioned, both in person and online.
△ Less
Submitted 21 September, 2021;
originally announced September 2021.
-
A bibliometric analysis of citation diversity in accessibility and HCI research
Authors:
Lucy Lu Wang,
Kelly Mack,
Emma McDonnell,
Dhruv Jain,
Leah Findlater,
Jon E. Froehlich
Abstract:
Accessibility research sits at the junction of several disciplines, drawing influence from HCI, disability studies, psychology, education, and more. To characterize the influences and extensions of accessibility research, we undertake a study of citation trends for accessibility and related HCI communities. We assess the diversity of venues and fields of study represented among the referenced and…
▽ More
Accessibility research sits at the junction of several disciplines, drawing influence from HCI, disability studies, psychology, education, and more. To characterize the influences and extensions of accessibility research, we undertake a study of citation trends for accessibility and related HCI communities. We assess the diversity of venues and fields of study represented among the referenced and citing papers of 836 accessibility research papers from ASSETS and CHI, finding that though publications in computer science dominate these citation relationships, the relative proportion of citations from papers on psychology and medicine has grown over time. Though ASSETS is a more niche venue than CHI in terms of citational diversity, both conferences display standard levels of diversity among their incoming and outgoing citations when analyzed in the context of 53K papers from 13 accessibility and HCI conference venues.
△ Less
Submitted 11 March, 2021;
originally announced March 2021.
-
What Do We Mean by "Accessibility Research"? A Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019
Authors:
Kelly Mack,
Emma McDonnell,
Dhruv Jain,
Lucy Lu Wang,
Jon E. Froehlich,
Leah Findlater
Abstract:
Accessibility research has grown substantially in the past few decades, yet there has been no literature review of the field. To understand current and historical trends, we created and analyzed a dataset of accessibility papers appearing at CHI and ASSETS since ASSETS' founding in 1994. We qualitatively coded areas of focus and methodological decisions for the past 10 years (2010-2019, N=506 pape…
▽ More
Accessibility research has grown substantially in the past few decades, yet there has been no literature review of the field. To understand current and historical trends, we created and analyzed a dataset of accessibility papers appearing at CHI and ASSETS since ASSETS' founding in 1994. We qualitatively coded areas of focus and methodological decisions for the past 10 years (2010-2019, N=506 papers), and analyzed paper counts and keywords over the full 26 years (N=836 papers). Our findings highlight areas that have received disproportionate attention and those that are underserved--for example, over 43% of papers in the past 10 years are on accessibility for blind and low vision people. We also capture common study characteristics, such as the roles of disabled and nondisabled participants as well as sample sizes (e.g., a median of 13 for participant groups with disabilities and older adults). We close by critically reflecting on gaps in the literature and offering guidance for future work in the field.
△ Less
Submitted 3 February, 2021; v1 submitted 11 January, 2021;
originally announced January 2021.
-
Organizing Family Support Services at ACM Conferences
Authors:
Audrey Girouard,
Jon E. Froehlich,
Regan Mandryk,
Mark Hancock
Abstract:
This article reflects on our experiences providing family-support services to a large, interdisciplinary ACM conference (CHI2018) including, the policy decisions, the challenges, and the successes. The article incorporates empirical data collected from pre- and post-conference surveys, observed use of the services, and aspirational aims for future conferences. We are discussing best practices and…
▽ More
This article reflects on our experiences providing family-support services to a large, interdisciplinary ACM conference (CHI2018) including, the policy decisions, the challenges, and the successes. The article incorporates empirical data collected from pre- and post-conference surveys, observed use of the services, and aspirational aims for future conferences. We are discussing best practices and recommendations to facilitate the implementation of child support services at other conferences. We believe our article will be of great interest to both practitioners and academics in expanding the inclusivity and family support provided by ACM conferences and beyond.
△ Less
Submitted 12 December, 2019;
originally announced December 2019.
