-
Assessment of Sign Language-Based versus Touch-Based Input for Deaf Users Interacting with Intelligent Personal Assistants
Authors:
Nina Tran,
Paige DeVries,
Matthew Seita,
Raja Kushalnagar,
Abraham Glasser,
Christian Vogler
Abstract:
With the recent advancements in intelligent personal assistants (IPAs), their popularity is rapidly increasing when it comes to utilizing Automatic Speech Recognition within households. In this study, we used a Wizard-of-Oz methodology to evaluate and compare the usability of American Sign Language (ASL), Tap to Alexa, and smart home apps among 23 deaf participants within a limited-domain smart ho…
▽ More
With the recent advancements in intelligent personal assistants (IPAs), their popularity is rapidly increasing when it comes to utilizing Automatic Speech Recognition within households. In this study, we used a Wizard-of-Oz methodology to evaluate and compare the usability of American Sign Language (ASL), Tap to Alexa, and smart home apps among 23 deaf participants within a limited-domain smart home environment. Results indicate a slight usability preference for ASL. Linguistic analysis of the participants' signing reveals a diverse range of expressions and vocabulary as they interacted with IPAs in the context of a restricted-domain application. On average, deaf participants exhibited a vocabulary of 47 +/- 17 signs with an additional 10 +/- 7 fingerspelled words, for a total of 246 different signs and 93 different fingerspelled words across all participants. We discuss the implications for the design of limited-vocabulary applications as a step**-stone toward general-purpose ASL recognition in the future.
△ Less
Submitted 22 April, 2024;
originally announced April 2024.
-
How Users Experience Closed Captions on Live Television: Quality Metrics Remain a Challenge
Authors:
Mariana Arroyo Chavez,
Molly Feanny,
Matthew Seita,
Bernard Thompson,
Keith Delk,
Skyler Officer,
Abraham Glasser,
Raja Kushalnagar,
Christian Vogler
Abstract:
This paper presents a mixed methods study on how deaf, hard of hearing and hearing viewers perceive live TV caption quality with captioned video stimuli designed to mirror TV captioning experiences. To assess caption quality, we used four commonly-used quality metrics focusing on accuracy: word error rate, weighted word error rate, automated caption evaluation (ACE), and its successor ACE2. We cal…
▽ More
This paper presents a mixed methods study on how deaf, hard of hearing and hearing viewers perceive live TV caption quality with captioned video stimuli designed to mirror TV captioning experiences. To assess caption quality, we used four commonly-used quality metrics focusing on accuracy: word error rate, weighted word error rate, automated caption evaluation (ACE), and its successor ACE2. We calculated the correlation between the four quality metrics and viewer ratings for subjective quality and found that the correlation was weak, revealing that other factors besides accuracy affect user ratings. Additionally, even high-quality captions are perceived to have problems, despite controlling for confounding factors. Qualitative analysis of viewer comments revealed three major factors affecting their experience: Errors within captions, difficulty in following captions, and caption appearance. The findings raise questions as to how objective caption quality metrics can be reconciled with the user experience across a diverse spectrum of viewers.
△ Less
Submitted 15 April, 2024;
originally announced April 2024.
-
ASL-Homework-RGBD Dataset: An annotated dataset of 45 fluent and non-fluent signers performing American Sign Language homeworks
Authors:
Saad Hassan,
Matthew Seita,
Larwan Berke,
Yingli Tian,
Elaine Gale,
Sooyeon Lee,
Matt Huenerfauth
Abstract:
We are releasing a dataset containing videos of both fluent and non-fluent signers using American Sign Language (ASL), which were collected using a Kinect v2 sensor. This dataset was collected as a part of a project to develop and evaluate computer vision algorithms to support new technologies for automatic detection of ASL fluency attributes. A total of 45 fluent and non-fluent participants were…
▽ More
We are releasing a dataset containing videos of both fluent and non-fluent signers using American Sign Language (ASL), which were collected using a Kinect v2 sensor. This dataset was collected as a part of a project to develop and evaluate computer vision algorithms to support new technologies for automatic detection of ASL fluency attributes. A total of 45 fluent and non-fluent participants were asked to perform signing homework assignments that are similar to the assignments used in introductory or intermediate level ASL courses. The data is annotated to identify several aspects of signing including grammatical features and non-manual markers. Sign language recognition is currently very data-driven and this dataset can support the design of recognition technologies, especially technologies that can benefit ASL learners. This dataset might also be interesting to ASL education researchers who want to contrast fluent and non-fluent signing.
△ Less
Submitted 8 July, 2022;
originally announced July 2022.
-
Closed ASL Interpreting for Online Videos
Authors:
Raja Kushalnagar,
Matthew Seita,
Abraham Glasser
Abstract:
Deaf individuals face great challenges in today's society. It can be very difficult to be able to understand different forms of media without a sense of hearing. Many videos and movies found online today are not captioned, and even fewer have a supporting video with an interpreter. Also, even with a supporting interpreter video provided, information is still lost due to the inability to look at bo…
▽ More
Deaf individuals face great challenges in today's society. It can be very difficult to be able to understand different forms of media without a sense of hearing. Many videos and movies found online today are not captioned, and even fewer have a supporting video with an interpreter. Also, even with a supporting interpreter video provided, information is still lost due to the inability to look at both the video and the interpreter simultaneously. To alleviate this issue, we came up with a tool called closed interpreting. Similar to closed captioning, it will be displayed with an online video and can be toggled on and off. However, the closed interpreter is also user-adjustable. Settings, such as interpreter size, transparency, and location, can be adjusted. Our goal with this study is to find out what deaf and hard of hearing viewers like about videos that come with interpreters, and whether the adjustability is beneficial.
△ Less
Submitted 5 September, 2019;
originally announced September 2019.
-
Artificial Intelligence Fairness in the Context of Accessibility Research on Intelligent Systems for People who are Deaf or Hard of Hearing
Authors:
Sushant Kafle,
Abraham Glasser,
Sedeeq Al-khazraji,
Larwan Berke,
Matthew Seita,
Matt Huenerfauth
Abstract:
We discuss issues of Artificial Intelligence (AI) fairness for people with disabilities, with examples drawn from our research on human-computer interaction (HCI) for AI-based systems for people who are Deaf or Hard of Hearing (DHH). In particular, we discuss the need for inclusion of data from people with disabilities in training sets, the lack of interpretability of AI systems, ethical responsib…
▽ More
We discuss issues of Artificial Intelligence (AI) fairness for people with disabilities, with examples drawn from our research on human-computer interaction (HCI) for AI-based systems for people who are Deaf or Hard of Hearing (DHH). In particular, we discuss the need for inclusion of data from people with disabilities in training sets, the lack of interpretability of AI systems, ethical responsibilities of access technology researchers and companies, the need for appropriate evaluation metrics for AI-based access technologies (to determine if they are ready to be deployed and if they can be trusted by users), and the ways in which AI systems influence human behavior and influence the set of abilities needed by users to successfully interact with computing systems.
△ Less
Submitted 2 September, 2019; v1 submitted 27 August, 2019;
originally announced August 2019.
