-
Sign Language-Based versus Touch-Based Input for Deaf Users with Interactive Personal Assistants in Simulated Kitchen Environments
Authors:
Paige DeVries,
Nina Tran,
Keith Delk,
Melanie Miga,
Richard Taulbee,
Pranav Pidathala,
Abraham Glasser,
Raja Kushlanagar,
Christian Vogler
Abstract:
In this study, we assess the usability of interactive personal assistants (IPAs), such as Amazon Alexa, in a simulated kitchen smart home environment, with deaf and hard of hearing users. Participants engage in activities in a way that causes their hands to get dirty. With these dirty hands, they are tasked with two different input methods for IPAs: American Sign Language (ASL) in a Wizard-of-Oz d…
▽ More
In this study, we assess the usability of interactive personal assistants (IPAs), such as Amazon Alexa, in a simulated kitchen smart home environment, with deaf and hard of hearing users. Participants engage in activities in a way that causes their hands to get dirty. With these dirty hands, they are tasked with two different input methods for IPAs: American Sign Language (ASL) in a Wizard-of-Oz design, and smart home apps with a touchscreen. Usability ratings show that participants significantly preferred ASL over touch-based apps with dirty hands, although not to a larger extent than in comparable previous work with clean hands. Participants also expressed significant enthusiasm for ASL-based IPA interaction in Netpromoter scores and in questions about their overall preferences. Preliminary observations further suggest that having dirty hands may affect the way people sign, which may pose challenges for building IPAs that natively support sign language input.
△ Less
Submitted 22 April, 2024;
originally announced April 2024.
-
Assessment of Sign Language-Based versus Touch-Based Input for Deaf Users Interacting with Intelligent Personal Assistants
Authors:
Nina Tran,
Paige DeVries,
Matthew Seita,
Raja Kushalnagar,
Abraham Glasser,
Christian Vogler
Abstract:
With the recent advancements in intelligent personal assistants (IPAs), their popularity is rapidly increasing when it comes to utilizing Automatic Speech Recognition within households. In this study, we used a Wizard-of-Oz methodology to evaluate and compare the usability of American Sign Language (ASL), Tap to Alexa, and smart home apps among 23 deaf participants within a limited-domain smart ho…
▽ More
With the recent advancements in intelligent personal assistants (IPAs), their popularity is rapidly increasing when it comes to utilizing Automatic Speech Recognition within households. In this study, we used a Wizard-of-Oz methodology to evaluate and compare the usability of American Sign Language (ASL), Tap to Alexa, and smart home apps among 23 deaf participants within a limited-domain smart home environment. Results indicate a slight usability preference for ASL. Linguistic analysis of the participants' signing reveals a diverse range of expressions and vocabulary as they interacted with IPAs in the context of a restricted-domain application. On average, deaf participants exhibited a vocabulary of 47 +/- 17 signs with an additional 10 +/- 7 fingerspelled words, for a total of 246 different signs and 93 different fingerspelled words across all participants. We discuss the implications for the design of limited-vocabulary applications as a step**-stone toward general-purpose ASL recognition in the future.
△ Less
Submitted 22 April, 2024;
originally announced April 2024.
-
How Users Experience Closed Captions on Live Television: Quality Metrics Remain a Challenge
Authors:
Mariana Arroyo Chavez,
Molly Feanny,
Matthew Seita,
Bernard Thompson,
Keith Delk,
Skyler Officer,
Abraham Glasser,
Raja Kushalnagar,
Christian Vogler
Abstract:
This paper presents a mixed methods study on how deaf, hard of hearing and hearing viewers perceive live TV caption quality with captioned video stimuli designed to mirror TV captioning experiences. To assess caption quality, we used four commonly-used quality metrics focusing on accuracy: word error rate, weighted word error rate, automated caption evaluation (ACE), and its successor ACE2. We cal…
▽ More
This paper presents a mixed methods study on how deaf, hard of hearing and hearing viewers perceive live TV caption quality with captioned video stimuli designed to mirror TV captioning experiences. To assess caption quality, we used four commonly-used quality metrics focusing on accuracy: word error rate, weighted word error rate, automated caption evaluation (ACE), and its successor ACE2. We calculated the correlation between the four quality metrics and viewer ratings for subjective quality and found that the correlation was weak, revealing that other factors besides accuracy affect user ratings. Additionally, even high-quality captions are perceived to have problems, despite controlling for confounding factors. Qualitative analysis of viewer comments revealed three major factors affecting their experience: Errors within captions, difficulty in following captions, and caption appearance. The findings raise questions as to how objective caption quality metrics can be reconciled with the user experience across a diverse spectrum of viewers.
△ Less
Submitted 15 April, 2024;
originally announced April 2024.
-
RTTD-ID: Tracked Captions with Multiple Speakers for Deaf Students
Authors:
Raja Kushalnagar,
Gary Behm,
Kevin Wolfe,
Peter Yeung,
Becca Dingman,
Shareef Ali,
Abraham Glasser,
Claire Ryan
Abstract:
Students who are deaf and hard of hearing cannot hear in class and do not have full access to spoken information. They can use accommodations such as captions that display speech as text. However, compared with their hearing peers, the caption accommodations do not provide equal access, because they are focused on reading captions on their tablet and cannot see who is talking. This viewing isolati…
▽ More
Students who are deaf and hard of hearing cannot hear in class and do not have full access to spoken information. They can use accommodations such as captions that display speech as text. However, compared with their hearing peers, the caption accommodations do not provide equal access, because they are focused on reading captions on their tablet and cannot see who is talking. This viewing isolation contributes to student frustration and risk of doing poorly or withdrawing from introductory engineering courses with lab components. It also contributes to their lack of inclusion and sense of belonging. We report on the evaluation of a Real-Time Text Display with Speaker-Identification, which displays the location of a speaker in a group (RTTD-ID). RTTD-ID aims to reduce frustration in identifying and following an active speaker when there are multiple speakers, e.g., in a lab. It has three different display schemes to identify the location of the active speaker, which helps deaf students in viewing both the speaker's words and the speaker's expression and actions. We evaluated three RTTD speaker identification methods: 1) traditional: captions stay in one place and viewers search for the speaker, 2) pointer: captions stay in one place, and a pointer to the speaker is displayed, and 3) pop-up: captions "pop-up" next to the speaker. We gathered both quantitative and qualitative information through evaluations with deaf and hard of hearing users. The users preferred the pointer identification method over the traditional and pop-up methods.
△ Less
Submitted 17 September, 2019;
originally announced September 2019.
-
Automatic Speech Recognition Services: Deaf and Hard-of-Hearing Usability
Authors:
Abraham Glasser
Abstract:
Nowadays, speech is becoming a more common, if not standard, interface to technology. This can be seen in the trend of technology changes over the years. Increasingly, voice is used to control programs, appliances and personal devices within homes, cars, workplaces, and public spaces through smartphones and home assistant devices using Amazon's Alexa, Google's Assistant and Apple's Siri, and other…
▽ More
Nowadays, speech is becoming a more common, if not standard, interface to technology. This can be seen in the trend of technology changes over the years. Increasingly, voice is used to control programs, appliances and personal devices within homes, cars, workplaces, and public spaces through smartphones and home assistant devices using Amazon's Alexa, Google's Assistant and Apple's Siri, and other proliferating technologies. However, most speech interfaces are not accessible for Deaf and Hard-of-Hearing (DHH) people. In this paper, performances of current Automatic Speech Recognition (ASR) with voices of DHH speakers are evaluated. ASR has improved over the years, and is able to reach Word Error Rates (WER) as low as 5-6% [1][2][3], with the help of cloud-computing and machine learning algorithms that take in custom vocabulary models. In this paper, a custom vocabulary model is used, and the significance of the improvement is evaluated when using DHH speech.
△ Less
Submitted 3 September, 2019;
originally announced September 2019.
-
Closed ASL Interpreting for Online Videos
Authors:
Raja Kushalnagar,
Matthew Seita,
Abraham Glasser
Abstract:
Deaf individuals face great challenges in today's society. It can be very difficult to be able to understand different forms of media without a sense of hearing. Many videos and movies found online today are not captioned, and even fewer have a supporting video with an interpreter. Also, even with a supporting interpreter video provided, information is still lost due to the inability to look at bo…
▽ More
Deaf individuals face great challenges in today's society. It can be very difficult to be able to understand different forms of media without a sense of hearing. Many videos and movies found online today are not captioned, and even fewer have a supporting video with an interpreter. Also, even with a supporting interpreter video provided, information is still lost due to the inability to look at both the video and the interpreter simultaneously. To alleviate this issue, we came up with a tool called closed interpreting. Similar to closed captioning, it will be displayed with an online video and can be toggled on and off. However, the closed interpreter is also user-adjustable. Settings, such as interpreter size, transparency, and location, can be adjusted. Our goal with this study is to find out what deaf and hard of hearing viewers like about videos that come with interpreters, and whether the adjustability is beneficial.
△ Less
Submitted 5 September, 2019;
originally announced September 2019.
-
Deaf, Hard of Hearing, and Hearing Perspectives on using Automatic Speech Recognition in Conversation
Authors:
Abraham Glasser,
Kesavan Kushalnagar,
Raja Kushalnagar
Abstract:
Many personal devices have transitioned from visual-controlled interfaces to speech-controlled interfaces to reduce costs and interactive friction, supported by the rapid growth in capabilities of speech-controlled interfaces, e.g., Amazon Echo or Apple's Siri. A consequence is that people who are deaf or hard of hearing (DHH) may be unable to use these speech-controlled devices. We show that deaf…
▽ More
Many personal devices have transitioned from visual-controlled interfaces to speech-controlled interfaces to reduce costs and interactive friction, supported by the rapid growth in capabilities of speech-controlled interfaces, e.g., Amazon Echo or Apple's Siri. A consequence is that people who are deaf or hard of hearing (DHH) may be unable to use these speech-controlled devices. We show that deaf speech has a high error rate compared to hearing speech, in commercial speech-controlled interfaces. Deaf speech had approximately a 78% word error rate (WER) compared to a hearing speech 18% WER. Our findings show that current speech-controlled interfaces are not usable by DHH people. Based on our findings, significant advances in speech recognition software or alternative approaches will be needed for deaf use of speech-controlled interfaces. We show that current speech-controlled interfaces are not usable by DHH people.
△ Less
Submitted 3 September, 2019;
originally announced September 2019.
-
Feasibility of Using Automatic Speech Recognition with Voices of Deaf and Hard-of-Hearing Individuals
Authors:
Abraham Glasser,
Kesavan Kushalnagar,
Raja Kushalnagar
Abstract:
Many personal devices have transitioned from visual-controlled interfaces to speech-controlled interfaces to reduce device costs and interactive friction. This transition has been hastened by the increasing capabilities of speech-controlled interfaces, e.g., Amazon Echo or Apple's Siri. A consequence is that people who are deaf or hard of hearing (DHH) may be unable to use these speech-controlled…
▽ More
Many personal devices have transitioned from visual-controlled interfaces to speech-controlled interfaces to reduce device costs and interactive friction. This transition has been hastened by the increasing capabilities of speech-controlled interfaces, e.g., Amazon Echo or Apple's Siri. A consequence is that people who are deaf or hard of hearing (DHH) may be unable to use these speech-controlled devices. We show that deaf speech has a high error rate compared to hearing speech, in commercial speech-controlled interfaces. Deaf speech had approximately a 78% word error rate (WER) compared to a hearing speech 18% WER. Our findings show that current speech-controlled interfaces are not usable by deaf and hard of hearing people. Therefore, it might be wise to pursue other methods for deaf persons to deliver natural commands to computers.
△ Less
Submitted 3 September, 2019;
originally announced September 2019.
-
Artificial Intelligence Fairness in the Context of Accessibility Research on Intelligent Systems for People who are Deaf or Hard of Hearing
Authors:
Sushant Kafle,
Abraham Glasser,
Sedeeq Al-khazraji,
Larwan Berke,
Matthew Seita,
Matt Huenerfauth
Abstract:
We discuss issues of Artificial Intelligence (AI) fairness for people with disabilities, with examples drawn from our research on human-computer interaction (HCI) for AI-based systems for people who are Deaf or Hard of Hearing (DHH). In particular, we discuss the need for inclusion of data from people with disabilities in training sets, the lack of interpretability of AI systems, ethical responsib…
▽ More
We discuss issues of Artificial Intelligence (AI) fairness for people with disabilities, with examples drawn from our research on human-computer interaction (HCI) for AI-based systems for people who are Deaf or Hard of Hearing (DHH). In particular, we discuss the need for inclusion of data from people with disabilities in training sets, the lack of interpretability of AI systems, ethical responsibilities of access technology researchers and companies, the need for appropriate evaluation metrics for AI-based access technologies (to determine if they are ready to be deployed and if they can be trusted by users), and the ways in which AI systems influence human behavior and influence the set of abilities needed by users to successfully interact with computing systems.
△ Less
Submitted 2 September, 2019; v1 submitted 27 August, 2019;
originally announced August 2019.
