-
A Machine Learning Approach for Predicting Upper Limb Motion Intentions with Multimodal Data in Virtual Reality
Authors:
Pavan Uttej Ravva,
Pinar Kullu,
Mohammad Fahim Abrar,
Roghayeh Leila Barmaki
Abstract:
Over the last decade, there has been significant progress in the field of interactive virtual rehabilitation. Physical therapy (PT) stands as a highly effective approach for enhancing physical impairments. However, patient motivation and progress tracking in rehabilitation outcomes remain a challenge. This work addresses the gap through a machine learning-based approach to objectively measure outc…
▽ More
Over the last decade, there has been significant progress in the field of interactive virtual rehabilitation. Physical therapy (PT) stands as a highly effective approach for enhancing physical impairments. However, patient motivation and progress tracking in rehabilitation outcomes remain a challenge. This work addresses the gap through a machine learning-based approach to objectively measure outcomes of the upper limb virtual therapy system in a user study with non-clinical participants. In this study, we use virtual reality to perform several tracing tasks while collecting motion and movement data using a KinArm robot and a custom-made wearable sleeve sensor. We introduce a two-step machine learning architecture to predict the motion intention of participants. The first step predicts reaching task segments to which the participant-marked points belonged using gaze, while the second step employs a Long Short-Term Memory (LSTM) model to predict directional movements based on resistance change values from the wearable sensor and the KinArm. We specifically propose to transpose our raw resistance data to the time-domain which significantly improves the accuracy of the models by 34.6%. To evaluate the effectiveness of our model, we compared different classification techniques with various data configurations. The results show that our proposed computational method is exceptional at predicting participant's actions with accuracy values of 96.72% for diamond reaching task, and 97.44% for circle reaching task, which demonstrates the great promise of using multimodal data, including eye-tracking and resistance change, to objectively measure the performance and intention in virtual rehabilitation settings.
△ Less
Submitted 15 May, 2024;
originally announced May 2024.
-
Analyzing Nursing Assistant Attitudes Towards Empathic Geriatric Caregiving Using Quantitative Ethnography
Authors:
Behdokht Kiafar,
Salam Daher,
Shayla Sharmin,
Asif Ahmmed,
Ladda Thiamwong,
Roghayeh Leila Barmaki
Abstract:
An emergent challenge in geriatric care is improving the quality of care, which requires insight from stakeholders. Qualitative methods offer detailed insights, but they can be biased and have limited generalizability, while quantitative methods may miss nuances. Network-based approaches, such as quantitative ethnography (QE), can bridge this methodological gap. By leveraging the strengths of both…
▽ More
An emergent challenge in geriatric care is improving the quality of care, which requires insight from stakeholders. Qualitative methods offer detailed insights, but they can be biased and have limited generalizability, while quantitative methods may miss nuances. Network-based approaches, such as quantitative ethnography (QE), can bridge this methodological gap. By leveraging the strengths of both methods, QE provides profound insights into need finding interviews. In this paper, to better understand geriatric care attitudes, we interviewed ten nursing assistants, used QE to analyze the data, and compared their daily activities in real life with training experiences. A two-sample t-test with a large effect size (Cohen's d=1.63) indicated a significant difference between real-life and training activities. The findings suggested incorporating more empathetic training scenarios into the future design of our geriatric care simulation. The results have implications for human-computer interaction and human factors. This is illustrated by presenting an example of using QE to analyze expert interviews with nursing assistants as caregivers to inform subsequent design processes.
△ Less
Submitted 14 May, 2024;
originally announced May 2024.
-
fNIRS Analysis of Interaction Techniques in Touchscreen-Based Educational Gaming
Authors:
Shayla Sharmin,
Elham Bakhshipour,
Behdokht Kiafar,
Md Fahim Abrar,
Pinar Kullu,
Nancy Getchell,
Roghayeh Leila Barmaki
Abstract:
Touchscreens are becoming increasingly widespread in educational games, enhancing the quality of learner experience. Traditional metrics are often used to evaluate various input modalities, including hand and stylus. However, there exists a gap in understanding the cognitive impacts of these modalities during educational gameplay, which can be addressed through brain signal analysis to gain deeper…
▽ More
Touchscreens are becoming increasingly widespread in educational games, enhancing the quality of learner experience. Traditional metrics are often used to evaluate various input modalities, including hand and stylus. However, there exists a gap in understanding the cognitive impacts of these modalities during educational gameplay, which can be addressed through brain signal analysis to gain deeper insights into the underlying cognitive function and necessary brain resources for each condition. This facilitates a more precise comparison between conditions. In this study, we compared the brain signal and user experience of using hands and stylus on touchscreens while playing an educational game by analyzing hemodynamic response and self-reported measures. Participants engaged in a Unity-based educational quiz game using both hand and stylus on a touchscreen in a counterbalanced within-subject design. Oxygenated and deoxygenated hemoglobin data were collected using fNIRS, alongside quiz performance scores and standardized and customized user experience questionnaire ratings. Our findings show almost the same performance level with both input modalities, however, the hand requires less oxygen flow which suggests a lower cognitive effort than using a stylus while playing the educational game. Although the result shows that the stylus condition required more neural involvement than the hand condition, there is no significant difference between the use of both input modalities. However, there is a statistically significant difference in self-reported measures that support the findings mentioned above, favoring the hand that enhances understanding of modality effects in interactive educational environments.
△ Less
Submitted 14 May, 2024;
originally announced May 2024.
-
A Large-Scale Feasibility Study of Screen-based 3D Visualization and Augmented Reality Tools for Human Anatomy Education: Exploring Gender Perspectives in Learning Experience
Authors:
Roghayeh Leila Barmaki,
Kangsoo Kim,
Zhang Guo,
Qile Wang,
Kevin Yu,
Rebecca Pearlman,
Nassir Navab
Abstract:
Anatomy education is an indispensable part of medical training, but traditional methods face challenges like limited resources for dissection in large classes and difficulties understanding 2D anatomy in textbooks. Advanced technologies, such as 3D visualization and augmented reality (AR), are transforming anatomy learning. This paper presents two in-house solutions that use handheld tablets or sc…
▽ More
Anatomy education is an indispensable part of medical training, but traditional methods face challenges like limited resources for dissection in large classes and difficulties understanding 2D anatomy in textbooks. Advanced technologies, such as 3D visualization and augmented reality (AR), are transforming anatomy learning. This paper presents two in-house solutions that use handheld tablets or screen-based AR to visualize 3D anatomy models with informative labels and in-situ visualizations of the muscle anatomy. To assess these tools, a user study of muscle anatomy education involved 236 premedical students in dyadic teams, with results showing that the tablet-based 3D visualization and screen-based AR tools led to significantly higher learning experience scores than traditional textbook. While knowledge retention didn't differ significantly, ethnographic and gender analysis showed that male students generally reported more positive learning experiences than female students. This study discusses the implications for anatomy and medical education, highlighting the potential of these innovative learning tools considering gender and team dynamics in body painting anatomy learning interventions.
△ Less
Submitted 31 January, 2024; v1 submitted 25 July, 2023;
originally announced July 2023.
-
Cognitive Engagement for STEM+C Education: Investigating Serious Game Impact on Graph Structure Learning with fNIRS
Authors:
Shayla Sharmin,
Reza Koiler,
Rifat Sadik,
Arpan Bhattacharjee,
Priyanka Raju Patre,
Pinar Kullu,
Charles Hohensee,
Nancy Getchell,
Roghayeh Leila Barmaki
Abstract:
For serious games on education, understanding the effectiveness of different learning methods in influencing cognitive processes remains a significant challenge. This study investigates the impact of serious games on graph structure learning. For this, we compared our in-house game-based learning (GBL) and video-based learning (VBL) methodologies by evaluating their effectiveness on cognitive proc…
▽ More
For serious games on education, understanding the effectiveness of different learning methods in influencing cognitive processes remains a significant challenge. This study investigates the impact of serious games on graph structure learning. For this, we compared our in-house game-based learning (GBL) and video-based learning (VBL) methodologies by evaluating their effectiveness on cognitive processes by oxygenated hemoglobin levels using functional near-infrared spectroscopy (fNIRS). We conducted a 2 x 1 between subjects preliminary study with twelve participants, involving two conditions: game and video. Both groups received equivalent content related to the basic structure of a graph, with comparable session lengths. The game group interacted with a quiz-based game, while the video group watched a pre-recorded video. The fNIRS was employed to capture cerebral signals from the prefrontal cortex, and participants completed pre- and post- questionnaires capturing user experience and knowledge gain. In our study, we noted that the mean levels of oxygenated hemoglobin were higher in the GBL group, suggesting the potential enhanced cognitive involvement. Our results show that the lateral prefrontal cortex (LPFC) has greater hemodynamic activity during the learning period. Moreover, knowledge gain analysis showed an increase in mean score in the GBL group compared to the VBL group. Although we did not observe statistically significant changes due to participant variability and sample size, this preliminary work contributes to understanding how GBL and VBL impact cognitive processes, providing insights for enhanced instructional design and educational game development. Additionally, it emphasizes the necessity for further investigation into the impact of GBL on cognitive engagement and learning outcomes.
△ Less
Submitted 7 March, 2024; v1 submitted 25 July, 2023;
originally announced July 2023.
-
Towards Anatomy Education with Generative AI-based Virtual Assistants in Immersive Virtual Reality Environments
Authors:
Vuthea Chheang,
Shayla Sharmin,
Rommy Marquez-Hernandez,
Megha Patel,
Danush Rajasekaran,
Gavin Caulfield,
Behdokht Kiafar,
Jicheng Li,
Pinar Kullu,
Roghayeh Leila Barmaki
Abstract:
Virtual reality (VR) and interactive 3D visualization systems have enhanced educational experiences and environments, particularly in complicated subjects such as anatomy education. VR-based systems surpass the potential limitations of traditional training approaches in facilitating interactive engagement among students. However, research on embodied virtual assistants that leverage generative art…
▽ More
Virtual reality (VR) and interactive 3D visualization systems have enhanced educational experiences and environments, particularly in complicated subjects such as anatomy education. VR-based systems surpass the potential limitations of traditional training approaches in facilitating interactive engagement among students. However, research on embodied virtual assistants that leverage generative artificial intelligence (AI) and verbal communication in the anatomy education context is underrepresented. In this work, we introduce a VR environment with a generative AI-embodied virtual assistant to support participants in responding to varying cognitive complexity anatomy questions and enable verbal communication. We assessed the technical efficacy and usability of the proposed environment in a pilot user study with 16 participants. We conducted a within-subject design for virtual assistant configuration (avatar- and screen-based), with two levels of cognitive complexity (knowledge- and analysis-based). The results reveal a significant difference in the scores obtained from knowledge- and analysis-based questions in relation to avatar configuration. Moreover, results provide insights into usability, cognitive task load, and the sense of presence in the proposed virtual assistant configurations. Our environment and results of the pilot study offer potential benefits and future research directions beyond medical education, using generative AI and embodied virtual agents as customized virtual conversational assistants.
△ Less
Submitted 24 January, 2024; v1 submitted 29 June, 2023;
originally announced June 2023.
-
MMASD: A Multimodal Dataset for Autism Intervention Analysis
Authors:
Jicheng Li,
Vuthea Chheang,
Pinar Kullu,
Eli Brignac,
Zhang Guo,
Kenneth E. Barner,
Anjana Bhat,
Roghayeh Leila Barmaki
Abstract:
Autism spectrum disorder (ASD) is a developmental disorder characterized by significant social communication impairments and difficulties perceiving and presenting communication cues. Machine learning techniques have been broadly adopted to facilitate autism studies and assessments. However, computational models are primarily concentrated on specific analysis and validated on private datasets in t…
▽ More
Autism spectrum disorder (ASD) is a developmental disorder characterized by significant social communication impairments and difficulties perceiving and presenting communication cues. Machine learning techniques have been broadly adopted to facilitate autism studies and assessments. However, computational models are primarily concentrated on specific analysis and validated on private datasets in the autism community, which limits comparisons across models due to privacy-preserving data sharing complications. This work presents a novel privacy-preserving open-source dataset, MMASD as a MultiModal ASD benchmark dataset, collected from play therapy interventions of children with Autism. MMASD includes data from 32 children with ASD, and 1,315 data samples segmented from over 100 hours of intervention recordings. To promote public access, each data sample consists of four privacy-preserving modalities of data; some of which are derived from original videos: (1) optical flow, (2) 2D skeleton, (3) 3D skeleton, and (4) clinician ASD evaluation scores of children, e.g., ADOS scores. MMASD aims to assist researchers and therapists in understanding children's cognitive status, monitoring their progress during therapy, and customizing the treatment plan accordingly. It also has inspiration for downstream tasks such as action quality assessment and interpersonal synchrony estimation. MMASD dataset can be easily accessed at https://github.com/Li-Jicheng/MMASD-A-Multimodal-Dataset-for-Autism-Intervention-Analysis.
△ Less
Submitted 1 October, 2023; v1 submitted 14 June, 2023;
originally announced June 2023.
-
Immersive Virtual Reality and Robotics for Upper Extremity Rehabilitation
Authors:
Vuthea Chheang,
Rakshith Lokesh,
Amit Chaudhari,
Qile Wang,
Lauren Baron,
Behdokht Kiafar,
Sagar Doshi,
Erik Thostenson,
Joshua Cashaback,
Roghayeh Leila Barmaki
Abstract:
Stroke patients often experience upper limb impairments that restrict their mobility and daily activities. Physical therapy (PT) is the most effective method to improve impairments, but low patient adherence and participation in PT exercises pose significant challenges. To overcome these barriers, a combination of virtual reality (VR) and robotics in PT is promising. However, few systems effective…
▽ More
Stroke patients often experience upper limb impairments that restrict their mobility and daily activities. Physical therapy (PT) is the most effective method to improve impairments, but low patient adherence and participation in PT exercises pose significant challenges. To overcome these barriers, a combination of virtual reality (VR) and robotics in PT is promising. However, few systems effectively integrate VR with robotics, especially for upper limb rehabilitation. This work introduces a new virtual rehabilitation solution that combines VR with robotics and a wearable sensor to analyze elbow joint movements. The framework also enhances the capabilities of a traditional robotic device (KinArm) used for motor dysfunction assessment and rehabilitation. A pilot user study (n = 16) was conducted to evaluate the effectiveness and usability of the proposed VR framework. We used a two-way repeated measures experimental design where participants performed two tasks (Circle and Diamond) with two conditions (VR and VR KinArm). We observed no significant differences in the main effect of conditions for task completion time. However, there were significant differences in both the normalized number of mistakes and recorded elbow joint angles (captured as resistance change values from the wearable sleeve sensor) between the Circle and Diamond tasks. Additionally, we report the system usability, task load, and presence in the proposed VR framework. This system demonstrates the potential advantages of an immersive, multi-sensory approach and provides future avenues for research in develo** more cost-effective, tailored, and personalized upper limb solutions for home therapy applications.
△ Less
Submitted 29 June, 2023; v1 submitted 21 April, 2023;
originally announced April 2023.
-
Virtual Therapy Exergame for Upper Extremity Rehabilitation Using Smart Wearable Sensors
Authors:
Lauren Baron,
Vuthea Chheang,
Amit Chaudhari,
Arooj Liaqat,
Aishwarya Chandrasekaran,
Yufan Wang,
Joshua Cashaback,
Erik Thostenson,
Roghayeh Leila Barmaki
Abstract:
Virtual Reality (VR) has been utilized for several applications and has shown great potential for rehabilitation, especially for home therapy. However, these systems solely rely on information from VR hand controllers, which do not fully capture the individual movement of the joints. In this paper, we propose a creative VR therapy exergame for upper extremity rehabilitation using multi-dimensional…
▽ More
Virtual Reality (VR) has been utilized for several applications and has shown great potential for rehabilitation, especially for home therapy. However, these systems solely rely on information from VR hand controllers, which do not fully capture the individual movement of the joints. In this paper, we propose a creative VR therapy exergame for upper extremity rehabilitation using multi-dimensional reaching tasks while simultaneously capturing hand movement from the VR controllers and elbow joint movement from a flexible carbon nanotube sleeve. We conducted a preliminary study with non-clinical participants (n = 12, 7 F). In a 2x2 within-subjects study (orientation (vertical, horizontal) x configuration (flat, curved)), we evaluated the effectiveness and enjoyment of the exergame in different study conditions. The results show that there was a statistically significant difference in terms of task completion time between the two orientations. However, no significant differences were found in the number of mistakes in both orientation and configuration of the virtual exergame. This can lead to customizing therapy while maintaining the same level of intensity. That is, if a patient has restricted lower limb mobility and requires to be seated, they can use the orientations interchangeably. The results of resistance change generated from the carbon nanotube sleeve revealed that the flat configuration in the vertical orientation induced more elbow stretches than the other conditions. Finally, we reported the subjective measures based on questionnaires for usability and user experience in different study conditions. In conclusion, the proposed VR exergame has the potential as a multimodal sensory tool for personalized upper extremity home-based therapy and telerehabilitation.
△ Less
Submitted 16 February, 2023;
originally announced February 2023.
