Showing 1–5 of 5 results for author: Kullu, P

A Machine Learning Approach for Predicting Upper Limb Motion Intentions with Multimodal Data in Virtual Reality

Authors: Pavan Uttej Ravva, Pinar Kullu, Mohammad Fahim Abrar, Roghayeh Leila Barmaki

Abstract: Over the last decade, there has been significant progress in the field of interactive virtual rehabilitation. Physical therapy (PT) stands as a highly effective approach for enhancing physical impairments. However, patient motivation and progress tracking in rehabilitation outcomes remain a challenge. This work addresses the gap through a machine learning-based approach to objectively measure outc… ▽ More Over the last decade, there has been significant progress in the field of interactive virtual rehabilitation. Physical therapy (PT) stands as a highly effective approach for enhancing physical impairments. However, patient motivation and progress tracking in rehabilitation outcomes remain a challenge. This work addresses the gap through a machine learning-based approach to objectively measure outcomes of the upper limb virtual therapy system in a user study with non-clinical participants. In this study, we use virtual reality to perform several tracing tasks while collecting motion and movement data using a KinArm robot and a custom-made wearable sleeve sensor. We introduce a two-step machine learning architecture to predict the motion intention of participants. The first step predicts reaching task segments to which the participant-marked points belonged using gaze, while the second step employs a Long Short-Term Memory (LSTM) model to predict directional movements based on resistance change values from the wearable sensor and the KinArm. We specifically propose to transpose our raw resistance data to the time-domain which significantly improves the accuracy of the models by 34.6%. To evaluate the effectiveness of our model, we compared different classification techniques with various data configurations. The results show that our proposed computational method is exceptional at predicting participant's actions with accuracy values of 96.72% for diamond reaching task, and 97.44% for circle reaching task, which demonstrates the great promise of using multimodal data, including eye-tracking and resistance change, to objectively measure the performance and intention in virtual rehabilitation settings. △ Less

Submitted 15 May, 2024; originally announced May 2024.

fNIRS Analysis of Interaction Techniques in Touchscreen-Based Educational Gaming

Authors: Shayla Sharmin, Elham Bakhshipour, Behdokht Kiafar, Md Fahim Abrar, Pinar Kullu, Nancy Getchell, Roghayeh Leila Barmaki

Abstract: Touchscreens are becoming increasingly widespread in educational games, enhancing the quality of learner experience. Traditional metrics are often used to evaluate various input modalities, including hand and stylus. However, there exists a gap in understanding the cognitive impacts of these modalities during educational gameplay, which can be addressed through brain signal analysis to gain deeper… ▽ More Touchscreens are becoming increasingly widespread in educational games, enhancing the quality of learner experience. Traditional metrics are often used to evaluate various input modalities, including hand and stylus. However, there exists a gap in understanding the cognitive impacts of these modalities during educational gameplay, which can be addressed through brain signal analysis to gain deeper insights into the underlying cognitive function and necessary brain resources for each condition. This facilitates a more precise comparison between conditions. In this study, we compared the brain signal and user experience of using hands and stylus on touchscreens while playing an educational game by analyzing hemodynamic response and self-reported measures. Participants engaged in a Unity-based educational quiz game using both hand and stylus on a touchscreen in a counterbalanced within-subject design. Oxygenated and deoxygenated hemoglobin data were collected using fNIRS, alongside quiz performance scores and standardized and customized user experience questionnaire ratings. Our findings show almost the same performance level with both input modalities, however, the hand requires less oxygen flow which suggests a lower cognitive effort than using a stylus while playing the educational game. Although the result shows that the stylus condition required more neural involvement than the hand condition, there is no significant difference between the use of both input modalities. However, there is a statistically significant difference in self-reported measures that support the findings mentioned above, favoring the hand that enhances understanding of modality effects in interactive educational environments. △ Less

Submitted 14 May, 2024; originally announced May 2024.

Cognitive Engagement for STEM+C Education: Investigating Serious Game Impact on Graph Structure Learning with fNIRS

Authors: Shayla Sharmin, Reza Koiler, Rifat Sadik, Arpan Bhattacharjee, Priyanka Raju Patre, Pinar Kullu, Charles Hohensee, Nancy Getchell, Roghayeh Leila Barmaki

Abstract: For serious games on education, understanding the effectiveness of different learning methods in influencing cognitive processes remains a significant challenge. This study investigates the impact of serious games on graph structure learning. For this, we compared our in-house game-based learning (GBL) and video-based learning (VBL) methodologies by evaluating their effectiveness on cognitive proc… ▽ More For serious games on education, understanding the effectiveness of different learning methods in influencing cognitive processes remains a significant challenge. This study investigates the impact of serious games on graph structure learning. For this, we compared our in-house game-based learning (GBL) and video-based learning (VBL) methodologies by evaluating their effectiveness on cognitive processes by oxygenated hemoglobin levels using functional near-infrared spectroscopy (fNIRS). We conducted a 2 x 1 between subjects preliminary study with twelve participants, involving two conditions: game and video. Both groups received equivalent content related to the basic structure of a graph, with comparable session lengths. The game group interacted with a quiz-based game, while the video group watched a pre-recorded video. The fNIRS was employed to capture cerebral signals from the prefrontal cortex, and participants completed pre- and post- questionnaires capturing user experience and knowledge gain. In our study, we noted that the mean levels of oxygenated hemoglobin were higher in the GBL group, suggesting the potential enhanced cognitive involvement. Our results show that the lateral prefrontal cortex (LPFC) has greater hemodynamic activity during the learning period. Moreover, knowledge gain analysis showed an increase in mean score in the GBL group compared to the VBL group. Although we did not observe statistically significant changes due to participant variability and sample size, this preliminary work contributes to understanding how GBL and VBL impact cognitive processes, providing insights for enhanced instructional design and educational game development. Additionally, it emphasizes the necessity for further investigation into the impact of GBL on cognitive engagement and learning outcomes. △ Less

Submitted 7 March, 2024; v1 submitted 25 July, 2023; originally announced July 2023.

Comments: 19 pages, 9 figures

Towards Anatomy Education with Generative AI-based Virtual Assistants in Immersive Virtual Reality Environments

Authors: Vuthea Chheang, Shayla Sharmin, Rommy Marquez-Hernandez, Megha Patel, Danush Rajasekaran, Gavin Caulfield, Behdokht Kiafar, Jicheng Li, Pinar Kullu, Roghayeh Leila Barmaki

Abstract: Virtual reality (VR) and interactive 3D visualization systems have enhanced educational experiences and environments, particularly in complicated subjects such as anatomy education. VR-based systems surpass the potential limitations of traditional training approaches in facilitating interactive engagement among students. However, research on embodied virtual assistants that leverage generative art… ▽ More Virtual reality (VR) and interactive 3D visualization systems have enhanced educational experiences and environments, particularly in complicated subjects such as anatomy education. VR-based systems surpass the potential limitations of traditional training approaches in facilitating interactive engagement among students. However, research on embodied virtual assistants that leverage generative artificial intelligence (AI) and verbal communication in the anatomy education context is underrepresented. In this work, we introduce a VR environment with a generative AI-embodied virtual assistant to support participants in responding to varying cognitive complexity anatomy questions and enable verbal communication. We assessed the technical efficacy and usability of the proposed environment in a pilot user study with 16 participants. We conducted a within-subject design for virtual assistant configuration (avatar- and screen-based), with two levels of cognitive complexity (knowledge- and analysis-based). The results reveal a significant difference in the scores obtained from knowledge- and analysis-based questions in relation to avatar configuration. Moreover, results provide insights into usability, cognitive task load, and the sense of presence in the proposed virtual assistant configurations. Our environment and results of the pilot study offer potential benefits and future research directions beyond medical education, using generative AI and embodied virtual agents as customized virtual conversational assistants. △ Less

Submitted 24 January, 2024; v1 submitted 29 June, 2023; originally announced June 2023.

Comments: Accepted to present at IEEE AIxVR 2024. 10 Pages, 6 Figures

MMASD: A Multimodal Dataset for Autism Intervention Analysis

Authors: Jicheng Li, Vuthea Chheang, Pinar Kullu, Eli Brignac, Zhang Guo, Kenneth E. Barner, Anjana Bhat, Roghayeh Leila Barmaki

Abstract: Autism spectrum disorder (ASD) is a developmental disorder characterized by significant social communication impairments and difficulties perceiving and presenting communication cues. Machine learning techniques have been broadly adopted to facilitate autism studies and assessments. However, computational models are primarily concentrated on specific analysis and validated on private datasets in t… ▽ More Autism spectrum disorder (ASD) is a developmental disorder characterized by significant social communication impairments and difficulties perceiving and presenting communication cues. Machine learning techniques have been broadly adopted to facilitate autism studies and assessments. However, computational models are primarily concentrated on specific analysis and validated on private datasets in the autism community, which limits comparisons across models due to privacy-preserving data sharing complications. This work presents a novel privacy-preserving open-source dataset, MMASD as a MultiModal ASD benchmark dataset, collected from play therapy interventions of children with Autism. MMASD includes data from 32 children with ASD, and 1,315 data samples segmented from over 100 hours of intervention recordings. To promote public access, each data sample consists of four privacy-preserving modalities of data; some of which are derived from original videos: (1) optical flow, (2) 2D skeleton, (3) 3D skeleton, and (4) clinician ASD evaluation scores of children, e.g., ADOS scores. MMASD aims to assist researchers and therapists in understanding children's cognitive status, monitoring their progress during therapy, and customizing the treatment plan accordingly. It also has inspiration for downstream tasks such as action quality assessment and interpersonal synchrony estimation. MMASD dataset can be easily accessed at https://github.com/Li-Jicheng/MMASD-A-Multimodal-Dataset-for-Autism-Intervention-Analysis. △ Less

Submitted 1 October, 2023; v1 submitted 14 June, 2023; originally announced June 2023.

Comments: 8 pages, 2 figures