-
Towards Enhanced Human Activity Recognition through Natural Language Generation and Pose Estimation
Authors:
Nikhil Kashyap,
Manas Satish Bedmutha,
Prerit Chaudhary,
Brian Wood,
Wanda Pratt,
Janice Sabin,
Andrea Hartzler,
Nadir Weibel
Abstract:
Vision-based human activity recognition (HAR) has made substantial progress in recognizing predefined gestures but lacks adaptability for emerging activities. This paper introduces a paradigm shift by harnessing generative modeling and large language models (LLMs) to enhance vision-based HAR. We propose utilizing LLMs to generate descriptive textual representations of activities using pose keypoin…
▽ More
Vision-based human activity recognition (HAR) has made substantial progress in recognizing predefined gestures but lacks adaptability for emerging activities. This paper introduces a paradigm shift by harnessing generative modeling and large language models (LLMs) to enhance vision-based HAR. We propose utilizing LLMs to generate descriptive textual representations of activities using pose keypoints as an intermediate representation. Incorporating pose keypoints adds contextual depth to the recognition process, allowing for sequences of vectors resembling text chunks, compatible with LLMs. This innovative fusion of computer vision and natural language processing holds significant potential for revolutionizing activity recognition. A proof of concept study on a Kinetics700 dataset subset validates the approach's efficacy, highlighting improved accuracy and interpretability. Future implications encompass enhanced accuracy, novel research avenues, model generalization, and ethical considerations for transparency. This framework has real-world applications, including personalized gym workout feedback and nuanced sports training insights. By connecting visual cues to interpretable textual descriptions, the proposed framework advances HAR accuracy and applicability, sha** the landscape of pervasive computing and activity recognition research. As this approach evolves, it promises a more insightful understanding of human activities across diverse contexts, marking a significant step towards a better world.
△ Less
Submitted 11 December, 2023;
originally announced December 2023.
-
Making Hidden Bias Visible: Designing a Feedback Ecosystem for Primary Care Providers
Authors:
Naba Rizvi,
Harshini Ramaswamy,
Reggie Casanova-Perez,
Andrea Hartzler,
Nadir Weibel
Abstract:
Implicit bias may perpetuate healthcare disparities for marginalized patient populations. Such bias is expressed in communication between patients and their providers. We design an ecosystem with guidance from providers to make this bias explicit in patient-provider communication. Our end users are providers seeking to improve their quality of care for patients who are Black, Indigenous, People of…
▽ More
Implicit bias may perpetuate healthcare disparities for marginalized patient populations. Such bias is expressed in communication between patients and their providers. We design an ecosystem with guidance from providers to make this bias explicit in patient-provider communication. Our end users are providers seeking to improve their quality of care for patients who are Black, Indigenous, People of Color (BIPOC) and/or Lesbian, Gay, Bisexual, Transgender, and Queer (LGBTQ). We present wireframes displaying communication metrics that negatively impact patient-centered care divided into the following categories: digital nudge, dashboard, and guided reflection. Our wireframes provide quantitative, real-time, and conversational feedback promoting provider reflection on their interactions with patients. This is the first design iteration toward the development of a tool to raise providers' awareness of their own implicit biases.
△ Less
Submitted 16 April, 2022;
originally announced April 2022.
-
Designing an AI Health Coach and Studying its Utility in Promoting Regular Aerobic Exercise
Authors:
Shiwali Mohan,
Anusha Venkatakrishnan,
Andrea Hartzler
Abstract:
Our research aims to develop interactive, social agents that can coach people to learn new tasks, skills, and habits. In this paper, we focus on coaching sedentary, overweight individuals (i.e., trainees) to exercise regularly. We employ adaptive goal setting in which the intelligent health coach generates, tracks, and revises personalized exercise goals for a trainee. The goals become incremental…
▽ More
Our research aims to develop interactive, social agents that can coach people to learn new tasks, skills, and habits. In this paper, we focus on coaching sedentary, overweight individuals (i.e., trainees) to exercise regularly. We employ adaptive goal setting in which the intelligent health coach generates, tracks, and revises personalized exercise goals for a trainee. The goals become incrementally more difficult as the trainee progresses through the training program. Our approach is model-based - the coach maintains a parameterized model of the trainee's aerobic capability that drives its expectation of the trainee's performance. The model is continually revised based on trainee-coach interactions. The coach is embodied in a smartphone application, NutriWalking, which serves as a medium for coach-trainee interaction. We adopt a task-centric evaluation approach for studying the utility of the proposed algorithm in promoting regular aerobic exercise. We show that our approach can adapt the trainee program not only to several trainees with different capabilities, but also to how a trainee's capability improves as they begin to exercise more. Experts rate the goals selected by the coach better than other plausible goals, demonstrating that our approach is consistent with clinical recommendations. Further, in a 6-week observational study with sedentary participants, we show that the proposed approach helps increase exercise volume performed each week.
△ Less
Submitted 11 September, 2020; v1 submitted 10 October, 2019;
originally announced October 2019.
