-
Contextual AI Journaling: Integrating LLM and Time Series Behavioral Sensing Technology to Promote Self-Reflection and Well-being using the MindScape App
Authors:
Subigya Nepal,
Arvind Pillai,
William Campbell,
Talie Massachi,
Eunsol Soul Choi,
Orson Xu,
Joanna Kuc,
Jeremy Huckins,
Jason Holden,
Colin Depp,
Nicholas Jacobson,
Mary Czerwinski,
Eric Granholm,
Andrew T. Campbell
Abstract:
MindScape aims to study the benefits of integrating time series behavioral patterns (e.g., conversational engagement, sleep, location) with Large Language Models (LLMs) to create a new form of contextual AI journaling, promoting self-reflection and well-being. We argue that integrating behavioral sensing in LLMs will likely lead to a new frontier in AI. In this Late-Breaking Work paper, we discuss…
▽ More
MindScape aims to study the benefits of integrating time series behavioral patterns (e.g., conversational engagement, sleep, location) with Large Language Models (LLMs) to create a new form of contextual AI journaling, promoting self-reflection and well-being. We argue that integrating behavioral sensing in LLMs will likely lead to a new frontier in AI. In this Late-Breaking Work paper, we discuss the MindScape contextual journal App design that uses LLMs and behavioral sensing to generate contextual and personalized journaling prompts crafted to encourage self-reflection and emotional development. We also discuss the MindScape study of college students based on a preliminary user study and our upcoming study to assess the effectiveness of contextual AI journaling in promoting better well-being on college campuses. MindScape represents a new application class that embeds behavioral intelligence in AI.
△ Less
Submitted 30 March, 2024;
originally announced April 2024.
-
MoodCapture: Depression Detection Using In-the-Wild Smartphone Images
Authors:
Subigya Nepal,
Arvind Pillai,
Weichen Wang,
Tess Griffin,
Amanda C. Collins,
Michael Heinz,
Damien Lekkas,
Shayan Mirjafari,
Matthew Nemesure,
George Price,
Nicholas C. Jacobson,
Andrew T. Campbell
Abstract:
MoodCapture presents a novel approach that assesses depression based on images automatically captured from the front-facing camera of smartphones as people go about their daily lives. We collect over 125,000 photos in the wild from N=177 participants diagnosed with major depressive disorder for 90 days. Images are captured naturalistically while participants respond to the PHQ-8 depression survey…
▽ More
MoodCapture presents a novel approach that assesses depression based on images automatically captured from the front-facing camera of smartphones as people go about their daily lives. We collect over 125,000 photos in the wild from N=177 participants diagnosed with major depressive disorder for 90 days. Images are captured naturalistically while participants respond to the PHQ-8 depression survey question: \textit{``I have felt down, depressed, or hopeless''}. Our analysis explores important image attributes, such as angle, dominant colors, location, objects, and lighting. We show that a random forest trained with face landmarks can classify samples as depressed or non-depressed and predict raw PHQ-8 scores effectively. Our post-hoc analysis provides several insights through an ablation study, feature importance analysis, and bias assessment. Importantly, we evaluate user concerns about using MoodCapture to detect depression based on sharing photos, providing critical insights into privacy concerns that inform the future design of in-the-wild image-based mental health assessment tools.
△ Less
Submitted 25 February, 2024;
originally announced February 2024.
-
Social Isolation and Serious Mental Illness: The Role of Context-Aware Mobile Interventions
Authors:
Subigya Nepal,
Arvind Pillai,
Emma M. Parrish,
Jason Holden,
Colin Depp,
Andrew T. Campbell,
Eric Granholm
Abstract:
Social isolation is a common problem faced by individuals with serious mental illness (SMI), and current intervention approaches have limited effectiveness. This paper presents a blended intervention approach, called mobile Social Interaction Therapy by Exposure (mSITE), to address social isolation in individuals with serious mental illness. The approach combines brief in-person cognitive-behavior…
▽ More
Social isolation is a common problem faced by individuals with serious mental illness (SMI), and current intervention approaches have limited effectiveness. This paper presents a blended intervention approach, called mobile Social Interaction Therapy by Exposure (mSITE), to address social isolation in individuals with serious mental illness. The approach combines brief in-person cognitive-behavioral therapy (CBT) with context-triggered mobile CBT interventions that are personalized using mobile sensing data. Our approach targets social behavior and is the first context-aware intervention for improving social outcomes in serious mental illness.
△ Less
Submitted 16 November, 2023;
originally announced November 2023.
-
Using Mobile Data and Deep Models to Assess Auditory Verbal Hallucinations
Authors:
Shayan Mirjafari,
Subigya Nepal,
Weichen Wang,
Andrew T. Campbell
Abstract:
Hallucination is an apparent perception in the absence of real external sensory stimuli. An auditory hallucination is a perception of hearing sounds that are not real. A common form of auditory hallucination is hearing voices in the absence of any speakers which is known as Auditory Verbal Hallucination (AVH). AVH is fragments of the mind's creation that mostly occur in people diagnosed with menta…
▽ More
Hallucination is an apparent perception in the absence of real external sensory stimuli. An auditory hallucination is a perception of hearing sounds that are not real. A common form of auditory hallucination is hearing voices in the absence of any speakers which is known as Auditory Verbal Hallucination (AVH). AVH is fragments of the mind's creation that mostly occur in people diagnosed with mental illnesses such as bipolar disorder and schizophrenia. Assessing the valence of hallucinated voices (i.e., how negative or positive voices are) can help measure the severity of a mental illness. We study N=435 individuals, who experience hearing voices, to assess auditory verbal hallucination. Participants report the valence of voices they hear four times a day for a month through ecological momentary assessments with questions that have four answering scales from ``not at all'' to ``extremely''. We collect these self-reports as the valence supervision of AVH events via a mobile application. Using the application, participants also record audio diaries to describe the content of hallucinated voices verbally. In addition, we passively collect mobile sensing data as contextual signals. We then experiment with how predictive these linguistic and contextual cues from the audio diary and mobile sensing data are of an auditory verbal hallucination event. Finally, using transfer learning and data fusion techniques, we train a neural net model that predicts the valance of AVH with a performance of 54\% top-1 and 72\% top-2 F1 score.
△ Less
Submitted 20 April, 2023;
originally announced April 2023.
-
A Survey of Passive Sensing in the Workplace
Authors:
Subigya Nepal,
Gonzalo J. Martinez,
Arvind Pillai,
Koustuv Saha,
Shayan Mirjafari,
Vedant Das Swain,
Xuhai Xu,
Pino G. Audia,
Munmun De Choudhury,
Anind K. Dey,
Aaron Striegel,
Andrew T. Campbell
Abstract:
As emerging technologies increasingly integrate into all facets of our lives, the workplace stands at the forefront of potential transformative changes. A notable development in this realm is the advent of passive sensing technology, designed to enhance both cognitive and physical capabilities by monitoring human behavior. This paper reviews current research on the application of passive sensing t…
▽ More
As emerging technologies increasingly integrate into all facets of our lives, the workplace stands at the forefront of potential transformative changes. A notable development in this realm is the advent of passive sensing technology, designed to enhance both cognitive and physical capabilities by monitoring human behavior. This paper reviews current research on the application of passive sensing technology in the workplace, focusing on its impact on employee wellbeing and productivity. Additionally, we explore unresolved issues and outline prospective pathways for the incorporation of passive sensing in future workplaces.
△ Less
Submitted 30 March, 2024; v1 submitted 9 January, 2022;
originally announced January 2022.
-
Transfer Learning for Activity Recognition in Mobile Health
Authors:
Yuchao Ma,
Andrew T. Campbell,
Diane J. Cook,
John Lach,
Shwetak N. Patel,
Thomas Ploetz,
Majid Sarrafzadeh,
Donna Spruijt-Metz,
Hassan Ghasemzadeh
Abstract:
While activity recognition from inertial sensors holds potential for mobile health, differences in sensing platforms and user movement patterns cause performance degradation. Aiming to address these challenges, we propose a transfer learning framework, TransFall, for sensor-based activity recognition. TransFall's design contains a two-tier data transformation, a label estimation layer, and a model…
▽ More
While activity recognition from inertial sensors holds potential for mobile health, differences in sensing platforms and user movement patterns cause performance degradation. Aiming to address these challenges, we propose a transfer learning framework, TransFall, for sensor-based activity recognition. TransFall's design contains a two-tier data transformation, a label estimation layer, and a model generation layer to recognize activities for the new scenario. We validate TransFall analytically and empirically.
△ Less
Submitted 12 July, 2020;
originally announced July 2020.
-
Jointly Predicting Job Performance, Personality, Cognitive Ability, Affect, and Well-Being
Authors:
Pablo Robles-Granda,
Suwen Lin,
Xian Wu,
Sidney D'Mello,
Gonzalo J. Martinez,
Koustuv Saha,
Kari Nies,
Gloria Mark,
Andrew T. Campbell,
Munmun De Choudhury,
Anind D. Dey,
Julie Gregg,
Ted Grover,
Stephen M. Mattingly,
Shayan Mirjafari,
Edward Moskal,
Aaron Striegel,
Nitesh V. Chawla
Abstract:
Assessment of job performance, personalized health and psychometric measures are domains where data-driven and ubiquitous computing exhibits the potential of a profound impact in the future. Existing techniques use data extracted from questionnaires, sensors (wearable, computer, etc.), or other traits, to assess well-being and cognitive attributes of individuals. However, these techniques can neit…
▽ More
Assessment of job performance, personalized health and psychometric measures are domains where data-driven and ubiquitous computing exhibits the potential of a profound impact in the future. Existing techniques use data extracted from questionnaires, sensors (wearable, computer, etc.), or other traits, to assess well-being and cognitive attributes of individuals. However, these techniques can neither predict individual's well-being and psychological traits in a global manner nor consider the challenges associated to processing the data available, that is incomplete and noisy. In this paper, we create a benchmark for predictive analysis of individuals from a perspective that integrates: physical and physiological behavior, psychological states and traits, and job performance. We design data mining techniques as benchmark and uses real noisy and incomplete data derived from wearable sensors to predict 19 constructs based on 12 standardized well-validated tests. The study included 757 participants who were knowledge workers in organizations across the USA with varied work roles. We developed a data mining framework to extract the meaningful predictors for each of the 19 variables under consideration. Our model is the first benchmark that combines these various instrument-derived variables in a single framework to understand people's behavior by leveraging real uncurated data from wearable, mobile, and social media sources. We verify our approach experimentally using the data obtained from our longitudinal study. The results show that our framework is consistently reliable and capable of predicting the variables under study better than the baselines when prediction is restricted to the noisy, incomplete data.
△ Less
Submitted 10 June, 2020;
originally announced June 2020.
