-
Towards RehabCoach: Design and Preliminary Evaluation of a Conversational Agent Supporting Unsupervised Therapy after Stroke
Authors:
Giada Devittori,
Mehdi Akeddar,
Alexandra Retevoi,
Fabian Schneider,
Viktoria Cvetkova,
Daria Dinacci,
Antonella Califfi,
Paolo Rossi,
Claudio Petrillo,
Tobias Kowatsch,
Olivier Lambercy
Abstract:
Unsupervised therapy after stroke is a promising way to boost therapy dose without significantly increasing the workload on healthcare professionals. However, it raises important challenges, such as lower adherence to therapy in the absence of social interaction with therapists. We present the initial prototype of RehabCoach, a novel smartphone-based app with conversational agent to support unsupe…
▽ More
Unsupervised therapy after stroke is a promising way to boost therapy dose without significantly increasing the workload on healthcare professionals. However, it raises important challenges, such as lower adherence to therapy in the absence of social interaction with therapists. We present the initial prototype of RehabCoach, a novel smartphone-based app with conversational agent to support unsupervised therapy. RehabCoach is designed to increase patients engagement and adherence to therapy and to provide information (e.g., about stroke, health) in an interactive and user-friendly manner. We report on the design and usability evaluation of the first prototype of RehabCoach, assessed by four stroke patients and five healthcare professionals, who interacted with the app in a single testing session. Task completion time and success rates were measured for 15 representative tasks, and participants assessed usability via questionnaires and a semi-structured interview. Results show that it was feasible for stroke patients to successfully interact with RehabCoach (task success $\geq$ 93 $\%$) without requiring extensive training. Participants positively rated the usability of RehabCoach (mean mHealth App Usability Questionnaire score: 1.3 for primary users, 1.4 for healthcare professionals, on a scale from 1 (positive evaluation) to 7). The feedback collected in this work opens the door to further enhance RehabCoach as an interactive digital tool to support unsupervised rehabilitation.
△ Less
Submitted 2 March, 2024;
originally announced March 2024.
-
"Are you okay, honey?": Recognizing Emotions among Couples Managing Diabetes in Daily Life using Multimodal Real-World Smartwatch Data
Authors:
George Boateng,
Xiangyu Zhao,
Malgorzata Speichert,
Elgar Fleisch,
Janina Lüscher,
Theresa Pauly,
Urte Scholz,
Guy Bodenmann,
Tobias Kowatsch
Abstract:
Couples generally manage chronic diseases together and the management takes an emotional toll on both patients and their romantic partners. Consequently, recognizing the emotions of each partner in daily life could provide an insight into their emotional well-being in chronic disease management. Currently, the process of assessing each partner's emotions is manual, time-intensive, and costly. Desp…
▽ More
Couples generally manage chronic diseases together and the management takes an emotional toll on both patients and their romantic partners. Consequently, recognizing the emotions of each partner in daily life could provide an insight into their emotional well-being in chronic disease management. Currently, the process of assessing each partner's emotions is manual, time-intensive, and costly. Despite the existence of works on emotion recognition among couples, none of these works have used data collected from couples' interactions in daily life. In this work, we collected 85 hours (1,021 5-minute samples) of real-world multimodal smartwatch sensor data (speech, heart rate, accelerometer, and gyroscope) and self-reported emotion data (n=612) from 26 partners (13 couples) managing diabetes mellitus type 2 in daily life. We extracted physiological, movement, acoustic, and linguistic features, and trained machine learning models (support vector machine and random forest) to recognize each partner's self-reported emotions (valence and arousal). Our results from the best models (balanced accuracies of 63.8% and 78.1% for arousal and valence respectively) are better than chance and our prior work that also used data from German-speaking, Swiss-based couples, albeit, in the lab. This work contributes toward building automated emotion recognition systems that would eventually enable partners to monitor their emotions in daily life and enable the delivery of interventions to improve their emotional well-being.
△ Less
Submitted 22 August, 2022; v1 submitted 16 August, 2022;
originally announced August 2022.
-
Development, Deployment, and Evaluation of DyMand -- An Open-Source Smartwatch and Smartphone System for Capturing Couples' Dyadic Interactions in Chronic Disease Management in Daily Life
Authors:
George Boateng,
Prabhakaran Santhanam,
Elgar Fleisch,
Janina Lüscher,
Theresa Pauly,
Urte Scholz,
Tobias Kowatsch
Abstract:
Dyadic interactions of couples are of interest as they provide insight into relationship quality and chronic disease management. Currently, ambulatory assessment of couples' interactions entails collecting data at random or scheduled times which could miss significant couples' interaction/conversation moments. In this work, we developed, deployed and evaluated DyMand, a novel open-source smartwatc…
▽ More
Dyadic interactions of couples are of interest as they provide insight into relationship quality and chronic disease management. Currently, ambulatory assessment of couples' interactions entails collecting data at random or scheduled times which could miss significant couples' interaction/conversation moments. In this work, we developed, deployed and evaluated DyMand, a novel open-source smartwatch and smartphone system for collecting self-report and sensor data from couples based on partners' interaction moments. Our smartwatch-based algorithm uses the Bluetooth signal strength between two smartwatches each worn by one partner, and a voice activity detection machine-learning algorithm to infer that the partners are interacting, and then to trigger data collection. We deployed the DyMand system in a 7-day field study and collected data about social support, emotional well-being, and health behavior from 13 (N=26) Swiss-based heterosexual couples managing diabetes mellitus type 2 of one partner. Our system triggered 99.1% of the expected number of sensor and self-report data when the app was running, and 77.6% of algorithm-triggered recordings contained partners' conversation moments compared to 43.8% for scheduled triggers. The usability evaluation showed that DyMand was easy to use. DyMand can be used by social, clinical, or health psychology researchers to understand the social dynamics of couples in everyday life, and for develo** and delivering behavioral interventions for couples who are managing chronic diseases.
△ Less
Submitted 16 May, 2022;
originally announced May 2022.
-
Emotion Recognition among Couples: A Survey
Authors:
George Boateng,
Elgar Fleisch,
Tobias Kowatsch
Abstract:
Couples' relationships affect the physical health and emotional well-being of partners. Automatically recognizing each partner's emotions could give a better understanding of their individual emotional well-being, enable interventions and provide clinical benefits. In the paper, we summarize and synthesize works that have focused on develo** and evaluating systems to automatically recognize the…
▽ More
Couples' relationships affect the physical health and emotional well-being of partners. Automatically recognizing each partner's emotions could give a better understanding of their individual emotional well-being, enable interventions and provide clinical benefits. In the paper, we summarize and synthesize works that have focused on develo** and evaluating systems to automatically recognize the emotions of each partner based on couples' interaction or conversation contexts. We identified 28 articles from IEEE, ACM, Web of Science, and Google Scholar that were published between 2010 and 2021. We detail the datasets, features, algorithms, evaluation, and results of each work as well as present main themes. We also discuss current challenges, research gaps and propose future research directions. In summary, most works have used audio data collected from the lab with annotations done by external experts and used supervised machine learning approaches for binary classification of positive and negative affect. Performance results leave room for improvement with significant research gaps such as no recognition using data from daily life. This survey will enable new researchers to get an overview of this field and eventually enable the development of emotion recognition systems to inform interventions to improve the emotional well-being of couples.
△ Less
Submitted 16 February, 2022;
originally announced February 2022.
-
BERT meets LIWC: Exploring State-of-the-Art Language Models for Predicting Communication Behavior in Couples' Conflict Interactions
Authors:
Jacopo Biggiogera,
George Boateng,
Peter Hilpert,
Matthew Vowels,
Guy Bodenmann,
Mona Neysari,
Fridtjof Nussbeck,
Tobias Kowatsch
Abstract:
Many processes in psychology are complex, such as dyadic interactions between two interacting partners (e.g. patient-therapist, intimate relationship partners). Nevertheless, many basic questions about interactions are difficult to investigate because dyadic processes can be within a person and between partners, they are based on multimodal aspects of behavior and unfold rapidly. Current analyses…
▽ More
Many processes in psychology are complex, such as dyadic interactions between two interacting partners (e.g. patient-therapist, intimate relationship partners). Nevertheless, many basic questions about interactions are difficult to investigate because dyadic processes can be within a person and between partners, they are based on multimodal aspects of behavior and unfold rapidly. Current analyses are mainly based on the behavioral coding method, whereby human coders annotate behavior based on a coding schema. But coding is labor-intensive, expensive, slow, focuses on few modalities. Current approaches in psychology use LIWC for analyzing couples' interactions. However, advances in natural language processing such as BERT could enable the development of systems to potentially automate behavioral coding, which in turn could substantially improve psychological research. In this work, we train machine learning models to automatically predict positive and negative communication behavioral codes of 368 German-speaking Swiss couples during an 8-minute conflict interaction on a fine-grained scale (10-seconds sequences) using linguistic features and paralinguistic features derived with openSMILE. Our results show that both simpler TF-IDF features as well as more complex BERT features performed better than LIWC, and that adding paralinguistic features did not improve the performance. These results suggest it might be time to consider modern alternatives to LIWC, the de facto linguistic features in psychology, for prediction tasks in couples research. This work is a further step towards the automated coding of couples' behavior which could enhance couple research and therapy, and be utilized for other dyadic interactions as well.
△ Less
Submitted 18 September, 2021; v1 submitted 2 June, 2021;
originally announced June 2021.
-
"You made me feel this way": Investigating Partners' Influence in Predicting Emotions in Couples' Conflict Interactions using Speech Data
Authors:
George Boateng,
Peter Hilpert,
Guy Bodenmann,
Mona Neysari,
Tobias Kowatsch
Abstract:
How romantic partners interact with each other during a conflict influences how they feel at the end of the interaction and is predictive of whether the partners stay together in the long term. Hence understanding the emotions of each partner is important. Yet current approaches that are used include self-reports which are burdensome and hence limit the frequency of this data collection. Automatic…
▽ More
How romantic partners interact with each other during a conflict influences how they feel at the end of the interaction and is predictive of whether the partners stay together in the long term. Hence understanding the emotions of each partner is important. Yet current approaches that are used include self-reports which are burdensome and hence limit the frequency of this data collection. Automatic emotion prediction could address this challenge. Insights from psychology research indicate that partners' behaviors influence each other's emotions in conflict interaction and hence, the behavior of both partners could be considered to better predict each partner's emotion. However, it is yet to be investigated how doing so compares to only using each partner's own behavior in terms of emotion prediction performance. In this work, we used BERT to extract linguistic features (i.e., what partners said) and openSMILE to extract paralinguistic features (i.e., how they said it) from a data set of 368 German-speaking Swiss couples (N = 736 individuals) who were videotaped during an 8-minutes conflict interaction in the laboratory. Based on those features, we trained machine learning models to predict if partners feel positive or negative after the conflict interaction. Our results show that including the behavior of the other partner improves the prediction performance. Furthermore, for men, considering how their female partners spoke is most important and for women considering what their male partner said is most important in getting better prediction performance. This work is a step towards automatically recognizing each partners' emotion based on the behavior of both, which would enable a better understanding of couples in research, therapy, and the real world.
△ Less
Submitted 18 September, 2021; v1 submitted 2 June, 2021;
originally announced June 2021.
-
Detecting Receptivity for mHealth Interventions in the Natural Environment
Authors:
Varun Mishra,
Florian Künzler,
Jan-Niklas Kramer,
Elgar Fleisch,
Tobias Kowatsch,
David Kotz
Abstract:
JITAI is an emerging technique with great potential to support health behavior by providing the right type and amount of support at the right time. A crucial aspect of JITAIs is properly timing the delivery of interventions, to ensure that a user is receptive and ready to process and use the support provided. Some prior works have explored the association of context and some user-specific traits o…
▽ More
JITAI is an emerging technique with great potential to support health behavior by providing the right type and amount of support at the right time. A crucial aspect of JITAIs is properly timing the delivery of interventions, to ensure that a user is receptive and ready to process and use the support provided. Some prior works have explored the association of context and some user-specific traits on receptivity, and have built post-study machine-learning models to detect receptivity. For effective intervention delivery, however, a JITAI system needs to make in-the-moment decisions about a user's receptivity. To this end, we conducted a study in which we deployed machine-learning models to detect receptivity in the natural environment, i.e., in free-living conditions.
We leveraged prior work regarding receptivity to JITAIs and deployed a chatbot-based digital coach~-- Ally~-- that provided physical-activity interventions and motivated participants to achieve their step goals. We extended the original Ally~app to include two types of machine-learning model that used contextual information about a person to predict when a person is receptive: a \textit{static model\/} that was built before the study started and remained constant for all participants and an \textit{adaptive model\/} that continuously learned the receptivity of individual participants and updated itself as the study progressed. For comparison, we included a \textit{control model\/} that sent intervention messages at random times. The app randomly selected a delivery model for each intervention message. We observed that the machine-learning models led up to a 40\% improvement in receptivity as compared to the control model. Further, we evaluated the temporal dynamics of the different models and observed that receptivity to messages from the adaptive model increased over the course of the study.
△ Less
Submitted 1 May, 2021; v1 submitted 16 November, 2020;
originally announced November 2020.
-
Improving Heart Rate Variability Measurements from Consumer Smartwatches with Machine Learning
Authors:
Martin Maritsch,
Caterina Bérubé,
Mathias Kraus,
Vera Lehmann,
Thomas Züger,
Stefan Feuerriegel,
Tobias Kowatsch,
Felix Wortmann
Abstract:
The reactions of the human body to physical exercise, psychophysiological stress and heart diseases are reflected in heart rate variability (HRV). Thus, continuous monitoring of HRV can contribute to determining and predicting issues in well-being and mental health. HRV can be measured in everyday life by consumer wearable devices such as smartwatches which are easily accessible and affordable. Ho…
▽ More
The reactions of the human body to physical exercise, psychophysiological stress and heart diseases are reflected in heart rate variability (HRV). Thus, continuous monitoring of HRV can contribute to determining and predicting issues in well-being and mental health. HRV can be measured in everyday life by consumer wearable devices such as smartwatches which are easily accessible and affordable. However, they are arguably accurate due to the stability of the sensor. We hypothesize a systematic error which is related to the wearer movement. Our evidence builds upon explanatory and predictive modeling: we find a statistically significant correlation between error in HRV measurements and the wearer movement. We show that this error can be minimized by bringing into context additional available sensor information, such as accelerometer data. This work demonstrates our research-in-progress on how neural learning can minimize the error of such smartwatch HRV measurements.
△ Less
Submitted 17 July, 2019;
originally announced July 2019.
