-
Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselors
Authors:
Alicja Chaszczewicz,
Raj Sanjay Shah,
Ryan Louie,
Bruce A Arnow,
Robert Kraut,
Diyi Yang
Abstract:
Realistic practice and tailored feedback are key processes for training peer counselors with clinical skills. However, existing mechanisms of providing feedback largely rely on human supervision. Peer counselors often lack mechanisms to receive detailed feedback from experienced mentors, making it difficult for them to support the large number of people with mental health issues who use peer couns…
▽ More
Realistic practice and tailored feedback are key processes for training peer counselors with clinical skills. However, existing mechanisms of providing feedback largely rely on human supervision. Peer counselors often lack mechanisms to receive detailed feedback from experienced mentors, making it difficult for them to support the large number of people with mental health issues who use peer counseling. Our work aims to leverage large language models to provide contextualized and multi-level feedback to empower peer counselors, especially novices, at scale. To achieve this, we co-design with a group of senior psychotherapy supervisors to develop a multi-level feedback taxonomy, and then construct a publicly available dataset with comprehensive feedback annotations of 400 emotional support conversations. We further design a self-improvement method on top of large language models to enhance the automatic generation of feedback. Via qualitative and quantitative evaluation with domain experts, we demonstrate that our method minimizes the risk of potentially harmful and low-quality feedback generation which is desirable in such high-stakes scenarios.
△ Less
Submitted 21 March, 2024;
originally announced March 2024.
-
What Makes Digital Support Effective? How Therapeutic Skills Affect Clinical Well-Being
Authors:
Anna Fang,
Wenjie Yang,
Raj Sanjay Shah,
Yash Mathur,
Diyi Yang,
Haiyi Zhu,
Robert Kraut
Abstract:
Online mental health support communities have grown in recent years for providing accessible mental and emotional health support through volunteer counselors. Despite millions of people participating in chat support on these platforms, the clinical effectiveness of these communities on mental health symptoms remains unknown. Furthermore, although volunteers receive some training based on establish…
▽ More
Online mental health support communities have grown in recent years for providing accessible mental and emotional health support through volunteer counselors. Despite millions of people participating in chat support on these platforms, the clinical effectiveness of these communities on mental health symptoms remains unknown. Furthermore, although volunteers receive some training based on established therapeutic skills studied in face-to-face environments such as active listening and motivational interviewing, it remains understudied how the usage of these skills in this online context affects people's mental health status. In our work, we collaborate with one of the largest online peer support platforms and use both natural language processing and machine learning techniques to measure how one-on-one support chats affect depression and anxiety symptoms. We measure how the techniques and characteristics of support providers, such as using affirmation, empathy, and past experience on the platform, affect support-seekers' mental health changes. We find that online peer support chats improve both depression and anxiety symptoms with a statistically significant but relatively small effect size. Additionally, support providers' techniques such as emphasizing the autonomy of the client lead to better mental health outcomes. However, we also found that some behaviors (e.g. persuading) are actually harmful to depression and anxiety outcomes. Our work provides key understanding for mental health care in the online setting and designing training systems for online support providers.
△ Less
Submitted 17 December, 2023;
originally announced December 2023.
-
Agent-based Simulation for Online Mental Health Matching
Authors:
Yuhan Liu,
Anna Fang,
Glen Moriarty,
Robert Kraut,
Haiyi Zhu
Abstract:
Online mental health communities (OMHCs) are an effective and accessible channel to give and receive social support for individuals with mental and emotional issues. However, a key challenge on these platforms is finding suitable partners to interact with given that mechanisms to match users are currently underdeveloped. In this paper, we collaborate with one of the world's largest OMHC to develop…
▽ More
Online mental health communities (OMHCs) are an effective and accessible channel to give and receive social support for individuals with mental and emotional issues. However, a key challenge on these platforms is finding suitable partners to interact with given that mechanisms to match users are currently underdeveloped. In this paper, we collaborate with one of the world's largest OMHC to develop an agent-based simulation framework and explore the trade-offs in different matching algorithms. The simulation framework allows us to compare current mechanisms and new algorithmic matching policies on the platform, and observe their differing effects on a variety of outcome metrics. Our findings include that usage of the deferred-acceptance algorithm can significantly better the experiences of support-seekers in one-on-one chats while maintaining low waiting time. We note key design considerations that agent-based modeling reveals in the OMHC context, including the potential benefits of algorithmic matching on marginalized communities.
△ Less
Submitted 20 March, 2023;
originally announced March 2023.
-
Modeling Motivational Interviewing Strategies On An Online Peer-to-Peer Counseling Platform
Authors:
Raj Sanjay Shah,
Faye Holt,
Shirley Anugrah Hayati,
Aastha Agarwal,
Yi-Chia Wang,
Robert E. Kraut,
Diyi Yang
Abstract:
Millions of people participate in online peer-to-peer support sessions, yet there has been little prior research on systematic psychology-based evaluations of fine-grained peer-counselor behavior in relation to client satisfaction. This paper seeks to bridge this gap by map** peer-counselor chat-messages to motivational interviewing (MI) techniques. We annotate 14,797 utterances from 734 chat co…
▽ More
Millions of people participate in online peer-to-peer support sessions, yet there has been little prior research on systematic psychology-based evaluations of fine-grained peer-counselor behavior in relation to client satisfaction. This paper seeks to bridge this gap by map** peer-counselor chat-messages to motivational interviewing (MI) techniques. We annotate 14,797 utterances from 734 chat conversations using 17 MI techniques and introduce four new interviewing codes such as chit-chat and inappropriate to account for the unique conversational patterns observed on online platforms. We automate the process of labeling peer-counselor responses to MI techniques by fine-tuning large domain-specific language models and then use these automated measures to investigate the behavior of the peer counselors via correlational studies. Specifically, we study the impact of MI techniques on the conversation ratings to investigate the techniques that predict clients' satisfaction with their counseling sessions. When counselors use techniques such as reflection and affirmation, clients are more satisfied. Examining volunteer counselors' change in usage of techniques suggest that counselors learn to use more introduction and open questions as they gain experience. This work provides a deeper understanding of the use of motivational interviewing techniques on peer-to-peer counselor platforms and sheds light on how to build better training programs for volunteer counselors on online platforms.
△ Less
Submitted 9 November, 2022;
originally announced November 2022.
-
Can You be More Social? Injecting Politeness and Positivity into Task-Oriented Conversational Agents
Authors:
Yi-Chia Wang,
Alexandros Papangelis,
Runze Wang,
Zhaleh Feizollahi,
Gokhan Tur,
Robert Kraut
Abstract:
Goal-oriented conversational agents are becoming prevalent in our daily lives. For these systems to engage users and achieve their goals, they need to exhibit appropriate social behavior as well as provide informative replies that guide users through tasks. The first component of the research in this paper applies statistical modeling techniques to understand conversations between users and human…
▽ More
Goal-oriented conversational agents are becoming prevalent in our daily lives. For these systems to engage users and achieve their goals, they need to exhibit appropriate social behavior as well as provide informative replies that guide users through tasks. The first component of the research in this paper applies statistical modeling techniques to understand conversations between users and human agents for customer service. Analyses show that social language used by human agents is associated with greater users' responsiveness and task completion. The second component of the research is the construction of a conversational agent model capable of injecting social language into an agent's responses while still preserving content. The model uses a sequence-to-sequence deep learning architecture, extended with a social language understanding element. Evaluation in terms of content preservation and social language level using both human judgment and automatic linguistic measures shows that the model can generate responses that enable agents to address users' issues in a more socially appropriate way.
△ Less
Submitted 29 December, 2020;
originally announced December 2020.
-
Leading the Collective: Social Capital and the Development of Leaders in Core-Periphery Organizations
Authors:
Benjamin Collier,
Robert Kraut
Abstract:
Wikipedia and open source software projects have been cited as canonical examples of collectively intelligent organizations. Both organizations rely on large crowds of contributors to create knowledge goods. The crowds that emerge in both cases are not flat, but form a core-periphery network in which a few leaders contribute a large portion of the production and coordination work. This paper explo…
▽ More
Wikipedia and open source software projects have been cited as canonical examples of collectively intelligent organizations. Both organizations rely on large crowds of contributors to create knowledge goods. The crowds that emerge in both cases are not flat, but form a core-periphery network in which a few leaders contribute a large portion of the production and coordination work. This paper explores the social network processes by which leaders emerge from crowd-based organizations.
△ Less
Submitted 16 April, 2012;
originally announced April 2012.
