Cactus: Towards Psychological Counseling Conversations using Cognitive Behavioral Theory

We conduct an experiment to compare the selection of CBT techniques between LLMs and psychological experts. We provide experts with 150 randomly selected client thoughts and patterns from PatternReframe to annotate appropriate CBT techniques. In Figure 2, GPT-4o exhibits a biased selection of CBT techniques, with Evidence-Based Questioning being utilized in almost half of all cases. This finding reveals that a divergence between LLMs and human experts exists. Further details are in Appendix B.1.

The role of a counselor is not to provide instant solutions but to help the client discover their own solutions through fostering a collaborative relationship. However, as shown in Figure 3, AI counselors often present reframed thoughts directly, such as “Remind yourself of positive moments.” This approach may encounter client resistance and hinder clients from independently reframing negative thoughts.

In Figure 3, the client states, “It might be labeling,” using psychological terms to explain their situation. It differs from real-life clients, who describe their reasons for seeking help with detailed and contextual stories without using psychological terminology.

AI clients consistently exhibit a positive attitude during counseling sessions (Serapio-García et al., 2023), which often results in compliant responses such as “I am willing to try it.” AI clients lack the diverse range of attitudes (e.g., strongly negative attitudes) that real-life clients often express.

To generate counseling dialogues, we use personas, thoughts, and patterns from the PatternReframe (Maddela et al., 2023) as contexts for simulating clients. We choose PatternReframe for two reasons: (1) the thoughts closely mirror those of actual clients, and (2) individuals with psychology backgrounds contributed to its creation through crowd-sourcing. Further details are provided in Appendix E.4.

The counselor simulation aims to enhance the effectiveness of counseling in multi-turn interactions. Initially, counselors receive reframed thoughts from PatternReframe, where negative thought patterns have been transformed. Their task is to select the top three CBT techniques that best support this reframing process, a fundamental goal of counseling. To ensure systematic counseling, counselors undergo a planning process that incorporates CBT techniques before sessions. Through the planning process, counselors effectively guide the clients toward the independent discovery of solutions rather than directly suggesting reframed thoughts.

To address the limitations of AI clients (section 2.2), we convert negative thoughts into detailed client narratives (Radford et al., 2019). Inspired by Schmidgall et al. (2024), which simulates patient agents with templates for demographics, clinical history, and symptoms we prompt LLMs to fill out an intake form for client modeling. An intake form includes information on the reason for seeking help, as well as details such as name, gender, and age. This process allows us to simulate specific clients and facilitate effective counseling sessions. Furthermore, to simulate a realistic and diverse counseling environment, we establish three distinct client attitudes (i.e., positive, neutral, negative) reflecting the variability in client behaviors (Li et al., 2023). Following Li et al. (2023), specific behaviors such as providing information or self-criticism are assigned to each attitude. More details are in Appendix E.1.

There are two methods for generating counseling dialogues: (1) assigning the roles of client and counselor to different models (i.e., two-agent mode); (2) providing the information of client and counselor to generate in a script (i.e., script mode). First, we conduct experiments comparing the two-agent mode (Zhou et al., 2023) and the script mode to determine which method generates more natural dialogue. Our findings indicate that the dialogue generated using the script mode is more natural and well-constructed. Detailed results are provided in Appendix E.3. These findings align with those of Zhou et al. (2024), further supporting our decision to utilize script mode for generating dialogues in counselor and client simulations using GPT-4o. Recognizing that typical counseling sessions last 30-60 minutes, we generate longer dialogues to better reflect real counseling and provide more comprehensive interactions, as shown in Table 1.

To ensure the quality of Cactus, we initially filter out dialogues exhibiting abnormal formats or an insufficient number of turns. Subsequently, we utilize the Cognitive Therapy Rating Scale (CTRS), a real-world metric used to assess the quality of CBT-based counseling, for dataset filtering (Beck, 2020). We select three items each from CTRS to evaluate general counseling skills and CBT-specific skills, with each criterion scoring between 0 and 6 points. Dialogues with an average score below 5 points for the 6 CTRS criteria are filtered out. Following this filtering process, 86.3% of the initial conversations remain, constituting the 31,577 conversations in Cactus. More details about filtering are in Appendix E.4.

The Cactus stands out for its large scale and diverse content, providing a comprehensive resource for training counselor agents. It consists of 31,564 dialogues with approximately 1 million utterances (Table 1). To ensure a broad spectrum of client personas, we incorporate diverse backgrounds such as the reason for therapy, age, and occupation (Appendix G.1). Additionally, we construct the dataset by considering three different attitudes identified in previous research, as well as the corresponding behaviors clients might exhibit during counseling (Li et al., 2023). This diversity enhances the realism and applicability of the dataset, making it a resource for develo** robust counselor agents.

To assess the relative quality of the dataset, we conduct human evaluations on Amazon Mechanical Turk (AMT), comparing Cactus with Psych8k and SmileChat. We randomly sample 100 dialogues from each dataset and evaluate them according to four criteria: (1) Helpfulness, (2) Coherence, (3) Empathy, and (4) Guidance. Further details are in Appendix G.1.

Despite possessing a fewer instance compared to SmileChat and being synthetically generated unlike Psych8k, which utilizes real counseling conversations, Figure 5 demonstrates that Cactus consistently outperforms both datasets across all evaluated metrics. As shown in Figure 6, Cactus shows the large performance gap in both helpfulness and empathy, highlighting the effectiveness of planning with CBT techniques for counseling.

We introduce CounselingEval, an evaluation framework designed to assess the counseling skills through multi-turn conversation simulations. Firstly, to model AI clients like real clients, we construct a set of client information, which includes intake form, attitude, and initial utterance and consists of a total of 450 instances.³³3Detailed information is provided in Appendix D.1. Then, following the Cognitive Therapy Rating Scale (CTRS) (Aarons et al., 2012), we devise a set of criteria to assess both general counseling and CBT-specific skills. Finally, we modify the Positive and Negative Affect Schedule (PANAS) (Watson et al., 1988) to assess the effectiveness of counseling from the client’s perspective, measuring changes in the client’s positive/negative emotions before/after counseling sessions. For evaluation, we use G-Eval (Liu et al., 2023b) to assess each criterion in the Likert-scale with a scoring rubric.

To evaluate the counselor agent’s counseling skills, we utilize CTRS, recognized as the gold standard for measuring counseling effectiveness (Aarons et al., 2012). CTRS includes criteria for evaluating both general counseling and CBT-specific skills. We select three criteria each for assessing general counseling skills and CBT-specific skills, tailored to our context.⁴⁴4Details regarding the rationale for selecting these criteria are provided in Appendix D.2. For each criterion, scores ranging from 0 to 6 are assigned. The metrics to assess general counseling skills are as follows: (1) Understanding: evaluating the counselor’s accuracy in understanding the client’s issues; (2) Interpersonal Effectiveness: measuring the counselor’s efficacy in maintaining a therapeutic relationship with the client; and (3) Collaboration: assessing the extent to which the counselor engages the client in collaborative goal-setting and decision-making. Additionally, the metrics to assess CBT-specific skills are as follows: (1) Guided Discovery: evaluating the counselor’s effectiveness in using guided discovery techniques to facilitate client self-reflection; (2) Focus: examining how well the counselor identifies and addresses the client’s key cognitions requiring change; and (3) Strategy: assessing the appropriateness of the counselor’s strategy for promoting change in the client’s problematic thoughts.

The effectiveness of counseling can also be assessed using the Positive and Negative Affect Schedule (PANAS), a commonly used tool for evaluating counseling outcomes. PANAS measures changes in a client’s positive and negative emotions before and after counseling sessions (Watson et al., 1988). Each item is rated on a five-point Likert scale, ranging from 1 to 5. Total scores for positive and negative emotions are calculated based on the sum of ratings, respectively. Counseling is deemed more effective if there is an increase in positive emotion scores and a decrease in negative emotion scores following the sessions. Given LLMs’ capability to comprehend others’ mental states (Kosinski, 2024), we leverage intake forms to infer the client’s emotional state before counseling and predict changes in their emotional state after receiving counseling.

For comparing the utility of Cactus compared to other datasets, we train LLaMA-2-chat-7B (Touvron et al., 2023) and LLaMA-3-Instruct-8B (Meta, 2024) on Psych8k Liu et al. (2023a) and SmileChat (Touvron et al., 2023). We use Cactus to train Camel (CBT-augmented counseling model) that can perform the following tasks: (1) Selecting the appropriate CBT technique and generating a counseling plan based on the information of client and reason for counseling; (2) Conducting psychological counseling sessions with the client according to the generated counseling plan. Details for training and inference are in the Appendix F.

The results of the general counseling metrics and counseling technique metrics are presented in Table 2. We observe that training with Cactus enhances both general counseling skills and the quality of counseling techniques. Despite SmileChat being trained on a larger dataset compared to Psych8k, Psych8k demonstrates better performance. This superior performance can be attributed to the higher quality of Psych8k, as it is based on real counseling conversations. Furthermore, the results of our models indicate that Cactus is a high-quality dataset, closely resembling real counseling scenarios.

To understand the effect of planning with CBT techniques on counseling, we conduct experiments using various methods with GPT-3.5-Turbo. As shown in Table 3, we observe that planning without CBT performs slightly better than not planning at all. However, planning with CBT techniques demonstrates a significant improvement over other methods. Especially, a big gap of performance in Strategy means that counselor agents are proficient in using evidence-based strategies to foster meaningful and lasting changes in the client’s thought patterns and behaviors.

Evaluating the effectiveness of counseling solely based on utterances may be insufficient. Therefore, we use PANAS to measure counseling efficacy by assessing changes in client emotions. Table 4 indicates that Camel shows strong effectiveness in enhancing positive emotions but has limitations in reducing negative emotions. This could be attributed to our approach of guiding clients to explore thought patterns from various perspectives rather than directly changing them.

To ensure the quality of Cactus, we go through the filtering process. we conduct an ablation study to evaluate the effectiveness of the filtering process. Figure 7 shows that the filtering process significantly improves counseling skills. When augmenting datasets through various counseling techniques, there is a possibility of choosing improper techniques. This result demonstrates the importance of eliminating such erroneous technique selections.

To validate the trustworthiness of LLM-based evaluations, we compare the correlation between the expert and the other evaluators (GPT-4o and non-expert). For non-expert evaluation, we conduct human evaluation on AMT. In Table 5, GPT-4o exhibits a higher correlation coefficient than non-expert evaluators. This indicates that LLM-based evaluations have the potential to effectively assess the effectiveness of counseling and can serve as an alternative method to replace expert evaluations, which often face challenges such as scalability and resource constraints.

After the expert evaluations, psychologists note that there have been quality differences, however, there are aspects that do not meet psychological standards, resulting in similar scores. Therefore, they suggest that a qualitative evaluation is more appropriate for comparison than a quantitative evaluation. The results of the quantitative evaluation are in Appendix G.3. For qualitative evaluation, three psychologists compare the counseling conversation between Psych8k-LLaMA3 and Camel-LLaMA3. As a result, Psych8k-LLaMA3 gives the impression of preaching rather than collaboratively exploring and finding a solution with the client. In other words, it only suggests a method without engaging in interactive counseling. In contrast, Camel-LLaMA3 shows superior counseling skills compared to other models by exploring the issues of the client and showing empathy for the client’s emotions. However, it exhibits limitations in its tendency to ask superficial questions and a lack of personalized exploration.

Counselors aim to understand their clients’ situations more accurately by facilitating better expression through empathy. Therefore, instead of being overly empathetic and anticipating beyond what the client expresses, counselors should empathize specifically with what the client discusses after hearing the details of the problem.

Rather than delivering one-sided solutions, counselors should guide clients to self-realization and find appropriate solutions themselves. This involves guiding clients through reflective questioning to help organize their thoughts. Previous research has focused on develo** counselors to give more empathetic and good answers. We would recommend that future counseling research shift its perspective to view counselors as questioners, moving away from the traditional role of counselors as answer providers.