Cactus: Towards Psychological Counseling Conversations using Cognitive Behavioral Theory (2024)

Abstract: Recently, the demand for psychological counseling has significantly increased as more individuals express concerns about their mental health. This surge has accelerated efforts to improve the accessibility of counseling by using large language models (LLMs) as counselors. To ensure client privacy, training open-source LLMs faces a key challenge: the absence of realistic counseling datasets. To address this, we introduce Cactus, a multi-turn dialogue dataset that emulates real-life interactions using the goal-oriented and structured approach of Cognitive Behavioral Therapy (CBT).We create a diverse and realistic dataset by designing clients with varied, specific personas, and having counselors systematically apply CBT techniques in their interactions. To assess the quality of our data, we benchmark against established psychological criteria used to evaluate real counseling sessions, ensuring alignment with expert evaluations.Experimental results demonstrate that Camel, a model trained with Cactus, outperforms other models in counseling skills, highlighting its effectiveness and potential as a counseling agent.We make our data, model, and code publicly available.

## introduction according to cognitive therapy, psychological problems arise when individuals irrationally interpret external events @xcite . therefore, the goal of a counselor is to identify and correct irrational and negative thinking patterns of the client through conversation, guiding clients to change their negative thoughts to positive thoughts (i.e., reframing thoughts) @xcite . cognitive behavioral therapy (cbt) uses reframing techniques tailored to the client's characteristics, making it one of the most widely used counseling strategies @xcite . for example, consider a client who thinks, "many consider me a nerd. i'm such a useless social person" as depicted in figure 1 . the client describes themselves negatively as a nerd and a useless social person, leading to an overly negative self-assessment. these unhelpful thought patterns are classified as labeling, contributing to their distorted self-perception. to reframe the negative thoughts of the client, the counselor can apply the cbt technique of alternative perspective. using alternative perspective helps the client move beyond these negative labels to develop a more accurate and positive self-view. recently, there has been growing interest in using closed-source models (e.g., chatgpt) as counselors @xcite due to their remarkable conversational ability @xcite . however, significant data privacy concerns emerge as these approaches send conversation data, which often contains highly personal information, to third-party providers. to address data privacy concerns, it is crucial to utilize an open-source model @xcite , where performance can be ensured by training on datasets that closely resemble real-world data. while using actual counseling data would be ideal, such dataset is not publicly available due to ethical concerns, especially the risk of individual identification @xcite . as a result, synthetic datasets that closely emulate real-world counseling scenarios present an alternative to address this issue. prior attempts to create synthetic counseling datasets have mainly focused on single-turn counseling strategies, significantly diverging from realworld counseling practices @xcite @xcite . in response, recent efforts have aimed to implement multi-turn strategies @xcite . nevertheless, these approaches often involve only 2-3 turns of dialogue or convert single-turn interactions into multi-turn conversations. therefore, there is a need for a more realistic approach to psychological counseling that ensures therapeutic consistency and progression across multiple turns. to tackle these challenges, we introduce cac-tus (cbt-augmented counseling chat corpus), a publicly available multi-turn realistic counseling dataset. by carefully guiding llms to simulate interactions between counselors and clients, cactus captures the depth and flow of psychological counseling. human evaluation results demonstrate that cactus surpasses the existing counseling datasets in terms of helpfulness and empathy. an assessment of a counselor's abilities needs to consider both their communication skills and the client's psychological changes before and after the counseling session. however, existing methods, such as automatic or single-turn evaluations, are not suitable for assessing a counselor's abilities @xcite . to this end, we introduce counselingeval, an evaluation framework that assesses the ability of counselor in the perspective of both counselor and client within multi-turn counseling dialogues. in counselingeval, we assume ai clients, which interact with the counselor (i.e., target for evaluation), using a set of client information. based on the multi-turn conver-sations between these virtual clients and the counselor, we evaluate the counseling conversations through a modified version of real-world evaluation methods including ctrs and panas @xcite . the results of counselingeval present the effectiveness of cactus through both llm-based evaluation and psychological expert evaluation. furthermore, our model camel, trained on cactus, also outperforms other baseline models in counseling abilities, suggesting that cactus serves as a valuable resource for enhancing the psychological abilities of open-source llms. ## design considerations for cactus we aim to develop a counseling dataset simulating real-world scenarios by simulating conversations between an ai counselor and client @xcite . before the conversation begins, we assume an initial intake session, where basic client information (e.g., name, age, reasons for seeking therapy) is shared, has already been conducted. during the conversation, the ai counselor guides clients through reframing their thoughts using cbt techniques, while personas from patternreframe @xcite help simulate ai clients. in the following sections, we discuss the challenges when utilizing llms in counseling dataset generation. the results of gpt-3.5-turbo are shown in figure 9 . ## cactus: a psychological counseling dataset using cbt we describe the dataset construction process of cactus, a psychological counseling dataset that can also be used for training a counselor agent. we carefully design the dataset construction process to overcome the llms' challenges presented in section 2. in this section, we outline the generation of dialogues with llms and the filtering process for realistic and specialized data. the overview of our approach is in figure 4 and an example of cactus is in appendix e. ## experiments the assessment for the quality of counseling needs to consider both the abilities of counselor and the psychological changes in the client. however, the existing methods, such as automatic or single-turn evaluations, are not suitable for evaluating the abilities of counselor @xcite . therefore, we propose an evaluation framework, counselingeval. ## conclusions we introduce cactus, a large-scale synthetic dataset of counseling dialogue. it aims to provide realistic multi-turn conversations by having the ai counselor and client exhibit real-life behaviors through simulation before sessions. including diverse client personas with varying counseling attitudes and generating conversations based on different counseling strategies counselors may use, results in a diverse dataset. camel, trained on cactus demonstrated high performance across all domains. additionally, we propose coun-selingeval, an evaluation framework that simulates dialogues between an ai counselor and ai clients modeling real clients. it applies established psychological criteria like ctrs and panas to these simulated dialogues, enabling human-aligned evaluation of counseling conversations.