CBT -Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapy (2025)

Abstract: There is a significant gap between patient needs and available mental health support today. In this paper, we aim to thoroughly examine the potential of using Large Language Models (LLMs) to assist professional psychotherapy. To this end, we propose a new benchmark, CBT-Bench, for the systematic evaluation of cognitive behavioral therapy (CBT) assistance. We include three levels of tasks in CBT-Bench: **I: Basic CBT knowledge acquisition**, with the task of multiple-choice questions; **II: Cognitive model understanding**, with the tasks of cognitive distortion classification, primary core belief classification, and fine-grained core belief classification; **III: Therapeutic response generation**, with the task of generating responses to patient speech in CBT therapy sessions.These tasks encompass key aspects of CBT that could potentially be enhanced through AI assistance, while also outlining a hierarchy of capability requirements, ranging from basic knowledge recitation to engaging in real therapeutic conversations. We evaluated representative LLMs on our benchmark. Experimental results indicate that while LLMs perform well in reciting CBT knowledge, they fall short in complex real-world scenarios requiring deep analysis of patients’ cognitive structures and generating effective responses, suggesting potential future work.

## introduction mental health conditions have reached alarming levels globally, with one in eight people affected, according to the world health organization (2023) 2 . there is a severe shortage of mental health professionals. in the u.s., more than 160 million people live in areas with insufficient mental health *equal contributions. 1 the data and code are released at https://github.com/ mianzhang/cbt-bench . in this work, we aim to thoroughly investigate the proficiency and potential of llms in supporting various facets and stages of professional mental health care. to this end, we propose cbt-bench, a systematic benchmark for evaluating cbt efficacy. cbt-bench is structured in three levels, from cbt knowledge recitation to the therapeutic responses generation in cbt sessions, providing a hierarchical assessment of cbt capabilities. to ensure the professionalism and high quality of our benchmark, we collaborate with domain experts (clinical psychologists, professors, and social workers) throughout the construction of cbt-bench. in level i, we aim to assess basic cbt knowledge acquisition. we propose cbt-qa, a new dataset of 220 multiple-choice questions. the qa pairs are collected from cbt exam questions for master of social work (msw) students as well as compositions from cbt experts, covering a wide range of cbt knowledge, including basic concepts, practical knowledge, and case studies. in level ii, we aim to assess cognitive model understanding. modeling how the cognitive components of the patients, such as the beliefs and thoughts, are connected, is at the core of cbt @xcite . in this work, we propose cbt-cd, a new dataset of cognitive distortion classification with 146 high-quality examples. we also propose two new tasks to assess the understanding of patients' beliefs: primary core belief classification and fine-grained core belief classification. for primary core belief classification, we construct cbt-pc, a new dataset of 184 examples with three primary core belief categories. for fine-grained core belief classification, we propose cbt-fc, a new dataset of 112 examples with nineteen fine-grained core belief categories. building ai models for such tasks to assist the cognitive modeling process has the great potential to enhance therapists' accuracy and productivity. in level iii, we aim to evaluate therapeutic response generation, the ultimate cbt efficacywhether the model can effectively respond to patient speech during cbt sessions like the therapists. due to the privacy constraints associated with real cbt session data, accessing extensive datasets is challenging. in collaboration with professors specialized in cbt education, we propose cbt-dp, using deliberate practice as outlined in boswell and constantino (2022) for our assessment. this methodology is traditionally employed in assessing cbt proficiency among graduate students. it encompasses a collection of exercises to respond to typical patient speeches across ten key aspects of cbt sessions, categorized into three levels of difficulty, totaling 156 distinct exercises. this approach ensures comprehensive coverage of critical and challenging scenarios likely to occur in real sessions. generating high-quality responses to patient speeches in cbt-dp can serve as an effective proxy for real cbt session efficacy. we experiment with six popular llms for level i and level ii in cbt-bench and find that 1) the models of large sizes are better at answering cbt knowledge questions; 2) simply making the models larger could not enhance their understanding ability of the cognitive model; 3) current llms struggle with detecting fine-grained cognitive disorders or core beliefs. a deeper analysis of the model performance for each class points out the main reasons for a wrong prediction and directions to improve the corresponding ability of the models. for the level iii task, we find that llms generally follow a rigid logical reasoning process but lack a crucial skill in psychotherapy-thinking and guiding from the patient's perspective to respect their autonomy and build rapport. these findings highlight significant limitations in applying llms to real-world psychotherapy practice and provide valuable insights for future research and development toward more accessible and efficient mental health care. ## related work our work is the first to systematically evaluate llms' ability to assist professional human therapists in the specialized field of cbt. the works directly related to our research include injecting domain knowledge of mental health into the models @xcite , cognitive disorder detection @xcite @xcite , negative thoughts recognition and reframing @xcite @xcite , and patient simulation @xcite or therapist simulation @xcite in therapeutic conversation. @xcite proved that with clear modeling of the cognitive model of patients, llms could act more like real patients. also, we believe that enhancing the understanding of the cognitive models of patients could be beneficial to interpretable medical decisions @xcite , which is a crucial step towards reliable and safe automated mental healthcare @xcite . in the proposed therapeutic response generation task, we leverage the feedback from professional therapists to assess the potential of llms responding like a therapist in real therapy sessions, while @xcite explored how to make llms roleplay patients with the feedback from professional therapists. ## cbt-bench in this section, we elaborate on how cbt-bench is constructed. we discuss level i, ii, and iii tasks in §3.1, §3.2, and §3.3, respectively. note that apart from collaborating with social work professors and clinical psychologists, who are our co-authors, we recruit cbt experts (clinical psychologists, social workers, etc.) from upwork 4 for all additional data annotation tasks (see §8 for recruiting details). ## conclusion in this paper, we propose a new benchmark, cbt-bench, to comprehensively evaluate the efficacy of assisting cbt with llms. cbt-bench contains three levels of tasks ranging from basic cbt knowledge acquisition to therapeutic response generation, totaling five new datasets constructed by domain experts. we experiment with representative llms and the results show that models achieve comparable or even better cognitive understanding ability, especially in qa tasks, but still lag behind humans in most generation tasks. we hope our new benchmark provides a meaningful testbed for future research in this area.