Evaluating Large Language Models for Enhancing Live Chat Therapy: A Comparative Study with Psychotherapists (2025)

Abstract: Large Language Models (LLMs) hold promise for addressing the shortage of qualified therapists in mental health care. While chatbot-based Cognitive Behavioral Therapy (CBT) tools exist, their efficacy in sensitive contexts remains underexplored. This study examines the potential of LLMs to support therapy sessions aimed at reducing Child Sexual Abuse Material (CSAM) consumption. We propose a Retrieval-Augmented Generation (RAG) framework that leverages a fine-tuned BERT-based retriever to guide LLM-generated responses, better capturing the multi-turn, context-specific dynamics of therapy. Four LLMs—Qwen2-7B-Instruct, Mistral-7B-Instruct-v0.3, Orca-2-13B, and Zephyr-7B-Alpha—were evaluated in a small-scale study with 14 domain-expert psychotherapists. Our comparative analysis reveals that, in certain scenarios, LLMs like Mistral-7B-Instruct-v0.3 and Orca-2-13B were preferred over human therapist responses. While limited by sample size, these findings suggest that LLMs can perform at a level comparable to or even exceeding that of human therapists, especially in therapy focused on reducing CSAM consumption. Our code is available online: https://git.tu-berlin.de/neha.deshpande/therapy_responses/-/tree/main

## introduction over the last few years, the rise of large language models (llms) has led to significant advancements in different and varied applications of online therapy. within the domain of therapy, motivational interviewing @xcite which is aimed at enhancing motivation to change problematic behaviors, and cognitive behavioral therapy (cbt) @xcite , which focuses on altering negative thought patterns, are two notable techniques explored by therapy chatbots @xcite . some cbt-based chatbots, in particular, use llms to address user queries on online forums or through therapy platforms that connect users with mental health professionals @xcite . while llm-powered chatbots can deliver helpful, empathetic, and structured responses based on cbt principles, they tend to be limited to pre-defined queries and struggle with dynamic, multi-turn conversations as highlighted by @xcite . these chatbots sometimes lack the empathy required for more specialized assistance, particularly for patients with severe mental health issues. the deployment of such fully automated mental health chatbots also raises significant ethical concerns, including risks of inadequate or harmful advice, exploitation of vulnerable users, and potential biases in recommendations @xcite @xcite . ## the present study this study is part of a broader initiative exploring online psychological intervention via chat for individuals seeking to reduce or stop consuming child sexual abuse material (csam) @xcite . in this intervention, conducted over four weeks, mental health professionals (referred to as "therapists") guide individuals (referred to as "patients") through weekly 50-minute sessions while monitoring csam use. this approach presents unique challenges, including ethical and legal concerns, stigma, and high dropout rates. in this paper, we present an important aspect of the study: testing the use of an ai assistant to support therapists in chat-based therapy sessions for consumers of csam. the goal is not to replace therapists but to use ai as a supportive tool to reduce their cognitive load and enhance the overall therapy experience. specifically, we focus on improving ai's human-like qualities by using ideal dialogue examples to generate therapistlike responses. our approach aims to replicate key therapist behaviors-such as appropriate response length, open-endedness, balanced empathy, and effective emotional support-critical for maintaining patient engagement in live-chat sessions especially in the domain of csam prevention. ## conclusion and future work this study highlights the feasibility of using llms for chat-based therapy in the context of csam prevention, emphasizing the need to balance model performance with privacy and safety considerations. the results suggest that models such as mistral-7b-instruct-v0.3 and orca-2-13b are capable of generating contextually appropriate and therapeutically relevant responses. these findings offer valuable insights into llm applications in sensitive domains, where patient messages may contain explicit content that many models struggle to interpret. although not yet suitable for direct patient use, the approach shows promise as a supportive tool for therapists, offering ai-generated suggestions to reduce their workload. our work emphasizes the importance of combining both retrieval and generation components to optimize model performance in therapy settings. the ablation study showed that the retrieval mechanism plays a crucial role in enhancing the effectiveness of response generation, while the generation component is essential for producing contextually appropriate responses that mimic therapists. these findings highlight the potential of llms to replicate or mimic therapist responses, emphasizing the importance of balancing both retrieval and generation in therapeutic applications, rather than relying exclusively on therapy principles. this was largely possible due to the availability of domain-specific therapy dialogues, which provided concrete examples that guided the llms in producing contextually appropriate responses. looking ahead, future work could improve retrieval via better data labeling or intent detection, and further explore prompt engineering, given its strong impact on model outputs. temperature settings for generating responses with llms also need investigation (especially in this domain), as they affect response creativity and coherence. expanding the study with larger, more diverse samples can offer deeper insights into model behavior across therapeutic contexts. using llm-as-a-judge approach could be valuable where expert therapists are scarce, but due to the sensitive nature of the dialogues, this would also require smaller locally run llms rather than gpt, which has strict content moderation. while llms show promise in areas like csam prevention, continued research is required to address challenges and ensure their safe and effective use.