Understanding the Therapeutic Relationship between Counselors and Clients in Online Text-based Counseling using LLM s (2024)

Abstract: Robust therapeutic relationships between counselors and clients are fundamental to counseling effectiveness. The assessment of therapeutic alliance is well-established in traditional face-to-face therapy but may not directly translate to text-based settings. With millions of individuals seeking support through online text-based counseling, understanding the relationship in such contexts is crucial.In this paper, we present an automatic approach using large language models (LLMs) to understand the development of therapeutic alliance in text-based counseling. We adapt a theoretically grounded framework specifically to the context of online text-based counseling and develop comprehensive guidelines for characterizing the alliance. We collect a comprehensive counseling dataset and conduct multiple expert evaluations on a subset based on this framework. Our LLM-based approach, combined with guidelines and simultaneous extraction of supportive evidence underlying its predictions, demonstrates effectiveness in identifying the therapeutic alliance. Through further LLM-based evaluations on additional conversations, our findings underscore the challenges counselors face in cultivating strong online relationships with clients. Furthermore, we demonstrate the potential of LLM-based feedback mechanisms to enhance counselors’ ability to build relationships, supported by a small-scale proof-of-concept.

## introduction globally, approximately one in five individuals experience mental health problems each year @xcite @xcite . owing to the high costs and geographical limitations associated with traditional face-to-face therapy, coupled with concerns about stigma @xcite , many individuals are turning to seek support through online text-based psychological counseling @xcite . however, in real-world scenarios of such counseling approaches, it remains largely unknown whether counselors and clients have established strong therapeutic alliances solely through textual communications. in psychological counseling, a positive relationship between counselors and clients is fundamental for achieving effective therapeutic outcomes @xcite @xcite . the robust therapeutic alliance signifies the cooperative relationship between counselors and clients, characterized by their shared therapeutic goals and their ability to engage together, within the context of an affective bond or positive attachment @xcite . the absence of regular and timely assessment and feedback may impede counselors in gradually nurturing of a solid rapport with clients, potentially leading to client dropout. traditionally, counselors rely on professional supervisors for feedback, a process that is often labor-intensive and time-consuming, resulting in delayed responses. natural language processing (nlp) systems capable of understanding the alliance could empower counselors with prompt and insightful feedback to enhance their practice. however, the current understanding of the alliance is primarily limited to traditional face-to-face, speechbased counseling @xcite @xcite , due to the scarcity of resources and methods tailored to text-based interactions. also, while previous nlp research has focused on the behaviors of individual participants, either counselors' strategies @xcite @xcite or clients' reactions @xcite @xcite , the mutual cognitive and affective agreement they reach through multi-turn interactions is the key factor influencing conversation outcomes @xcite . in this paper, we present an effective automatic approach using large language models (llms) to understand the establishment of therapeutic alliances in online text-based counseling @xcite . we propose a conceptual framework with detailed guidelines to characterize the alliance in conversations, adapting theories and scales of therapeutic relationships from face-to-face therapy to text-only interactions. the framework includes three dimensions: goal-setting consensus, approaches to goals, and cultivation of affective bonds (figure 1 ). the guidelines facilitate identifying observed elements in textual conversations corresponding to each framework component. we then collect a large-scale text-based counseling dataset from an online platform. using our proposed framework and guidelines, trained experts annotate a subset of sessions with high inter-rater reliability. we employ prompt tuning to enable llms to apply these guidelines in understanding the alliance within texts. additionally, we use the chain-of-thought (cot) process @xcite to help models identify supportive evidence for their evaluations (as shown in the reason part of figure 1 ). experimental findings show that integrating precise guidelines and cot significantly enhances llms' ability to understand the alliance, ensuring consistency and alignment with experts. we use the best-performing model on the remaining unannotated sessions to show a positive correlation between the alliance and favorable counseling outcomes. our findings highlight that counselors, including experienced ones, may struggle to build deeper connections as counseling progresses. this underscores the need for evaluation and feedback mechanisms to enhance counseling effectiveness. our small-scale proof-of-concept demonstrates that llm-based feedback can offer counselors insights to better understand their alliances with clients and improve their relationship-building skills. our research is designed to leverage the understanding and reasoning capabilities of llms to comprehensively explore the development of critical components of therapeutic alliance through text-only interactions. llms for mental health analysis. as the emergence of llms showcasing advanced text understanding and reasoning capabilities, recent research has explored to leverage llms in mental health analysis @xcite . most studies focuses on analyzing users' posts published in social media platforms to predict their personality traits @xcite , sentiment @xcite , and mental health conditions @xcite @xcite , including anxiety, depression, suicide ideation, and others. besides, several studies investigate multi-turn mental health counseling conversations to improve efficiency in psychological counseling @xcite @xcite . @xcite utilize llms to summarize counseling sessions across specific components such as patients' symptoms and history, patient discovery, and reflection, aiding counselors in treatment planning. ## framework and guidelines for measuring therapeutic alliance to understand the alliance between counselors and clients in psychological counseling, we adapt the existing therapeutic alliance definitions and scales to the context of online text-only counseling. to facilitate accurate understanding based on this framework, we carefully design specific guidelines in collaboration with counseling psychology experts. ## data collection to validate the feasibility of our proposed framework, we collect counseling conversations between professional counselors and actual clients, and carefully annotate a subset of these conversations according to the framework. ## llm evaluation with annotated data and proposed guidelines, we conduct zero-shot experiments to prompt advanced llms including glm-4, claude-3, chatgpt and gpt-4 to understand the therapeutic relationships reflected in text-only conversations. ## llm-based insights into text-based mental health counseling we employ the best-performing model (i.e., gpt-4 with detailed guidelines and cot) to study how the therapeutic alliance impacts online text-based psychological counseling. we utilize the model to predict the alliance in the remaining unannotated sessions. we investigate how counselors' experience and counseling progress influence the alliance strength, and examine its correlation with counseling outcomes. furthermore, we analyze common patterns in counselors' interactions that affect the establishment of the therapeutic relationship. additionally, we showcase the efficacy of llm-based feedback for counselors through a simple proof-ofconcept demonstration. counselors' counseling experience ̸ = abilities to establish relationships. we explore whether counselors with more experience find it easier to establish therapeutic relationships with clients. nine counselors are grouped by their counseling experience: primary (≤2 years), intermediate (3-8 years), and advanced (≥10 years). we calculate average working alliance scores for each counselor across all sessions to gauge their relationship-building pro- figure 3 : the average alliance scores for all counselors and counselors with varied experience levels. long-term communications ̸ = stronger alliance. we divide clients' counseling sessions into three phases-early, middle, and late-and compare the counselor-client relationship across these stages. our findings indicate that the relationship does not significantly deepen over time. specifically, there is only a marginal increase in affective connections, while agreement on counseling goals and approaches remains constant. further analysis shows that nearly 50% of client-counselor pairs experience either a decline or no change in the strength of the therapeutic alliance, with less than 3% improving by at least one level within our framework. this emphasize the challenges counselors face in enhancing relationship-building skills without adequate feedback. 1994; @xcite . here, we utilize clients' self-reported ratings on the outcome rating scale (ors) @xcite to gauge the effectiveness of each counseling session. the ors evaluates various aspects including clients' individual physical and mental well-being, interpersonal relationships, social role functioning and overall well-being, with scores ranging from 0 to 100 for each aspect. pearson correlation analyses between total working alliance scores and ors dimensions show significant correlations (r ≈ 0.30, @xmath0.001). this indicates that a stronger working alliance may be associated with more favorable conversation outcomes. additional details are provided in appendix d.2. ## conclusion we adapted a framework with crafted guidelines, a dataset, and llm-based approaches to understand the working alliance between counselors and clients in online text-based counseling. our findings indicate that integrating detailed guidelines and cot prompting enables llms to assess the working alliance with underlying rationales effectively. furthermore, we underscore the importance of the working alliance in online counseling and demonstrate the utility of llm-based feedback in enhancing counselors' understanding of their client relationships and providing valuable insights for improvement.