Towards Understanding Counseling Conversations: Domain Knowledge and Large Language Models (2024)

Abstract: Understanding the dynamics of counseling conversations is an important task, yet it is a challenging NLP problem regardless of the recent advance of Transformer-based pre-trained language models. This paper proposes a systematic approach to examine the efficacy of domain knowledge and large language models (LLMs) in better representing conversations between a crisis counselor and a help seeker. We empirically show that state-of-the-art language models such as Transformer-based models and GPT models fail to predict the conversation outcome. To provide richer context to conversations, we incorporate human-annotated domain knowledge and LLM-generated features; simple integration of domain knowledge and LLM features improves the model performance by approximately 15%. We argue that both domain knowledge and LLM-generated features can be exploited to better characterize counseling conversations when they are used as an additional context to conversations.

## introduction online counseling has become a more significant part of mental health services over the last couple of decades as younger generations feel more emotionally safe with digital communication @xcite . although building therapeutic relationships and social presence through written communication may exhibit significant challenges compared to in-person services @xcite ), text or chat based counseling services are irreplaceable; nearly 50% of the united states population reside in a mental health shortage area where there are less than two psychiatrists per 100,000 residents @xcite . the conversation dynamics and therapeutic relationship between mental health providers and clients have been actively studied in the health science field, mainly analyzing mutual trust @xcite , empathy @xcite , social presence @xcite , and rapportbuilding @xcite . despite its importance, there's relatively little work done in analyzing linguistic components of counseling conversations and characterizing them to better understand the conversation dynamics. throughout this research, we aim to propose a systematic approach to better characterize counseling conversations. we hypothesize that the current state-of-the-art language models contain insufficient knowledge of the counseling domain in their parameters. motivated by existing works using external knowledge for solving tasks such as question answering @xcite , commonsense reasoning @xcite , and language generation @xcite , this paper studies whether additional knowledge helps characterize counseling conversations. we suggest two different ways of obtaining this additional knowledge: human annotation and large language model (llm) prompting. in this paper, we measure the level of understanding counseling conversations by predicting conversation outcomes, i.e., whether the help seeker would feel more positive after the conversation or not. we empirically show that transformer-based classifiers as well as state-of-the-art llms exhibit sub-optimal performances despite their strong ability on many downstream tasks. the paper then describes how domain knowledge is obtained in order to further emphasize the counselor's strategic utterances and the help seeker's perspectives. we show that the additional knowledge helps pre-trained language models better fit the dataset and perform well in predicting the conversation outcomes-simple integration of the knowledge and feature ensembling improves the model performance by approximately 15%. we further analyze the efficacy of different features and explain how these features help classifiers better predict the outcome. ## counseling conversation analysis in chat based services for crisis counseling, a help seeker starts a session seeking help and a counselor replies to it. there are two speakers in these chat sessions, a help seeker and a counselor. following previous works in analyzing such conversations @xcite , we aim to analyze counseling conversations by observing two different levels of features-utterance level features and session level features. utterance level features examine the characteristics of conversation turns (i.e. messages), whereas session level features consider different aspects that can be found throughout the whole conversation. ## models we implement baseline models with the conversation text and integrate varying features to evaluate their efficacy. ## experimental settings very little difference exists between 'positive' and 'neutral' conversation outcomes. we combine these two classes and make the task as a binary classification task (i.e. 'negative' v. 'non-negative'). to evaluate and compare different models, we compute macro f1 scores and the recall values of the minority class (i.e. 'negative' class). models can achieve a satisfactory macro f1 score by minimally assigning minority class to test instances. in such cases, these models will score low recall on the minority class. however, models with higher recall on the 'negative' class are more desirable in a real use case, as they identify more instances where the help seekers do not feel positive, and one can further assess what can be done alternatively. the reported results are from distilbert-baseuncased classifier which works the best among all bert based classifiers we implemented. conversation text includes @xmath0. we use the union of d small and d large as our main dataset, d all , with 60/20/20 splits of training, evaluation, and testing sets. all models are experimented with 10-fold cross validation. ## discussion in this section, we further diagnose the model outputs and their relatedness to the features. ## related work several recent nlp works looked at analyzing counseling conversations and predicting their outcomes @xcite @xcite @xcite . similar to our approach, several work relied on domain knowledge to identify counseling strategies and conversational actions @xcite @xcite . for example, @xcite employed behavioral codes of clients and therapists to provide real-time feedback to a therapist about the category of the current utterance and suggest the next category to apply. other works analyzed the conversational style of counselors, how it changes over time @xcite and the emotional support they provide @xcite . for example, @xcite proposed an empathybased approach in understanding counseling conversations between a help seeker and peer supporters on talklife and r/depression subreddits @xcite . @xcite worked on guiding dialog models with emotional support strategy chains using 7cups dataset @xcite . the authors evaluated the framework on blenderbot @xcite and di-alogpt @xcite . as counseling conversation analysis has been improving with the help of more representative language models over time, our research poses the initial attempt to utilize llms for reasoning about features relevant to conversational dynamics, and their relatedness to conversation outcomes. ## conclusion we study the dynamics of conversations between crisis counselors and help seekers. transformerbased models and the chatgpt fail to predict whether the help seeker feels positive after the conversation. to better characterize counseling conversations, we integrate domain-specific knowledge, human-annotated utterance level features identifying counseling strategies, and llm generated session level features portraying help seekers' perspectives. we show that ensembling additional features improves performance in predicting conversation outcomes. analyses suggest that the features lead the model to focus more on the counselor's strategyrelated utterances, and better represent lengthy conversations with session level features.