PAL to Lend a Helping Hand: Towards Building an Emotion Adaptive Polite and Empathetic Counseling Conversational Agent (2023)

Abstract: The World Health Organization (WHO) has significantly emphasized the need for mental health care. The social stigma associated with mental illness prevents individuals from addressing their issues and getting assistance. In such a scenario, the relevance of online counseling has increased dramatically. The feelings and attitudes that a client and a counselor express towards each other result in a higher or lower counseling experience. A counselor should be friendly and gain clients’ trust to make them share their problems comfortably. Thus, it is essential for the counselor to adequately comprehend the client’s emotions and ensure client’s welfare, i.e. s/he should adapt and deal with the clients politely and empathetically to provide a pleasant, cordial and personalized experience. Motivated by this, in this work, we attempt to build a novel Polite and empAthetic counseLing conversational agent PAL to lay down the counseling support to substance addict and crime victims. To have client’s emotion-based polite and empathetic responses, two counseling datasets laying down the counseling support to substance addicts and crime victims are annotated. These annotated datasets are used to build PAL in a reinforcement learning framework. A novel reward function is formulated to ensure correct politeness and empathy preferences as per client’s emotions with naturalness and non-repetitiveness in responses. Thorough automatic and human evaluation showcase the usefulness and strength of the designed novel reward function. Our proposed system is scalable and can be easily modified with different modules of preference models as per need.

## introduction the global prevalence of mental health disorders is estimated to affect 20% of the world population @xcite . the covid-19 pandemic has further jeopardized the mental well-being of individuals @xcite . further, the limited availability of experts @xcite and the overwhelming healthcare burden @xcite have created a substantial demand for counseling services. counselors are faced with a diverse audience, including individuals struggling with substance addiction and crime victims, who often experience mental health challenges such as anxiety, depression, and post-traumatic stress @xcite @xcite . therefore, pertinent and timely counseling support is crucial. in light of this, it is imperative to explore the potential benefits that counseling could derive from the advancements in artificial intelligence (ai) and natural language processing (nlp). in order to ensure effective counseling, active listening, combined with accurate interpretation of the client's meaning and emotions is vital @xcite . this demonstrates care and understanding, fostering a sense of being heard and valued, hence, facilitating deeper self-exploration for the client. the essential elements of effective support include politeness and empathy @xcite @xcite . previous studies indicate that politeness plays a critical role in facilitating supportive interactions @xcite and aids in the disclosure of sensitive information @xcite . quantitative data supports the idea that empathetic interactions play a significant role in reducing issues in mental health support @xcite and are essential for building a strong therapeutic bond and connection @xcite . thus, a counseling conversational agent should adapt its support to users' mental state whilst being polite and empathetic towards them. in figure 1 , the agent's polite and empathetic response in the green box fosters higher user engagement and satisfaction compared to the counseling response in the red box. furthermore, the agent's response should align with the client's emotions -offering reassurance and compassion when they are sad and showing understanding and providing encouragement when they are afraid. this paper introduces a novel client's emotionadaptive polite and empathetic counseling dialogue system, named pal to create a comfortable, engaging and cordial atmosphere during counseling. for this, we create a novel counseling conversational dataset in english utilizing the recently released emoinhindi @xcite dataset and annotate it with defined emotion, politeness, and empathy labels. we also annotate the highquality and low-quality counseling conversations dataset @xcite with emotion, politeness and empathy label for our work. finally, to build our system, we first warm-start with a supervised fine-tuning (sft) of a language model on these datasets. then, this trained sft model is finetuned in a reinforcement learning (rl) framework using preference models to ensure preferences of utterance-emotion politeness (and empathy) consistency, adaptiveness, and correctness by designing six preference rewards. the key contributions and/or the distinguishing attributes of our current work can be summarized as: (i) prepared two large-scale counseling conversational datasets annotated with three distinct labels, viz. emotion, politeness and empathy; (ii) proposed pal -a novel emotion-adaptive polite and empathetic counseling conversational agent fine-tuned in an rl setting; (iii) designed a novel reward function consisting of six preference re-wards and two generic rewards to generate engaging, fluent and interactive client's emotion-adaptive polite and empathetic responses; (iv) trained six transformer-based robust preference models comprising of two generation models and four classifiers to compute preference rewards; (v) extensive automatic and human evaluation is performed with novel preference-specific and language qualityspecific metrics to demonstrate the effectiveness of the proposed system. ## related work recent research has focused on developing dialogue systems for online mental health support, investigating approaches such as contextspecific adaptation, response diversity @xcite @xcite @xcite , linguistic development of counselors @xcite , conversational engagement patterns @xcite , moderation analysis @xcite , therapeutic action detection @xcite , and cognitive restructuring identification @xcite . in our current work, we prioritize two crucial aspects of conversation, namely politeness and empathy, which are fundamental components of counseling @xcite @xcite . previous research has demonstrated that perceiving conversational agents as caring and polite can enhance empathy and encourage the disclosure of personal and sensitive information @xcite @xcite . for instance, @xcite and @xcite applied brown and levinson's politeness strategies @xcite to create dialogue templates for mental health "mood log" and develop a polite persuasive dialogue system, respectively. the agent's polite demeanor reflects a sense of care and personal investment, resembling a human companion. prior works @xcite advocate incorporating emotion as an extra input in generation frameworks. this can be achieved by including emotion either during decoding phase or using it as a reward to assist models in generating responses that align with the user's emotions. the role of empathy in therapy has been explored in @xcite , while studies in @xcite have investigated methods to induce empathy in mental health support conversations. further, several research studies have endeavoured to understand and create computational techniques for recognizing empathy in various contexts, such as face-to-face therapy @xcite ) and text-based peerto-peer support systems @xcite . existing studies primarily focus on analyzing social media posts and activities to detect different mental health concerns and provide support to users. they often emphasize incorporating empathy into the agent's responses to instill motivation and hope in the user. in contrast, our current work aims to develop a counseling dialogue agent that can adapt its politeness and empathy based on the user's emotions. ## dataset we perform the experiments on two different counseling datasets, viz. english translation of @xcite dataset and high-quality and low-quality counseling conversations dataset @xcite . ## proposed methodology to build our proposed system, first, to foster natural language interaction between the counseling agent and the client, we get a warm-start by training a client's emotion-informed dialogue model (eidm) in a supervised learning framework. then, to generate the client's emotion-adaptive polite and distinct emotion class merged emotion class caring, sentimental caring hopeful hopeful proud, content content surprised surprised afraid, apprehensive, anxious afraid confident, prepared confident grateful proud trusting proud joyful, impressed joy disappointed, guilty, embarrassed, sad, lonely sad angry, annoyed, jealous angry a multi-turn dialogue with a client's and counseling agent's utterances can be represented as @xmath0, cl 0 , ..., ca i , cl i , ..., ca t -1 , cl t -1 }, where ca i and cl i give the counseling agent's and client's i th response out of t turns. let the client's current utterance with corresponding emotion, e @xmath1, .., e m } (m is the number of emotion classes), be represented as cl e @xmath2. further, let ρ ca and ρ cle representing agent's and client's utterances with corresponding emotions distributions, respectively. now for a given context, an emotioninformed dialogue model (emid) ρ θ (d) is trained by minimizing the kl-divergence @xcite between the predicted and ground truth utterances probability distributions. ρ θ (d) can be expressed as: ## results and analysis first, all six preference models results are shown i.e., two generation models and four classifiers. thereafter, the results of the proposed system pal are stated and compared with three baselines, viz. lm (gpt-2-medium), ardm, eidm and three variants of pal, pal-r (pal with zero reward), pal-gr (pal without generic rewards) and pal-pr (pal without preference rewards). for both the datasets -epe-eneih and epe-hlcc, tables 5 and 4 show the evaluation results of all the preference models. scores of ppl, bleu, met and r-2-f1 in ## conclusion and future direction in this work, we proposed an emotion-adaptive polite and empathetic counseling conversational agent pal to offer higher engagement in e-counseling sessions and resolve the issues faced by the clients. to this end, we prepared two counseling conversational datasets: epe-eneih and epe-hlcc annotated with emotion, politeness and empathy labels. these datasets are used to train pal in an rl framework. a novel reward function is designed to ensure the preferences of emotion-adaptive politeness and empathy in generated responses. our extensive automatic and human evaluation results conclude the requirement and effectiveness of designed rewards considering all three aspects -emotion, politeness and empathy over supervised fine-tuned based models. it is found that pal is able to efficiently generate engaging emotion-adaptive polite and empathetic responses. in future, we would like to incorporate domain and common-sense knowledge into the responses to facilitate more realistic and engaging counseling conversations.