Empathetic Persuasion: Reinforcing Empathy and Persuasiveness in Dialogue Systems (2022)

Abstract: Persuasion is an intricate process involving empathetic connection between two individuals. Plain persuasive responses may make a conversation non-engaging. Even the most well-intended and reasoned persuasive conversations can fall through in the absence of empathetic connection between the speaker and listener. In this paper, we propose a novel task of incorporating empathy when generating persuasive responses. We develop an empathetic persuasive dialogue system by fine-tuning a maximum likelihood Estimation (MLE)-based language model in a reinforcement learning (RL) framework. To design feedbacks for our RL-agent, we define an effective and efficient reward function considering consistency, repetitiveness, emotion and persuasion rewards to ensure consistency, non-repetitiveness, empathy and persuasiveness in the generated responses. Due to lack of emotion annotated persuasive data, we first annotate the existing Persuaion For Good dataset with emotions, then build transformer based classifiers to provide emotion based feedbacks to our RL agent. Experimental results confirm that our proposed model increases the rate of generating persuasive responses as compared to the available state-of-the-art dialogue models while making the dialogues empathetically more engaging and retaining the language quality in responses.

## introduction while conversing with persuasive dialogue agents, on top of fluent and meaningful response generation, a high quality conversation is often derived by understanding and acknowledging implied feelings towards the conversing partner. people are more likely to engage in the conversation when they are motivated with empathetic responses. these persuasive responses can be associated with differ-ent emotions in consonance with the way people perceive and think about the world. for instance, in figure 1 , while the strike-through response is persuasive, the green box response may be more engaging, as it connects with the end-user and acknowledges the underlying emotion of caring. in this work, we investigate different generic and task specific rewards to reinforce a dialogue agent to generate fluent, persuasive and empathetic responses. in recent studies on personalized conversational agents @xcite @xcite , it is suggested that adopting different human oriented chatbot identities or conversational strategies can significantly affect the responses of users and make the conversations more engaging. these dialogue agents greatly improved the user-targeted personalization. for instance, @xcite include user sentiment to make an effective useradaptive system. @xcite takes both finegrained token-level and coarse-grained sentencelevel emotions to generate the responses. @xcite designed different rewards to reinforce politeness in a dialogue agent's responses. but, there is a subtle dependency between the different personalization techniques, such as empathy, sentiment and persuasion which can be used to generate better human-like responses. therefore, we focus to incorporate emotions to generate more engaging and persuasive utterances. due to the paucity of available data and dynamic nature of attitude and emotions of users in an ongoing dialogue, it is a hard task to model a personalized dialogue agent in a supervised learning (sl) framework which can generalize to different users in different situations. these mle-loss based models tend to suffer from exposure bias. therefore, lately researchers have focused on rl to fine-tune these models due to its ability to learn from user interactions and improve based on user's feedbacks in the form of rewards @xcite @xcite @xcite ). an rl based dialogue agent treats dialog planning as a sequential decision problem and focuses on long-term rewards to decide the next action which helps in enhancing the performance compared to the earlier systems @xcite . recently, there had been an effort made by @xcite to refine an mle-loss based language model without user simulators to generate persuasive responses. they focused to penalize repetitive and inconsistent utterance generation when persuading a persuadee. our current work focuses on incorporating emotions to engage the end users empathetically as well as to persuade them for donation. we first design a reward function consisting of generic rewards i.e. consistency and repetitiveness, and the task specific rewards i.e. emotion and persuasiveness to explicitly assess the quality of a generated response as per consistency, repetition, emotion and persuasion. we then train a policy via rl to maximize the score given by our reward function. the policy generates a response at each turn, and is updated using the proximal policy optimisation (ppo) algorithm @xcite based on the reward assigned to the entire generated response by the defined reward function . as there is no relevant data that provides both emotion and persuasion, we extend the existing persuasionforgood @xcite dataset to infuse empathy in the form of emotions. we assess the adequacy, fluency, empathy and persuasiveness of the generated responses from our rl-based model using both automatic as well human evaluation metrices. our core contributions are four-fold: historically, there had been attempts made to model persuasions. petty and cacioppo's elaboration likelihood model (elm) @xcite argues that a person's persuasion depends on the varying degrees of thoughts of processing information and persuasive context. friestad and wright's persuasion knowledge model (pkm) suggests that there is a inter-relationship between scientific persuasion knowledge and everyday persuasion knowledge @xcite . further, @xcite suggests that incorporation of personal factors with the persuasive information can enhance individual's motivation towards persuasive messages. recently, due to the increasing need for social chatbots, modelling empathy and persuasion has attracted much attention in the community. @xcite have proposed a empatheticdia-logues dataset to generate empathetic dialogues grounded in emotional situations. to recognize user emotions and generate empathetic responses, @xcite developed an end-to-end dialogue system, caire. @xcite predicted the politeness of utterances in goal-oriented conversations. while @xcite modelled argument sequences in social media to predict the persuasiveness, the research reported in @xcite identified different persuasion strategies using a hierarchical neural network. @xcite proposed a multi-turn persua-sionforgood dataset annotated with different persuasion strategies to model the persuasion classification. using the same dataset, @xcite randomly assigned 790 participants to different conditions to conduct an online study that whether they can be persuaded by a chatbot for charity donation or not. @xcite considered personality traits in single-turn persuasion dialogues and found that personality factors such as emotional arguments on social and political issues can affect belief change, with conscientious, and can convince more people. recently, @xcite trained a mle loss based language model while @xcite fine-tuned this mle loss based model in an rlframework to generate persuasive responses. but, these research works focused on generating persuasive responses alone, whereas the persuasion, in itself, covers a vast domain space with different end-user attitudes. further, a persuasive utterance cannot ensure engagement of user in an ongoing dialogue unless the user is connected emotionally with the cause s/he is persuaded for. therefore, our work focuses here on the stylistic and engaging dialogue generation by incorporating empathy with persuasion. to the best of our knowledge, there has been no prior research that incorporated emotions for persuasive dialogue generation. ## datasets and experiments we experiment and analyze to what extent our rlbased fine-tuning improves the persuasive response generation through both automatic and human evaluation. first, the datasets used in our experiments are introduced (section 4.1). then the implementation details for the proposed rl-based system are provided in section 4.2. due to space restrictions, the implementation details of emotion and persuasion strategy classifier are given in the section a.1 of the appendix. finally, the details of automatic and human evaluation metrics are provided in section 4.3. ## discussion we present an example of user's interaction with our proposed rl-emo-per and ardm models in our proposed approach has certain limitations. speaker rl-emo-per model generated utterances (n-@xmath3, @xmath4, @xmath5, @xmath6, @xmath7, lets talk about some of the issues we face on a daily basis. how are you? user alright, i am doing good. agent do you think there are people who have been out of touch with the world for too long? user yes i think so, we are so involved in ourselves. agent you are right, i know. i feel like it has become so important to me to help others and to be a part of the solution. what do you think about that? user i think it is very important to help people in need whenever one can. agent do you like children? user i do like children. my sister has three children and i love to play with them. agent have you ever heard of the organization save the children? user i have not heard about them. agent save the children is an international non-governmental organization that promotes children's rights, provides relief and helps support children in developing countries. user oh they are doing god's work. which countries do they operate in? agent oh yes! they are doing all over the world. the amount of children dying every day due to lack of access to health care is staggering. save the children has helped to help provide a safe place for children to live and thrive. user do they operate here in india as well? agent yes, they are active in the us, too. user oh very well. agent so what are you going to do today? user i am not sure if i will donate. i need more information about the charity. agent the research team will collect all donations and send it to save the children. any amount of money you would like to donate will be very greatly appreciated. user how much can i donate? agent i'm thinking @xmath01 dollar. agent thank you so much, so much appreciated! user you are welcome agent your welcome, i hope you have a wonderful day! user thank you for the information. you too enjoy your day. speaker ardm model generated utterance (n-@xmath8, @xmath9, @xmath10, @xmath11, well i'm just going to hit the survey link user okay agent you can choose any donation amount from @xmath12). user what is this donation for? agent to help the children. user thats a very good cause agent it's a very noble cause user can you tell me more about the charity? agent its called save the children and it help kids in war zones. user how much can i donate? agent i am willing to donate @xmath20.5 dollars. agent ok cool user thank you. agent you are very kind user thanks agent you are very kind sometimes our model generates out of the context entities, such as in reply to 'do they operate here in india as well?', the model responds with 'yes, they are active in the us, too'. it can be due to the fact that defined reward function can not possibly cover the crucial aspects of an ideal conversation due to the lack of world knowledge present in the model. ## conclusion development of persuasive dialogue agents to generate empathetic responses is still in its nascent stage due to the lack of modelling the changing attitudes of individuals. further, generative models only with mle loss may lead to exposure bias and tend to generate generic responses. therefore, to connect with end-users empathetically and generate goal oriented-responses, we propose here an rl-based dialogue generation framework adopting ppo method to fine-tune the model. to force the agent to generate more empathetic and persuasive responses, we define an efficient and effective reward function considering two generic rewards, viz. consistency and repetitiveness and two taskspecific rewards i.e. emotion reward -which forces the agent towards empathetic responses and persuasive reward -which forces the agent to generate persuasive responses. automatic and human evaluation results demonstrate that by just adding extra reward of emotion, our model is able to achieve state-of-the-art result in a complex task like persuasion, and generate consistent, non-repetitive, empathetic and persuasive responses 4 . in future, we would like to model persuasion in healthcare domain considering factors, such as effectiveness (providing evidence-based persuasions to the needed) and safety (avoiding harm to people for whom the persuasion is intended).