Facilitating Multi-turn Emotional Support Conversation with Positive Emotion Elicitation: A Reinforcement Learning Approach (2023)

Abstract: Emotional support conversation (ESC) aims to provide emotional support (ES) to improve one’s mental state. Existing works stay at fitting grounded responses and responding strategies (e.g., question ), which ignore the effect on ES and lack explicit goals to guide emotional positive transition. To this end, we introduce a new paradigm to formalize multi-turn ESC as a process of positive emotion elicitation. Addressing this task requires finely adjusting the elicitation intensity in ES as the conversation progresses while maintaining conversational goals like coherence. In this paper, we propose Supporter, a mixture-of-expert-based reinforcement learning model, and well design ES and dialogue coherence rewards to guide policy’s learning for responding. Experiments verify the superiority of Supporter in achieving positive emotion elicitation during responding while maintaining conversational goals including coherence.

## introduction emotional support (es) aims to reassure a person to recover from emotional distress and improve one's mental state @xcite . it is a manifestation of emotional intelligence in social interactions @xcite . endowing es into social dialogue systems for building helpful and trustful agents is an emerging trend @xcite . to achieve this goal, a typical practice is modeling empathy, which aims to perceive and understand the situation and feelings of others @xcite . yet, the empathetic conversation @xcite is inherently deficient in providing es as (1) lack of consideration of multi-turn conversation. just making empathetic responses in each single dialogue turn leads to ignoring the user's feedback and mental state changes in multi-turn my school was closed without any prior warning due to the pandemic. i understand you. i would also have been really yeah! i don't even know what is going to happen with our final. that is really upsetting and stressful. maybe you can talk to your parents or a close friend about this, they are also frustrated due to the same experience. yes, you are right! i am not the only one affected by the pandemic. come on! i believe that you will definitely be able to overcome the difficulties and find a solution! yeah, definitely! thanks for your support, i feel better now. figure 1 : a simplified multi-turn esc example between the user (left) and agent (right). the agent progressively adjusts the intensity of empathy and elicitation to achieve the goal of improving the user's mental state. to this end, we introduce multi-turn esc with positive emotion elicitation, a new paradigm aims to progressively empathize and elicit users to reach a better mental state through multi-turn conversation. addressing this task is challenging (an example is in figure 1 ): first, in a realistic multi-turn esc, the user's emotions often transit towards positive (e.g., the user's emotion starts with negative and ends with positive, i.e., "my school was closed" → "i feel better now") with fluctuation (e.g., the user's negative emotions in the first two turns gradually deepen, i.e., "my school was closed" → "i don't even know"), which requires the agent to equip with the mechanism dealing with complex situations to respond satisfactorily @xcite . second, for es, the es response requires a delicate balance between empathy and elicitation. only empathizing without eliciting falls into a negative emotional cycle, while the opposite setting brings a sense of distance in communication. they need to be progressively and purposefully adjusted in ongoing interactions, e.g., the agent expresses empathy of varying emotional polarity (@xmath0. third, for language expression, the es response purposefully elicits positive emotions but should not undermine general conversational goals like coherence. making an eliciting response that is out of the dialogue context, e.g., replacing "i understand you. i would ... happened to me." with "come on! i believe ... find a solution!", may cause users to resent and block useful feedback. in this paper, we propose supporter foot_0 to facilitate multi-turn emotional support conversation with positive emotion elicitation using a mixtureof-expert(moe) based reinforcement learning(rl). moe designs heuristic experts associated with specific tasks to learn diverse semantics by characterizing dialogue context, where: (1) to cope with the user's emotional fluctuation in the ongoing conversation, experts are devised as positive and negative experts as a whole; (2) to inspire es of responding, the emotion experts of moe are designed to predict the user's emotional states that are possibly transited to; (3) to inspire the expression of responding, the keyword experts of moe are designed to predict the keywords that maintain the dialogue coherence. with experts as candidates, our rl agent learns conversational semantic encoding policy and purposefully selects experts with expert selection policy for response generation. to achieve the goal of positive emotion elicitation during responding while maintaining conversational goals like coherence, we optimize policy by carefully constructing the rewards: (1) es rewards consider the conversation progress to dynamically adjust the elicitation intensity of positive emotion; (2) dialogue coherence rewards involve keyword-level and sentencelevel guides to finely maintain coherence. our contributions are summarized as follows: emotional support conversation to remedy above deficiencies, @xcite design esc for providing es in interactions. our work is related to existing works on esc but differs in task definition as we focus on enhancing the elicitation effect of positive emotion of responses instead of responding strategy prediction (e.g., question) and grounded response generation. although fusing knowledge @xcite and planning strategy @xcite are beneficial for wordoverlap metrics (e.g., bleu), we argue whether the gains serve to es is opaque and less convincing due to lacking corresponding evaluation mechanisms. ## preliminaries at the t-th turn of dialogue, given dialogue context c @xmath1, y 1 , . . . , x t-1 , y t-1 , x t }, our goal is to generate the response y t which serves to improve the user's mental state. to equip this ability, the response generation process should achieve specific goals related to es and language expression. ## methodology in figure 2 , our supporter takes dialogue context as input to construct state sequence, which is encoded by a dialogue encoder as the conversational semantic encoding policy. the mixture-of-expert associated with emotion and keyword prediction tasks characterize state semantics to yield action candidates of the expert selection policy, which are purposefully selected for inducing state update. we use the updated state to generate response and further optimize the policy by measuring how well the response reaches the goal of es and dialogue coherence with the well-designed parallel rewards. ## conclusions in this paper, we introduce a new paradigm to formalize multi-turn esc as a process of positive emotion elicitation and propose an moe-based rein-forcement learning model supporter with welldesigned es and dialogue coherence rewards. extensive experiments verify the superiority of our model in providing effective es for positive emotion elicitation while maintaining conversational goals including coherence. our work will facilitate future work to develop esc with positive emotion elicitation for improving the users' mental state.