CDL : Curriculum Dual Learning for Emotion-Controllable Response Generation (2020)

Abstract: Emotion-controllable response generation is an attractive and valuable task that aims to make open-domain conversations more empathetic and engaging. Existing methods mainly enhance the emotion expression by adding regularization terms to standard cross-entropy loss and thus influence the training process. However, due to the lack of further consideration of content consistency, the common problem of response generation tasks, safe response, is intensified. Besides, query emotions that can help model the relationship between query and response are simply ignored in previous models, which would further hurt the coherence. To alleviate these problems, we propose a novel framework named Curriculum Dual Learning (CDL) which extends the emotion-controllable response generation to a dual task to generate emotional responses and emotional queries alternatively. CDL utilizes two rewards focusing on emotion and content to improve the duality. Additionally, it applies curriculum learning to gradually generate high-quality responses based on the difficulties of expressing various emotions. Experimental results show that CDL significantly outperforms the baselines in terms of coherence, diversity, and relation to emotion factors.

## introduction infusing emotions into dialogue systems can make conversational agents more human-like and benefit the interaction between human and machine @xcite @xcite . in some reallife scenarios, we need to customize and control the agent's emotion so that the agent can express a specific one. for example, in psychological counseling, the agent is supposed to express sadness to * yang feng is the corresponding author. recently, a framework called emotional chatting machine (ecm) @xcite was proposed to address the emotion factor in a controlled manner, which focuses on generating a response with a specific emotion (example 1 in intuitively, for a good pair of query and response, they should be in a tight relationship and have equal qualities. then, both the query-to-response mapping and response-to-query mapping would be easier and more natural. on the contrary, it is hard for a safe response to reach the original query through back-generation, neither on the content level nor the emotion level. at the same time, the difficulties of producing various emotions are different, especially in a noisy and uneven-quality dataset. therefore, we can evaluate the response based on the feedback from the backward process to improve the coherence @xcite @xcite and try to learn from easy to hard data to generate appropriate and emotion-rich responses. in this paper, we propose a new framework for emotion-controllable response generation named curriculum dual learning (cdl). we take the learning of response and query generation with emotions as a dual task, and use the duality to model the mutual relation between them. the forward and backward models are trained alternatively via reinforcement learning (rl). rewards designed here aim to encourage both emotion expression and content consistency. specifically, emotion expression can be either explicit (embodied in some obvious emotion words) or implicit (reflected by the organization of the entire sentence). for example, "i am happy to meet her again" is explicit with the word "happy", while "it seems like i have eaten the honey" is implicit, but the happiness can be felt when we consider the sentence as a whole. based on these features, we use the accuracy of emotion classification of sentences and the proportion of emotion words as feedbacks for explicit and implicit emotions, respectively. for content consistency, we apply the reconstruction probability as the measurement of coherence (section 3.1). furthermore, in order to better utilize samples of multiple emotions from the noisy and uneven-quality dataset, we incorporate the curriculum learning (section 3.2) into our dual learning framework (section 3.3). experimental results on both automatic and human evaluations show that for a given query and an emotion category, our cdl can successfully express desired emotion as well as keep the response informative and coherent to the query. ## background for emotion-controllable response generation, given a query q and an emotion category e r , the goal is to generate a response r that is not only meaningful, but also in accordance with the desired emotion. emotional chatting machine (ecm) @xcite addresses the emotion factor using three new mechanisms: emotion category embedding, internal memory, and external memory. specifically, 1) emotion category embedding models the high-level abstraction of emotion expression by embedding emotion categories, and concatenates corresponding embedding to the input at each decoding step. 2) internal memory captures the change of implicit internal emotion states with read and write gates, 3) external memory applies an external emotion vocabulary to express emotion explicitly, and finally assigns different generation probabilities to emotion and generic words. the loss function on one training sample (q, r) (@xmath0, q 2 , ..., q n , @xmath1, r 2 , ..., r m ) is defined as: ## cdl for emotion-controllable response generation since our cdl method is a combination of dual learning (dl) and curriculum learning (cl), we first present the main components of dl, including states, actions, policy and reward, then introduce the plausibility of curriculum learning. finally, we describe the training algorithm of cdl. ## experiments in this section, we conduct experiments to evaluate our proposed method. we first introduce some empirical settings, including dataset, hyperparameters, baselines, and evaluation measures. then we illustrate our results under both automatic and human evaluations. finally, we give out some cases generated by different models and do further analyses over our method. ## related work responses generated by traditional open-domain dialogue systems are usually safe and generic. to produce diverse and informative responses, researchers tried to either import latent variables for model construction @xcite @xcite or utilize some extra knowledge, e.g., sentence types, personas, emotions, documents and knowledge triples/graphs @xcite @xcite @xcite . in this paper, we mainly touch on two branches of research: emotional response generation and dual learning in nlp. ## conclusion in this paper, we propose a new framework curriculum dual learning (cdl) for generating emotional responses in a controlled manner. since existing methods in this field only focus on the emotion expression of target label but fail to consider the emotion of queries, the safe response problem deteriorates and hurts the content consistency. cdl utilizes two kinds of rewards to enhance emotion and content simultaneously via dual learning. besides, with the support of curriculum learning, it can be more efficient. experimental results show that cdl can generate fluent, coherent, informative as well as emotional responses.