Diversifying Neural Dialogue Generation via Negative Distillation (2022)

Abstract: Generative dialogue models suffer badly from the generic response problem, limiting their applications to a few toy scenarios. Recently, an interesting approach, namely negative training, has been proposed to alleviate this problem by reminding the model not to generate high-frequency responses during training. However, its performance is hindered by two issues, ignoring low-frequency but generic responses and bringing low-frequency but meaningless responses. In this paper, we propose a novel negative training paradigm, called negative distillation, to keep the model away from the undesirable generic responses while avoiding the above problems. First, we introduce a negative teacher model that can produce query-wise generic responses, and then the student model is required to maximize the distance with multi-level negative knowledge. Empirical results show that our method outperforms previous negative training methods significantly.

## introduction in the past few years, data-driven response generation @xcite @xcite has achieved impressive performance, drawing continuously increasing attention from academia and industry. conventionally, with the guidance of maximum likelihood estimation (mle), neural dialogue models are expected to maximize the probability of generating the corresponding reference given any query. unfortunately, due to the many-to-one phenomenon (see however, inspired by negative training @xcite , we argue that it is also necessary to tell the dialogue model what not to say. to alleviate the problem of generic responses, @xcite negatively updates the parameters when identifying the high-frequency responses. @xcite punishes the behaviors of generating repetitive or high-frequency tokens by using the unlikelihood objective @xcite . although the negative-training based methods enhance the diversity of responses, there still exists two drawbacks: first, they regard high-frequency tokens or utterances as negative candidates. however, the high-frequency response problem is only a sub-problem of the generic response problem @xcite . it means that the responses that are low-frequency but generic will escape from punishment. even worse, we have observed that some generic responses followed by a low-frequency but meaningless subsequence can avoid being identified as high-frequency, which inevitably sacrifices the fluency of responses (see analysis). second, these methods ignore the implicit negative knowledge in neural networks that characterizes negative candidates at multiple levels. we contend that it is more effective to conduct negative training with richer information (e.g., hierarchical representation). to tackle the above problems and further improve the diversity of responses, we propose a novel negative training paradigm called negative distillation (nd). conventional knowledge distil- specifically, we first collect a negative training set by using a filtering method called source entropy @xcite . this filtering method can retrieve all many-to-one cases of the raw dataset. note that the "one" is usually a generic response. then, we train a dialogue model on the above subset as the negative teacher. given queries, the negative teacher can provide a set of negative candidates (i.e., generic and dull responses) that the student is prone to generate, which avoids the first drawback mentioned before. therefore, the student obtains query-wise bad behaviors for negative distillation. to conduct the negative update holistically, we design two negative objectives, including soft unlikelihood loss on the prediction layer and reverse square error on the intermediate layer. in this way, the negative distillation fully exploits multilevel negative knowledge to force the student to generate non-generic responses. our contributions are summarized as follows: it constructs query-wise generic responses as the negative candidates. ## method in this section, we first introduce the negative teacher, then describe the negative distillation on the prediction layer and the intermediate layer, respectively, and finally present the progressive optimization objective. algorithm 1 shows the whole training details. ## related work diversity dialogue learning there are two lines of work for solving the generic response prob-lem: one line promotes the diversity from positive view, which is outside of our work. specially, previous work includes mmi @xcite , gan @xcite , cvae @xcite , bt @xcite , face @xcite , adalabel @xcite , @xcite , and nucleus sampling @xcite . the other line alleviates the generic response problem using negative training. he and glass (2020) regards frequent response problem as a sub-problem of the generic response problem and conduct negative update for the high-frequency responses during training. @xcite focuses on high-frequency tokens rather than tokens and punishes them by using the unlikelihood objective @xcite . both of them handle the generic response problem only from the angle of reducing frequency, thus can not capture all the features of generic replies. negative training for dialogue learning negative training for retrieval-based dialogue learning has been previously extensively studied @xcite , while we focus on the dialogue generation in this work. he and glass (2020) uses negative training to prevent generic and malicious responses in dialogue models. @xcite generalizes unlikelihood to dialogue generation for improving repetition, specificity and coherence. @xcite proposes implicit unlikelihood training to minimize repetition. our work proposes a new negative training paradigm aimed at improving the diversity of dialogue responses while avoiding the problem of poor consistency and fluency of previous work. ## conclusion we present a novel negative training paradigm to improve the diversity of dialogue responses. it formulates the conventional negative training as a knowledge distillation process, which is rarely explored before. the negative teacher can produce the corresponding generic and dull responses given any query, which naturally avoids problems that hinder previous negative training methods. besides, we further boost the performance of negative distillation by exploiting richer information, i.e., multilevel features. extensive experiments validate the superiority of our proposed method compared with prior negative training work. a limitation of our work is that we only focus on the generic response problem. for future work, we will extend the proposed negative distillation to handle other generation problems, such as inconsistency and lacking personas or emotions.