Negative Training for Neural Dialogue Response Generation (2020)

Abstract: Although deep learning models have brought tremendous advancements to the field of open-domain dialogue response generation, recent research results have revealed that the trained models have undesirable generation behaviors, such as malicious responses and generic (boring) responses. In this work, we propose a framework named “Negative Training” to minimize such behaviors. Given a trained model, the framework will first find generated samples that exhibit the undesirable behavior, and then use them to feed negative training signals for fine-tuning the model. Our experiments show that negative training can significantly reduce the hit rate of malicious responses, or discourage frequent responses and improve response diversity.

## introduction end-to-end dialogue response generation can be formulated as a sequence-to-sequence (seq2seq) task: given a dialogue context, the model is asked to generate a high-quality response. in recent years, deep learning models, especially seq2seq language generation models @xcite , have brought significant progress to the field of dialogue response generation. however, recent research has revealed undesirable behaviors of seq2seq models that are side effects of standard maximum likelihood estimation (mle) training, such as the generic (boring) response problem @xcite , vulnerability to adversarial attacks @xcite , and the malicious (egregious) response problem @xcite . in this work, we propose and explore the negative training framework to correct unwanted behaviors of a dialogue response generator. during negative training, we first find or identify input-output pairs for a trained seq2seq model that exhibit some undesirable generation behavior, treat them as "bad examples," and use them to feed negative training signals to the model. correspondingly, we regard the training data as "good examples" and standard mle training as "positive training". the idea of negative training is inspired from the way parents might teach their children to use language by incorporating both positive and negative training signals. for example, when teaching children how to use "love" and "hate", in addition to using positive examples like "i love apples but i hate bananas", they might also point out that saying "i hate you" to someone is considered impolite. in this work, negative training is used to address the malicious response problem and the frequent response problem (to be described in section 3.2 and 3.3) in open-domain dialogue response generation. in our experiments, we show that negative training can significantly reduce the hit rate for malicious responses, or discourage frequent responses and greatly improve response diversity. ## model formulation in this work we adopt recurrent neural network (rnn) based encoder-decoder seq2seq models @xcite @xcite , which are widely used in nlp applications like dialogue response generation @xcite , machine translation @xcite , etc. we use @xmath0, x 2 , ..., x n } to denote onehot vector representations of the input sequence, which serves as context or history information (e.g. the previous utterance), @xmath1, y 2 , ..., y m } 1 to denote scalar indices of the corresponding reference target sequence, and v as the vocabulary. we use θ to represent the parameters for the seq2seq model, and p θ (y|x) as the model's generative distribution. on the encoder side, every x t will be first mapped into its corresponding word embedding x emb t . then {x emb t } are input to a long-short term memory (lstm) @xcite rnn to get a sequence of latent representations {h enc t } foot_1 . for the decoder, at time t, similarly y t is first mapped to y emb t . then a context vector c t , which is supposed to capture useful latent information of the input sequence, needs to be constructed. we adopt the "attention" mechanism for context vector construction: first an attention mask vector a t (which is a distribution) on the input sequence is calculated to decide which part to focus on, then the mask is applied to the latent vectors to construct c t : c @xmath2. we use the formulation of the "general" type of global attention, described in @xcite , to calculate the mask. during baseline training, standard mle training with stochastic gradient descent (sgd) is used to minimize the negative log-likelihood (nll) of the reference target sentence given the input sentence in the data: we consider two popular ways of decoding (generating) a sentence given an input: greedy decoding and sampling. in practice for dialogue response generation, greedy decoding will provide stable and reproducible outputs, but is severely affected by the generic response problem. sampling will provide more diverse but less predictable responses, and thus give rise to the malicious response problem. ## experiments we conduct experiments on three publicly available conversational dialogue data-sets: ubuntu, switchboard, and opensubtitles. to save space, descriptions of the data-sets are provided in appendix b. ## related works the malicious response problem and the gibbsenum algorithm to find trigger inputs @xcite originates from a large body of work on adversarial attacks for deep learning models, with continuous input space (e.g. image classification) @xcite , or discrete input space (e.g. sentence classification, or data-set tie baseline neg-train ubuntu 64.6% 14.0% 21.3% switchboard 45.1% 18.3% 36.4% opensubtitles 58.3% 19.0% 22.6% the malicious target list used in this work is very similar to the one used in @xcite . we propose to add a test target list to test the generalization of negative training. further, we show that the training list can be effectively augmented by utilizing a paraphrase model. in this work, we propose a definition for the frequent response problem, as a sub-problem of the generic response problem @xcite . much research work has devoted to alleviate the generic response problem in end-to-end dialogue response generation, @xcite use the maximal mutual information (mmi) objective, and propose to utilize an auxiliary lm to penalize the generic response during decoding. closely related to this work, sophisticated training frameworks based on gan @xcite have also been shown to be effective, where techniques such as variational information maximization or reward for every generation step (regs) are proposed to improve gan training. however, in our experiments it is shown that a vanilla gan approach gives unsatisfactory results. whether negative training foot_9 is complementary to these frameworks is worth investigating in future work. finally, note that the concept of negative training in this work is very different to the negative samples in word2vec training @xcite . the negative samples in word2vec training are used to prevent the training from being trivial, and is usually chosen randomly. in this work, the negative samples are carefully chosen to exhibit some particular undesirable behavior of the model, and is then used to correct such behavior. ## conclusion in this work, we propose the negative training framework to correct undesirable behaviors of a trained neural dialogue response generator. the algorithm involves two major steps, first input-output pairs that exhibit bad behavior are identified, and then are used for fine-tuning the model as negative training examples. we also show that negative training can be derived from an overall objective (eq. ( 2 )) to minimize the expected risk of undesirable behaviors. in our experiments, we apply negative training to the malicious response problem and the frequent response problem and get significant improvement for both problems.