Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training (2020)

Abstract: Generative dialogue models currently suffer from a number of problems which standard maximum likelihood training does not address. They tend to produce generations that (i) rely too much on copying from the context, (ii) contain repetitions within utterances, (iii) overuse frequent words, and (iv) at a deeper level, contain logical flaws. In this work we show how all of these problems can be addressed by extending the recently introduced unlikelihood loss (Welleck et al., 2019) to these cases. We show that appropriate loss functions which regularize generated outputs to match human distributions are effective for the first three issues. For the last important general issue, we show applying unlikelihood to collected data of what a model should not do is effective for improving logical consistency, potentially paving the way to generative models with greater reasoning ability. We demonstrate the efficacy of our approach across several dialogue tasks.

## introduction open-ended tasks such as dialogue reveal a number of issues with current neural text generation methods. in more strongly grounded tasks such as machine translation and image captioning, current encoder-decoder architectures provide strong performance, where mostly word-level decisions are often taken correctly by the model. however, critical failings are exposed in less constrained generation: reliance on repetitive copying and overuse of frequent words, and an inability to maintain logical coherence. the former shows the learning objective is faulty in that it cannot match simple statistics of the training data, while the latter touches more to the heart of artificial intelligence: work done while at facebook ai research (fair). these models do not understand what they are saying. for example, figure 1 shows how the 345mparameter gpt2 model @xcite can give high probability to contradictory generations. in this work, we show how the recently introduced unlikelihood objective @xcite can be generalized to remedy these problems. unlikelihood is a technique developed for removal of repetition in language model completions, and works by adding an extra term to the objective that forces repetitions to have low probability, alleviating the degenerative problems highlighted in @xcite . in fact, unlikelihood can be seen as a much more general framework, as we will see. we first generalize unlikelihood to a different domain: dialogue, where we measure statistics of the training distribution in terms of contextual copies, within-utterance repeats, and vocabulary usage. we then develop loss functions that control these statistics, providing improved metrics on several tasks. secondly, we show how the same tools can be used to address deeper semantic issues in such models. by leveraging existing natural language inference (nli) data @xcite as supervision against poor quality generations, we train models that assign low probability to generating incoherent and contradictory text. overall, our approach yields more consistent dialogue models across several axes, and provides a promising framework for further advances. code and pre-trained models will be made available. † 2 dialogue unlikelihood training dialogue generation dialogue generation consists in predicting an utterance @xmath0, . . . , y |y| ) given a context @xmath1, . . . , s k , u 1 , . . . , u t } that consists of initial context sentences s 1:k (e.g., scenario, knowledge, personas, etc.) followed by dialogue history utterances u 1:t from speakers who take consecutive turns. likelihood training given a dataset @xmath2, y (i) )} derived from a collection of humanhuman interactions, the standard approach to generative training for dialogue tasks is maximum likelihood estimation (mle), that minimizes: ## related work our work provides new applications of unlikelihood training @xcite , showing that unlikelihood offers a general framework for improving generative models, and in particular dialogue models. outside of that work, the use of negative training in dialogue retrieval, rather than generation, has been previously extensively studied, see e.g. @xcite . in the area of generative dialogue, a number of works have focused on improving the standard likelihood training approach. closer to our work is that of @xcite which developed the approach of negative training to prevent generic and malicious responses in dialogue models. in terms of improving repetition and specificity, a recent alternative approach is that of control @xcite @xcite . nucleus sampling @xcite can help to remove generic or repetitive utterances at the expense of accuracy, but was shown to be inferior to beam blocking, which in turn was shown to be inferior to unlikelihood in @xcite . in terms of dialogue coherence, @xcite showed that retrieval, but not generative models, could be improved with nli as a rescorer, while @xcite multi-tasked with nli. the work of @xcite has also studied improving narrative flow with a discriminative rescorer, but in that case for generated language. in our work, the improvements are tightly integrated into the training of the model itself. ## experiments in all of our experiments we employ a large pre-trained seq2seq transformer @xcite as our base model, which we then fine-tune for particular tasks with the objectives outlined in section 2 and specified in each experiment below. following previous work @xcite , we pre-train our model on dialogue data, using a previously existing reddit dataset extracted and obtained by a third party and made available on pushshift.io, training to generate a comment conditioned on the full thread leading up to the comment, spanning ∼ 2200m training examples. our transformer model consists of an 8 layer encoder, 8 layer decoder with 512-dimensional embeddings and 16 attention heads, and is based on the parlai implementation of @xcite . the model was trained with a batch size of 3072 sequences for approximately 3m updates using a learning rate of 5e-4, and an inverse square root scheduler. this pre-training took approximately two weeks using 64 nvidia v100s. ## conclusion generating consistent and coherent human-like dialogue is a core goal of natural language research. we studied several aspects that contribute to that goal, defined metrics to measure them, and proposed algorithms that improve them, mitigating some of the failings of maximum likelihood training, the current dominant approach. our method defines objective functions under the umbrella of unlikelihood: during training, we wish to make inconsistent dialogue unlikely by lowering the probability of such events occurring. this makes generative models repeat themselves less, copy the context less, and use more rare words from the vocabulary -closer to matching human statistics. further, utilizing supervised datasets with labeled coherent and incoherent utterances and applying unlikelihood yields measurably improved levels of coherence with respect to the aspect measured, in this case contradiction. future work could apply this same technique with other supervised data, e.g. correcting causal or commonsense reasoning errors @xcite . repetition model ppl f1 context label human --.160 .0006 mle baseline 8.3 .373 .582 .002 ul (context only) 8.8 .345 .270 .001 ul (label only) 8.3 .371 .645 .000 ul (context + label) 8.5 .358 .445 .003