Multi-Referenced Training for Dialogue Response Generation (2021)

Abstract: In open-domain dialogue response generation, a dialogue context can be continued with diverse responses, and the dialogue models should capture such one-to-many relations. In this work, we first analyze the training objective of dialogue models from the view of Kullback-Leibler divergence (KLD) and show that the gap between the real world probability distribution and the single-referenced data’s probability distribution prevents the model from learning the one-to-many relations efficiently. Then we explore approaches to multi-referenced training in two aspects. Data-wise, we generate diverse pseudo references from a powerful pretrained model to build multi-referenced data that provides a better approximation of the real-world distribution. Model-wise, we propose to equip variational models with an expressive prior, named linear Gaussian model (LGM). Experimental results of automated evaluation and human evaluation show that the methods yield significant improvements over baselines.

## introduction open-domain dialogue modeling has been formulated as a seq2seq problem since @xcite and @xcite borrowed machine translation (mt) techniques @xcite to build dialogue systems, where a model learns to map from one context to one response. in mt, one-to-one mapping is a reasonable assumption since an mt output is highly constrained by its input. though we may use a variety of expressions to translate the same input sentence, these different translations still highly overlap with each other lexically and semantically (see the translation example in figure 1 ), and learning from one output reference is often sufficient for training a good mt system @xcite . in dialogues, however, the same input can be continued with multiple diverse outputs which are different in both the used lexicons and the expressed semantic meanings (see the dialogue example in figure 1 ). learning from barely one output reference ignores the possibility of responding with other valid outputs and is thus insufficient for building a good dialogue system. the current dialogue modeling paradigm is largely derived from mt research, and it trains dialogue models with one output reference given each input. in this paper, we will investigate why single-referenced training harms our dialogue models and how to apply multi-referenced training. ## why multi-referenced training matters? in the training of a response generation model, we expect to model the real probability distribution p (y |x) with model probability distribution p θ (y |x) for each context x, where θ is the model parameters. in most scenarios, however, we can only rely on a data set @xmath0, y emprical training objective as a result, we optimize a model to match the model probability distribution and the data probability distribution. from the view of kullback-leibler divergence (kld), we can see it as to minimize d kl (p d ||p θ ): the resulting objective is the negative log likelihood (nll) loss function commonly used in the implementation of dialogue models. ## proposal: enhancing data for multi-referenced training to enhance the training data, we try to close the gap between p d (y |x) and p (y |x). since all probability mass is on a single response in p d (y |x), the gap can be closed by assigning some mass to other valid responses. we use a finetuned gpt2 md to generate n hypotheses as valid responses, and let the probability mass to be assigned to them uniformly. it results in p φ (y |x) wherein n elements have 1 n probability. the new training objective is: training with the new loss function can be achieved by directly replacing the ground-truth responses in the training data with the hypotheses. 4 sequences generated by beam search often highly overlap both lexically and semantically @xcite . therefore, we use nucleus sampling with top probability 0.95 @xcite to generate 100 hypotheses as for each context in the training data. ## proposal: enhancing model for multi-referenced training we have previously seen the hred's performance gain when we increase the number of hypotheses from 1 to 20, but it starts to degrade when we further increase the number to 100. a conjecture is that the model's capacity is insufficient to learn too complex input-output relations. ## human evaluation besides automated evaluation, we also conduct human evaluation to provide a more accurate assessment of model performance. we sample 100 dialogues randomly from the test data and generate responses using 3 models (hred, vhred gmm5 , vhred lgm5 ) trained on 2 types of data (the 1-gt data and the 100-hypotheses data). we ask 4 amazon mturk human workers to annotate the sensibleness and the specificity of the 600 (context, response) pairs. the collected data reach good inter-rater agreement (krippendorff's α > 0.6). then we calculate the average of the two metrics (ssa, @xcite as introduced in section 4.3. the results of the human evaluation are given in ## analysis 8.1 combining ground truth and hypotheses one issue that readers may be concerned about is whether it is better to combine ground truth with hypotheses than to use them separately. we take the vhred lgm20 as an example and conduct experiments using mixed training data. as shown in ## conclusion in this work, we analyzed the training objective of dialogue response generation models from the view of distribution distance as measured by kullback-leibler divergence. the analysis showed that single-referenced dialogue data cannot characterize the one-to-many feature of open-domain dialogues and that multi-referenced training is necessary. towards multi-referenced training, we first proposed to enhance the training data by replacing every single reference with multiple hypotheses generated by a finetuned gpt2, which provided us with a better approximation of the real data distribution. secondly, we proposed to equip variational dialogue models with an expressive prior, named linear gaussian model (lgm), to capture the oneto-many relations. the automated and human eval-dialogue example #422 floor context utterance a i'm so hungry. shall we go eat now, rick? b sure. where do you want to go? are you in the mood for anything in particular? a how about some dumplings? i just can't get enough of them. b [to be predicted] k response utterance 4 tables tables tables there any any any any pale, medium rare. 0 ok. i don't think we have any soup at the moment. 8 i've heard that some dumplings are really good. but i don't know what to eat. 1 ok. i'll go to the restaurant.