Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweight (2020)

Abstract: Current state-of-the-art neural dialogue models learn from human conversations following the data-driven paradigm. As such, a reliable training corpus is the crux of building a robust and well-behaved dialogue model. However, due to the open-ended nature of human conversations, the quality of user-generated training data varies greatly, and effective training samples are typically insufficient while noisy samples frequently appear. This impedes the learning of those data-driven neural dialogue models. Therefore, effective dialogue learning requires not only more reliable learning samples, but also fewer noisy samples. In this paper, we propose a data manipulation framework to proactively reshape the data distribution towards reliable samples by augmenting and highlighting effective learning samples as well as reducing the effect of inefficient samples simultaneously. In particular, the data manipulation model selectively augments the training samples and assigns an importance weight to each instance to reform the training data. Note that, the proposed data manipulation framework is fully data-driven and learnable. It not only manipulates training samples to optimize the dialogue generation model, but also learns to increase its manipulation skills through gradient descent with validation samples. Extensive experiments show that our framework can improve the dialogue generation performance with respect to various automatic evaluation metrics and human judgments.

## introduction open-domain dialogue generation, due to its potential applications, is becoming ubiquitous in the community of natural language processing. current end-to-end neural dialogue generation models @xcite @xcite , 2017) are primarily built following the data-driven paradigm, that is, these models mimic the human conversations by training on the large-scale queryresponse pairs. as such, a reliable training corpus that exhibits high-quality conversations is the crux of building a robust and well-behaved dialogue model. unfortunately, owing to the subjectivity and open-ended nature of human conversations, the quality of the collected human-generated dialogues varies greatly @xcite , which hampers the effectiveness of data-driven dialogue models: 1) effective conversation samples are quite insufficient. to glean some insights on the data quality of dialogue corpus, we choose the queryrelatedness to take a glimpse of the data quality. in dialogue corpus, some conversations are quite coherent, where the queries and responses are wellcorrelated, while others are not. query-relatedness measures the semantic similarities between the query and its corresponding response in the embedding space and ranges from 0 to 1. when reviewing dailydialog @xcite , we find that only 12% conversation samples are of relatively high queryrelatedness scores (> 0.6). without adequate reliable training samples, the neural dialogue model is prone to converge to a sub-optimal point. 2) meanwhile, noisy and even meaningless conversa-tion samples frequently appear. as @xcite reported, "i don't know" appears in over 113k sentences in the training corpus opensubtitles @xcite . such kind of noisy conversation data prevails in neural dialogue model training, and vitally impedes the model learning. therefore, effective dialogue learning requires not only more reliable learning samples, but also fewer noisy samples. in this work, as illustrated in figure 1 , we propose a novel learnable data manipulation framework to proactively reshape the data distribution towards reliable samples by augmenting and highlighting effective learning samples as well as reducing the weights of inefficient samples simultaneously. specifically, to generate more effective data samples, the data manipulation model selectively augments the training samples in terms of both word level and sentence level, using masked language models such as bert @xcite and back-translation @xcite technique. to reduce the weights of inefficient samples from the original training samples and the augmented samples, the data manipulation model assigns an importance weight to each sample to adapt the sample effect on dialogue model training. it gives out higher importance weights to critical learning samples and lower weights to those inefficient samples. furthermore, different from most previous data augmentation or data weighting studies @xcite @xcite , which are unaware of the target model states during augmentation or weighting, our data manipulation framework not only manipulates training samples to optimize the dialogue generation model, but also learns to increase its manipulation skills through gradient descent with validation samples. we apply the proposed data manipulation framework on several state-of-the-art generation models with two real-life open-domain conversation datasets and compare with the recent data manipulation approaches in terms of 13 automatic evaluation metrics and human judgment. experiment results show that our data manipulation framework outperforms the baseline models over most of the metrics on both datasets. ## data manipulation for neural dialogue generation the proposed data manipulation framework tackles the problem of un-even quality data by inducing the model learning from more effective dialogue samples and reducing effects of those inefficient samples simultaneously. in particular, as illustrated in figure 2 , it manipulates and reshapes the data distribution for neural dialogue model learning in mainly three stages: first, each batch of training samples are selectively augmented to generate more variant samples; and then, all the samples, including the original samples and the augmented samples, are assigned with instance weights indicating their importance regarding current learning status; finally, the weighted samples are fed into the neural dialogue model to induce the model learning from more effective training instances. note that, although we describe the framework in three components for ease of understanding, in fact, the whole framework can be trained in an endto-end manner. as a result, the data manipulation network is capable of not only manipulating training samples to optimize the dialogue generation model, but also learning to increase its manipulation skills through gradient descent with validation samples. we first introduce the augmentation and weighting strategies for data manipulation in §2.1 and §2.2, and then describe how the neural dialogue generation model learns from the manipulated samples in §2.3. parameters estimation for the data manipulation model is elaborated in §2.4. ## related work existing approaches to improving neural dialogue generation models mainly target on building more powerful learning systems, using extra information such as conversation topics @xcite , persona profile @xcite , user emotions @xcite , or out-sourcing knowledge @xcite . another popular framework for dialogue generation is variational autoencoder @xcite @xcite , in which a latent variable is introduced to benefit the dialogue model with more diverse response generation. contrasted with previous researches, we investigate to improve the dialogue model from a different angle, i.e., adapting the training examples using data manipulation techniques. data augmentation is an effective way to im-prove the performance of neural models. to name a few, @xcite propose to generate more utterances by introducing noise to the decoding process. @xcite ; @xcite demonstrate that contextual augmentation using label-conditional language models helps to improve the neural networks classifier on text classification tasks. sennrich et al. (2016) boost neural machine translation models using back-translation. xie et al. (2017); andreas (2020) design manuallyspecified strategies for data augmentation. hou et al. (2018) utilize a sequence-to-sequence model to produce diverse utterances for language understanding. li et al. (2019); niu and bansal (2019) propose to generate sentences for dialogue augmentation. compared with previous augmentation approaches for dialogue generation, augmented sentences in our framework are selectively generated using the pretrained models and the augmentation process is additionally fine-tuned jointly with the training of dialogue generation. regarding data weighting, past methods @xcite @xcite @xcite have been proposed to manage the problem of training set biases or label noises. @xcite propose to enhance the retrieval-based dialog system with a weighting model. @xcite likewise design a matching network to calibrate the dialogue model training through instance weighting. @xcite investigate curriculum learning to adapt the instance effect on dialogue model training according to the sample complexity. whereas our proposed framework learns to reweight not only the original training examples but also the augmented examples. another differ-ence is that, we directly derive data weights based on their gradient directions on a validation set, instead of separately training a external weighting model. @xcite claim that high-entropy utterances in the training set lead to those boring generated responses and thus propose to ameliorate such issue by simply removing training instances with high entropy. although data filtering is a straightforward approach to alleviate the problem of noisy data, the informative training samples remain untouched and insufficient. whereas our method holds the promise of generating more valid training data and alleviating the negative noises in the meantime. note that either data augmentation or instance reweighting can be considered band-aid solution: simply augmenting all training data risks introducing more noisy conversations as such low-quality examples prevail in human-generated dialogues, whilst adapting the sample effect merely by instance reweighting is also suboptimal since effective training samples remain insufficient. the proposed learning-to-manipulate framework organically integrates these two schemes, which collectively fulfill the entire goal. ## conclusion in this work, we consider the automated data manipulation for open-domain dialogue systems. to induce the model learning from effective instances, we propose a learnable data manipulation model to augment effective training samples and reduce the weights of inefficient samples. the resulting data manipulation model is fully end-to-end and can be trained jointly with the dialogue generation model. experiments conducted on two public conversation datasets show that our proposed framework is able to boost the performance of existing dialogue systems. our learning-to-manipulate framework for neural dialogue generation is not limited to the elaborately designed manipulation skills in this paper. future work will investigate other data manipulation techniques (e.g., data synthesis), which can be further integrated to improve the performance.