Filtering Noisy Dialogue Corpora by Connectivity and Content Relatedness (2020)

Abstract: Large-scale dialogue datasets have recently become available for training neural dialogue agents. However, these datasets have been reported to contain a non-negligible number of unacceptable utterance pairs. In this paper, we propose a method for scoring the quality of utterance pairs in terms of their connectivity and relatedness. The proposed scoring method is designed based on findings widely shared in the dialogue and linguistics research communities. We demonstrate that it has a relatively good correlation with the human judgment of dialogue quality. Furthermore, the method is applied to filter out potentially unacceptable utterance pairs from a large-scale noisy dialogue corpus to ensure its quality. We experimentally confirm that training data filtered by the proposed method improves the quality of neural dialogue agents in response generation.

## introduction some million-scale datasets such as movie scripts and social media posts have become available in recent years for building neural dialogue agents @xcite . such large-scale datasets can be expected to improve the performance of dialogue response generation models based on deep neural networks (dnns) since the combination of dnns and large-scale training datasets has led to considerable performance improvement in many sentence generation tasks @xcite @xcite . in contrast to the quantity of the data, the quality of the data has often been problematic. for example, opensubtitles @xcite , the most widely used largescale english dialogue corpus, was constructed by collecting two consecutive lines of movie subtitles under the simplified assumption that one line of a movie subtitle is one utterance and the next line is the next utterance follow it. inevitably, this corpus includes unacceptable utterance pairs from the viewpoint of a conversational sequence, e.g., caused by scene switching or flashback. several previous studies have identified such flaws and reported that the corpus is noisy @xcite @xcite , where noisy refers to unacceptable utterance pairs in this context. figure 1 shows the result of our experimental investigation regarding the acceptability rate of the utterance pairs in the opensubtitles corpus. 2 it can be noticed from the figure that only half of the utterance pairs can be considered acceptable (i.e., were rated with score 5: strongly agree or 4: agree), and over 25% of utterance pairs are clearly unacceptable (i.e., were rated with score 1: strongly disagree or 2: disagree) from the human perspective. 3 with this situation, a straightforward research question arises, namely, can we further improve the performance of neural response generation models by ablating unacceptable utterance pairs from training data? to the best of our knowledge, no previous study has explicitly focused on this question. thus, the goal of this paper is to provide an answer to this question. furthermore, it is not clear whether and how one can effectively discover unacceptable utterance pairs within largescale training datasets. this study explores a way of constructing a scoring method for filtering noisy data filtering to improve the performance of response generation models. to achieve the set goals, we started with a review of previous arguments about the criteria for identifying appropriate utterances in dialogues and designed our scoring function that is consistent with reflects as much of the community's consensus as possible. in particular, the proposed scoring method estimates the quality of utterance pairs based on the following two aspects: (i) the connectivity between source and target utterances and (ii) their content relatedness (section 4). the contributions of this study are the following: then, an utterance pair can be denoted as we refer to (x, y). let d be a dataset that comprising a set of utterance pairs, @xmath0, y)}. then, the task can be formulated as ablating unacceptable utterance pairs from d to obtain a less noisy subset d ⊆ d, hereinafter referred to as filtered dataset. d can then be used to train response generation models. this paper refers to this process as noisy data filtering, where noisy means unacceptable utterance pairs in this context. furthermore, we establish a function s : @xmath1, y) ∈ d. ## background response generation using noisy data. the following two approaches are widely used to address the problem of dialogue response generation noisy dialogue corpora. according to the model approach, models are trained while handling noise at the same time. for example, @xcite proposed a method with a calibration framework and demonstrated its effectiveness on a chinese corpus. according to the data approach, training data are pre-processed with the aim of improving their quality before training models. in this study, we take the data approach in light of the success of noisy parallel corpus filtering in machine translation (mt). additionally, it has become a reasonable strategy to reduce the size of training data since enormous dialogue data has been available. @xcite 's method is most relevant to our study in that it cleanses dialogue corpora. however, the main goal of their method is to eliminate generic, or boring, responses, whereas the goal of the method proposed here is to eliminate unacceptable utterance pairs. this difference in goals leads to the essential difference in filtering strategies. effectiveness of filtering noisy data in neural machine translation. researchers in the field of neural machine translation (nmt) have recognized that collecting high-quality training data to be equally or even more important than exploring sophisticated model architectures @xcite @xcite . techniques used in neural response generation and nmt are nearly identical; e.g., sequenceto-sequence models @xcite and transformers @xcite are often used as base model architectures. we hypothesize that high-quality filtered dialogue data can also improve the performance of dialogue response generators. however, the straightforward application of methods proposed for filtering noisy data in nmt may not work well due to the different nature of nmt and neural response generation tasks. in particular, mt data have one-to-one (ignoring paraphrases) correspondence in source and target sentences, whereas dialogues have many-to-many mappings @xcite . the experiments presented in this paper provide an answer to whether nmt filtering methods can perform well in dialogue response generation. ## requirements to utterance pairs in this section, we investigate the requirements that should be satisfied by an acceptable utterance pair. ## proposed method as per the discussion in the previous section, each acceptable utterance pair should satisfy the following criteria: ## experiments: response generation this section reports on the effectiveness of the proposed method for filtering noisy data in neural response generation. ## multilingual availability while the proposed method s c+r was tested on an english corpus, it can potentially work for other languages as well. to demonstrate this, we selected japanese dialogue data as another case study. 16 the linguistic phenomena in japanese are quite different from those in english, thus making this japanese dataset. we prepare the japanese dialogue data from japanese opensubtitles @xcite containing roughly 3m lines. we obtain 1,893,477 utterance pairs as our training data, which excludes our test and validation data. foot_14 ## relationship with evaluation metric the proposed method s c+r maps an utterance pair to a score (scalar value) in terms of the quality of dialogue. that is, formally, our method is similar to the reference-free automatic evaluation metrics for dialogue agents; both of them evaluate the response given an input utterance and also map into a score. recently, the novel reference-free metrics for evaluating generated responses such as usr @xcite or maude @xcite ware developed. while it is possible to use them as a scoring method for filtering noisy data, in theory, there are some concerns with applying them in practice. one is the difference of the data of interest; since evaluation metrics are intended for responses generated as dialogue, i.e., somewhat valid dialogue data, it is unclear whether they also work for apparently noisy data. another one is the difference of desired properties; evaluation metrics need to be sensitive to "how good is it?" while the filtering requires to detect "is it a dialogue?" with high accuracy. it would be interesting to investigate the effectiveness of reference-free metrics for noisy dialogue data filtering tasks, and vice versa. we leave these investigations for future work. in contrast, reference-based metrics require a reference response (i.e., ground truth) when they calculate scores; such metrics include the traditional overlap-based bleu, rouge, meteor, embedding-based metrics @xcite , and neural network-based ruber @xcite and adem @xcite thus, these methods cannot straightforwardly be considered as alternatives to the proposed method, which aims at filtering. ## conclusion in light of the success of noisy corpus filtering in neural machine translation, we attempted to filter out unacceptable utterance pairs from large dialogue corpora in an unsupervised manner. the proposed scoring method estimates the quality of utterance pairs by focusing on the two crucial aspects of dialogue, namely, the connectivity and content relatedness of utterance pairs. we demonstrated that our scoring method has a higher correlation with human judgment than recently proposed methods. furthermore, we provided empirical evidence that our method improves the performance of a response generation model by removing unacceptable utterance pairs from its training data. we hope that this study will facilitate discussions in the dialogue response generation community regarding the issue of filtering noisy corpora.