Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining (2020)

Abstract: There is an increasing focus on model-based dialog evaluation metrics such as ADEM, RUBER, and the more recent BERT-based metrics. These models aim to assign a high score to all relevant responses and a low score to all irrelevant responses. Ideally, such models should be trained using multiple relevant and irrelevant responses for any given context. However, no such data is publicly available, and hence existing models are usually trained using a single relevant response and multiple randomly selected responses from other contexts (random negatives). To allow for better training and robust evaluation of model-based metrics, we introduce the DailyDialog++ dataset, consisting of (i) five relevant responses for each context and (ii) five adversarially crafted irrelevant responses for each context. Using this dataset, we first show that even in the presence of multiple correct references, n-gram based metrics and embedding based metrics do not perform well at separating relevant responses from even random negatives. While model-based metrics perform better than n-gram and embedding based metrics on random negatives, their performance drops substantially when evaluated on adversarial examples. To check if large scale pretraining could help, we propose a new BERT-based evaluation metric called DEB, which is pretrained on 727M Reddit conversations and then finetuned on our dataset. DEB significantly outperforms existing models, showing better correlation with human judgments and better performance on random negatives (88.27% accuracy). However, its performance again drops substantially when evaluated on adversarial responses, thereby highlighting that even large-scale pretrained evaluation models are not robust to the adversarial examples in our dataset. The dataset1 and code2 are publicly available.

## introduction open-domain conversational systems are increasingly in demand for several applications ranging from personal digital assistants to entertainers for recreation. while several automated dialogue agents such as siri, alexa, cortana, and google assistant have been built and deployed, there is no good automatic evaluation metric to measure the quality of their conversations. researchers have usually adopted n-gram based metrics @xcite @xcite or embedding based metrics @xcite @xcite to compare the model's response with a single reference. these metrics assume that a valid response should be semantically or lexically similar to the reference without taking the context of the conversation into consideration. however, in open domain conversations, a given context can have a wide range of possible responses that may be lexically and semantically very different from each other. for example, the context, ''i like dancing and swimming, what about you?'' can be responded to with ''i paint in my free time'' or ''i do not have time for hobbies right now'', both of which are valid responses. as a result, n-gram and word embedding based metrics, which rely on lexical and/or semantic match, correlate very weakly with human judgments for dialogue evaluation @xcite . given the shortcomings of context-agnostic n-gram and embedding based metrics, the focus has now shifted to building neural network-based, trainable dialogue evaluation models @xcite @xcite . such models are trained to identify whether a given response can be considered as a valid continuation of the given context or not. in other words, the model should (i) assign a high score to all relevant responses no matter how diverse they are and (ii) assign a low score to all irrelevant responses, preferably with a clear margin of separation from relevant responses. although there exist several opendomain dialogue datasets @xcite @xcite that are used for training dialogue response generation systems, they are not suitable for training and testing such evaluation models. this is because these datasets have only a single relevant response and no irrelevant responses. irrelevant responses can of course be generated by sampling random utterances from other contexts, but such examples typically do not have any overlap with the context and hence are easier for the model to distinguish from relevant responses (as we will show in our results later). we refer to the randomly sampled responses as random negatives. some efforts have been made to build dialog datasets with multiple relevant responses (i.e., multiple references), but these datasets are either very small (1000 contexts) @xcite or automatically constructed from reddit conversations, hence, potentially noisy @xcite . further, these datasets do not have any carefully crafted adversarial irrelevant responses. we define an adversarial irrelevant response as one that has a significant word overlap with the context but is still an irrelevant response (hence harder to identify than randomly selected irrelevant examples, which may not have any relation to the context). to overcome this limitation of existing datasets, we propose a large scale multi-reference dataset, dailydialog++, which is an extension of the dailydialog dataset. in particular, for each of the 19k contexts derived from dailydialog, we collect an additional 5 reference responses with the help of human annotators. further, for ∼11k contexts in dailydialog, we also ask human annotators to carefully craft irrelevant responses that have a significant word overlap with the context. this dataset will be made publicly available and help towards better training and more robust evaluation of dialogue evaluation metrics. using this dataset, we extensively evaluate a wide range of n-gram-based and embeddingbased metrics. in particular, we compute (i) the correlation of these metrics with binary human judgments and (ii) the accuracy obtained by using the scores assigned by the metrics to classify relevant/irrelevant responses. the performance of these metrics improves when presented with multiple references as opposed to a single reference, but they still leave a lot to be desired. on the other hand, most model-based evaluation metrics, when trained and evaluated using multiple relevant and random negative responses, perform significantly better than the n-gram-based and embedding-based methods. however, their performance drops substantially on the adversarial examples in our dataset. lastly, one could argue that dialog evaluation metrics could be improved by pretraining on large amounts of data. to check if this is indeed the case, we propose a new bertbased evaluation metric called deb (dialog evaluation using bert), which is pretrained on 727m reddit conversations. indeed, this model performs significantly better on random negatives with an accuracy of 88.27% in distinguishing the positive and random negative responses. it also correlates well with human judgments on responses generated by five dialog generation systems @xcite @xcite . in particular, the spearman rank correlation between human scores and deb scores is 0.52 at the response level scores and 0.70 at the system level scores, calculated by aggregating the scores on all responses by each system. however, once again, when evaluated on adversarial examples from our dataset, its performance drops substantially, underscoring that even large-scale pretrained models are not robust to adversarial examples. ## proposed dataset our goal was to build a dataset with manually created multiple relevant and adversarial irrelevant responses. for this, we wanted to start with an existing base dataset that already has one relevant response for every context, and then extend it to include multiple responses. for the base dataset, we considered several popular datasets such as twitter @xcite , reddit @xcite , open subtitles @xcite , nps chat @xcite , personachat @xcite , and dailydialog @xcite . of these, twitter and reddit are generally considered noisy, so we chose not to use either of them as the base dataset. similarly, open subtitles and nps chat did not have speaker-aligned utterances, and hence were not suitable for our purposes. we found that the dailydiaog dataset was clean, humanwritten, readily available, and covered a diverse set of generic topics such as ordinary life, school life, tourism, attitude & emotion, relationship, health, work, politics, culture & education, and finance. it contains a total of 13k conversations with an average of 8 turns between exactly 2 speakers. alternatively, we could have also chosen personachat, which is of a similar size and also contains chit-chat style conversations, but we chose the antecedent dailydialog dataset. for shorter conversations in dailydialog (having less than 8 turns) we collected multiple relevant responses only for the last utterance. for longer conversations (having 8 turns or more), we divided the conversation into two or more smaller chunks and collected multiple relevant responses for the last utterance in every chunk. in this way, from the 13k conversations foot_2 in dailydialog, we were able to create 19k subconversations with multiple relevant responses for the last utterance in each sub-conversation or context. the responses were created by inhouse annotators. each context was shown to 2-3 annotators, and each of them was asked to generate 1-3 alternative responses for the last utterance, capping the total number of alternative responses to 5 (in addition to the one response already available in dailydialog). the annotators were strictly instructed to avoid short generic responses @xcite , and write longer meaningful responses containing at least 8-10 words. these responses were then verified (and if needed, corrected and re-validated) by a different set of annotators. ## existing metrics in this section, we present a brief overview of the existing automatic metrics used for dialogue evaluation. the existing metrics can be broadly classified into two categories, namely, (i) untrained metrics, and (ii) trained metrics. untrained evaluation metrics, usually adopted from the nlg literature, use a predefined formula to compare the candidate response with a reference without taking the context into account. on the other hand, trained metrics are usually trained specifically for the task of dialogue response evaluation to identify valid and invalid responses for a given context. ## dialogue evaluation using bert in the last two years, considerable success in nlp has been driven by large pretrained transformerbased models @xcite @xcite . these models are typically trained with a language model objective and leverage large amounts of unlabeled data. however, none of the trained metrics discussed in the previous section leverage pretraining on largescale dialogue corpora. with the hope that such pretraining should help dialog evaluation models also, we introduce deb (dialog evaluation using bert) which is trained using a masked language model objective (similar to bert) and a modified next response prediction objective. we set up the the task of next response prediction as one of identifying whether the given response is a valid next response for the given context. formally, given a context @xmath0, . . . , w c n } and a response @xmath1, . . . , w r m }, we first pass the concatenated sequence @xmath2, w c 1 , . . . , w c n , [sep], w r 1 , . . . , w r m } through the bert transformer and obtain h @xmath3, the last-layer activations corresponding to the special [cls] token. we then make our final next response predictions as follows: ŷ = softmax(wh cls ), where @xmath4. we use cross entropy loss with binary targets for the next-response prediction. in addition, we use the standard masked language model objective by randomly masking 15% of the words in c and r. note that the proposed model is a straightforward extension of the standard bert model used for language modeling. we do not claim any novelty on this front. the key contribution here is to assess if pretraining on large-scale dialogue corpora improves the performance of dialogue evaluation metrics. existing bert-based evaluation metrics @xcite do not use such pretraining on any large-scale, domain-related corpora. in other words, they do not leverage the more successful recipe of (i) pretraining with a masked language modeling objective and (ii) finetuning with a taskspecific objective (dialog evaluation in this case). the idea behind deb is to check if this successful recipe can be replicated for dialog evaluation, making use of the dialogues in the large-scale reddit corpus. ## experimental setup our goal is to check if the adversarial responses in our dataset, which are specifically crafted to target context-dependent model-based metrics (such as adem, ruber, bert+dnn, and deb), indeed affect the performance of such models. to do so, we first need to benchmark the models' performance on random negatives and then check if the performance drops when evaluated on adversarial examples. hence, in this section, we describe (i) the process of creating and validating such random negatives and (ii) the process used for training model-based metrics. we randomly divide our dataset into train (80% contexts), validation (10% contexts), and test (10% contexts) splits. note that adversarial negatives are not used for training or finetuning the models unless explicitly specified. ## results in this section, we compare the performance of different dialog evaluation metrics in separating relevant references from (i) random negatives, (ii) synthetically crafted adversarial irrelevant responses (explained below), and (iii) manually crafted adversarial irrelevant responses (as in our dailydialog++ dataset). ## discussion in this section, we do further analysis of deb. ## generalization to other datasets in this section, we investigate how well the different model-based metrics trained on dailydialog++ generalize to other datasets that are not seen during training. we evaluate the 3 unreferenced models, bert+dnn, ruber, and deb, which require only context and candidate response as inputs on these 3 datasets. twitter: microsoft research social media conversation corpus @xcite contains a curated list of 3-turn twitter conversations, all of which are human-verified as good responses. personachat: the dialogues in personachat @xcite are associated with welldefined personalities of the speakers involved. we consider the verified human-human chat logs, released by @xcite , as positive examples. we conducted human evaluations to compare the extent to which the model-based metrics agree with human judgements. we randomly sampled 100 contexts from the test set of the dailydialog++ dataset and obtained the responses generated by each of the above models. annotators were shown a context-response pair and were asked to rate how human-like the response is with respect to the context, on a scale of 0-3. the annotators were asked to check for both fluency and coherence. a total of 15 in-house annotators participated in the human evaluation study. the annotators were computer science graduates competent in english. each context-response pair was rated by 5 annotators and the final score was obtained by averaging the 5 scores. we also obtained scores at the system level by aggregating the scores for each model. in ## related work we point the reader to @xcite for an excellent survey of existing datasets containing single reference responses. recently, there has been some effort to create datasets containing multiple references but these datasets are either too small (around 1,000 contexts) @xcite or noisy @xcite . we have already reviewed all the existing dialog metrics in section 3 and hence we do not discuss them again here. instead, we quickly mention existing works which critically examine dialog evaluation metrics. for example, @xcite show that existing n-gram based metrics do not correlate well with human judgements for dialog evaluation. we report similar results but additionally show that the correlation improves in the presence of multiple references. @xcite have critically examined adem and shown that in most cases it produces a score close to 2.5 (on a scale of 1 to 5) and hence does not clearly separate relevant and irrelevant responses. lastly, we also mention a very recent work, @xcite , which has pretrained a large scale transformer on reddit corpus for building conversation systems. however, their focus is on dialog generation and not on evaluation metrics. we propose a multi-reference open-domain dialogue dataset with multiple relevant responses and adversarial irrelevant responses. we perform an extensive study of the existing dialogue evaluation metrics using this dataset and also propose a new transformer-based evaluator pretrained on large-scale dialogue datasets. we identify the strengths and weaknesses of such a model through studies of its performance on untrained and synthetically modified data. we find deb to be easily adaptable to other opendomain dialogue datasets. we also present the scope of the adversarial responses in our dataset towards bringing out better evaluation metrics, since all the current models do not perform well on those unless explicitly trained.