u BLEU : Uncertainty-Aware Automatic Evaluation Method for Open-Domain Dialogue Systems (2020)

Abstract: Because open-domain dialogues allow diverse responses, basic reference-based metrics such as BLEU do not work well unless we prepare a massive reference set of high-quality responses for input utterances. To reduce this burden, a human-aided, uncertainty-aware metric, ΔBLEU, has been proposed; it embeds human judgment on the quality of reference outputs into the computation of multiple-reference BLEU. In this study, we instead propose a fully automatic, uncertainty-aware evaluation method for open-domain dialogue systems, υBLEU. This method first collects diverse reference responses from massive dialogue data and then annotates their quality judgments by using a neural network trained on automatically collected training data. Experimental results on massive Twitter data confirmed that υBLEU is comparable to ΔBLEU in terms of its correlation with human judgment and that the state of the art automatic evaluation method, RUBER, is improved by integrating υBLEU.

## introduction there has been increasing interest in intelligent dialogue agents such as apple siri, amazon alexa, and google assistant. the key to achieving higher user engagement with those dialogue agents is to support open-domain non-task-oriented dialogues to return a meaningful response for any user input. the major challenge in developing open-domain dialogue systems is that existing evaluation metrics for text generation tasks, such as bleu @xcite , correlate poorly with human judgment on evaluating responses generated by dialogue systems @xcite . in open-domain dialogues, even though responses with various contents and styles are acceptable @xcite , only a few responses, or often only one, are available as reference responses in evaluation datasets made from actual conversations. it is, therefore, hard for these reference-based metrics to consider uncertain responses without writing additional reference responses by hand ( § 2). to remedy this problem, @xcite proposed ∆bleu ( § 3), a human-aided evaluation method for text generation tasks with uncertain outputs. the key idea behind ∆bleu is to consider human judgments on reference responses with diverse quality in bleu computation. although ∆bleu correlates more strongly with human judgment than bleu does, it still requires human intervention. therefore it cannot effectively evaluate open-domain dialogue systems in a wide range of domains. to remove the human intervention in ∆bleu, we propose an automatic, uncertainty-aware evaluation metric, υbleu. this metric exploits reference responses that are retrieved from massive dialogue logs and rated by a neural network trained with automatically collected training data ( § 4). we first retrieve diverse response candidates according to the similarity of utterances to which the responses were directed. we then train a neural network that judges the quality of the responses by using training data automatically generated from utterances with multiple responses. we also propose integrating υbleu into the state of the art evaluation method, ruber @xcite ( § 2) to advance the state of the art by replacing its reference-based scorer. using our method, we experimentally evaluated responses generated by dialogue systems such as a retrieval-based method @xcite and a generation-based method @xcite using twitter dialogues ( § 5). our method is comparable to ∆bleu in terms of its correlation with human judgment, and when it is integrated into ruber @xcite , it substantially improves that correlation ( § 6). our contributions are the followings: ## related work this section introduces recent studies on evaluating open-domain dialogue systems. we focus here on model-agnostic methods than can evaluate the quality of a response for a given utterance. 1 for evaluation of dialogue systems, researchers have adopted existing evaluation metrics for other text generation tasks such as machine translation and summarization. unfortunately, referencebased metrics such as bleu @xcite and rouge @xcite correlate poorly with human judgment on evaluating dialogue systems @xcite . this is because only a few responses, or often only one, can be used as reference responses when actual conversations are used as datasets, even though responses in open-domain dialogues can be diverse @xcite . to consider uncertain responses in open-domain dialogues, @xcite attempted to collect multiple reference responses from dialogue logs for each test utterance-response pair. @xcite improved that method by manually rating the augmented reference responses and used the ratings to perform discriminative bleu evaluation, as detailed later in § 3.2. @xcite created multiple reference responses by hand for the daily dialogue dataset @xcite . although the last two studies empirically showed that the use of human-rated or -created reference responses in evaluation improves the correlation with human judgment, it is costly to create such evaluation datasets for various domains. as for evaluation methods, adem @xcite learns an evaluation model that predicts human scores for given responses by using large-scale human-rated responses that are originally generated by humans or dialogue systems. the drawback of that method is the cost of annotation to train the 1 perplexity is sometimes used to evaluate dialogue systems @xcite . it is only applicable, however, to generation-based dialogue systems, so we do not discuss it here, like @xcite . evaluation model. moreover, the evaluation model has been reported to overfit the dialogue systems used for generating the training data. ruber @xcite is an automatic evaluation method that combines two approaches: its referenced scorer evaluates the similarity between a reference and a generated response by using the cosine similarity of their vector representations, while its unreferenced scorer, trained by negative sampling, evaluates the relevance between an input utterance and a generated response. @xcite showed that use of bert embedding @xcite in pretrained vectors improves the unreferenced scorer but not the referenced scorer in ruber. the referenced scorer is similar to ∆bleu in that they both are referenced-based evaluation metrics. we later confirm that the referenced scorer in ruber underperforms our method, and we thus propose replacing it with our method ( § 5.5). ## preliminaries this section reviews ∆bleu @xcite , a human-aided evaluation method for text generation tasks with uncertain outputs, after explaining the underlying metric, bleu @xcite . ## proposed method: υbleu this section describes our approach to the problems of ∆bleu described in § 3.2. to remove the cost of human judgments of extended references, we propose using a neural network trained on automatically collected training data to rate each of the retrieved responses (figure 1 , § 4.2). in addition, to diversify the extended reference responses in terms of content and style, we propose a relaxed response retrieval approach using continuous vector representations of utterances only ( § 4.1). ## experimental settings this section describes how to evaluate our method for evaluating open-domain dialogue systems. using utterances from twitter ( § 5.1), responses written by humans, and responses obtained by dialogue systems ( § 5.2), we evaluated our method in terms of its correlation with human judgment ( § 5.3-5.5). ## results metric spearman's ρ pearson's r max min max min ∆bleu .366 .300 .360 .294 υbleu .330 .281 .394 .332 ruber unref. & ref. scorer .339 .206 .325 .193 ref. scorer only .188 .071 .075 .016 unref. scorer only .342 .225 .336 .217 unref. & υbleu .435 .323 .450 .338 human .773 .628 .778 .607 next, examples ## conclusions we have proposed a method to remove the need for costly human judgment in ∆bleu @xcite and obtain an automatic uncertainty-aware metric for dialogue systems. our proposed υbleu rates diverse reference responses retrieved from massive dialogue logs by using a neural network trained with automatically-collected training data, and it uses the responses and the scores to run ∆bleu. experimental results on massive twitter dialogue data revealed that υbleu is comparable to human-aided ∆bleu, and that, by integrating it into ruber, the state of the art method for evaluating open-domain dialogue systems, we can improve the correlation with human judgment. we will release all code and datasets (tweet ids) to promote the reproducibility of our experiments. 4 the readers are referred to our code to evaluate their dialogue systems for their native languages.