Evaluating Dialogue Generation Systems via Response Selection (2020)

Abstract: Existing automatic evaluation metrics for open-domain dialogue response generation systems correlate poorly with human evaluation. We focus on evaluating response generation systems via response selection. To evaluate systems properly via response selection, we propose a method to construct response selection test sets with well-chosen false candidates. Specifically, we propose to construct test sets filtering out some types of false candidates: (i) those unrelated to the ground-truth response and (ii) those acceptable as appropriate responses. Through experiments, we demonstrate that evaluating systems via response selection with the test set developed by our method correlates more strongly with human evaluation, compared with widely used automatic evaluation metrics such as BLEU.

## introduction automatic evaluation for open-domain dialogue generation systems has a potential for driving their research and development because of its high reproducibility and low cost. however, existing automatic evaluation metrics, such as bleu @xcite , correlate poorly with human evaluation @xcite . this poor correlation arises from a nature of dialogue, that is, there are many acceptable responses to an input context, known as the one-to-many problem @xcite . to tackle this problematic issue, we focus on evaluating response generation systems via response selection. in this task, systems select an appropriate response for a given context from a set of response candidates. each candidate has the label that indicates whether the candidate is appropriate response for the given context. traditionally, response selection has been used to evaluate retrieval-based dialogue systems @xcite . we consider applying this task to driving the research for dialogue generation repository context: do you have a car? ground-truth: yes, i have a car. ## 1 4 figure 1 : overview of the construction method of our test set. first, we retrieve only utterances related to the ground-truth response from a repository. then, we remove acceptable utterances by human evaluation. generally, false response candidates are randomly sampled from a repository @xcite , which causes two problems: (i) unrelated false candidates and (ii) acceptable utterances as false. the first problem is that randomly sampled false candidates are often too far from ground-truth responses. consider the case where for a given context "do you have a car?", a response candidate "i play tennis." is ran-domly sampled. systems can easily recognize this candidate as a false one because there are no related content words between them. such excessive easiness is not preferable because the performance gap between good and inferior systems tends to be small. the second problem is that there is no guarantee that randomly sampled candidates are always unacceptable ones. for example, "i don't know." is often sampled as a false response because this phrase often occurs in open-domain dialogues. this phrase can be regarded as acceptable for various contexts. these two problems make general response selection test sets unreliable. in this work, we propose a method to construct response selection test sets with well-chosen false candidates (figure 1 ). first, we retrieve only utterances related to the ground-truth response. then we remove acceptable utterances by human evaluation. through experiments, we demonstrate that automatic evaluation using the test set developed by our method correlates more strongly with human evaluation, compared with widely used automatic evaluation metrics such as bleu. our empirical results indicate that response selection with wellchosen false candidates can be a valid option for evaluating response generation systems. we will release the test set used in the experiments. foot_0 2 related work automatic evaluation metrics various metrics have been proposed for automatic evaluation of dialogue systems, such as bleu, meteor (banerjee and lavie, 2005), rouge @xcite , greedy matching @xcite , and vector extrema @xcite . these metrics evaluate the quality of the responses generated by systems. however, this is challenging due to the oneto-many problem. for example, adem, a metric proposed by @xcite , is easily fooled by adversarial examples (responses) @xcite . to remedy one-to-many problem, we focus on evaluating systems via response selection. response selection test sets with human labels one popular test set for response selection is douban conversation corpus in chinese @xcite . in this test set, each response candidate has a manually annotated label that indicates whether or not the candidate is appropriate for the given context. although this test set is similar to ours, there are some differences between the purposes and procedure of test set designs. the purpose of creating their test set is to simulate and evaluate retrieval-based dialogue systems. thus, all the candidates in this corpus are retrieved by using the context as queries, as retrieval-based systems do. in this paper, we develop an english response selection test set with human labels to evaluate dialogue generation systems. one of the salient differences from douban conversation corpus is the procedure of retrieving false candidates. we retrieve false candidates using the ground-truth responses. by this method, we can more certainly collect false candidates that are related to ground-truth responses and facilitate error analysis as described in section 4.3. ## experiments we test whether the automatic evaluation of response generation systems on our test set correlates with human evaluation. ## conclusion in this paper, we focused on evaluating response generation systems via response selection. to evaluate systems properly via response selection, we proposed a method to construct response selection test sets with well-chosen false candidates. specifically, we proposed to construct test sets filtering out some types of false candidates: (i) those unrelated to the ground-truth response and (ii) those acceptable as appropriate responses. we demonstrated that evaluating systems via response selection with the test sets developed by our method correlates more strongly with human evaluation, compared with that of widely used metrics such as bleu. in the future, we will provide labels that indicate "why this candidate is false" for false candidates in our test set, so that one can easily detect weak points of systems through error analysis.