Towards Holistic and Automatic Evaluation of Open-Domain Dialogue Generation (2020)

Abstract: Open-domain dialogue generation has gained increasing attention in Natural Language Processing. Its evaluation requires a holistic means. Human ratings are deemed as the gold standard. As human evaluation is inefficient and costly, an automated substitute is highly desirable. In this paper, we propose holistic evaluation metrics that capture different aspects of open-domain dialogues. Our metrics consist of (1) GPT-2 based context coherence between sentences in a dialogue, (2) GPT-2 based fluency in phrasing, (3) n -gram based diversity in responses to augmented queries, and (4) textual-entailment-inference based logical self-consistency. The empirical validity of our metrics is demonstrated by strong correlations with human judgments. We open source the code and relevant materials.

## introduction learning to communicate is a key capacity of intelligent agents. research on enabling a machine to have meaningful and natural conversations with humans plays a fundamental role in developing artificial general intelligence, as can be seen in the formulation of turing test @xcite . recently open-domain or non-task-oriented dialogue systems have attracted a surge of research interest @xcite @xcite @xcite @xcite . evaluating models of open-domain dialogue generation in an efficient manner poses a significant challenge in developing dialogue systems. the prevalent method of open-domain dialogue evaluation is human-based rating with a given rubric. context of conversation speaker a: hey, what do you want to do tonight? speaker b: why don't we go see a movie? model response speaker a: nah, let's do something active. reference response speaker a: yeah, the film about turing looks great! context of conversation speaker a: do you have maps of downtown area? speaker b: yes, here you are. speaker a: how much is it? model response speaker b: i've no idea, i don't have any maps of the area. when various variations in the model and sets of hyper-parameters are needed, the labor-intensive human evaluation is deemed impracticable. this key drawback may hinder the research progress and render the human evaluation approach not scalable. previous automatic evaluation metrics generally focus on the quality of the dialogue generation: context coherence and fluency. word-overlap metrics @xcite @xcite or ad-hoc classifiers @xcite are designed for measuring the quality. in open-domain dialogue, the relation between two utterances is more critical as shown in the first example of in this work, we propose holistic metrics that evaluate distinctive aspects of generated dialogues. specifically, we consider (1) context coherence of a dialogue: the meaningfulness of a response within the context of prior query, (2) language fluency of generated responses: the quality of phrasing relative to a human native speaker, (3) response diversity of a set of generated sentences: the variety in meaning and word choice of responses, and (4) logical self-consistency: the logical consistency of utterances from a dialogue agent. both context coherence and response fluency (quality metrics) can naturally be captured by metrics based on strong language models like gpt-2 @xcite . therefore, we propose to recruit and fine-tune gpt-2 as a basis of our quality metrics. with regard to response diversity and logical selfconsistency, we propose to measure them under augmented utterances with controlled paraphrasing. we leverage two effective approaches to generate augmented utterances: word substitution and text generator with a k-best decoder. moreover, we utilize n-gram based entropy to capture response diversity and entailment based approach to capture logical self-consistency. our experiments show that the proposed metrics strongly correlate with human judgments. moreover, our augmented datasets allow for a more accurate and straightforward hu-man annotation, significantly improving the agreement between human evaluation. we release the code and relevant materials as open-source contribution to pave the way towards further research. ## prior art heuristic-based metrics have been shown to align well with human judgments and widely applied in various language generation tasks. for machine translation, bleu @xcite computes n-gram precision, whereas meteor @xcite takes into account both precision and recall. for summarization, rouge @xcite also considers both precision and recall by calculating f-measure. these n-gram based metrics are well-suited for the generation tasks that are more source-determined or low conditional entropy such as translation, image captioning, and summarization. some dialogue studies adopted these metrics to evaluate the quality of generated conversation responses @xcite @xcite . they nevertheless are not suitable for open-ended generations or high conditional entropy tasks like dialogue generation where a diverse range of generations is acceptable conditional on a query. indeed, @xcite conducts extensive empirical studies on these metrics (e.g., bleu, meteor, and rouge) to test their effectiveness on evaluating dialogue generation and find limited relation between these automatic metrics and human judgments. the word-overlap metrics (e.g., bleu) fail to capture the semantic similarity between model and reference responses. the following works leverage the distributed representation learned in neural network models to capture semantic similarity among context, model response, and reference response. @xcite collect a dataset of human scores and train a hierarchical recurrent neural network (rnn) to predict human-like scores to input responses given the context, resulting in an automatic metric that has a medium level correlation with human judgments. obtaining this metric however requires a large dataset of human-annotated scores, thus rendering this approach less flexible and extensible. @xcite proposes a referenced metric and unreferenced metric blended evaluation routine (ruber) for open-domain dialogue systems. this blended metric is a combination of two metrics. a referenced metric measures the similarity between model-generated and reference responses on the basis of word-embeddings. an unreferenced metric captures the relevance between the query and response. it is obtained by training a neural network classifier to determine whether a response is appropriate. the positive examples are the references, while the negative examples are reference responses randomly chosen from the dataset, hence avoiding the need of human-annotated data. after training, the softmax score is utilized to measure whether the generated response is coherent with the query. attempting to improve ruber, @xcite explores to use contextualized embeddings from bert. the bert-based unreferenced metric improves over the word-embedding-based ruber unreferenced metric. interestingly, they show that the combined metric has a reduced correlation with human judgments than the unreferenced metric alone. although this finding is counterintuitive, it is consistent with the characteristics of open-domain dialogue that a range of diverse responses is reasonable given a query. hence a response can be acceptable to human annotators even if it does not align well with the reference either in terms of word-overlap or semantic embedding. ## conclusion this paper provides a holistic and automatic evaluation method for open-domain dialogue models. in contrast to prior art, our means of evaluation captures not only the quality of generation, but also the diversity and logical consistency of responses. we recruit gpt-2 as a strong language model to evaluate the context coherency and response fluency. for response diversity and logical self-consistency, we propose to measure these two aspects under augmented utterances with controlled paraphrasing. we leverage two effective approaches to generate augmented utterances: word substitution and text generator with k-best decoder. moreover, we utilize n-gram based entropy to capture response diversity and entailment based approach to measure logical self-consistency. the proposed metrics show a strong correlation with human judgments. it is our hope the proposed holistic metrics may pave the way towards the comparability of open-domain dialogue models.