GRADE : Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems (2020)

Abstract: Automatically evaluating dialogue coherence is a challenging but high-demand ability for developing high-quality open-domain dialogue systems. However, current evaluation metrics consider only surface features or utterance-level semantics, without explicitly considering the fine-grained topic transition dynamics of dialogue flows. Here, we first consider that the graph structure constituted with topics in a dialogue can accurately depict the underlying communication logic, which is a more natural way to produce persuasive metrics. Capitalized on the topic-level dialogue graph, we propose a new evaluation metric GRADE, which stands for Graph-enhanced Representations for Automatic Dialogue Evaluation. Specifically, GRADE incorporates both coarse-grained utterance-level contextualized representations and fine-grained topic-level graph representations to evaluate dialogue coherence. The graph representations are obtained by reasoning over topic-level dialogue graphs enhanced with the evidence from a commonsense graph, including k-hop neighboring representations and hop-attention weights. Experimental results show that our GRADE significantly outperforms other state-of-the-art metrics on measuring diverse dialogue models in terms of the Pearson and Spearman correlations with human judgments. Besides, we release a new large-scale human evaluation benchmark to facilitate future research on automatic metrics.

## introduction coherence, what makes dialogue utterances unified rather than a random group of sentences, is an essential property to pursue an open-domain * equal contribution. ## related work automatic evaluation for open-domain dialogue systems is difficult since there are many appropriate responses for a dialogue context under the open-domain setting, known as the one-to-many problem @xcite . initially, the statistic-based metrics in language generation tasks are adopted for dialogue evaluation, such as bleu @xcite , meteor @xcite and rouge @xcite . these metrics use statistical rules to measure the surface similarity between generated responses and reference responses. for example, bleu computes the geometric average of the n-gram precisions. however, they can not cope with the one-to-many problem and have weak correlations with human judgements @xcite . in recent years, learning-based metrics have increasingly attracted interest from researchers. adem proposed by @xcite achieves higher correlations with human judgements than the statistic-based metrics, which is trained with human-annotated data in a supervised manner. however, it is time-consuming and expensive to obtain large amounts of annotated data. to reduce the cost of obtaining annotated data, @xcite trained their metric ruber with auto-constructed negative samples in an unsupervised manner. with the advances of the pre-trained language model, bert @xcite has been adopted for dialogue or nlg evaluation. @xcite proposed bert-ruber, which outperforms ruber significantly by incorporating bert embeddings. bertscore @xcite performs soft-overlap between candidate and reference sentences by using bert embeddings directly without fine-tuning, and has been shown to correlate with human judgment robustly. besides, @xcite introduced bleurt by further training regular pre-trained bert with an elaborate pre-training scheme and fine-tuning on small amounts of rating data, which yields superior results. note that our model differs from the above learning-based metrics in two folds. first, our metric is trained with high-quality negative samples that are similar to the ground truths in both lexical and semantic aspects instead of randomly sampling. second, different levels of representations are considered in our grade, especially the fine-grained topic-level graph representation. ## grade metric in this paper, we focus on designing an evaluation metric that can automatically assess the coherence of responses produced by dialogue models. formally, given a dialogue context @xmath0, • • • , c m } and a response @xmath1, • • • , r n }, where each c k is a token in the context and each r k is a token in the response, our goal is to learn a function f : (c, r) → s that predicts the coherence score s. as illustrated in figure 2 , our grade predicts a coherence score s between a context c and a response r in three steps: (1) producing the utterance-level contextualized representation v c (section 3.1); (2) generating the topic-level graph representation v g (section 3.2 and section 3.3); ## conclusion and discussion in this paper, we proposed grade (graphenhanced representations for automatic dialogue evaluation), a novel metric for dialogue coherence evaluation of open-domain dialogue systems. empirical results show that grade has stronger correlations with human judgements and can generalize to other unseen chit-chat datasets. besides, we also release a new large-scale human evaluation bench-mark to facilitate future research on automatic metrics. a limitation of grade is the inconsistency between the training objective (relative ranking) and the expected behavior (absolute scoring). specifically, the ranking loss we adopted only requires good responses to be ranked higher than bad responses, which is a relatively loose constraint compared with the absolute scoring that humans do. therefore, grade may deviate from the human scoring criterion and fail to quantify the dialogue responses accurately, and that the human correlation results fluctuate over different runs. overall, to develop a dialogue metric that can quantify in a more human-like manner, it is critical to reducing the gap between the training objective and the model behavior we truly care about.