F ine D -Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022)

Abstract: Recent model-based reference-free metrics for open-domain dialogue evaluation exhibit promising correlations with human judgment. However, they either perform turn-level evaluation or look at a single dialogue quality dimension. One would expect a good evaluation metric to assess multiple quality dimensions at the dialogue level. To this end, we are motivated to propose a multi-dimensional dialogue-level metric, which consists of three sub-metrics with each targeting a specific dimension. The sub-metrics are trained with novel self-supervised objectives and exhibit strong correlations with human judgment for their respective dimensions. Moreover, we explore two approaches to combine the sub-metrics: metric ensemble and multitask learning. Both approaches yield a holistic metric that significantly outperforms individual sub-metrics. Compared to the existing state-of-the-art metric, the combined metrics achieve around 16% relative improvement on average across three high-quality dialogue-level evaluation benchmarks.

## introduction in the study of generative dialogue systems, we heavily rely on some reference-based static metrics, such as bleu @xcite , to measure improvements during system development and to compare various model variants. these metrics still need improvement due to their poor correlations with human judgment @xcite and poor interpretability @xcite . recently, model-based reference-free metrics @xcite represent one of the ways to address the limitations of static reference-based metrics. although such metrics exhibit promising correlations with human evaluation, most of them @xcite @xcite @xcite @xcite target turn-level evaluation, i.e., they focus on single-response quality, such as contextual relevance and naturalness. when evaluating a multi-turn human-chatbot dialogue, turn-level metrics do not model the dialogue in totality, but frame it as a set of context-response pairs. they assign scores to every chatbot responses in the dialogue. hence, an aggregation strategy is required to derive the single dialogue-level metric score, such as taking the average of all the responselevel scores. both prior works @xcite and our experimental results in §5 suggest that such an approach yields sub-optimal dialogue-level evaluation. the reason may be that turn-level metrics do not model the dependency among utterances within multi-turn interactions, it is difficult for them to spot errors that are only obvious after observing the entire conversation @xcite . there are some metrics that perform multi-turn evaluation. however, they focus only on a single dimension, such as coherence or overall impression @xcite @xcite . when evaluating a dialogue, they assign a single score to quantify one aspect of dialogue quality. as pointed out in @xcite , dialogue quality is inherently multi-faceted. by breaking down the quality of the dialogue into multiple fine-grained dimensions, we may provide a more interpretable and descriptive dialogue evaluation. with such an interpretable metric, dialogue researchers know exactly which aspect of the dialogue system to improve. to this end, we propose a multi-dimensional metric, dubbed fined-eval 2 , which consists of specialized sub-metrics. each sub-metric targets a specific fine-grained dimension and all sub-metrics are trained in a self-supervised manner without reliance on any human annotations. to develop fined-eval, our first step is to identify the dimensions for metric design. it is a wellknown phenomenon that human judges do not provide completely independent assessments for various fine-grained dimensions. for instance, @xcite analyzes the human ratings with respect to (w.r.t.) different fine-grained dimensions on four text generation tasks and has observed moderate correlations for most dimension pairs. intuitively, we want to select dimensions that are less correlated such that our metric can holistically capture the dialogue quality from different perspectives. the selection process is guided by an analysis on fine-grained human ratings of dialogue-level evaluation data ( §2). through the analysis, we want to cluster the dimensions into relatively independent dimension groups and then, select representative dimensions from different dimension groups. next, we propose dimension-specific strategies for training the sub-metrics. ( §3.3). the submetrics, which target the representative dimensions, can also be applied to evaluate other dimensions in their respective dimension groups. furthermore, both @xcite and @xcite highlight that the combination of different metrics leads to better correlations with human evaluation than individual specialized metrics. we are motivated to explore how to combine the sub-metrics into a unified one. specifically, both the metric ensemble and multitask learning @xcite are examined ( §3.4). finally, in the experiments ( §5), we demonstrate that (1) the sub-metrics highly correlate with human judgment for their target dimensions. (2) the scores assigned by fined-eval are more interpretable than the existing metrics. (3) with either metric ensemble or multitask learning, fined-eval significantly outperforms existing state-of-the-art metrics as well as individual sub-metrics on three high-quality dialogue-level evaluation benchmarks. ## experiments & analysis in this section, we conduct the main analysis based on the following research questions (rq): (1) are dialogue-level metrics better than turn-level metrics for multi-turn dialogue evaluation? (2) does our 3 that all dialogue-level metrics perform significantly better than the turn-level metrics across different quality dimensions, this observation is inline with conclusions from previous works @xcite . however, usl-h and dscore outperform deam on on dstc9-interactive ( rq 2. in the sub-metrics section of rq 3. we can observe in ## related work evaluation is a long-lasting problem in dialogue system research @xcite @xcite . in open-domain dialogue evaluation, @xcite shows that commonly-adopted metrics, such as bleu @xcite , can be misleading due to their poor correlations with human judgment. recently, interests in automatic evaluation of open-domain dialogue systems have intensified with the introduction of reference-free model-based evaluation paradigm 12 . most of them focus on turnlevel response quality @xcite @xcite @xcite . despite their promising correlations with human evaluation, such metrics are insufficient for dialogue-level assessment. our fined-eval targets dialogue-level evaluation specifically. in addition, existing works on model-based dialogue-level metrics @xcite @xcite focus very much on a single quality dimension. on the contrary, fined-eval is capable of multidimensional evaluation and it can provide more fine-grained and interpretable scores. the idea of decomposing overall dialogue quality into fine-grained dimensions has been explored in prior works @xcite @xcite for turn-level evaluation. however, its application on dialogue-level evaluation is under-explored, our work serves to bridge this gap. ## conclusion in this paper, we propose fined-eval, a multidimensional dialogue-level evaluation metric. fined-eval consists of three specialized submetrics, which targets three fine-grained dialogue quality respectively, including coherence, likability, and topic depth. each specialized sub-metric is trained with a pairwise ranking objective on dialogue pairs that are curated according to the corresponding dimension-specific strategies. two variants of fined-eval are proposed to combine the sub-metrics into a holistic metric. one variant is based on metric ensemble and the other is based on multitask learning. we have empirically demonstrated that fined-eval strongly correlate with human evaluation for different dialogue quality dimensions as well as exhibits strong generalization across different evaluation datasets.