RADE : Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue (2023)

Abstract: Evaluating open-domain dialogue systems is challenging for reasons such as the one-to-many problem, i.e., many appropriate responses other than just the golden response. As of now, automatic evaluation methods need better consistency with humans, while reliable human evaluation can be time- and cost-intensive. To this end, we propose the Reference-Assisted Dialogue Evaluation (RADE) approach under the multi-task learning framework, which leverages the pre-created utterance as reference other than the gold response to relief the one-to-many problem. Specifically, RADE explicitly compares reference and the candidate response to predict their overall scores. Moreover, an auxiliary response generation task enhances prediction via a shared encoder. To support RADE, we extend three datasets with additional rated responses other than just a golden response by human annotation. Experiments on our three datasets and two existing benchmarks demonstrate the effectiveness of our method, where Pearson, Spearman, and Kendall correlations with human evaluation outperform state-of-the-art baselines.

## introduction open-domain dialogue system, which focuses on non-goal-oriented chitchat, may converse on a broad range of arbitrary topics. recent years have witnessed rapid advances in natural language generation @xcite @xcite , boosting the development of opendomain dialogue systems. conversations with such systems resemble human-human interactions as various responses might fit the context, given that users often do not have a specific goal beyond enjoying the conversation. evaluating these conversations is thus challenging because of the socalled one-to-many problem @xcite ; see figure 1 responses with different semantics fit the context while there is only one golden response. the most common practice of dialogue evaluation is done with reference-based metrics, which compare the generated response with a pre-created response, commonly referred to as the golden standard @xcite . the reference-based metrics calculate the similarity between the generated and gold responses at either lexical level (e.g., rouge @xcite , bleu @xcite ) or semantic level (e.g., bertscore @xcite , adem @xcite ). however, these metrics ignore the one-to-many nature of open-domain dialogues. as illustrated at the bottom of figure 1 , the generated response "amazon is good but expensive ..." expresses the opposite semantics to the golden response "i shop online..." and is therefore considered a non-good response by the reference-based metrics. therefore, these metrics may need a higher consistency with humans. recently, multi-reference methods and referencefree methods are proposed to address the drawback of reference-based metrics. the former explicitly annotates multiple references for dialogue @xcite , whereas the latter discards the golden response in the evaluation and achieves high cor-relations with human judgments @xcite . however, drawback still exists in these two classes of methods. multi-reference methods are costly and hard to generalize to different datasets, while referencefree methods are often unstable and vulnerable to data-induced biases foot_0 . to overcome the weakness of existing evaluation methods and further resolve the one-to-many problem, we propose a new technique, namely reference-assisted dialogue evaluation (rade). rade considers the pre-created response as a reference instead of the golden standard. to support rade, we design a new human annotation task to extend existing datasets, which includes metric decompose and pairwise annotation, where a pre-scored golden response is paired with generated responses for rating following a unified rating score. the final scores are arrived at by aggregating ratings with a weighted sum from different sub-metrics. the human annotation collects labels for three high-quality datasets with 10,112 dialogues, which correspond to three downstream open-domain dialogue system tasks, i.e., chitchat, empathetic dialogue, and personal chat. these multi-domain datasets make rade more robust when generalizing to cross-domain evaluation scenarios while having a better task-specific performance. we propose a rade model under the multitask learning framework for automatic evaluation based on the newly collected datasets. specifically, rade first explicitly encodes the relation between dialogue context and generated response with reference assistance. then rade discriminates whether the reference or response fits the context better and predicts the scores for each utterance. to relieve the one-to-many problem, we augment rade with a joint response generation task where rade learns to generate the reference responses to better perceive the range of candidate responses. extensive experiments on our three benchmarks demonstrate that rade achieves the best correlations with human judgment. we also examine two existing usr benchmark @xcite where rade outperforms the state-of-the-art methods, e.g., pushing the pearson correlation coefficient to 48% (6.8% absolute improvement) and spearman correlation coefficient to 46.6% (4.3% absolute improvement). experiments also verify the generalizability of our proposed method. our contributions can be summarized as follows: (1) we propose the reference-assisted evaluation method, i.e., rade, for open-domain dialogue evaluation; (2) we design a new human annotation task and collect three new dialogue evaluation datasets; (3) experiments on our benchmarks and two existing benchmarks verify the effectiveness and robustness of the proposed methods; (4) we release three new benchmarks and the pre-trained evaluation model to facilitate future research on dialogue evaluation. ## task formulation in this work, we propose two tasks: (1) extending the existing datasets by human annotation, and (2) leveraging the rated references collected in (1) to enhance automatic evaluation. human annotation human annotation aims to extend existing datasets with multiple rated responses to facilitate automatic evaluation. given a dialogue context c, which is always paired with a golden response (denoted as reference) r h , we employ the generation models, e.g., blender-bot @xcite , to generate one more response r a . we then assign a fixed overall score or derive from existing datasets to the reference as s h . the annotators are instructed to rate r a as s a , following the same scale while taking the reference as a benchmark. the annotators are also asked to revise the reference score s h if s h is inappropriate. ## human annotation our human annotation task aims to rate the candidate responses following a pre-scored reference as a benchmark. since there are multiple perspectives to assess the response, we simplify by sorting the possible aspects into two categories: the general view and the task-specific view. as listed in ## reference-assisted automatic evaluation we propose rade, a reference-assisted automatic dialogue evaluation method under the framework of multi-task learning. compared with reference-based methods that evaluate based on the distance between the golden and generated response, the proposed rade explicitly discriminates whether the reference or candidate response fits the dialogue context better. to relieve the one-to-many problem, we augment rade with a joint response generation task, which aims to perceive the range of feasible candidate responses. to improve the performance of rade with the limited dataset, we propose a two-stage training strategy, including cross-domain pre-training and task-specific finetune. ## discussions the impact of the training data scale. to explore the minimum data scale required for our method, we train rade using different amounts of randomly sampled annotated data. we observe a minor degradation in rade's performance as the amount of data decreases. for example, when training on 2,400 examples from the empa-theticdialogue dataset, rade(ts) achieves pearman'@xmath0.837 and spearman'@xmath1.829; whereas with 1,200 examples, it obtains pearman'@xmath2.807 and spearman'@xmath3.806. all results are averaged over three runs. moreover, we find that rade outperforms all baselines with only 800 training examples in three datasets, respectively. ## conclusion we have presented a new reference-assist dialogue evaluation (rade) method to address the one-tomany problem when evaluating open-domain dialogue systems. rade evaluates the response generated by open-domain dialogue agents with the assistance of reference response. in addition, we have curated the reference-assisted dialogue evaluation datasets by expanding three existing datasets via a pairwise human annotation. the extended datasets contain over 10k dialogues. extensive experiments on three extended datasets and two existing benchmarks have verified the effectiveness and robustness of the proposed methods and their generalizability.