A Comprehensive Assessment of Dialog Evaluation Metrics (2021)

Abstract: Automatic evaluation metrics are a crucial component of dialog systems research. Standard language evaluation metrics are known to be ineffective for evaluating dialog. As such, recent research has proposed a number of novel, dialog-specific metrics that correlate better with human judgements. Due to the fast pace of research, many of these metrics have been assessed on different datasets and there has as yet been no time for a systematic comparison between them. To this end, this paper provides a comprehensive assessment of recently proposed dialog evaluation metrics on a number of datasets. In this paper, 23 different automatic evaluation metrics are evaluated on 10 different datasets. Furthermore, the metrics are assessed in different settings, to better qualify their respective strengths and weaknesses. This comprehensive assessment offers several takeaways pertaining to dialog evaluation metrics in general. It also suggests how to best assess evaluation metrics and indicates promising directions for future work.

## introduction evaluation is a crucial component of the research process. evaluation metrics are used to shine light on the best models and thus they strongly influence the research directions of a field. standard automatic language evaluation metrics (e.g., bleu, meteor) have been shown to be ineffective for evaluating dialog @xcite . to this end, recent research has proposed a number of automatic metrics specifically designed to evaluate dialog @xcite @xcite @xcite @xcite @xcite . these metrics address the weaknesses of the standard language evaluation metrics. they have also been shown to better correlate with human judgement. however, most of these metrics were devel-oped at about the same time and were evaluated on different datasets. as such, there has not yet been a consistent comparison amongst them. this paper describes an assessment of these metrics on several different datasets. it quantifies the relative performance and reveals strengths and weaknesses of each metric. due to the fact that standard automatic metrics have been shown to be ineffective for dialog evaluation @xcite , dialog research typically relies on human evaluation. while useful and very informative, human evaluation is expensive and time-consuming. given these considerations, at present it is usually only used for a final evaluation. yet, during the development process, automatic evaluation metrics are essential since they are used to optimize the model design and the choice of hyperparameters. in order to be relevant and meaningful, these metrics must better correlate with human judgment so that they serve as a meaningful proxy for human judgements during the development process. this paper discusses the assessment of several recently-proposed automatic evaluation metrics for dialog over several datasets. these datasets have human annotations that measure the quality of the responses. furthermore, this paper includes analysis experiments that qualify the performance of the metrics across different settings. this assessment measures the performance of the metrics (1) on both the turn level and the dialog level, (2) for different dialog lengths, (3) for different dialog qualities (e.g., coherence, engaging), (4) for different types of response generation models (i.e., generative, retrieval, simple models and state-ofthe-art models), and (5) exploring combinations of different metrics. all of the code created for this assessment is open-sourced 1 to facilitate the easy assessment of 16 future evaluation metrics on a number of datasets. ## overview of automatic metrics in this paper, we studied 23 automatic metrics either designed for evaluating dialogues or general natural language generation (nlg) tasks. in most metrics, triplet ranking loss is used as a training objective, and it requires the model to give higher scores to appropriate responses. bert @xcite and its variant roberta @xcite are the most popular pretrained language models utilized in state-of-the-art metrics, followed by gpt-2 language model @xcite . dailydialog @xcite and per-sonachat @xcite are two widely-used datasets for training metrics. briefly, conversation topics of dailydialog are about day-to-day life, while personachat consists of dialogs where each participant is assigned a persona and the goal is to become familiar with the other individual. wordoverlap metrics are ineffective for dialog @xcite largely due to the one-to-many nature of dialog @xcite . thus, reference-free metrics have been proposed to circumvent the oneto-many problem. amongst the 24 metrics assessed here, we have 11 reference-free evaluation metrics. we refer readers to appendix a for detailed discussion of aforementioned characteristics of assessed metrics. ## testing datasets in this work, we test metrics with the quality annotated datasets: usr @xcite , grade @xcite , holisticeval @xcite , fed @xcite , dstc6 @xcite and dstc9 @xcite . concretely, each sample in the dataset consists of a dialog context, a generated response and a quality score. optionally, a groundtruth response may also be included. due to the space constraint, in this section we only provide a high-level overview of the used datasets. the complete list and discussion of characteristics of used datasets is provided in appendix b. in general, these datasets are constructed using the following steps: ## experiments this section describes the assessment of the evaluation metrics. wherever possible, the released pretrained model was used to reproduce results. since ruber, bert-ruber and pone do not release their pretrained models, those models were trained on dailydialog. for usr, we use the released model which is trained on topicalchat. for predic-tiveengage, the original paper combined the model with ruber for the best performance. in this assessment only the engagement score model is used for comparison to the other metrics. flowscore only scores dialogs with more than 3 utterances, so it cannot be used on some of the quality-annotated data. ## conclusions this paper provides a comprehensive assessment of various automatic evaluation metrics for dialog. many different evaluation metrics have been proposed in recent years. the results indicate several important directions for future work on the assessment of dialog evaluation metrics: 1) many metrics rely on pretrained language models, and the analysis in this paper highlights the impacts of them. researchers could further explore the impact of choice of pretrianed models. 2) unsurprisingly, metrics generally perform better on the data they were trained on. therefore, methods to adapt metrics to new domains should be an important research direction. 3) simple combinations of metrics yield promising results. future work should explore more sophisticated ways for combining metrics. tiancheng zhao, ran zhao, and maxine eskenazi. 2017a. learning discourse-level diversity for neural dialog models using conditional variational autoencoders. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 654-664, vancouver, canada. association for computational linguistics. tiancheng zhao, ran zhao, and maxine eskenazi. 2017b. learning discourse-level diversity for neural dialog models using conditional variational autoencoders. arxiv preprint arxiv:1703.10960. wangchunshu zhou and ke xu. 2020. learning to compare for better training and evaluation of open domain natural language generation models. in proceedings of the aaai conference on artificial intelligence, volume 34, pages 9717-9724. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in proceedings of the ieee international conference on computer vision, pages 19-27.