Unsupervised Evaluation of Interactive Dialog with D ialo GPT (2020)

Abstract: It is important to define meaningful and interpretable automatic evaluation metrics for open-domain dialog research. Standard language generation metrics have been shown to be ineffective for dialog. This paper introduces the FED metric (fine-grained evaluation of dialog), an automatic evaluation metric which uses DialoGPT, without any fine-tuning or supervision. It also introduces the FED dataset which is constructed by annotating a set of human-system and human-human conversations with eighteen fine-grained dialog qualities. The FED metric (1) does not rely on a ground-truth response, (2) does not require training data and (3) measures fine-grained dialog qualities at both the turn and whole dialog levels. FED attains moderate to strong correlation with human judgement at both levels.

## introduction evaluation metrics often define the research direction of a field. as dialog systems begin to demonstrate human-level performance, the development and adoption of meaningful and interpretable automatic evaluation measures is essential @xcite . since standard metrics (e.g., bleu, meteor) have been shown to be ineffective for dialog @xcite , human evaluation is often used. however, it is typically only used as a final evaluation since it is costly. during development, systems are generally optimized for poorly correlated automatic metrics which can result in sub-par performance @xcite . automatic metrics must be meaningful and interpretable so that they can be used to compare dialog systems, understanding their respective strengths and weaknesses, and effectively guide dialog research. dialog evaluation is difficult for several reasons: (1) the one-to-many nature of dialog @xcite makes word-overlap metrics ineffective for scoring valid responses that deviate from the ground-truth @xcite . (2) dialog quality is inherently multifaceted @xcite and an interpretable metric should measure several qualities (e.g., interesting, relevant, fluent). (3) dialog systems have begun to be evaluated in an interactive setting @xcite where a real user has a back-and-forth conversation with a system. interactive evaluation is not constrained to a static corpus and better captures the performance of a system in a realistic setting. however, the existing automatic metrics compare to a ground-truth response, making them unsuitable for assessing interactive conversations. to address these three problems, this paper presents the fed metric (fine-grained evaluation of dialog) which assesses eighteen qualities of dialog without relying on a reference response. first, a dataset of human quality annotations is collected for the human-system (meena and mitsuku) and human-human conversations released by @xcite . dialogs are annotated at both the turn level and the dialog level for eighteen fine-grained dialog qualities. this fed dataset can be used to benchmark the performance of automatic metrics relative to human judgement. analysis of this data provides insight into the qualities of dialog that are most important to human annotators. it therefore highlights the qualities that should be the focus of attention in dialog research. the fed dataset is intended only for evaluating automatic metrics relative to human judgement. it does not consist of any training data. as such, this paper addresses the task of developing an automatic evaluation metric which (1) does not compare to a reference response, (2) assesses eighteen different qualities of dialog and (3) relies on no training data or supervision. this paper is the first, to the best of our knowledge, to address this important and challenging problem. the fed metric described here leverages a massively pre-trained model, dialogpt @xcite , which can generate practically human-level responses. @xcite assert that pretrained models implicitly capture world knowledge and can therefore perform common-sense reasoning. similarly, we posit that dialogpt has implicitly captured some notion of dialog quality and can therefore be used for dialog evaluation. @xcite assessed the quality of a system utterance in an interactive setting by looking at the following user response. the proposed evaluation metric is based on the same intuition. given a system response, its quality is measured by computing the likelihood that dialogpt will respond to it with a particular follow-up utterance (e.g., "that is really interesting!"). dialogpt is more likely to respond in this way to what it believes is an interesting system response. a set of follow-up utterances is constructed for each of the eighteen qualities and the likelihoods of these follow-up utterances are used to measure dialog quality. the fed metric obtains moderate to strong correlation with human judgement for turn-level and dialog-level evaluation without any training data or ground-truth response. analysis in this paper demonstrates that through large-scale pre-training, dialogpt has implicitly captured some notion of dialog quality. these results suggest that pretrained models can be leveraged to further improve dialog evaluation. the contributions of this paper are as follows: (1) the fed dataset was collected for fine-grained evaluation of interactive dialog, with annotations for eighteen dialog qualities at both the turn-and the dialog-level. (2) analysis of the fed dataset identifies the dialog qualities most important to human annotators. (3) dialogpt is shown to implicitly capture an understanding of dialog quality. (4) the fed metric has moderate to strong correlation with human judgement by leveraging dialogpt, without training data or reference responses. ## data collection a dataset of human quality annotations was collected to assess automatic metrics by measuring correlation with human judgements. @xcite collected a set of conversations foot_0 between a human and two open-domain dialog systems, meena @xcite and mitsuku foot_1 . in addition, they also released humanhuman dialogs collected in the same environment where one of the humans was selected to play the role of the system. we annotated a subset of these conversations with human quality judgements to create the fed dataset. workers on amazon mechanical turk (amt) annotated 40 human-meena conversations, 44 human-mitsuku conversations and 40 human-human conversations. for each conversation, three system responses were hand-selected to be annotated at the turn level, presented to the worker sequentially. then the worker was shown the entire conversation and annotated on the dialog level. five workers annotated each conversation. they did not know which system was involved in a conversation, since all mentions of the system name were replaced with the word "system." since dialog quality is inherently multi-faceted it is important to measure several different qualities of dialog. eighteen fine-grained dialog qualities are measured in the fed dataset: eight at the turn level and ten at the dialog level. ## data analysis the fine-grained nature of the fed dataset is grounds for a rich analysis. first, inter-annotator agreement is evaluated for all of the dialog qualities. next, the dataset is used to better understand the comparative strengths and weaknesses of the three systems (mitsuku, meena, human). finally, detailed analysis of the data provides insight into the fine-grained qualities that most strongly contribute to the annotators' overall impression. ## methods the fed (fine-grained evaluation of dialog) metric is an automatic evaluation metric for dialog which (1) does not need to compare to a reference response, (2) measures eighteen fine-grained qualities of dialog, and (3) does not use training data. capturing a diverse set of fine-grained qualities without supervision is an especially challenging problem. the development of the fed metric is motivated by two areas of prior work: (1) pre-trained language models and their capabilities and (2) the use of follow-up utterances as a means of evaluation. ## conclusion this paper introduces the fed dataset and the fed metric. the fed dataset is constructed by annotating a set of interactive conversations with eighteen fine-grained dialog qualities. the fed metric can be used to measure fine-grained qualities of dia- this paper sets the groundwork for several directions of future work. (1) the fed dataset can be used to benchmark automatic evaluation metrics on eighteen fine-grained dialog qualities. (2) building on this paper, future work could identify mechanisms that further leverage pre-trained models for dialog evaluation. (3) future work can explore strategies for extending the fed metric beyond open-domain chit-chat conversations to goal oriented dialog. (4) the fed metric can be used to evaluate, analyze and improve dialog systems.