Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocols (2020)

Abstract: As conversational AI-based dialogue management has increasingly become a trending topic, the need for a standardized and reliable evaluation procedure grows even more pressing. The current state of affairs suggests various evaluation protocols to assess chat-oriented dialogue management systems, rendering it difficult to conduct fair comparative studies across different approaches and gain an insightful understanding of their values. To foster this research, a more robust evaluation protocol must be set in place. This paper presents a comprehensive synthesis of both automated and human evaluation methods on dialogue systems, identifying their shortcomings while accumulating evidence towards the most effective evaluation dimensions. A total of 20 papers from the last two years are surveyed to analyze three types of evaluation protocols: automated, static, and interactive. Finally, the evaluation dimensions used in these papers are compared against our expert evaluation on the system-user dialogue data collected from the Alexa Prize 2020.

## introduction most successful automated dialogue systems follow task-oriented dialogue management methodology, which defines an explicit goal that the system is seeking to fulfill through the conversation with the user @xcite . recently, the research in chat-oriented dialogue management has experienced a substantial increase in popularity. unlike task-oriented dialogues, where the success is generally measured as ability to complete the goal of the task, evaluation of chat-oriented dialogues is much less straightforward, since the conversational goals can be highly subjective @xcite . the evaluation of chat-oriented dialogue systems has been typically accomplished through the use of automated metrics and human evaluation (section 2). automated evaluation requires no human labor once the evaluation script is written (section 3). for automated evaluation to be a reliable measurement of the dialogue system quality, however, it needs to be shown to be a close approximation of human judgements (section 4). unfortunately, commonly used automated metrics correlate weakly with human judgments, indicating poor utility of such metrics @xcite . human evaluation has become more commonplace in recent dialogue system works; however, it presents its own challenges. for one, it is time-consuming and expensive to obtain human judgments. more critically, there is a lack of standardized protocol for such human evaluation, which makes it challenging to compare different approaches to one another. there have been many previous attempts at standardizing dialogue system evaluations. a major limitation has been their focus on task-oriented dialogue systems, which does not translate well to chat-oriented dialogue systems @xcite . previous works which have included chat-oriented evaluations have lacked comprehensive coverage over the many varieties of such evaluation procedures that are currently in use. instead, the emphasis has rested primarily on automated metrics at the expense of detailed analysis of human evaluation @xcite . at this stage in conversational ai, it is probable that automated and human metrics reveal different aspects of dialogue systems @xcite . it would be remiss to focus on a single evaluation category when assessing the state of the field. for this reason, our work aims to fill in the gaps of previous dialogue system evaluation surveys by identifying and comparing human evaluation protocols for chat-oriented dialogue systems. to this end, we present a comparative analysis of the evaluations used for chat-oriented dialogue systems over the past several years. since the field of conversational ai has experienced a rapid growth in these years, it presents a unique opportunity to observe and assess which evaluation metrics have been most widely adopted by the larger community in this period of expeditious development. we provide a detailed survey of both automated and human evaluations in order to present the most accurate depiction of the current evaluation protocols. however, our in-depth analysis is limited to that of the human evaluations due to the abundance of previous work in automated metric analysis. as such, we defer to such work as @xcite , @xcite al. (2019), and ghazarian et al. (2019) for more detail on automated metrics. as a part of our analysis, we also present a case study of real human-machine dialogues which explores the significance of different human evaluation metrics in terms of overall user satisfaction through an expert analysis. as a result of our work, the most commonly used evaluation metrics in contemporary literature -both automated and human -are revealed in detail and our findings towards the prevalence, impact, and applicability of human evaluation metrics are illustrated. ## evaluation protocols for a holistic understanding of current evaluation protocols on dialogue systems, we have carefully selected 20 relevant papers since 2018, primarily from top-tier venues, and synthesized their methods. these papers focus on open domain (or nontask-oriented) dialogue, and employ a variety of approaches including: 1 interactive evaluation is also done by human, although the evaluator assesses the quality of the dialogue after directly interacting with the dialogue system (section 2.3). ## analysis of automated evaluation ground truth response similarity most commonly used automated metrics focus on assessing how well system responses match the ground truth human responses, using word overlap (bleu, rouge) or embedding similarity. context coherence embedding similarities between dialogue contexts and system responses have been used to quantitatively assess the relevance between the system responses and the preceding dialogue history (coherence, embedding). response diversity other widespread metrics assess the diversity of the system responses in order to determine the amount of repetition and generic content in the system responses (distinct, entropy, inertia, entity score). language model fitness generative models are usually evaluated in terms of how well they learn to model the language of the dialogues in their training corpus (perplexity). ## analysis of human evaluation while automated evaluation measures dimensions of dialogue objectively, human evaluation captures the subjective assessment from the user's point of view. regardless of the exact method chosen, all human evaluations involve gathering external annotators who answer questions regarding the dialogues resulting from a dialogue system. ## case study: alexa prize 2020 this section presents a case study of the significance of the proposed dialogue dimensions in it is worth mentioning that the following findings are taken as only a preliminary analysis, strongly considering the low agreement between the expert and interactive evaluations on oq, which will be discussed shortly (section 5.2). this disparity between the expert and human user evaluations renders it difficult to convey a convincing conclusion regarding the significance of the evaluation dimensions. however, we hope this work begins the momentum to investigate the importance of such evaluation dimensions in overall human perception of dialogue quality. ## conclusion and future work in this paper, we provide an extensive background and the current states on the three types of dialogue system evaluation protocols, automated, static, and interactive. our analysis shows that static evaluation is the dominating human evaluation used in the most recent dialogue system works, although it has several concerning limitations, some of which are exemplified through our case study. we propose a set of eight dialogue dimensions that encapsulate the evaluations of previous studies without redundancy. as a result of our case study, we find preliminary evidence that the dimensions of relevance, proactivity, informativeness, and engagingness are likely to be contributing factors to the overall perception of dialogue quality. our future work will build upon these findings to develop a thorough understanding of the necessary dialogue dimensions for comprehensive interactive evaluation of dialogue systems. through an analysis based on large-scale user studies, we look to propose an evaluation protocol that captures the human judgement of dialogue quality through precise formulation of evaluation dimensions, in order to enable targeted dialogue system advancements.