Don’t Forget Your ABC ’s: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems (2023)

Abstract: Despite tremendous advancements in dialogue systems, stable evaluation still requires human judgments producing notoriously high-variance metrics due to their inherent subjectivity. Moreover, methods and labels in dialogue evaluation are not fully standardized, especially for open-domain chats, with a lack of work to compare and assess the validity of those approaches. The use of inconsistent evaluation can misinform the performance of a dialogue system, which becomes a major hurdle to enhance it. Thus, a dimensional evaluation of chat-oriented open-domain dialogue systems that reliably measures several aspects of dialogue capabilities is desired. This paper presents a novel human evaluation method to estimate the rates of many{pasted macro ‘LN’} dialogue system behaviors. Our method is used to evaluate four state-of-the-art open-domain dialogue systems and compared with existing approaches. The analysis demonstrates that our behavior method is more suitable than alternative Likert-style or comparative approaches for dimensional evaluation of these systems.

## introduction recent work in human-computer chat has made remarkable progress. multi-turn open-domain (mtod) models are capable of holding engaging conversations with humans @xcite . however, there remain a number of challenges facing mtod chatbots such as hallucinations @xcite , commonsense violations @xcite , and consistency issues @xcite . a significant obstacle for research that addresses these challenges is the difficulty in formulating an appropriate evaluation methodology due to the inherent subjectivity in determining chat quality @xcite . since existing automatic evaluation metrics have been shown to be biased measures of chat quality (liu et al., 2016; sai et al., 2019; deriu * contributed equally to this work as first authors. @xcite , evaluation using human judgments is standard, although the type of human judgments varies widely across works @xcite . overall, there are few works comparing and assessing the validity of various human evaluation methods. the result of this gap in the literature is that the relative sensitivity, interpretability, and importance of the metrics used to evaluate chat models are not well understood. a dimensional approach for evaluating chat models that measures different aspects of chat quality would surely aid progress @xcite . however, to our knowledge, no work has investigated the coverage of a comprehensive set of evaluation metrics. consequently, existing chat model evaluation results provide an incomplete picture of the strengths and weaknesses of mtod chatbots. this paper addresses these limitations of previous work through the following three contributions: by presenting a detailed picture of mtod chatbot performance and standard methods to evaluate them, we aid future work's efforts to further understand and improve human-computer interaction. our evaluation platform, analyses, and data are available at https://github.com/emorynlp/ chatevaluationplatform . ## chatbots to evaluate the strengths and weaknesses of mtod models, we select the chatbots for our study using a two-stage process: (1) a literature review to identify chatbot candidates, and (2) a pilot evaluation to select the final set of bots for our full study. literature review to promote diversity among the selected chatbots, we focus our review on four popular themes of the human-computer chat: (1) knowledge-grounded chat, (2) empathetic chat, (3) self-consistent chat, and (4) general open-domain chat with large pre-training resources like reddit. candidate chatbots are selected from each theme using the following criteria: this review yields the 6 chatbot candidates in ## conversation collection the conversation dataset used for the full study is collected using human interactors in a text-based conversation setting. 46 undergraduates are recruited as interactors. each interactor is compensated with a @xmath020/hr foot_5 . we allow evaluators to annotate up to 60 conversations per task. our final evaluation dataset consists of 400 conversations, each with results for all 40 labels. 8 additionally, a randomly-selected subset of 100 conversations (and 50 of the conversation pairs) is evaluated a second time by a different surger in order to measure iaa. ## chatbot evaluation to evaluate the strengths and weaknesses of our 4 selected chatbots, we present results for the 400 collected conversations across all abc-eval metrics (figure 4 and figure 5 ), likert dialogue metrics (figure 6 ), likert turn metrics (figure 8 ), and comparative metrics (figure 7 ). we focus our discussion on the final set of abc-eval metrics since they performed best in our metric analysis. the results highlight the notable recent progress in human-computer chat, as the vast majority of chatbot turns are interpretable, relevant responses to the dialogue context. less than 1% of re11 see appendix f for detailed cost results. despite these strengths, our results also show several clear directions for improvement. commonsense violations are present in about 15-20% of the bots' responses. consistency issues are prevalent across all bots: self-contradictions, partner contradictions, and redundancies appear in about 5% of the bots' responses overall. also, all chatbots have a substantial rate of violating natural dialogue structure: about 10% of responses are judged as ignoring the user, and depending on the chatbot, around 10-20% of responses include irrelevant contributions to the dialogue. additionally, 5-15% of the chatbots' responses show a lack of empathy or other emotional misunderstandings. the reality of these observed rates of problematic behaviors is that, in most 30-turn conversations with these chatbots, a human interactor is likely to experience several issues that impact conversation quality. con d emo d eng d gra d inf d pro d qua d rel d 3 4 5 bart-fid-rag emora blender2 blender-decode figure 6: average dialogue likert ratings of the conversations, with 95% student's t confidence intervals. con t emo t eng t gra t inf t pro t qua t rel t 3 4 5 bart-fid-rag emora blender2 blender-decode ## conclusion as illustrated here, dialogue quality is a complex construct with many dimensions. depending on the approach, dialogue systems can have markedly different weaknesses among these quality dimensions. our research highlights several outstanding challenges, especially regarding the relevance, consistency, common-sensibility, and emotional understanding of chat model responses. our analyses not only demonstrate that these four dimensions have a high impact on conversation quality, but also that current chatbots have substantial response error rates in these areas. to efficiently address the challenges facing opendomain dialogue models, we need a reliable, dimensional evaluation method; however, our results show that popular evaluations such as dialoguelevel likert and comparative methods may not be suitable. the presented abc-eval serves as a promising alternative in this direction. although the popular dialogue-level likert evaluation method may be the most cost-effective and robust method for measuring overall dialogue quality, we recommend that researchers additionally use the final set of abc-eval metrics, or a subset relevant to their scientific goals, to evaluate specific strengths and weaknesses of new chat models. overall, we hope future work can use insights from our study to make better-informed decisions about which evaluation method to use, and to tackle the challenges facing current chatbots.