Achieving Reliable Human Assessment of Open-Domain Dialogue Systems (2022)

Abstract: Evaluation of open-domain dialogue systems is highly challenging and development of better techniques is highlighted time and again as desperately needed. Despite substantial efforts to carry out reliable live evaluation of systems in recent competitions, annotations have been abandoned and reported as too unreliable to yield sensible results. This is a serious problem since automatic metrics are not known to provide a good indication of what may or may not be a high-quality conversation. Answering the distress call of competitions that have emphasized the urgent need for better evaluation techniques in dialogue, we present the successful development of human evaluation that is highly reliable while still remaining feasible and low cost. Self-replication experiments reveal almost perfectly repeatable results with a correlation of r=0.969 . Furthermore, due to the lack of appropriate methods of statistical significance testing, the likelihood of potential improvements to systems occurring due to chance is rarely taken into account in dialogue evaluation, and the evaluation we propose facilitates application of standard tests. Since we have developed a highly reliable evaluation method, new insights into system performance can be revealed. We therefore include a comparison of state-of-the-art models (i) with and without personas, to measure the contribution of personas to conversation quality, as well as (ii) prescribed versus freely chosen topics. Interestingly with respect to personas, results indicate that personas do not positively contribute to conversation quality as expected.

## introduction evaluation of open-domain dialogue is particularly challenging and has been cited in high-profile competitions as a known open problem @xcite . challenges arise primarily from the fact that in real-world conversations there exists such a vast number of possible appropriate responses. subsequently, dialogue evaluation that relies on comparison with pre-created reference dialogues incur substantial false-negative rates as many appropriate responses are unfairly penalized simply for not corresponding closely with references. in addition, evaluation faces further challenges with respect to the ability to fully take into account dialogue history. 1 in this paper, we present a new method of opendomain dialogue evaluation based on human assessment of live conversations with models that avoids the need for pre-created reference dialogues and ensures full familiarity with dialogue history, ticking two important boxes in terms of validity. although live human evaluation of models has the advantage of being highly valid, reliability unfortunately cannot be assumed and developing methods of evaluation for language tasks that achieve high rater consistency has been challenging, often resulting in low levels of agreement between annotators @xcite @xcite @xcite . despite challenges in this respect, our proposed method provides highly reliable evaluation, achieving a correlation of @xmath0.969 in selfreplication experiments. additionally, the evaluation can be carried out cheaply and on a large scale through strict quality controlled crowd-sourcing, as well as including score standardization for fairer ranking of competing models. we make the data and code publicly available to aid future research. 2 ## problems in past evaluations a common issue occurs that can potentially impact the validity of results is filtering the set of systems to be evaluated via automatic metric scores. since metric scores are known to be a poor substitute for human assessment, this only results in the possibility that the best system according to human judges is inadvertently filtered out at this stage. for example, convai2 @xcite ranked models firstly using automatic metrics before top models according to metric scores were assessed by crowd-sourced workers on mechanical turk, while similarly in the sixth dialog system technology challenge (dstc6) systems were filtered according to metric scores prior to human evaluation. in terms of the live evaluation, competitions such as convai2 report such evaluations as highly challenging, with many of the resulting dialogues reported to be senseless, offensive, or simply not in line with instructions and ultimately live evaluation results have been discarded. despite challenges, competitions that operate in the public domain, making data and evaluation techniques available to researchers (such as ourselves) should be applauded for such efforts. on the other hand, competitions that (for one reason or another) do not release data and evaluation techniques into the public domain have reported relative success in terms of human evaluation. however until such methods can be accessed and independently verified through replication studies, they will unfortunately have little impact . the first amazon alexa socialbot grand challenge required human assessors to score how coherent and engaging conversations were on a 1-5 rating scale by two distinct groups: volunteer amazon employees (experts), and general alexa users (crowds) @xcite , are reported to achieve a correlation of overall scores for the two types of human assessors at 0.93. the absolute average rating across all chatbots was reported to be 20% lower for experts compared to general users. in an additional effort to evaluate models, conversational user experience, coherence, engagement, domain coverage, topical diversity, and conversational depth were assessed (1-5 scale), with combined scores reported to correlate with those of general users at @xmath1.66. in addition to methods and data not being publicly available, correlations are difficult to interpret since no detail is provided about the number of judgments on which the correlation is calculated for example. in addition to competitions that generally aim to include human evaluation of systems, automatic metrics are often proposed for dialogue evaluation, themselves requiring a human evaluation data set on which to evaluate the proposed metric. however, inappropriate statistics are often applied. for example, @xcite propose a holistic metric to automatically evaluate four distinct aspects of dialogue, and a human evaluation experiment is deployed on mechanical turk using a 1-5 rating scale. the mean correlation between human assessors is reported as @xmath2.61. however, mean correlations are unfortunately difficult to interpret, since correlation coefficients are not additive , averages calculated in the usual way cannot be assumed to reflect central tendency, and unfortunately, the distribution of correlations is not reported @xcite . @xcite propose usr (un-supervised and reference-free), an unsupervised model that predicts the quality of dialog for a range of criteria using various rating scales: understandable (0-1 rating scale), natural (1-3), maintains context (1-3), interesting (1-3), uses knowledge (0-1); overall quality (1-5). despite human evaluation being carried out by experts inter-annotator agreement levels varied depending on criteria being measured, ranging from as low as 0.298. additionally, although correlations between human assessments are reported as significant at @xmath3.01, despite such statistics often being reported for correlations, they are unfortunately not very meaningful in terms of their impact on correlation interpretation and can be somewhat misleading. contrary to common expectations, even small effect sizes (low r) can produce very low p-values (strong significance) in such tests. aiming to achieve a significant correlation is an extremely low bar to reach in terms of consistency, since a low p-value in this case simply rejects the null hypothesis that the correlation is zero. in addition to the above issues, human evaluation of dialogue systems rarely take into account the fact that differences in performance can occur simply by chance. the method of human evaluation we propose provides a means of applying standard tests for statistical significance to avoid concluding differences that are highly likely to have occurred simply by chance. ## crowd-sourcing reliable human assessment of open-domain dialogue crowd-sourcing with highly accurate quality control provides a potential mechanism to ensure the three most important criteria that makes an eval-uation meaningful while still remaining feasible: validity, reliability and scalability. subsequently, we ask crowd-workers to carry out live text-based chat with models prior to that same worker also rating the quality of the immediately preceding conversation. ## meta-evaluation in order to assess the reliability of the proposed method of human evaluation, we carry out a metaevaluation in which we firstly examine individual human assessor consistency, before conducting a self-replication experiment. a number of models are required to function as a sample set of test systems, and for this purpose we employ available pre-trained models from parlai: foot_3 poly-encoder transformer @xcite , bi-encoder transformer @xcite , sequence to sequence @xcite , key-value memory networks @xcite , and a lstmbased model @xcite . within the evaluation setting of convai2, each model is with a persona consisting of approximately five textual statements to emulate a personality. however, to increase the number of models and to provide an interesting comparison, we additionally include a version of each of the above models without any persona, resulting in 10 competing models. hits were posted on the crowd-sourcing platform amazon mechanical turk. 5 firstly, and in order to evaluate the open-domain models in as realistic a setting as possible, we allow workers to choose the topic of conversation and input their chosen topic in a text field. the open nature of conversations should be noted however as something that influences the difficulty of producing consistent results in our self-replication experiment. the fact that we allow human assessors to freely choose the topic of conversation means that differences in ratings could result from legitimate differences in performance when different topics are chosen by human assessors. we nonetheless test our evaluation allowing the user to choose the topic as this is part of our core aim for developing evaluation of dialogue truly in the open domain. besides choosing a topic, we additionally asked workers to input their opinion of the topic they chose to discuss with models, categorizing the topic as either liked, ambivalent about it, or disliked. for example, if the topic they chose to discuss was dogs, we were curious to know if this was motivated by the fact that the worker liked or disliked dogs or indeed that they had chosen to discuss something they had no particular feeling about. ## persona contribution to system performance since we have verified the reliability of the human evaluation, we take a closer look at the results and investigate dialogue quality when models employ a persona. results in ## evaluating with prescribed topics in contrast to the initial experiment in which workers were permitted to choose the topic of conversation, we further investigate the performance of models in a slightly easier setting where the topic under discussion is known to the model, by selecting a statement from its persona, which we refer to as an ice-breaker topic statement. an ice-breaker topic statement is then provided to human assessors at the beginning of each conversation, and the assessor is instructed to talk about this topic with the model. we therefore provide the topic of conversation to workers in the form of an ice-breaker topic statement, corresponding to a randomly selected persona statement belonging to the agent. again, we run this experiment on mturk, this time contrasting results for our initial data collection run where workers freely chose a topic with one in which workers were instructed to talk about the ice-breaker statement with models. numbers of workers who participated in the icebreaker run are provided in ## conclusion development of reliable evaluation of open-domain dialogue has been highlighted as a known openproblem. we overcome previous challenges and provide a new human evaluation methodology shown as highly consistent, with results for models correlating at @xmath4.969 in two separate data collection runs. our evaluation has the advantage of highly accurate quality control of crowd-sourcing, differences in scoring strategies to be ironed out via score standardization, applicability of standard significance testing while increasing the reliability of results. normal degraded user: hello, how are you? user: how are you? bot: i am good , a little tired but good ! bot: i do not have a job. i have broken have a job. user: i am tired as well, but good too user: sorry about that? bot: i am tired too , but from being out all day . bot: thanks. i the love watching old. user: what do you like doing? user: tell me about it? bot: i enjoy movies , reading books , and even just doing nothing bot: that is cool, do you i supposed to i you travel? user: i love reading books user: i do not like travelling bot: i enjoy reading too ! what is your favorite book ? bot: i teach meet you, sorryy elementary school. user: i love the novel "voice of the heart" user: okay, do you like to swim? bot: i have been wanting to read that , i love reading too bot: do you live sports are u skin am in a house. user: you should read too user: i love sports, especially soccer bot: i do love to read when i have time bot: i like but ishuold go, books user: what book was the last book you read? user: how often do you read? bot: the notebook , a reread lol bot: i are your think u r i am a pilot , so i do not have much time . user: i haven't read that user: i envy you, i would love to be a pilot bot: i read it yesterday . what do you do for fun ? bot: i i am 4 felicia! user: i love going hiking user: i am ben bot: i have not gone hiking a while bot: oh, how nice. do you do for you have? user: i think you should user: i do not understand bot: i should bot: i is your favorite don't i am sure you will find out soon. figure 4: a typical human-model conversation (left) and a human quality-degraded model conversation (right) where random response and meaning distortion techniques have been applied to degraded model responses. 1. your task is to have 6 conversations with a chatbot, and a different chatbot will talk to you in each conversation. 2. before each conversation, you should think of a topic to talk about with the chatbot (your choice of topic). you will be asked to enter this topic before the conversation starts. 3. the current topic will be displayed to you throughout the conversation. 4. if the chatbot changes the topic to a new one, you should record this by updating the conversation topic using the topic button (bottom left). 5. you are also allowed to change the topic, you should use the same button to do this (bottom left). 6. at the end of each conversation, you should tell us what you think about the chatbot. 7. in each conversation, you should type in a minimum of 10 inputs/sentences. 8. the purpose of these hits where you will generate conversations with chatbots is to test how realistic their conversations are with users. in order for your data to be useful to us we require that your half of the conversation is also realistic. for example, your data will not be useful to us if you do the following: user: hi bot: hi user: hi bot: hi .. and so on. 9. another example, if you are too repetitive or your responses are not appropriate given what the chatbot has just said, this will not be a useful test for them. for example, the following conversation is not ok: user: hi bot: hi user: wow (not appropriate response) bot: i saw a good movie last night user: wow (repetitive) bot: do you like football? user: i have two children and one dog. (not appropriate response)