Beyond User Self-Reported L ikert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation (2020)

Abstract: Open Domain dialog system evaluation is one of the most important challenges in dialog research. Existing automatic evaluation metrics, such as BLEU are mostly reference-based. They calculate the difference between the generated response and a limited number of available references. Likert-score based self-reported user rating is widely adopted by social conversational systems, such as Amazon Alexa Prize chatbots. However, self-reported user rating suffers from bias and variance among different users. To alleviate this problem, we formulate dialog evaluation as a comparison task. We also propose an automatic evaluation model CMADE (Comparison Model for Automatic Dialog Evaluation) that automatically cleans self-reported user ratings as it trains on them. Specifically, we first use a self-supervised method to learn better dialog feature representation, and then use KNN and Shapley to remove confusing samples. Our experiments show that CMADE achieves 89.2% accuracy in the dialog comparison task.

## introduction open-domain dialog system evaluation is one of the most difficult challenges in the dialog community. open-domain chatbots have a user-centric goal: to provide human with enjoyable user experience. however, user experience is difficult to quantify due to bias and variance among different users. previous research has optimized on automatic dialog evaluation metrics such as blue @xcite , which measures the difference between the generated responses and the reference responses. due to the contrast between the one-tomany nature of open-domain conversations and the limited number of available references, such met-rics correlate poorly with human judgments @xcite @xcite . designing a fully automatic dialog evaluation metric is still an open research problem. currently, both academia and industry @xcite @xcite rely on human ratings to evaluate open-domain dialogs. following the ubiquitous application of likert scores in survey research like online reviews @xcite and consumer satisfaction @xcite , a common practice of human evaluation on dialogs is to ask either a third-person rater or the chatbot user to report a likert score. however, concerns have been raised about the validity of likert score-based ratings. kulikov et al. @xcite observe high bias and variance of likert scores. such issue is more severe in real-world commercial dialog systems like alexa social chatbot @xcite , because the real-world users have neither monetary incentive nor necessary annotation training to calibrate their ratings. to explore the validity of likert score based dialog evaluation, we first perform a large-scale data analysis of 3,608 collected real-world humanmachine dialogs along with their self-reported likert scale ratings from amazon alexa prize challenge @xcite @xcite . one noticeable property of the ratings is its j-shape skew distribution: nearly half of the dialogs are rated with the highest likert score. the prevalence of such extreme distribution of ratings has long been observed by the business research community in variable aspects of reallife @xcite @xcite . although we could tell which dialog system is better by running statistical test on a large number of noisy ratings, it is difficult to locate dialogs with bad performance reliably to improve dialog system quality. in this paper, we take on the challenge of calibrating a large number of noisy self-reported user ratings to build better dialog evaluation models. we formulate the task as to first denoise the self-reported user ratings and then train a model on the cleaned ratings. we design cmade (comparison model for automatic dialog evaluation), a progressive three-stage denoising pipeline. we first perform a self-supervised learning to obtain good dialog representations. we then fine-tune cmade on smoothed self-reported user ratings to improve the dialog representation while preventing the network from overfitting on noisy ratings. finally, we apply data shapley to remove noisy training data, and fine-tune the model on the cleaned training set. our experiments show that cmade is able to successfully identify noisy training data and achieves 89.2% in accuracy and 0.787 in kappa on a test set with unseen expert-rated dialog pairs. ## related work open-domain dialog system evaluation is a longlasting challenge. it has been shown that previous automatic dialog evaluation metrics correlate poorly with human judgments @xcite @xcite . a wellknown reason is that these automatic dialog evaluation metrics rely on modeling the distance between the generated response and a limited number of references available. the fundamental gap between the open-ended nature of the conversations and the limited references @xcite is not addressed in methods that are lexical-level based @xcite @xcite , embedding based @xcite , or learning based @xcite . given the aforementioned limitations, likertscore based rating is the de-facto standard for current dialog research and social conversational systems such as in amazon alexa prize challenge @xcite . various forms of evaluation settings have been explored to better measure human judgments. single-turn pairwise comparison @xcite is primarily used for comparing two dialog systems. each system predicts a single utterance given the static "gold" context utterance from human-human logs. although such a/b test setting is robust to annotator score bias, it cannot capture the multiturn nature of dialogs. a more complete multi-turn evaluation is typically measured with a likert scale for the full dialog history, where either a third-person rater or the chatbot user @xcite reports a likert score on user experience @xcite , engagement @xcite or appropriateness @xcite . however, as observed in @xcite @xcite likert scores suffer from bias and variance among different users. different from previous empirical observations, we conduct a large-scale quantitative and qualitative data analysis of likert score based ratings. to address the issue of likert scores, the alexa team proposed a rule-based ensemble of turn-granularity expert ratings @xcite , and automatic metrics like topical diversity @xcite and conversational breadth. acute-eval @xcite ) makes a small-scale attempt to use multi-turn pair-wise comparison to rank different chatbots. given the ubiquity and simplicity of likert scores based evaluation, instead of proposing an alternative measure, we take on the challenge of denoising likert scores with minimal expert annotations introduced (one order of magnitude smaller). different from @xcite , our proposed expert annotation scheme is for comparing the dialogs within the same chatbot. ## problem formulation suppose the training set d train consists of data points d @xmath0, y i )} n train 1 where x i is a dialog and y i is the noisy self-reported user ratings. we define a strict partial order relationship where x i x j means that dialog x i provides a better user experience than dialog x j . note that y @xmath1.3, § 3.4). the test set d test consists of n test dialog pairs along with their binary pair-wise comparison labels d @xmath2, x test j , z test i,j )} i,@xmath3, where z test i,j is annotated by experts and indicates whether dialog a provides a better user experience than dialog b, i.e., z test i,@xmath4. the development set d dev has a similar structure. following the structure of the expert annotated pairs, we formulate our model m (φ, f ) as a pairwise dialog predictor with a similar architecture as ranknet @xcite . for a dialog pair (x i , x j ), the model predicts an un-normalized score o i , o @xmath5: ## conclusion the ultimate chatbot evaluation metric should be user-centric, as chatbots are there to provide human with an enjoyable experiences. previously likertscore based self-reported rating is the de-facto standard for current dialog evaluation . however, our analysis indicates that self-reported dialog ratings are skewed (j-shape), noisy and insensitive due to bias and variance among different users. we propose a three-stage denoising pipeline cmade to reduce self-reported ratings and, at the same time, build an automatic comparison-based automatic dialog quality predictor. cmade's results highly correlate with expert judgments on pair-wise dialog comparison ratings (89.2% agreement, 0.787 user: the whole harry potter series sys: whoops i think i misheard you. can you repeat the movie name please? user: all the harry potter series sys: ah, i don't know any movies by that name. sorry if i sometimes confuse you saying something else with a movie name ... ... sys: oh, okay. anyways, i really enjoyed watching black panther and wonder woman recently. what's your favorite superhero movie? user: super man sys: what was that? can you please say the movie name again? user: stop i gotta go