Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents (2022)

Abstract: At the heart of improving conversational AI is the open problem of how to evaluate conversations. Issues with automatic metrics are well known (Liu et al., 2016), with human evaluations still considered the gold standard. Unfortunately, how to perform human evaluations is also an open problem: differing data collection methods have varying levels of human agreement and statistical sensitivity, resulting in differing amounts of human annotation hours and labor costs. In this work we compare five different crowdworker-based human evaluation methods and find that different methods are best depending on the types of models compared, with no clear winner across the board. While this highlights the open problems in the area, our analysis leads to advice of when to use which one, and possible future directions.

## introduction any comprehensive analysis of the performance of an open-domain conversational model must include human evaluations: automatic metrics can capture certain aspects of model performance but are no replacement for having human raters judge how adept models are at realistic and interesting conversation @xcite @xcite . unfortunately, human evaluations themselves must be carefully constructed in order to capture all the aspects desired of a good conversationalist. any evaluation technique must evaluate over many turns of a conversation in order to detect emergent faults such as repetitiveness or contradiction, while techniques that rely solely on a single evaluation at the end of a conversation may fail to take into account changes in model performance over its span. further, techniques that rate model performance on a likert scale may suffer from inconsistencies in subjective numerical ratings across evaluations of different models @xcite . when comparing various human evaluation methods to assess which works best, we find that each has success and failure cases, leading us to conclude that human evaluation is still an open problem. in this work, we analyze a representative set of human evaluation techniques. first, we compare per-turn evaluations, where ratings are given after every model response, and per-dialogue evaluations, where ratings are collected solely at the end of the conversation. per-turn evaluations have the advantage of being more fine-grained, encouraging annotators to focus on small differences; however, the quality of a conversation is more than the sum of its parts, and global per-dialogue evaluations can capture this better. second, we consider pairwise methods, where two models are compared directly by an annotator, to single-model methods, where the annotator sees and rates only one model at a time. both approaches can be either per-turn or per-dialogue. for example, in pairwise per-turn evaluation, a crowdworker chats with a dialogue agent, and after each of the worker's messages, they must choose between two possible responses from the agent, one from each of two different models. the pairwise approach can spot subtle differences apparent when comparing responses, and it can mitigate problems with distribution shift that occur in absolute scoring. single-model approaches, however, can work well when direct comparison is not paramount. we compare all of these different techniques for evaluating dialogue models in three different settings, and we contrast their individual strengths. we find that: these findings, while highlighting the difficulty of human evaluation, also provide guidance on which method might be best to use in these different circumstances, as well as possible future work. in particular, investigating the best way to merge pairwise and single-model, per-turn and per-dialogue benefits into a single method could be a fruitful direction. we also analyze the interpretability of these approaches when collecting human written explanations. we have released code for these evaluation techniques in the parlai framework. foot_0 ## existing work this work concerns itself with evaluation of opendomain dialogue, which, unlike more restricted domains such as question-answering and goaloriented conversations, may not have a precise goal, and no widely accepted evaluation technique for it currently exists @xcite @xcite . automatic metrics are relatively fast, efficient, and reproducible, but many of them have been shown to "correlate very weakly with human judgement" @xcite , see also @xcite ), and the best way to create a reliable automatic metric is still up for debate @xcite . in this work we focus on human evaluation, and in particular on employing crowdworkers, which has an advantage over utilizing trained experts @xcite or deployment @xcite of allowing for a larger pool of evaluators and for ensuring alignment with research goals, respectively. however, the use of crowdworkers itself has a number of pitfalls to avoid as well @xcite . particular instruction wording choices to crowdworkers have a large effect on the quality of conversations and resulting evaluations @xcite . wording can direct workers to evaluate specific facets, such as general "get to know each other" chitchat @xcite , getting a bot to generate unsafe utterances @xcite , and instructing crowdworkers to be adversarial vs. not @xcite . one can also pick from a variety of specific questions when asking crowdworkers to rate conversations, including asking about interestingness, making sense, fluency @xcite , sensibleness, specificity @xcite , toxicity, and bias @xcite , and the exact phrasing of these questions can have a large impact on sensitivity @xcite . standard evaluation protocols have a single human both converse with a model and rate that conversation in the same task, but other methods have a rater rate pre-existing conversations between a human and a model or between a pair of models @xcite . these latter techniques allow for efficient reuse of existing conversational data, and have shown to be useful experimentally @xcite , but it may be harder for evaluators to rate conversations that they have not been involved in. another choice when designing evaluation protocols is whether conversations are rated individually, e.g., with likert-score ratings @xcite , see more in appendix a), or pairwise by comparing models @xcite . likert scoring suffers from weaknesses such as potential per-annotator bias @xcite and drift in the distribution of errors over time @xcite , but is more efficient than pairwise comparisons in that new models' ratings can be compared to those of older models without having to re-collect those older models' ratings. lastly, evaluation techniques differ in whether they collect ratings on each turn of the conversation @xcite or only at the end of the conversation, as in acute-eval @xcite . whole-conversation techniques can work well if the quality of a conversation is assumed to be more than just the sum of its parts, but could perhaps suffer due to the primacy effect and recency effect that appear when more weight is given to information presented at the start and end of the rating session, respectively @xcite @xcite . see appendix a for a more thorough assessment of related works. ## conclusion in this work we compare the extent to which different evaluation techniques are able to measure performance differences between dialogue models, and we show instances in which the performance varies between per-turn techniques and perdialogue techniques, and between pairwise techniques and single-model techniques. a completely exhaustive analysis of the cases in which each technique is most appropriate would require measurement on many more pairs of models than the three studied here, and would likely require a dramatic scaling-up of labor for crowdworkers. nevertheless, the results shown here demonstrate the difficulty in anointing one evaluation technique as superior to all others regardless of the models being compared, and they suggest that a combination of techniques, or else a different technique entirely, may be necessary for optimal measurement of differences among models. a more universally ideal technique would likely need to investigate model performance per-turn but still be able to give an overall judgment of model quality across a conversation in order to capture elements of performance that manifest clearest in a single response vs. in aggregate. we demonstrate that combining evaluation scores from per-turn and per-dialogue techniques can bridge the gap in the performance differences between the two, but that this does not outperform either individual technique in all cases, at least in the way that we combined them. future improvements may also come from exploring other ways to amplify the weak signal from models with only slight performance differences such as blenderbot3b and blenderbot90m, perhaps by training workers to select responses based on general measures of conversational quality, as opposed to content that appeals to their personal interests. improving sensitivity to roughly equivalent pairs of models such as these should in turn enable the comparison of models whose performance differences are smaller still. while this work has concentrated on evaluating techniques that enable differentiability (one can differentiate between models) with efficiency (with less annotator hours), there are other desirable qualities as well. some of these in particular are diversity of conversations @xcite , repeatability of experiments, and explainability of results @xcite . while there is some discussion of the latter two topics in our experiments, these topics are fully deserving of a more thorough analysis than is provided here. yi-ting yeh, maxine eskenazi, and shikib mehri. 2021. a comprehensive assessment of dialog evaluation metrics. in the first workshop on evaluations and assessments of neural conversation systems, pages 15-33. saizheng zhang, emily dinan, jack urbanek, arthur szlam, douwe kiela, and jason weston. 2018. personalizing dialogue agents: i have a dog, do you have pets too? in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2204-2213.