We Need to Consider Disagreement in Evaluation (2021)

Abstract: Evaluation is of paramount importance in data-driven research fields such as Natural Language Processing (NLP) and Computer Vision (CV). Current evaluation practice largely hinges on the existence of a single “ground truth” against which we can meaningfully compare the prediction of a model. However, this comparison is flawed for two reasons. 1) In many cases, more than one answer is correct. 2) Even where there is a single answer, disagreement among annotators is ubiquitous, making it difficult to decide on a gold standard. We argue that the current methods of adjudication, agreement, and evaluation need serious reconsideration. Some researchers now propose to minimize disagreement and to fix datasets. We argue that this is a gross oversimplification, and likely to conceal the underlying complexity. Instead, we suggest that we need to better capture the sources of disagreement to improve today’s evaluation practice. We discuss three sources of disagreement: from the annotator, the data, and the context, and show how this affects even seemingly objective tasks. Datasets with multiple annotations are becoming more common, as are methods to integrate disagreement into modeling. The logical next step is to extend this to evaluation.

## introduction evaluation is of paramount importance to natural language processing (nlp) and computer vision (cv). automatic evaluation is the primary mechanism to drive and measure progress due to its simplicity and efficiency @xcite . however, * authors in alphabetical order. figure 1 : what is the ground truth? examples from vqa v2 @xcite and @xcite today's evaluation practice for virtually all nlp tasks concerned with a fundamental aspect of language interpretation-pos tagging, word sense disambiguation, named entity recognition, coreference, relation extraction, natural language inference, or sentiment analysis-is seriously flawed: the candidate hypotheses of a system (i.e., its predictions) are compared against an evaluation set that is assumed to encode a "ground truth" for the modeling task. yet this evaluation model is outdated and needs reconsideration. the notion of a single correct answer ignores the subjectivity and complexity of many tasks, and focuses on "easy", low-risk evaluation, holding back progress in the field. we discuss three sources of disagreement: from the annotator, the data, and the context. the underlying assumption of the current approach is that the evaluation set represents the best possible approximation of the truth about a given phenomenon, or at least a reasonable one. this ground truth is usually obtained by developing an annotation scheme for the task aiming to achieve the highest possible agreement between human annotators @xcite . disagreements between annotators are either reconciled by hand or aggregated (particularly in the case of crowdsourced annotations) to extract the most likely or agreed-upon choices @xcite @xcite . this aggregated data is referred to as "gold standard" (see @xcite for an in-depth analysis of annotation methodology). however, there is plenty of evidence that gold labels are an idealization, and that unreconcilable disagreement is abundant. figure 1 shows two examples from cv and nlp. this is particularly true for tasks involving highly subjective judgments, such as hate speech detection @xcite or sentiment analysis @xcite . however, it is not a trivial issue even in more linguistic tasks, such as part-of-speech tagging @xcite , word sense disambiguation @xcite , or coreference resolution @xcite . systematic disagreement also exists in image classification tasks, where labels may overlap @xcite . disagreement and task difficulty and subjectivity also challenge traditional agreement measures @xcite . high agreement is typically used as a proxy for data quality. however, it obscures possible sources of disagreement @xcite . we summarize some of the evidence on disagreement in section 2. the need for metrics not based on the assumption that a gold standard exists has long been accepted for end-to-end tasks, particularly those involving an aspect of natural language generation, such as conversational agents, machine translation, surface realisation, image captioning, or summarization. metrics such as bleu for machine translation/generation, rouge for summarization, or ndcg for ranking web searches all support more than one gold standard reference. shared tasks in this areas (particularly on paraphrasing), have also considered the role of disagreement in their evaluation metrics @xcite . variability in the annotation is a feature of many such tasks (see, e.g., @xcite for agreement issues in generated text evaluation) even though many corpora still may come with single references due to data collection costs. high agreement is disfavored, and even bears risks of non-natural, highly homogenized system outputs for generation tasks @xcite . the main argument of this position paper is that we should recognize that the same issues, if perhaps in less extreme version, apply to the analysis tasks we discuss here. in recent years, proposals have been put forward to consider the disagreement as informative content that can be leveraged to improve task performance @xcite @xcite . @xcite and @xcite investigated the impact of disagreement-informed data on the quality of nlp evaluation, and found it to be beneficial and providing complementary information, as further discussed in section 3. this led them to organize a first shared task on learning from disagreement and providing non-aggregated benchmarks for evaluation @xcite . in contrast with this trend, bowman and dahl (2021) recently proposed to study biases and artifacts in data to eliminate them. beigman @xcite adopt a slightly softer stance, proposing to only evaluating on "easy" (as in, highly agreed upon) instances. based on the evidence about the prevalence of disagreement in nlp judgments, we argue against this approach. first, it leads to information loss in the attempt to reducing noise in the data. second, it is unnecessary: while evaluation methods that include disagreement are not yet established, several methodologies already do exist. removing the disagreement might lead to better evaluation scores, but it fundamentally hides the true nature of the task we are trying to solve. ## disagreement in nlp in this section, we outline three possible sources of disagreement. afterward, we describe how disagreement has been studied in objective and arguably more subjective tasks in nlp. ## evaluation in light of disagreement while the research mentioned in the previous section questions the assumption that a single 'hard' label (a gold label) exists for every item in a dataset, the models proposed for learning from multiple interpretations are still largely evaluated under this assumption, using 'hard' measures like accuracy or class-weighted f1 @xcite . abandoning the gold standard assumption requires the ability to evaluate a system's output also over instances on which annotators disagree. there is no consensus yet on this form of evaluation, but a few proposals have been used already. in fact, a way of performing soft evaluation exists which is a natural extension of current practice in nlp. this is to evaluate ambiguity-aware models by treating the probability distribution of labels they produce as a soft label, and comparing that to a full distribution of labels, instead of a 'one-hot' approach. this can be done using, for example, cross-entropy, although other options also exist. this approach was adopted in, inter alia, @xcite @xcite . @xcite tested this approach on image classification tasks, generating the soft label by transforming the item annotation distribution using standard normalization. @xcite employed this form of soft metric evaluation for nlp, also comparing different ways to obtain a soft label from the raw data. they use soft metrics to compare the classifiers' distribution to the human-derived label distributions, complementing traditional hard evaluation measures. basile (2020) suggested a more extreme evaluation framework, where a model is required to produce different outputs encoding the individual annotators' labels. the predictions are then individually evaluated against the single annotations, rather than against an aggregated gold standard. this proposal aims at fostering the design of 'inclusive' models with respect to diverse backgrounds in highly subjective tasks. while evaluating with disagreement is not yet widely adopted, methods for doing so exist. in the rest of this section, we discuss the two aforementioned approaches more in detail. ## conclusion in this position paper, we argue against the current prevalent evaluation practice of comparing against a single truth. this method has allowed automated evaluation, sped up model selection and development, and resulted in good evaluation scores. however, those scores hide the truth about the state of our models: many tasks are complex and subjective. assuming a single truth for the sake of evaluation amounts to a gross oversimplification of inherently complex matters. we further reject the notion that we should remove annotation noise from datasets. instead, we propose to embrace the complex and subjective nature of task labels. we show how disagreement from the annotator, the data, and the context, affects even seemingly objective tasks. research already shows that incorporating this disagreement leads to better training performance. we suggest that it can do the same for evaluation. the datasets already exist, all we need is to use them. it might not produce the same nice high scores we have gotten used to. but it will provide an honest assessment of how good our models are, and do justice to the complexity of the subject we are trying to model.