Would you describe a leopard as yellow? Evaluating crowd-annotations with justified and informative disagreement (2020)

Abstract: Semantic annotation tasks contain ambiguity and vagueness and require varying degrees of world knowledge. Disagreement is an important indication of these phenomena. Most traditional evaluation methods, however, critically hinge upon the notion of inter-annotator agreement. While alternative frameworks have been proposed, they do not move beyond agreement as the most important indicator of quality. Critically, evaluations usually do not distinguish between instances in which agreement is expected and instances in which disagreement is not only valid but desired because it captures the linguistic and cognitive phenomena in the data. We attempt to overcome these limitations using the example of a dataset that provides semantic representations for diagnostic experiments on language models. Ambiguity, vagueness, and difficulty are not only highly relevant for this use-case, but also play an important role in other types of semantic annotation tasks. We establish an additional, agreement-independent quality metric based on answer-coherence and evaluate it in comparison to existing metrics. We compare against a gold standard and evaluate on expected disagreement. Despite generally low agreement, annotations follow expected behavior and have high accuracy when selected based on coherence. We show that combining different quality metrics enables a more comprehensive evaluation than relying exclusively on agreement.

## introduction would you say leopards are yellow? most likely, some people would while others would not. both interpretations are valid, as the interpretation depends on a person's boundaries for the properties 'yellow' and 'brown'. selecting only one judgment would disregard the vagueness of the expression, a phenomenon at the heart of lexical semantics. at the same time, most people would probably agree that wine can be red without having to think about it. a high number of semantic annotation tasks is characterized by unclear, difficult, ambiguous and vague examples. annotation, in particular when distributed among a crowd, has the potential of capturing different interpretations, conceptualizations and perspectives and can thus provide highly relevant semantic information. existing evaluation and label extraction methods, however, still heavily rely on agreement between annotators, which implies a single correct interpretation. finished datasets rarely provide indications about difficulty and ambiguity on the level of annotated units. the explanatory power of nlp experiments that aim to evaluate or analyze models depends on the informativeness of the data. this is particularly relevant for experiments which specifically aim to understand models better, such as the tradition of diagnostic experiments @xcite .traditional error analyses could also benefit substantially from test sets which contain information about phenomena with a likely impact on model performance. furthermore, knowing whether model-errors are similar to human disagreements can yield important insights about models. for instance, an analysis of natural language inference models shows that classifiers do not necessarily capture the same type of ambiguity and uncertainty as reflected in the annotations @xcite . error analysis often require manual annotation and tend to focus on small and not representative subsections of test sets @xcite . we argue that the behavior of human annotators can provide rich information which should be exploited, rather than reduced to single labels. information about (dis)agreement is a by-product of the original annotation effort and thus comes for free. it can form the basis of an error analysis or, in the case of our data, should be used to draw informative conclusions from diagnostic experiments. such experiments crucially depend on the quality and informativeness of the underlying data @xcite . in this paper, we present an approach to crowd-annotation for a diagnostic dataset which attempts to tackle these limitations. the dataset is meant to test which semantic properties are captured by distributional word representations. the task is designed to trigger fine-grained semantic judgements of potentially ambiguous examples. the behavior of ambiguous words in distributional semantic models is not well understood and thus particularly interesting @xcite @xcite . we investigate to what extent existing and new quality metrics indicate annotation accuracy on the one hand and ambiguity and difficulty of annotation units on the other hand. we evaluate our task from three perspectives: (1) comparison against an expert-annotated gold standard, (2) a task-specific coherence metric independent of agreement and (3) evaluation in terms of inter-annotator agreement metrics compared to predefined expectations about agreement and disagreement. in particular, we aim to investigate (1) how we can exploit the strengths and weaknesses of various suggested metrics to select and aggregate labels provided by the crowd, ( 2 ) to what degree disagreement among workers occurs in cases where it is expected and legitimate and (3) which metrics are suitable for detecting annotation units with legitimate and informative disagreement. 1 disagreement has been shown to indicate ambiguous cases when measured with the crowdtruth framework @xcite . however, we are not aware of work which compares different (dis)agreement and difficulty metrics. to the best of our knowledge, there is no study which tests how well different metrics can be used to identify ambiguous annotation units in a set of units annotated in terms of expected and legitimate disagreement. we show that the metrics we use give complementary insights and can be used to filter and aggregate labels in a way that produces highquality annotations. despite a relatively low inter-annotator-agreement, we show that worker behavior follows our expectations about agreement and disagreement and that high-quality labels can be extracted from the annotations, in particular for cases where we expect worker agreement. the remainder of this paper is structured as follows: after reviewing related work (section 2), we introduce the use-case of a diagnostic dataset (section 3) and describe the annotation task (section 4). we present our expert-annotated gold standard in section 5 and different quality metrics in section 6. the results of our experiments are described in section 7, followed by a discussion and conclusion. ## related work recent annotation studies recognize that ambiguity, vagueness and varying degrees of difficulty are inherent to semantic phenomena @xcite @xcite @xcite . @xcite demonstrate that the fundamental task of natural language inferencing contains large proportions of instances with multiple valid interpretations and argue that this phenomenon is central to the task rather than an aspect which can be disregarded. @xcite show that even experts disagree on a difficult semantic annotation task and that interpretations are likely to vary due to differences in conceptualizations, which are in themselves justified and cannot simply be disregarded as 'mistakes'. information about ambiguity and difficulty is crucial for a number of areas in nlp. while traditional experiments tend to accept the level of informativeness provided by standardly used corpora, the rather recent trend of model analysis @xcite particularly highlights the necessity of informative data. such approaches test to what extent uninterpretable, machine or deep learning-based models capture certain aspects of information. they are highly experimental and crucially depend on the quality and informativeness of the diagnostic dataset @xcite . in some cases, establishing a clean dataset is less problematic, for example when considering relatively clear and uncontested aspects of linguistic analysis (e.g. part-of-speech information @xcite ) or when creating entirely artificial data @xcite . approaches which aim to capture information about semantics (such as embedding analysis @xcite ), however, are much more complex as ambiguity, vagueness and differences in required knowledge are by no means marginal phenomena and cannot simply be disregarded. furthermore, the role of ambiguity in the behavior of word embeddings is not fully understood yet @xcite . we have designed an annotation task to analyze how different aspects of word meaning are represented in distributional representations @xcite . in this paper, we investigate how we can measure the quality of the annotations and capture valid disagreement, which is crucial information for the diagnostic experiments we want to conduct @xcite . the task is similar to that of @xcite , but uses basic yes-no questions so that it is suitable for crowd-annotations. it includes more fine-grained semantic judgments and intentionally ambiguous words. we can thus expect even more disagreement than already observed in @xcite . despite the central nature of phenomena triggering disagreement in annotation tasks, we are not aware of evaluation methods that do not mainly rely on agreement. traditionally, annotations by a few annotators who worked on the same units are evaluated in terms of kappa scores (usually cohens's kappa) and tasks with varying workers annotating the same units (usually crowd tasks) in terms of krippendorff's alpha @xcite . the crowdtruth framework suggested in @xcite and @xcite offers a more fine-grained view by distinguishing the levels of workers, units and labels, rather than reducing the entire task to a single score. the goal is to distinguish meaningful disagreements (i.e. agreements by reliable annotators) from noise (i.e. disagreement or agreement by generally unreliable annotators). the framework provides scores for workers, annotation units (clear units receive a high score, units triggering disagreement between reliable annotators a low score), labels and associations between units and labels. the scores can be used to aggregate labels and for identifying unclear annotation units, as for instance shown in @xcite and @xcite . other approaches attempt to discover disagreeing but valid interpretations in annotations based on clustering @xcite and gaussian modeling @xcite . while these approaches provide valuable insights, we focus on transparent and simple methods for quality assessment which do not require a large volume of data. ## use case: a dataset for diagnostic experiments in this section, we present our use-case, which requires particularly high-quality data. annotations should be provided by a crowd, rather than experts, as we are interested people's general perception rather than expert judgments. though we focus on a task with these specific characteristics, we believe that the general approach presented in this paper can also yield important insights in other, perhaps more traditional annotation scenarios. experiments in the tradition of model analysis require informative and high-quality data, as they aim to discover general tendencies about what kind of information models can capture. @xcite propose a dataset for diagnostic experiments on word embeddings using property-concept pairs annotated with fine-grained semantic judgments. the dataset is meant to test whether a semantic property (e.g. 'flying') is encoded by embedding representations or not. this can be investigated by testing whether positive (e.g. 'seagull', 'airplane', 'bee') and negative candidate concepts (e.g. 'penguin', 'train', 'ant') can be distinguished purely based on their embedding. the examples should not only be used to test whether a specific semantic property is encoded in embeddings, but, beyond this, help to uncover underlying factors determining whether a property is reflected in a distributional representation of a concept or not. therefore, the concept-property pairs should be annotated with semantic relations reflecting various linguistic factors. each concept-property pair can be connected by one or more of a total of ten relations (for instance expressing types of typicality or whether there can be variation in instances of concepts). the semantic relations can be grouped with respect to the subset of conceptinstances a property applies to (most to all, some, or few to no instances of a concept). this enables diagnostic experiments with positive and negative examples. we encountered the problem of annotation evaluation given expected disagreement while compiling this dataset as it contains a high number of ambiguous instances and instances of varying degrees of difficulty, for which disagreement can be valid and meaningful. in an ideal scenario, our analysis of annotations can (1) provide an overall indication of annotation quality which does not purely rely on agreement and (2) distinguish different types of disagreement. at the most coarse-grained level, it should distinguish justified disagreement from noise (caused by mistakes or spammers). ## annotation task the goal of the annotation task is to annotate property-concept pairs with relations. to make the task simple and suitable for a crowd of untrained workers, we turned it into a binary-decision task. this means that a single annotation unit consists of a property-concept-relation triple. this results in ten annotation units per concept-property pair. as the relations have rather abstract names, we translate them to natural language statements describing a property and a concept. the following statement is an example of a description expressing the property-concept-relation triple black-rhino-variability limited: you can find (a/an) rhino which is black. black is one of a few possible colors (a/an) rhino usually has. there is only a limited range of possible colors. participants are asked to indicate whether they agree with a given statement about a property and a concept. more example statements are listed in you can find (a/an) t-shirt which is white. white is one of many possible colors (a/an) t-shirt usually has. the range of colors is almost unlimited. rare i think (a/an) wine glass can is made of plastic, but this is rare or uncommon. impossible i think it is impossible for (a/an) corpse to be alive. we split the dataset into batches of around 70 descriptions. a worker who is proficient in english would need about 10 minutes per batch. while some statements may be difficult to judge and therefore take more time, most are expected to be rather intuitive and easy to answer. annotators were paid based on the uk minimum wage. each unit was annotated by 10 workers. to enable regular quality checks, we always include the full range of descriptions associated with a property-concept pair in the same batch. this enables us to check whether answers contradict each other. it has the disadvantage that the diversity of property-concept pairs in a batch is low. we monitored the quality of the annotations during the annotation process and used inter-mediate worker evaluations to 'recruit' good annotators. rather than rejecting low-quality submissions, we developed a 'whitelist' approach. prolific enables researchers to distribute studies exclusively among a pre-selected group of workers. we test whether workers contradict themselves in their answers (explained in more detail below), for instance by judging a property as typical of a concept and at the same time stating that it is unusual of the concept. as we do not know how much legitimate disagreement could be expected in a single batch, we decide to rely on an agreement-independent metric rather than inter-annotator agreement. ## a gold standard for accuracy and expected agreement we establish a gold standard to evaluate (1) the accuracy of annotations extracted based on different quality metrics and (2) the ability of different metrics to identify justified and potentially meaningful disagreement. the authors of the paper annotated a subset of already annotated units. the units for expert annotation were selected from units with high, medium and low agreement. agreement was established by calculating krippendorff's alpha on the level of concept-property pairs (each pair has up to ten units). 4 this resulted in a set of 154 units (containing 19 property-concept pairs and 11 different properties). the inter-annotator agreement before discussion was 0.51 and 0.72 after discussion (averaged pairwise cohen's kappa). we count all units in which agreement between experts could not be reached as units with expected disagreement. these units (23 in total) are excluded from the gold standard for label accuracy, as there are no incorrect answers in these cases. we also indicated whether we expected the crowd to disagree for legitimate reasons. examples of such disagreements are shown in ## quality metrics we experiment with three types of quality metrics: we consider traditional inter-annotator agreement, quality scores in the crowdtruth framework and our own, task-specific coherence metric. the metrics assess different aspects of the annotated dataset, as explained below. ## results in this section, we present the results of our analysis. section 7.1 presents a general overview and statistics about the collected annotations. in section 7.2, we show the results of our evaluation against the gold standard in terms of label accuracy, followed by our evaluation with respect to expected agreement and disagreement (section 7.3). finally, we test how well different quality metrics are able to identify units with legitimate disagreement (section 7.4). ## discussion in this paper, we have attempted to fill the gap between heavy emphasis of inter-annotator agreement on the one hand and justified disagreement on the other hand. semantic annotation tasks have been acknowledged to contain ambiguous, difficult, vague and possibly confusing examples which are likely to trigger disagreement. while some approaches may still see these cases as marginal, we argue that they are a vital part of many linguistic phenomena and can yield important insights. in this paper, we have illustrated an approach for a dataset used in model analysis experiments. the tradition of model analysis methods places strong emphasis on the quality and soundness of datasets and the phenomena indicated by disagreement are particularly relevant for our task. however, we argue that datasets used in other experiments should be held to similarly high standards. the explanatory power of evaluation datasets for semantic tasks in general could be improved by explicitly containing information about disagreement. we have shown that, for our particular use-case, the agreement-based metrics should not be used as the sole indicator of quality. our results show that a task-inherent coherence check can yield important insights and serve as a valuable basis for discarding noisy annotations. while we have only shown this for our use-case, we believe that the principle can be applied to other annotation tasks as well. for example, we could imagine that the principle of logical coherence checks can be applied to a semantic role-labeling task. predicates with contradictory semantic roles (based on the idea of selectional preferences) can be used as an indication of either noisy annotations or ambiguous annotation units. even tasks that are particularly drawn to high disagreement, such as tasks in the domain of sentiment annotation, could benefit from such checks. in hate speech identification, it could be considered to check if (1) the same annotator uses opposing labels for very similar instances and (2) annotators completely contradict one another on the same instances (rather than just disagreeing about the boundaries of categories (such as 'positive' and 'neutral'). we do not intend to disregard the complex nature of such a task; other contextual factors, such as the background of the annotators, can also trigger contradictions. taking these factors into account can yield further useful insights when interpreting (differences in) annotations. we believe that considering the interaction between these factors and logical checks can provide a valuable tool for analyzing and processing annotations. while the approach presented here can be taken as a first step, there are still a number of limitations and remaining challenges. most importantly, it would be highly valuable if the existing metrics could be combined in such a way that we could use them for the identification of different types of disagreements. for instance, it is relevant whether workers disagree because some have more specialized knowledge than others or because the annotation unit under consideration is indeed ambiguous. it could be considered to combine different metrics in such a way that they can distinguish between disagreement due to noise, disagreement because of differences in knowledge and disagreement due to real ambiguity. a possible way to achieve this could be to use the different metrics as features in a machine learning system. this research direction would require a larger volume of expert annotated gold data. ## conclusion despite the limitations discussed above, we draw the following conclusions: (1) absolute thresholds for inter-annotator agreement and aggregated scores over all annotations disregard the nature of a difficult semantic task with ambiguous and vague instances. rather, evaluations should focus on whether agreement can be found in cases where agreement can be expected. our evaluation against expected agreement and disagreement shows that worker-behavior is in line with our expectations despite overall low interannotator agreement. (2) the results indicate that a simple, coherence-based task-specific worker-quality check yields accurate labels, even on datasets with low inter-annotator agreement. the advantage of this check is that it does not require high volumes of data to be accurate, but can be used with only a handful of annotated units. we expect that similar checks can also be established for other tasks. such checks can be a cheap but high-impact approach, as they can be designed in such a way that they adhere to what is important in a particular task. in our case, good workers should understand questions and not contradict themselves. this is more important than that they agree with other workers. (3) high inter-annotator agreement is not necessarily a requirement for obtaining high-quality labels. our evaluation shows that the highest f1-score on the expert-annotated gold standard was achieved by a filtering and aggregation method which does not result in the highest alpha score on the remaining labels. (4) while our approach to identifying legitimate disagreements is preliminary, we observe that a simple, proportional agreement metric on a dataset filtered for contradictory answers yields the best results. this research provides the groundwork for establishing the exact status of individual annotation units and thereby establish whether the information and quality is sufficient for experiments with computational linguistic models. as the size of our dataset increases, we plan to take the next steps towards such a fine-grained assessment.