Interrater Disagreement Resolution: A Systematic Procedure to Reach Consensus in Annotation Tasks (2021)

Abstract: We present a systematic procedure for interrater disagreement resolution. The procedure is general, but of particular use in multiple-annotator tasks geared towards ground truth construction. We motivate our proposal by arguing that, barring cases in which the researchers’ goal is to elicit different viewpoints, interrater disagreement is a sign of poor quality in the design or the description of a task. Consensus among annotators, we maintain, should be striven for, through a systematic procedure for disagreement resolution such as the one we describe.

## introduction a growing body of literature signals a thorny issue with assessing general progress in the field of natural language processing (nlp) as part of artificial intelligence. benchmarks that are considered 'general', and are widely used as standards to assess nlp systems' performance, turn out to be rather specific, and hence of more limited significance than commonly acknowledged @xcite . good performance on specific benchmarks does not guarantee good performance across the board @xcite ; ethayarajh and jurafsky 2020): it only helps with gaining understanding of how certain systems work for those specific benchmarks. in order to claim progress across the board, one would need to evaluate system performance on a certain reasoned series of such specific benchmarks, that is, results on a host of "more focused and explicitly defined problems" @xcite . to enact this, one would need a ground truth for the evaluation of each specific task-cum-dataset, including ground truths in expert domains. ground truth construction is challenging. in this paper we focus on the process of constructing ground truths via semantic annotations tasks. recent studies stress the intrinsic difficulty of semantic annotation due to vagueness and ambiguity @xcite @xcite . importantly, some argue that interpretative disagreements due to different conceptualizations or perspectives cannot be seen as just 'mistakes' @xcite . it is our tenet that in ground truth construction differences in conceptualizations or perspectives can and must be explicitly specified as an integral part of annotation tasks; moreover, interrater disagreement is not necessarily due to inherent ambiguities in the data, but at least in part to the annotation task being underspecified, in particular as to the right context to consider. take annotation tasks involving relatedness or similarity judgments, which are key types of judgment for nlp evaluation. similarity is not a property of two things by themselves in isolation: it is always judged by a specific standard, and by weighing properties of the things compared in different ways, according to a context @xcite . when people judge by different standards foot_0 , disagreement arises as a matter of course -and is especially likely when annotating texts of high conceptual density, as this requires a lot of prior knowledge and interpretation. in order to get comparable and meaningful annotations, judgment standards need to be aligned and made extremely transparent. in this paper we propose a six-step systematic procedure for interrater disagreement resolution in which conceptual alignment figures as one of the steps. the procedure is designed to facilitate the resolution of interrater disagreement that fre-quently arises in annotation tasks in which multiple annotators participate. the emergence of disagreement in annotation tasks is valuable information, albeit of a negative type: barring cases in which the researchers' goal is none other than eliciting disagreement, interrater disagreement, we maintain, is a sign of poor quality in the design or the description of a task. in ground truth construction, consensus among annotators should be striven for. the procedure applies to a wide range of annotation tasks, namely every task involving the application of one or more concepts to a unit of annotation (a fragment of text, such as a paragraph or a sentence, or a more artificial unit, such as a string with a length of n characters). we hold that the benefit of a systematic procedure of resolving interrater disagreement is twofold: first, such a procedure leads to the construction of reliable and well-grounded datasets, and second, it ensures that the resolution proceeds in a non-arbitrary fashion allowing for proper documentation and replicability of the data. ## related work computational research: interrater agreement, dataset creation and ground truths standard methods for measuring interrater agreement and reliability @xcite such as (cohen's) kappa @xcite ) and krippendorff's alpha @xcite output a single score to represent the agreement between different raters. methods such as the crowdtruth framework @xcite give a more detailed disagreement analysis, though only in post-annotation phase. similarly, @xcite mention that disagreement cannot simply be treated as noise and propose a post-annotation method for identifying different valid interpretations annotators may use to come to different conclusions. by contrast, we take disagreement analysis and resolution as internal to the annotation procedure. @xcite stress difficulties with annotation due to ambiguity or vagueness in language while studying cases in which disagreement between different annotators is expected and multiple answers are legitimate. our focus is datasets that are meant to be used as ground truths. in ground truth construction, we argue, it is necessary to resolve cases of disagreement (disagreement resolution phase, see step 5 below), and, more importantly, dispel the ambiguities that cause disagree-ment (if ambiguity is the cause of the disagreement) by task specification, either by redesigning the task or by making the annotation guidelines more precise (conceptual alignment phase, see step 2 below). we do recognize that genuine disagreement might exist due to e.g. ambiguity in language in existing datasets (see also, @xcite ), but we see legitimate disagreement as having a specific meaning: it is either a signal that further resolution is needed (through annotation task redesign or guideline redefinition), or it is the possible result of a task specifically designed to chart or elicit instances of disagreement, as in @xcite or @xcite . we offer a procedure by which annotators can avoid disagreement due to unclarity of the task, accurately discern the reason for disagreement whenever it arises, and make a deliberate decision on how these cases should be annotated. any differences between 'people's beliefs about the world' (or the data), we say, should be explicitly integrated in task design such that annotators are required to judge according to a certain perspective or set of beliefs, and not from an absolute point of view. we agree with @xcite that disagreement between annotators cannot simply be seen as noise in the data supposedly due to lowquality annotations. however, while they divide the annotations into consistent units to get sets of consistent gold labels, we argue that in ground truth construction the variety of human judgments can and should be narrowed down to exactly one type by specification of the task. in our case, the process of identifying reasons for disagreement is part of the annotation process, which allows for resolution of disagreement and thereby a dataset suitable for use as a ground truth for the task at hand. in @xcite , a general method for constructing expert-controlled ground truths for concept-focused domains is proposed, and the construction for an actual ground truth for a philosophical corpus is described. disagreement resolution is mentioned, and one example of resolution is reported, but no explicit general methodology for disagreement resolution is offered. it has been emphasized that the conditions under which a dataset has been created need to be properly documented to allow for reproducibility and replicability @xcite @xcite . language models are known to pick up and reinforce exist-ing biases in data (see, e.g., @xcite . @xcite offer instructions on how to document data using data statements to help reproducibility and replicability, bring existing biases to the surface and improve representation in future dataset creation. the procedure we propose asks for explicit decisions from raters after deliberation. this requirement makes the conditions of dataset creation clear, thus allowing proper documentation. philosophy peer disagreement is a topic of investigation in philosophy, in particular in the subfield of social epistemology. a large amount of literature exists on issues concerning both peer disagreement (e.g. @xcite and group decision making in the face of such disagreement (e.g. @xcite , but resolution procedures that aid in moving from peer disagreement to unanimously agreed upon results are not proposed, and are in general '[...] at best rare in scientific contexts.' (de ridder, 2014). one of the scarce examples is gius and jacke's (2017) procedure for resolving interrater disagreement in literary corpus annotation. although similar in approach, our work improves on the latter in terms of applicability: we intend our procedure to be fit for all annotation tasks that involve the application of one or more concepts to units of annotation, while gius & jacke focus on tasks within literary analysis exclusively. note that annotation tasks in which concepts are applied to units of annotation are frequent: any task involving the identifying of instances of any concept qualifies. for example, in our validation example in section 5.2 the annotation task requires annotators to identify wide-scope claims in the text of journal articles (that is, instances of the concept of wide-scope claim). ## ground truths and interrater agreement in pivovarov and elhadad (2012) a cohen's kappa of 0.68 is "accepted as representing a substantial amount of agreement between annotators". by contrast, in @xcite the initial interrater agreement of 0.65 was taken as a starting point to reach further consensus. when the aim of the annotation is e.g. to get an overview of the variety of ways in which people interpret statements, then interrater agreement need only be high on statements for which there is only one obvious interpretation and so agreement is expected. however, when the annotations are supposed to establish a ground truth, interrater agreement, we argue, should be 1. one strategy used for getting the interrater agreement on the ground truth to 1, is to discard disputed annotation(s) (see, e.g., @xcite ). but clearly this is loss of valuable information: for the purpose of training and evaluating a computational system we want to be as specific as possible as to what its output needs to be; by tossing out disputed annotation we underspecify what the right output on the matter is. consider one of the examples in @xcite : "missiles ex-plode received the labels some, most and all. it is likely that the some interpretation quantifies over missiles which actually explode, while the most/all interpretation considers the potential of a missile to explode". for ground truth construction, it is necessary to specify whether an annotator should e.g. take an actual or potential interpretation, to prevent annotators from making arbitrary choices or introducing unknown biases. so, if an annotation data set is to be used as a ground truth, agreement should be the aim. when disagreement arises, it is important to identify why it arises, and make well-grounded decisions on how to deal with it. in the next section, we will outline a procedure for annotation through which different reasons for disagreement can be identified and which specifies directions for resolution of each of these types of disagreements. the procedure results in a reproducible dataset by forcing annotators to make well-grounded, and thereby traceable decisions on their annotations. note that traceability makes the procedure relevant to all annotations, not just ground truth construction. the annotation procedure supposes what we call an 'annotation toolbox' consisting of (i) the annotation task or question, (ii) the guidelines specifying the instructions for annotation and (iii) some kind of definition or characterisation of the key concepts involved (see step 2). fixing the definitions and characterisations of these concepts is essential to the conceptual alignment of annotators and for subsequent use of the resulting annotations. the use of the annotation toolbox also facilitates disagreement resolution insofar as annotators can refer to elements of the toolbox to give a justification for their scoring. this also means that if disagreement cannot be resolved by referring to elements of the toolbox, the toolbox is incomplete, or in any case insufficient as a basis for annotation. in this case, further expert research might be necessary to supplement the annotation toolbox. based on the newly supplemented annotation toolbox, previous annotations might have to be redone, for there is no guarantee that these would end up receiving the same scoring. if such a resolution or supplementation is deemed impossible, the annotation cannot be completed and cannot lead to a dataset that is suitable as a ground truth. ## the annotation procedure what follows is a description of the steps of the annotation procedure (see flowchart in figure 1 ). throughout this description we will talk of 'scoring' as the act of annotating a single unit. this is intended to also refer to types of annotation that are more adequately called 'categorizing', 'labelling' or otherwise. note that with the exception of cases in which step 0-2 is performed by the same group of researchers as step 3-5 (see, e.g., section 5.1 in which the annotation procedure of @xcite is described), the annotators should be under close supervision of the researchers formulating the research question, and those setting the annotation task and guidelines, throughout all steps of the procedure. ## test cases by way of illustration and validation, in this section we outline two different user applications of the procedure we have observed, by two nonoverlapping teams of domain expert annotators. the first application concerns a study of a complex, rich philosophical concept in the complete corpus of the works of a specific author. in this case, the annotators worked through the entire procedure. the second application concerns a study of the methodological justification given to widescope claims in academic literature. although the corpus used in the second case is also from the field of philosophy, the annotation task is generic, and could have been performed on any type of scholarly article. the second team set up the research (step 0), annotation task and guidelines (step 1), but they did not settle on the meaning of all key concepts (step 2) before annotation. for the first case we will give examples for each of the reasons for disagreement, while for the second case we will focus on an issue due to the lack of conceptual alignment. ## further applications we have shown how the procedure applies to the two test cases discussed in section 5. however, our procedure is not limited to cases of that type. concepts are involved in any type of annotation task, and any concept necessitates both interpretation and conceptual alignment. consider the case of herbelot and vecchi (2016) again: "missiles explode received the labels some, most and all.". suppose we want to construct a ground truth of property-object pairs. the example shows that the guidelines should specify whether to use an actual or potential interpretation of property possession. note, though, that settling for an interpretation often won't be enough: while annotating under a potential interpretation, the issue may arise whether objects should have the potential to have a property actually (most do, but some are faulty) or teleologically (all). by our procedure, these ambiguities become apparent, and disagreement can be resolved. the two test cases of section 5 both have academics as annotators, but this is no intrinsic requirement of our procedure. for some linguistic tasks, being a native speaker of the relevant language is enough expertise to be able to grasp and apply the concepts involved in the task. another matter is the common practice of resorting to crowdsourcing platforms foot_5 to construct large, non-academic annotation datasets. the practice is useful, but ill-suited to accommodate the type of disagreement resolution we envisage. our take is that even though it might not always be possible to adopt the entire procedure for ground truth construction, we see no fundamental, theoretical problems with its application in a wide variety of cases. ## conclusion and further work in this paper we proposed a six-step systematic procedure for annotation focused on disagreement resolution. we argued that disagreement is the result of poorly specified tasks or guidelines, or of insufficient conceptual alignment among annotators. to avoid incomplete datasets unfit for use as ground truths, we set up the procedure in such a way that the identification and non-arbitrary resolution of different types of disagreement is facilitated. disagreement resolution by a clearly defined procedure results in more reliable and well-grounded datasets. by identifying the cause of disagreement and giving appropriate instructions for resolution for each type of disagreement, our procedure ensures that the resolution proceeds in a non-arbitrary fashion allowing for proper documentation and increasing replicability of the data. we have validated the effectiveness and the importance of our annotation procedure by two test cases. the first case shows that conceptual alignment by itself does not guarantee that annotators make no mistakes or only come across clarified concepts, indicating the need for disagreement resolution after annotation. the second case emphasizes the importance of task clarification and conceptual alignment prior to annotation. without this, the likeliness increases of having to redo annotations due to different construals of terms influencing both conflicting and agreed-upon annotations. in further work we aim to collect more use cases to test the applicability of the procedure to more varied types of annotations. moreover, we want to consider in more depth the interplay of step 0-2 and further elaborate on the idea of key concept at step 2.