Why Don’t You Do It Right? Analysing Annotators’ Disagreement in Subjective Tasks (2023)

Abstract: Annotators’ disagreement in linguistic data has been recently the focus of multiple initiatives aimed at raising awareness on issues related to ‘majority voting’ when aggregating diverging annotations. Disagreement can indeed reflect different aspects of linguistic annotation, from annotators’ subjectivity to sloppiness or lack of enough context to interpret a text. In this work we first propose a taxonomy of possible reasons leading to annotators’ disagreement in subjective tasks. Then, we manually label part of a Twitter dataset for offensive language detection in English following this taxonomy, identifying how the different categories are distributed. Finally we run a set of experiments aimed at assessing the impact of the different types of disagreement on classification performance. In particular, we investigate how accurately tweets belonging to different categories of disagreement can be classified as offensive or not, and how injecting data with different types of disagreement in the training set affects performance. We also perform offensive language detection as a multi-task framework, using disagreement classification as an auxiliary task.

## introduction the development of benchmark datasets based on the reconciliation of annotators' disagreement has been a standard practice within the nlp community for decades. however, in the last few years this paradigm has been questioned @xcite , since aggregating discording labels is based upon the assumption that texts have a single interpretation and that annotators' disagreement is something that should be corrected. on the contrary, it may convey useful information on the task, the data and the annotators themselves @xcite . for example, annotators may disagree because of poorly described guidelines, or because they interpret in different ways a text based on their background and beliefs. the text may be ambiguous because of a lack of context, or annotators may simply be working poorly, without paying much attention to the task they should perform on the text @xcite . having a better understanding of the reasons behind annotators' disagreement could lead to better annotation guidelines and provide insights into the advantages and limits of developing nlp systems able to deal with disagreement @xcite . it would also contribute to a better understanding of how disagreement interferes with systems' performance. finally, automatically detecting instances that are likely to obtain discording labels could help researchers in creating linguistic datasets with more or less challenging examples @xcite . in this work, we contribute to the research line on annotators' disagreement by first presenting a taxonomy, where we classify possible reasons behind discording annotations in subjective tasks, i.e. tasks admitting diverse valid beliefs about what the correct data labels should be @xcite . we also annotate part of an existing dataset for offensive language detection @xcite in order to assess the validity of our categorisation. we further perform several experiments aimed at addressing the following research questions: i) what are the most challenging categories of disagreement to classify in a task of abusive language detection? ii) what is the effect of including specific categories of disagreement in the training set? iii) can multi-task learning be effectively used for offensive language detection using disagreement classification as auxiliary task? are these results consistent across disagreement categories? evaluation results can contribute to ongoing studies investigating the negative impact of disagreement on systems performance @xcite @xcite by providing a more fine-grained view on disagreement types. the annotated data used in our experiments can be obtained as an extension of @xcite 's dataset, at this link: https://github. com/dhfbk/annotators-agreement-dataset . ## related work despite all the efforts to minimize inter-annotator disagreement, every large-scale project involving linguistic annotation has to deal with cases of diverging perceptions among human workers, especially in tasks where human sensibility is at play. however, disagreement due to interpretation divergences is also found in text annotation tasks usually perceived as objective, such as part-of-speech tagging @xcite , semantic role labeling @xcite or word sense disambiguation @xcite . in subjective tasks, the hypothesis that a single truth exists for all instances has been debated in several past works @xcite @xcite . indeed, many researchers suggested that disagreement has to be treated as a signal, and not as noise @xcite @xcite . for this reason, @xcite has recently proposed to use human label variation rather than the term disagreement to capture the fact that two or more views may sometimes be plausible in text annotation. along the same line, different approaches have been proposed with the primary aim of amending the traditional way in which disagreement is dealt with (i.e., to treat it as noise and discard it from the training set), thus demonstrating that disagreement provides insights into human perception, linguistic data as well as classification systems. concerning the integration of disagreement in nlp classifiers, few approaches have been proposed so far (for an overview, see @xcite ). beside aggregating judgments based on majority voting, which has the clear shortcoming of reducing different voices and points of view in favour of the dominant one, some approaches consider disagreement as an indicator of the diffi-culty of the instances to be annotated @xcite . past works have proposed to train separate classifiers, one for each annotator, and build an ensemble classifier that makes a prediction when all classifiers agree on the class label @xcite or to adopt a multi-task architecture using a shared representation to model annotator disagreements @xcite . similarly, fornaciari et al. (2021) use soft-labels (i.e., probability distributions over the annotator labels) as an auxiliary task in a multi-task neural model. as an alternative, disagreement can also be excluded from the data by training the model only on items that show high agreement @xcite or, conversely, models can be trained and evaluated only on disagreement-raising items @xcite . concerning the analysis of annotators' behaviour on subjective tasks, past works showed that disagreement is to be expected @xcite . this is confirmed by @xcite , showing that around 30% of the instances in a twitter corpus for sentiment analysis are controversial, thus likely to lead to disagreement. they also propose to merge them in a new class of sentiment called "complicated", so that they are not excluded from the data. @xcite show that age and personality are factors that greatly influence annotators' perception of offensive content. @xcite show that disagreement can be reduced and annotation quality increased by combining text representations of labeled annotators' opinions with their demographic traits. @xcite deal with disagreement in hate speech annotation by partitioning the annotators into clusters reflecting more uniform subjective judgments. as regards the categorisation of disagreement in subjective tasks, one of the first studies on this topic was presented @xcite , where in a metaphor annotation task a distinction is made between annotators' attentions slips and genuine disagreement. @xcite identify three main sources of disagreement: individual differences related to annotators' background and beliefs, stimulus characteristics, i.e. inherent ambiguity of texts, and context. @xcite further extend this categorisation by listing the following reasons behind disagreement: annotation errors, imprecise or vague annotation scheme, context-dependent text ambiguity, introduced in @xcite , item difficulty and annotators' subjectivity. in our proposed taxonomy, we rely on previous works in the choice of categories, keeping the distinction between inherent text ambiguity and context-dependent one, as well as annotators' subjectivity. we further specify these categories by proposing subtypes that take into account different linguistic phenomena. a recent work introducing a taxonomy of disagreement has also been presented in @xcite , which focuses on the task of natural language inference. interestingly, the authors propose a three-layered taxonomy which has some high-level overlaps with ours, for example our ambiguity class can be roughly mapped onto uncertainty in sentence meaning, and subjectivity onto annotator behavior. the lower categories, however, are in some cases very task-specific. the analysis of task-specific and task-independent categories of disagreement may be an interesting research direction to explore in the future. ## a taxonomy of disagreement as a first step towards a better understanding of the reasons behind annotators' disagreement, we propose a taxonomy of disagreement for subjective tasks, which is built starting from past categorisations presented in the literature and iteratively adding subtypes based on the analysis of examples in existing datasets. an overview of the taxonomy is reported in figure 1 . the taxonomy includes four macro-categories, which are further specified through more fine-grained subtypes. for each category we report also selected examples, all taken from the dataset of disagreement presented in @xcite (see a more detailed description of the dataset in section 4). in the following, we illustrate each category in detail. ## data annotation and analysis we apply the taxonomy illustrated above to the dataset of disagreement in abusive language presented in @xcite , which contains more than 10k english tweets labeled as offensive or not offensive by five crowd-workers, and covers three topics: covid-19, us presidential elections and black lives matter movement. the dataset has been designed so as to include a balanced set of tweets with full agreement (a ++ ), with partial agreement (a + class, 4 matching labels versus 1) and with disagreement (a 0 class, 2 vs. 3 labels) and has been released divided into a balanced training and test split. we manually annotate all a + and a 0 tweets in the test set (1,756 in total), plus a portion of tweets from the training set (809 tweets). a total of 2,574 tweets is annotated, divided in 1,518 for the a 0 agreement level and 1,056 for the a + agreement level. annotation was performed by a trained linguist, and during the process the initial taxonomy was adjusted by refining the category definitions or introducing new ones when needed. a second linguist annotated a sample of 200 tweets divided equally into a 0 and a + following the annotation guidelines (see appendix b). cohen's kappa is 0.591, which corresponds to a moderate agreement. after computing agreement, a discussion and adjudication phase was conducted. based on the annotator's feedback, we foresaw the possibility to assign multiple labels to the same tweet when the source of disagreement could refer to two or more categories in the taxonomy, considering however the first label as the most probable interpretation. a multi-category annotation scheme was adopted for the same reason also in @xcite . in in case of multiple labels, we include the first one in the statistics. however, tweets annotated as belonging to more than one category are around 40%, showing that disagreement is often due to a mix of different linguistic phenomena. in figure 2 we report how the subtypes are distributed in the annotated dataset. as shown in previous studies, swear words are often a signal for a hateful attitude, but they are also used in casual contexts with positive social functions @xcite . this double interpretation is likely the main reason why the swearing subtype is very frequent in our dataset of disagreement. concerning personal bias, its relevance is probably related to the fact that the annotated tweets deal with controversial topics such as us american elections, black lives matter and covid-19. for instance, few tweets targeting trump were labeled as not offensive by annotators, who were likely to be biden supporters, and vice versa. if we compare the above statistics with the analysis reported in jiang and de marneffe (2022) on disagreement in natural language inference (nli), we observe that the most frequent causes of disagreement are task-specific: in our dataset, they are mostly due to annotators' subjectivity and their perception of what is offensive, while in nli they often stem from the underspecified meaning of lexical items in the sentence pairs to be labeled, or in the probabilistic nature of the inferred content. ## experiments in the next subsections we describe a series of experiments conducted on the annotated dataset, aimed at analysing the relationship between annotators' disagreement and various aspects of automatic offensive language detection. for all the experiments described below, we use the machamp v0.2 toolkit (van der goot et al., 2021), a classification tool that allows easy implementation of transformers-based classification tasks and supports single-task and multi-task learning. for all the experiments we employ bert-base uncased @xcite (110m parameters) and perform 20 restarts. we keep the default hyperparameter setting of machamp, i.e. max seq length 128, batch size 32, 0.3 dropout, 10 epochs. all the results reported in the following subsections are the average values of 20 runs. all experiments are run on a nvidia quadro rtx 5000 gpu. ## conclusions in this work, we first introduced a two-layered taxonomy for the classification of annotators' disagreement in subjective tasks, consisting in four main categories and a number of subtypes aimed at covering different linguistic phenomena. we then annotated part of an existing dataset developed to study disagreement with the above classes and subtypes. a first analysis shows that the subjectivity class is the prevalent one in the dataset, and that personal bias and swearing are two major reasons leading to frequent cases of disagreement among annotators for the task of offensive language detection. secondly, we run several experiments to gain novel insights into disagreement phenomena. in particular, we investigate whether a system for offensive language detection is more prone to wrong classification on specific classes of disagreement. our results show that the presence of sarcasm, analogy and false assertions negatively affects classifier performance, while swearing obtains the best classification results, despite showing a bias in favour of offensive tweets. furthermore, cases with high disagreement are generally more difficult to classify than those with mild disagreement for all categories except for analogy. in a second experiment, we show that adding instances of missing information to the training set, even if they belong to the a 0 class, has a positive effect on the classification of this specific class, while this is not true for ambiguity, probably because it contains more heterogeneous data. finally, we show that adding disagreement information as an auxiliary task in a multi-task setting, having offensive language detection as the main task, is not generally better, and has different effects on the three above classes. as regards tweets annotated with multiple labels, we observe that it is rather frequent to find more than one cause of disagreement (around 40% of the tweets in our dataset). we performed a preliminary experiment (not reported in this paper) comparing the performance of the best system configuration on single-label and multi-label examples, and we observed no significant difference. we will further investigate this aspect and compare single-and multi-label items in detail in the future. in general, we hope that this work can contribute to the ongoing debate on the importance of considering, and not removing, disagreement when creating datasets and when developing classifiers. furthermore, we advocate for a differentiation of the types of disagreement, showing that their presence in training and test data can have different effects on classification.