Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement (2021)

Abstract: Since state-of-the-art approaches to offensive language detection rely on supervised learning, it is crucial to quickly adapt them to the continuously evolving scenario of social media. While several approaches have been proposed to tackle the problem from an algorithmic perspective, so to reduce the need for annotated data, less attention has been paid to the quality of these data. Following a trend that has emerged recently, we focus on the level of agreement among annotators while selecting data to create offensive language datasets, a task involving a high level of subjectivity. Our study comprises the creation of three novel datasets of English tweets covering different topics and having five crowd-sourced judgments each. We also present an extensive set of experiments showing that selecting training and test data according to different levels of annotators’ agreement has a strong effect on classifiers performance and robustness. Our findings are further validated in cross-domain experiments and studied using a popular benchmark dataset. We show that such hard cases, where low agreement is present, are not necessarily due to poor-quality annotation and we advocate for a higher presence of ambiguous cases in future datasets, in order to train more robust systems and better account for the different points of view expressed online.

## introduction when creating benchmarks for nlp tasks through crowd-sourcing platforms, it is important to consider possible issues with inter-annotator agreement. indeed, crowd-workers do not necessarily have a linguistic background and are not trained to perform complex tasks, thus jeopardizing benchmark quality. furthermore, some crowd-workers try to maximize their pay by supplying quick answers that have nothing to do with the correct label. this issue has been tackled in the past by proposing approaches to control for annotators' expertise and reliability @xcite , trying to identify spammers and mitigate their effect on annotation, or by repeating labeling on targeted examples @xcite . however, not all tasks are the same: while in some cases, like for instance postagging or parsing, disagreement among annotators is more likely due to unclear annotation guidelines and can usually be reconciled through adjudication, full annotators' agreement should not be necessarily enforced in social computing tasks, whose goal is to study and manage social behavior and organizational dynamics, especially in virtual worlds built over the internet @xcite . in these taskswhich include offensive language detection among others -subjectivity, bias and text ambiguity play an important role @xcite , and being an inherent component of the task they should be measured and analysed rather than discarded @xcite . indeed, instead of aiming for a global consensus on what constitutes verbal abuse on social media, we investigate the impact of different degrees of disagreement, how classifiers behave with ambiguous training and test data, and the role of disagreement in current shared tasks. more specifically, we first collect and annotate three datasets of english tweets covering different domains, to test if agreement among a pool of generic classifiers can be considered a proxy for annotator agreement. we then focus on how annotator agreement (both in training and test set) impacts classifiers' performance, considering domainspecific and generic classifiers as well as in-domain and out-of-domain experiments. we also show that low agreement examples -no matter how difficult they can be -still provide useful signal for training offensive language detection systems and do not represent random annotations. so "coin-flipping" or example removal seems not to be the right strategy to solve these disagreement cases. then, we measure disagreement in the english test set of the last offenseval shared task @xcite , and analyse to what extent the high performance achieved by most participating systems is related to high agreement in annotation. we release the new annotated datasets upon request, 1 including more than 10k tweets covering three domains. the messages have been labeled with 50k crowd-worker judgements and annotated with agreement levels. to our knowledge, this represents the first dataset explicitly created to cover different agreement levels in a balanced way.we also advocate for the release of more datasets like the one we propose, especially for highly subjective tasks, where the need to include different points of view should be accounted for. ## related work while there has been an extensive discussion on minimal standards for inter-annotator agreement to ensure data quality @xcite @xcite , recently an increasing number of works argue that disagreement is unavoidable because language is inherently ambiguous @xcite , proposing ways to tackle annotators' disagreement when building training sets @xcite . @xcite , for example, identify a set of criteria to select informative yet unambiguous examples for predictive modeling in a sentiment classification task. @xcite analyse the impact that annotation noise can have on active learning approaches. other works along this line investigate the impact of uncertain or difficult instances on supervised classification @xcite , @xcite show that including hard cases in training data results in poorer classification of easy data in a word classification task. along the same lines, @xcite show that filtering instances with low agreement improve classifier performance in four out of five tasks. both works observe that the presence of such instances lead to misclassifications. several approaches have been presented that implement strategies to deal with disagreement when training classifiers for diverse tasks. in most cases, disagreement has been treated as a consequence of low annotation quality, and addressed through 1 see ethics statement section for further details methodologies aimed at minimising the effects of noisy crowdsourced data. @xcite , for example, present a bayesian sequence combination approach to train a model directly from crowdsourced labels rather than aggregating them. they test their approach on tasks such as ner where disagreement is mainly due to poor annotation quality. other works have focused instead on uncertainty in pos-tagging, integrating annotators' agreement in the modified loss function of a structured perceptron @xcite . also @xcite propose an approach to automatically distinguish the good and the unreliable annotators and capture their individual biases. they propose a novel crowd layer in deep learning classifiers to train neural networks directly from the noisy labels of multiple annotators, using only backpropagation. other researchers have suggested to remove hard cases from the training set @xcite because they may potentially lead to poor classification of easy cases in the test set. we argue instead that disagreement is inherent to the kind of task we are going to address (i.e. offensive language detection) and, in line with recent works, we advocate against forced harmonisation of annotators' judgements for tasks involving high levels of subjectivity @xcite . among recent proposals to embrace the uncertainty exhibited by human annotators, @xcite propose a novel metric to evaluate social computing tasks that disentangles stable opinions from noise in crowd-sourced datasets. @xcite , instead, divide the annotators into groups based on their polarization, so that different gold standard datasets are compiled and each used to train a different classifier. compared to existing works, our contribution is different in that we are interested mainly in the process of dataset creation rather in evaluation metrics or classification strategies. indeed, our research is guided mainly by research questions concerning the data selection process, the composition of datasets and the evaluation using controlled levels of agreement. to this purpose, we create the first dataset for offensive language detection with three levels of agreement and balanced classes, encompassing three domains. this allows us to run comparative in-domain and out-of-domain evaluations, as well as to analyse existing benchmarks like the offenseval dataset @xcite using the same approach. while few crowd-sourced datasets for toxic and abusive language detection have been released with disaggregated labels @xcite , they have not been created with the goal of analysing disagreement, therefore no attention has been paid to balance the number of judgments across different dimensions, like in our case. ## data selection and annotation in our study, we focus on three different domains, which have been very popular in online conversations in 2020: covid-19, us presidential elections and black lives matter (blm) movement. after an empirical analysis of online discussions, a set of hashtags and keywords for each domain are defined (e.g. #covid19, #election202, #blm). then, using twitter public apis, tweets in english containing at least one of the above keywords are collected in a time span between january and november 2020 (for more details about data collection see @xcite . from this data collection, we randomly select 400,000 tweets (around 130,000 for each domain), which we then pre-process by splitting hashtags into words using the ekphrasis tool @xcite and then replacing all mentions to users and urls with user and url respectively. ## classification experiments after collecting information on human agreement on tweets covering three different domains, we aim at assessing the impact of (dis)agreement on classifier behaviour. to this end, we create several balanced configurations of the datasets, so to control for the effect of agreement level, label distribution and domain topic. we first split the data into a training and test set of 75% and 25% for each domain. then, to control for the effect of training data size, we further downsample all sets to the smallest one, so that each agreement sample is equally represented (a ++ , a + , a 0 ). in this way, we obtain 3 sets of training data -one per ambiguity level -containing 900 tweets each. every set further contains 300 tweets from each domain, half for offensive label and half for non-offensive label so to control also for the effect of label distribution across domains and agreement levels. ## experiments on offenseval dataset our experiments show that when training and test data include tweets with different agreement levels, classification of offensive language is still a challenging task. indeed, our classification results reported in results, reported in to understand whether this skewness can be generalised, i.e. if this sample distribution might be representative of a population distribution, we also estimate the distribution of agreement levels in the initial pool of data (around 400k tweets) we collected using us election, blm and covid-related hashtags (section 3). 3 the estimate of the distri- as a final analysis, we collect the runs submitted to offenseval and compute the f1 score of each of these systems over the three levels of agreement separately. overall, we consider all runs that in the task obtained @xmath1.75, i.e. 81 runs out of 85. results are reported in ## discussion and conclusions we have presented a data annotation process and a thorough set of experiments for assessing the effect of (dis)agreement in training and test data for offensive language detection. we showed that an ensemble of classifiers can be employed to preliminarily select potentially unambiguous or challenging tweets. by analysing these tweets we found that they represent real cases of difficult decisions, deriving from interesting phenomena, and are usually not due to low-quality annotations. we also found that these challenging data are minimally present in a popular benchmark dataset, accounting for higher system performance. we believe that such hard cases should be more represented in benchmark datasets used for evaluation of hate speech detection systems, especially in the test sets, so to develop more robust systems and avoid overestimating classification performance. this goal can be achieved by integrating the common practice of oversampling the minority offensive class with the oversampling of minority agreement classes. from a multilingual perspective, we also noted that at offenseval 2020 the best performing systems on arabic scored 0.90 f1 with a training set of 8k tweets, 0.85 on greek with less than 9k tweets, and 0.82 on turkish despite having more than 32k examples for training. this shows that the amount of training data is not sufficient to ensure good classification quality, and that also in this case a study on disagreement levels could partly explain these differences (this is further corroborated by the fact that for turkish the lowest overall interannotator agreement score was reported). as future work, we plan to develop better approaches to classify (dis)agreement, in order to ease oversampling of low agreement classes. preliminary experiments (not reported in this paper) show that the task is not trivial, since supervised learning with lms such as bert does not work properly when trying to discriminate between ambiguous and not ambiguous tweets. indeed, bertbased classification performed poorly both in the binary task (ambiguous vs. not ambiguous) and in the three-way one (offensive vs. not offensive vs. ambiguous). this suggests that ambiguity is a complex phenomenon where lexical, semantic and pragmatic aspects are involved, which are difficult to capture through a language model. this corpus, together with the experiments presented in this paper, will hopefully shed light onto the important role played by annotators' disagreement, something that we need to understand better and to see as a novel perspective on data. indeed, if we want to include diversity in the process of data creation and reduce both the exclusion of minorities' voices and demographic misrepresentation @xcite , disagreement should be seen as a signal and not as noise.