On the Interaction between Annotation Quality and Classifier Performance in Abusive Language Detection (2021)

Abstract: Abusive language detection has become an important tool for the cultivation of safe online platforms. We investigate the interaction of annotation quality and classifier performance. We use a new, fine-grained annotation scheme that allows us to distinguish between abusive language and colloquial uses of profanity that are not meant to harm. Our results show a tendency of crowd workers to overuse the abusive class, which creates an unrealistic class balance and affects classification accuracy. We also investigate different methods of distinguishing between explicit and implicit abuse and show lexicon-based approaches either over- or under-estimate the proportion of explicit abuse in data sets.

## introduction in recent years, annotation quality has come under closer scrutiny, especially for subjective classification tasks that rely on human judgement. investigations of unintended bias in abusive language data sets have demonstrated that they are susceptible to sampling and annotation bias @xcite . although this work provides some guidance for reducing the effects of unintended bias from sampling, it does not provide a clear path forward for mitigating annotation bias. as we need effective ways to curb online hate, we definitely need reliable data sets with high-quality annotations for abusive language detection. in this paper, we compare annotations by untrained crowd workers with annotations by experts. our examination demonstrates that labeling differences between crowd workers and experts change the class distribution in the data set and affect classifier performance. we also compare methods for determining explicit and implicit abuse in the data set, and how this affects the interpretation of machine learning experiments. our paper is structured as follows: sec. 2 explains our research questions, sec. 3 describes prior work on data sets, annotation procedures and quality, and classification schemes for abusive language, sec. 4 describes our data sets and methodology, sec. 5 discusses our insights into the interaction of annotation quality and classifier performance, and sec. 6 investigates the interaction of explicit and implicit abuse and the interpretation of results. we conclude in sec. 7. ## research questions we started our investigation by reviewing a random sample of 1 000 posts from the kaggle competition jigsaw unintended bias in toxicity classification foot_0 . our initial inspection showed that many posts that were considered abusive by crowd annotators (wrt. the final classification used by the competition) were open to interpretation. the examples below show typical "abusive" posts where different interpretations are possible. 1. i do love yataimura maru's ramen? it is a perfect food for portland's long winter. and pdx does kick a little ass. the first example is a positive review of a ramen restaurant in portland that also contains profanity. the second example also uses profanity, but is doing so as a part of a meta-comment about the filters used by the platform. finally, the last post is a criticism of an apartment complex. although an insult is used, it is directed toward an object and not individuals. these examples show two common tendencies by crowd workers: interpreting profanity as abusive without considering the context and not distinguishing between insults and criticism directed at people and objects. these observations led us to our first research question: rq1: how does annotation quality affect characteristics of the data set and subsequently the performance of machine learning approaches? more specifically, we address the following questions: rq1.1: how does annotation quality affect the distribution of abusive and non-abusive posts? rq1.2: based on a detailed annotation scheme that distinguishes varieties of non-abuse, how do crowdsourced and expert annotations differ? rq1.3: how do annotations by crowd workers and experts influence classification results? we then turn to the issue of explicit vs. implicit abuse. it is generally accepted that explicit abuse is easier to detect automatically @xcite . however, the method that we use to determine whether a post is explicitly or implicitly abusive will result in different splits of the data, and different results of how well a classifier performs on either class. explicit abuse is often identified via lexicons of abusive expressions. automatically created lexicons, such as the one by @xcite , have good coverage, but may overestimate the abusiveness of terms while manually curated lists, such as the one by @xcite , are more reliable in their selection of abusive terms but may lack coverage. this leads to our second question: rq2: how does the method of identifying explicit abuse influence the distribution in the data set and subsequently the interpretation of the classifier's performance? annotation schemes one way to maintain annotation quality is to create clear annotation guidelines with rich taxonomies @xcite . many scholars have developed annotation schema and guidelines that describe different types of abuse in order to better characterize abusive content. @xcite evaluated 7 abuse categories (e.g., offensive, abusive, hateful, aggressive, cyberbullying, spam, and normal), which were then merged into four (e.g., abusive, hateful, spam, and normal) when they found overlap between categories. @xcite created a 3-tier scheme in which annotators decided whether a post was abusive, targeted, and whether the target was an individual, a group, or other. @xcite distinguished hateful content from the casual use of profanity by creating three categories: hateful, offensive (but not hateful), and neither. current methods overwhelmingly focus on the labeling of abusive posts, often at the expense of accuracy on non-abusive posts. investigating unintended bias recent work on abusive language detection has looked at sampling bias in the data. sampling methods are required to increase the amount of abusive posts in data sets. however, the specific sampling methods used have been shown to create bias. @xcite document this bias. razo and kübler (2020) build on their work and find that the source of the text (twitter, wikepedia, etc.) has more influence on the bias of the data set than the sampling method. ## investigating annotation quality as discussed in section 2, a cursory inspection of the data sets showed that there was a considerable amount of posts that were annotated as abusive (based on jigsaw's definition of the challenge), which the expert annotators found questionable. this does not only mean that the classifier learns a model that is disposed towards classifying too many posts as abusive, it also raises the question of whether a more consistent annotation would improve classification results or make the task more difficult to learn (since profanity would have to be disambiguated between abusive and colloquial use). for this reason, we decided to re-annotate the abusive portions of the two data sets. we first present the new annotation scheme in section 5.1, then we describe the resulting changes to the data set in section 5.2 and on the classifier in section 5.3. ## investigating explicit vs. implicit abuse now we turn to the distinction between explicit and implicit abuse. it is generally accepted that explicit abuse is easier to detect than implicit abuse. however, making this distinction is not a simple task. in general, lexicons of abusive words are used to determine the explicitly abusive posts; a post is considered explicit abuse if one of the lexicon words occurs in the post. @xcite describe a method for automatically extending a base lexicon into a larger lexicon of abusive words. their base list contains 551 words, their extended list 2 989 words. @xcite show that both the base and the extended lexicon cover a large proportion of posts that were labeled nonabusive by crowd workers. they manually checked the base lexicon and reduced it to 151 words. since we now have expert annotations for the abusive posts, we can investigate how the distribution of explicit and implicit posts in the two gold standards differs from those based on the three lexicons. we compare the proportions of posts from each gold standard labeled as explicit and implicit abuse with the lexicon approaches used by @xcite and @xcite . however, the similarity in proportions raises the question whether the wiegand base lexicon and the manual annotations choose the same posts, or just the same proportion of posts. we checked the overlap of posts that were labeled explicit or implicit by both (not shown in table). for the topic sample, 75.96% of explicit posts annotated by experts can be found in the posts extracted using the base lexicon. the random sample shares a smaller proportion of explicit posts (66.94%) than the topic sample. there is also a smaller proportion of overlap between annotations and the base lexicon among implicit posts. 24.90% of posts from the topic sample and 32.89% of posts from the random sample can be found in the implicit posts using the base lexicon method. this shows very clearly how different the samples of explicit and implicit abuse are based on the different methods. we then investigate how these different decisions affect classification results. for this, we use the same classification results from section 5.3, and we evaluate the subsets against our expert annotations. the subsets consist of only explicitly or implicitly abusive posts, based on either expert annotations or the base lexicon. the results are shown in ## conclusion and future work our investigation illustrates the effect of diminished annotation quality on machine learning performance. crowd workers and expert annotators disagreed on approximately a third of the posts originally labeled abusive. disagreement often occurred with profanity and when targets were not individuals. the method for identifying implicit and explicit abuse leads to significant discrepancies between the explicit and implicit classes and affects evaluation. our work shows the need to improve annotation quality. this is only the tip of the iceberg, though. verbal abuse can only be identified within a cultural context, but there exist so many different subcultures that any annotator, independent of their being sensitized to the nuances of abuse, may not be able to identify abuse if they are not part of that subculture. we will investigate using annotators with different backgrounds along with methods to distinguish between disagreement based on inattention or lack of sensitization from disagreement based on cultural backgrounds.