Identifying and Measuring Annotator Bias Based on Annotators’ Demographic Characteristics (2020)

Abstract: Machine learning is recently used to detect hate speech and other forms of abusive language in online platforms. However, a notable weakness of machine learning models is their vulnerability to bias, which can impair their performance and fairness. One type is annotator bias caused by the subjective perception of the annotators. In this work, we investigate annotator bias using classification models trained on data from demographically distinct annotator groups. To do so, we sample balanced subsets of data that are labeled by demographically distinct annotators. We then train classifiers on these subsets, analyze their performances on similarly grouped test sets, and compare them statistically. Our findings show that the proposed approach successfully identifies bias and that demographic features, such as first language, age, and education, correlate with significant performance differences.

## introduction according to the online harassment report published by pew research center, "four-in-ten americans have personally experienced online harassment, and 62% consider it a major issue." @xcite . online environments such as social media and discussion forums have created spaces for people to express their opinions and viewpoints, but this comes at the cost of hateful, offensive, and abusive content. moderating this content manually requires a lot of staff and large amounts of hand-curated policies, which generated much interest in automatic content moderation systems that make use of recent advances in machine learning @xcite . one challenge of training machine learning systems is the demand for large amounts of labeled * these authors contributed equally to this work. data. hence, many researchers use crowdsourcing platforms to annotate their data sets @xcite @xcite , although having expert annotators has proven to improve the quality of annotations @xcite . such crowdsourcing approaches, however, exposes hate speech detection systems to annotator bias. hateful behavior can take many forms @xcite , making it harder to obtain a clean, common definition of hate speech, and resulting in subjective and biased annotations. biases in the annotations are then absorbed and reinforced by the machine learning models, causing systematically unfair systems @xcite . therefore, it is not surprising that a large body of work has identified and mitigated this bias @xcite @xcite . we already know that people with particular demographic characteristics (e.g., black, disabled, or younger people) become more frequently targets of hate @xcite ). an aspect that is sparsely investigated in this context is the relation between annotators' demographic features and a potential bias in the data set. we want to fill this gap by addressing the following research question: how do annotators' demographic features such as gender, age, education and first language impact their annotations of hateful content? to answer this question, we conduct the following exploratory study: we sample balanced subsets of data that are labeled by demographically distinct annotators. we then train classifiers on these subsets, analyze their performances on similarly split test sets, and compare them statistically. ## related work since unintended bias in hate speech datasets can impair the model's performance @xcite and fairness @xcite , a lot of recent work has been done to investigate this phenomenon @xcite . some work examined racial bias @xcite @xcite , others explored gender bias (gold and zesch, 2018), aggregation bias @xcite and political bias @xcite . the type of bias we are examining in this study is the annotator bias. @xcite studied the influence of annotator expertise on classification models and found that systems trained on expert annotations outperform those trained on amateur annotations, confirming and extending the results from ross et al. (2017). geva et al. (2019) showed that model performance improves when exposed to annotator identifiers, which suggests that annotator bias needs to be considered when creating hate speech models. @xcite studied the difference between annotations of crowd workers from 50 countries and found those differences highly significant. @xcite examined the effect of the gender of the annotators on the performance of classifiers. @xcite studied the similarities in the behaviour of the annotators to reveal biases that they bring into the data. to the best of our knowledge, no one has developed a method to identify annotator bias based on multiple demographic characteristics of the annotators and measure its impact on the classification performance. ## data we used the personal attack corpora from wikipedia's detox project @xcite , which contains 115,864 labeled comments from wikipedia on whether the comment contains a form of personal attack. the labels are the following (wikimedia, n.d.): the comments were labeled by 4,053 crowdworkers. for 2,190 of them, we have the demographic information. for each of these annotators we have the following demographic features: ## methodology we address the research question by training classification models on data from demographically distinct groups and comparing their performances foot_0 . the hypothesis is that a statistically significant difference between the classifiers' performances indicates an annotator bias related to the studied demographic feature. in the first step, we group the annotators by their demographic features, such as gender, age, education level, and native language. for each of those features, we create m + 1 datasets where m is the number of different values a demographic feature can take, e.g. for gender m could be equal to 2 if we only consider male and female annotators. all datasets have the same comments, but with different labels aggregated from annotators belonging to each different group. the additional dataset ( +1) has labels aggregated from annotators belonging to all groups. it serves as a control group. we call this dataset the mixed dataset. we measured the inter-rater agreement within each group using krippendorff's alpha (hayes and krippendorff, 2007). in the second step, we split the datasets into train and test sets, and train 20 classifiers for each group on the group's training set and report f1 scores for all test sets. we train 20 classifiers to get multiple data points for each group's classifier and then apply the kolmogorov-smirnov test to examine whether they are significantly different 2 . the null hypothesis in this context is that the two samples are drawn from the same distribution. if we can reject the null hypothesis (@xmath0.05) for a certain demographic feature, this will be evidence that annotators belonging to different groups of feature values hold different norms and are bringing in different biases into their annotations. concerning the classification model, we chose to make use of recent advancements in transfer learning and employ distilbert as a classifier due to the limited number of data points annotated by each group. distilbert @xcite is a smaller and faster distilled version of bert @xcite . in the context of abusive language detection, it provides a comparable performance @xcite . we used the base uncased version of distilbert (distilbert-base-uncased) with a maximum sequence length of 100, a learning rate of 5 × 10 -6 , and 1cycle learning rate policy (smith, 2018) and trained each classifier for 2 epochs. 2 we trained 20 classifiers only for practical constraints. ## results in this section, we report the results of our experiments for each demographic feature. the results comprise the inter-rater agreement of the annotators in the different groups, the averaged f1 scores of the trained classifiers, the sensitivity and specificity of the classifiers as charts, and the p-values generated by the kolmogorov-smirnov tests. ## discussion in light of our results, we can conclude that the gender of the annotator does not bring a significant bias in annotating personal attacks in the studied dataset. however, when @xcite explored the role of gender in offensive content annotations, they established a distinguishable difference between males and females. we think this is related to the nature of the annotation task itself. to investigate other tasks, our approach can further be applied in future work on the other data sets provided by wikipedia's detox project @xcite such as aggressiveness and toxicity to investigate the effects of gender for those tasks. when it comes to the first language of the annotators, it seems that native english speakers are gen-erally better at identifying personal attacks in comments. the results also suggest that non-natives could not capture attack in comments that natives found to contain attack. in addition, age groups and education levels of the annotators also seem to play a notable role in how attacks are perceived. training a classifier on aggregated labels from all groups, even if the data is balanced between groups, does not seem to be fair to all groups involved. although we have only explored the demographic features provided by the data set and grouped some of them for reasons dictated by the data size, we think other features (e.g., race, ethnicity, and political orientation), different within feature groupings and feature intersections might produce new biases. while exploring all possible demographic features prior to building models is simply infeasible, the set of studied features can be determined per task. our approach demonstrated how particular training sets labeled by different groups of people can be used to identify and measure bias in data sets. these biases are never constant or static even within one group, for what counts as hateful is always subjective. in consequence, having only one version of ground truth is bound to produce biased systems. it is inevitable that training models on biased datasets produces systems that amplify those biases, whether these biases are exclusionary, prejudicial, or historical. therefore and due to the conflicting and ever-changing definitions of hate speech among communities, we urge researchers in the hate speech domain to examine their data sets closely and thoroughly in order to understand their limitations and consequences. ## conclusion this work explored bias in hate speech classification models where the task is inherently controversial and annotators' demographic data might influence the labels. we demonstrate how particular demographic features might bias the models in ways that are important to look into prior to using such models in production. we explored the performance of classification models trained and tested on different training and test data splits, in order to identify the fairness of these classifiers and the biases they absorb. we hope that our proposed method for identifying and measuring annotator bias based on annotators' demographic characteris-tics will help to build fairer hate speech classifiers.