Hate Speech Classifiers Learn Normative Social Stereotypes (2023)

Abstract: Social stereotypes negatively impact individuals’ judgments about different groups and may have a critical role in understanding language directed toward marginalized groups. Here, we assess the role of social stereotypes in the automated detection of hate speech in the English language by examining the impact of social stereotypes on annotation behaviors, annotated datasets, and hate speech classifiers. Specifically, we first investigate the impact of novice annotators’ stereotypes on their hate-speech-annotation behavior. Then, we examine the effect of normative stereotypes in language on the aggregated annotators’ judgments in a large annotated corpus. Finally, we demonstrate how normative stereotypes embedded in language resources are associated with systematic prediction errors in a hate-speech classifier. The results demonstrate that hate-speech classifiers reflect social stereotypes against marginalized groups, which can perpetuate social inequalities when propagated at scale. This framework, combining social-psychological and computational-linguistic methods, provides insights into sources of bias in hate-speech moderation, informing ongoing debates regarding machine learning fairness.

## study 1: text annotation here, we investigate the effect of individuals' social stereotypes on their hate speech annotations. specifically, we aim to determine whether novice annotators' stereotypes (perceived warmth and/or competence) of a mentioned social group lead to higher rate of labeling text as hate speech and higher rates of disagreement with other annotators. we conduct a study on a nationally stratified sample (in terms of age, ethnicity, gender, and political orientation) of us adults. first, we ask participants to rate eight us-relevant social groups on different stereotypical traits (e.g., friendliness). then, participants are presented with social media posts mentioning the social groups and are asked to label the content of each post based on whether it attacks the dignity of that group. we expect the perceived warmth and/or competence of the social groups to be associated with participants' annotation behaviors, namely, their rate of labeling text as hate speech and disagreeing with other annotators. participants to achieve a diverse set of annotations, we recruited a relatively large (@xmath0,228) set of participants in a us sample stratified across participants' gender, age, ethnicity, and political ideology through qualtrics panels. 1 after filtering participants based on quality-check items (described below), our final sample included 857 american adults (381 male, 476 female) ranging in age from 18 to 70 (@xmath1.7, @xmath2.4) years, about half democrats (50.4%) and half republicans (49.6%), with diverse reported race/ethnicity (67.8% white or european american, 17.5% black or african american, 17.7% hispanic or latino/latinx, 9.6% asian or asian american). stimuli to compile a set of stimuli items for this study, we selected posts from the gab hate corpus (ghc; @xcite , which includes 27,665 social-media posts collected from the corpus of gab.com @xcite , each annotated for their hate speech content by at least three expert annotators. we collected all posts with high disagreement among the ghc's (original) annotators (based on equation 1 for quantifying item disagreement) which mention at least one social group. we searched for posts mentioning one of the eight most frequently targeted social groups in the ghc: ( 1 ## study 2: ground-truth generation the high levels of inter-annotator disagreements in hate speech annotation @xcite can be attributed to numerous factors, including annotators' varying perception of the hateful language, or ambiguities of the text being annotated @xcite . however, aggregating these annotations into single ground-truth labels disregards the nuances of such disagreements @xcite and even leads to disproportionate representation of individual annotators in annotated datasets @xcite . here, we explore the effect of normative social stereotypes, as encoded in language, on the aggregated hate labels provided in a large annotated dataset. annotated datasets of hate speech commonly represent the aggregated judgments of annotators rather than individual annotators' annotation behaviors. therefore, rather than being impacted by individual annotators' self-reported social stereotypes (as in study 1), we expect aggregated labels to be affected by normative social stereotypes. here, we rely on semantic representations of social groups in pre-trained language models, known to encode normative social stereotypes and biases of large text corpora @xcite . figure 3 illustrates the methodology of study 2. data we analyzed the ghc @xcite in study 1) which includes 27,665 social-media posts labeled for hate speech content by 18 annotators. this dataset includes 91,967 annotations in total, where each post is annotated by at least three coders. based on our definition of item disagreement in equation 1, we computed the inter-annotator disagreement and the majority vote for each of the posts and considered them as dependent variables in our analyses. ## discussion here, we integrate theory-driven and data-driven approaches @xcite to investigate human annotators' and normative social stereotypes as a source of bias in hate speech datasets and classifiers. in three studies, we combine social psychological frameworks and computational methods to make theory-driven predictions about hate-speech-annotation behavior and empirically test the sources of bias in hate speech classifiers. overall, we find that hate speech annotation behaviors, often assumed to be objective, are impacted by social stereotypes, and that this in turn adversely influences automated content moderation. in study 1, we investigated the association between participants' self-reported social stereotypes against 8 different social groups, and their annotation behavior on a small subset of social-media posts about those social groups. our findings indicate that for novice annotators judging social groups as competent is associated with a higher tendency toward detecting hate and lower disagreement with other annotators. we reasoned that novice annotators prioritize protecting the groups they perceive as warm and competent. these results can be interpreted based on the behaviors from intergroup affect and stereotypes framework @xcite : groups judged as competent elicit passive facilitation (i.e., obligatory association), whereas those judged as lacking competence elicit passive harm (i.e., ignoring). here, novice annotators might tend to ''ignore'' social groups judged to be incompetent and not assign ''hate speech'' labels to inflammatory posts attacking these social groups. however, study 1's results may not uncover the pattern of annotation biases in hate speech datasets as data curation efforts rely on annotator pools with imbalanced representation of different socio-demographic groups @xcite and data selection varies among different datasets. in study 2, we examined the role of social stereotypes in the aggregation process, where expert annotators' disagreements are discarded to create a large dataset containing the ground-truth hate-speech labels. we demonstrated that, similar to study 1, texts that included groups stereotyped to be warm and competent were highly agreed upon. however, unlike study 1, posts mentioning groups stereotyped as incompetent are more frequently marked as hate speech by the aggregated labels. in other words, novice annotators tend to focus on protecting groups they perceive as competent; however, the majority vote of expert annotators tend to focus on common targets of hate in the corpus. we noted two potential reasons for this disparity (1) novice and expert annotators vary in their annotation behaviors; in many cases, hate speech datasets are labeled by expert annotators who are thoroughly trained for this specific task @xcite , and have specific experiences that affect their perception of online hate @xcite . ghc annotators were undergraduate psychologist research assistants trained by first reading a typology and coding manual for studying hate-based rhetoric and then passing a curated test of about thirty messages designed for assessing their understanding of the annotation task @xcite . therefore, their relatively higher familiarity with and experience in annotating hate speech, compared to annotators in study 1, led to different annotation behaviors. moreover, dataset annotators are not usually representative of the exact population that interacts with social media content. as pointed out by @xcite , understanding the socio-cultural factors of an annotator pool can shed light on the disparity of our results. in our case, identities and lived experiences can significantly vary between participants in study 1 and ghc's annotators in study 2, which impacts how annotation questions are interpreted and responded to. (2) social groups with specific stereotypes have imbalanced presence in hate speech datasets; while in study 1, we collect a balanced set of items with equal representation for each of the 8 social groups, social media posts disproportionately include mentions of different social groups, and the frequency of each social group being targeted depends on multiple social and contextual factors. to empirically demonstrate the effect of social stereotypes on supervised hate speech classifiers, in study 3, we evaluated the performance and biased predictions of such models when trained on an annotated dataset. we used the ratio of incorrect predictions to operationalize the classifiers' unintended bias in assessing hate speech toward specific groups @xcite . study 3's findings suggested that social stereotypes of a mentioned group, as captured in large language models, are significantly associated with biased classification of hate speech such that more falsepositive predictions are generated for documents that mention groups that are stereotyped to be cold and incompetent. however, we did not find consistent trends in associations between social groups' warmth and competence stereotypes and false-negative predictions among different models. these results demonstrate that false-positive predictions are more frequent for the same social groups that evoked more disagreements between annotators in study 2. similar to @xcite , these findings challenge supervised learning approaches that only consider the majority vote for training a hate speech classifier and dispose of the annotation biases reflected in interannotator disagreements. it should be noted that while study 1 assesses social stereotypes as reported by novice annotators, studies 2 and 3 rely on a semantic representation of such stereotypes. since previous work on language representation have shown that semantic representations encode socially embedded biases, in studies 2 and 3 we referred to the construct under study as normative social stereotypes. our comparison of results demonstrated that novice annotators' self-reported social stereotypes impact their annotation behaviors, and the annotated datasets and hate speech classifiers are prone to being affected by normative stereotypes. our work is limited to the english language, a single dataset of hate speech, and participants from the us. given that the increase in hate speech is not limited to the us, it is important to extend our findings in terms of research participants and language resources. moreover, we applied scm to quantify social stereotypes, but other novel theoretical frameworks such as the agent-beliefs-communion model @xcite can be applied in the future to uncover other sources of bias. recent research efforts argue that annotators' disagreements should not be treated solely as noise in data @xcite and call for alternative approaches for considering annotators as independent sources for informing the modeling process in subjective tasks @xcite . such efforts tend to improve data collection @xcite and the modeling process in various tasks, such as detecting sarcasm @xcite , humor @xcite , sentiment @xcite , and hate speech @xcite . for instance, @xcite introduced a method for modeling individual annotators' behaviors rather than their majority vote. in another work, @xcite clustered annotators into groups with high internal agreement (similarly explored by @xcite and redefined the task as modeling the aggregated label of each group. our findings especially help such efforts by providing a framework for incorporating annotators' biases into hate speech classifiers. measuring hate speech detection bias when propagated into the modeling process, biases in the annotated hate speech datasets cause groupbased biases in predictions @xcite and lack of robustness in results @xcite . specifically, previous research has shed light on unintended biases @xcite , which are generally defined as systemic differences in performance for different demographic groups, potentially compounding existing challenges to fairness in society at large @xcite . while a significant body of work has been dedicated to mitigating unintended biases in hate speech (and abusive language) classification @xcite @xcite @xcite @xcite @xcite @xcite , the choice of the exact bias metrics is not consistent within all these studies. as demonstrated by @xcite , various bias metrics can be considered as different parametrizations of a generalized metric. in hate speech detection in particular, disproportionate false predictions, especially false positive predictions, for marginalized social groups have often been considered as an indicator of unintended bias in the model. this is due to the fact that hate speech, by definition, involves a social group as the target of hate, and the disproportionate mentions of specific social groups in hateful social media content have led to imbalance datasets and biased models. ## conclusion our findings suggest that hate speech classifiers trained on human annotations will acquire particular prediction biases toward historically marginalized groups. our results have two specific and direct implications: first, one way to decrease unintended bias in classifiers is to study and diversify annotation teams, so that annotators come from different ideological and social backgrounds; by doing so, coders may not agree on a label to (over)protect an ingroup or ''ally'' groups (leading to false positives), or to ignore actually hateful content targeting ''blameworthy'' groups (leading to false negatives). second, supervised learning approaches may benefit from modeling annotation biases, which are reflected in inter-annotator disagreements, rather than the current practice, which is to treat them as unexplained noise in human judgment, to be disposed of through annotation aggregation. our work is an example of how well-established social scientific theories can be applied to explain the novel interactions between algorithms and people. large amounts of data that are being constantly recorded in everchanging socio-technical environments call for integrating social scientific theories for assessing the novel technologies and their associated social impacts.