Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection (2022)

Abstract: The perceived toxicity of language can vary based on someone’s identity and beliefs, but this variation is often ignored when collecting toxic language datasets, resulting in dataset and model biases. We seek to understand the *who*, *why*, and *what* behind biases in toxicity annotations. In two online studies with demographically and politically diverse participants, we investigate the effect of annotator identities (*who*) and beliefs (*why*), drawing from social psychology research about hate speech, free speech, racist beliefs, political leaning, and more. We disentangle *what* is annotated as toxic by considering posts with three characteristics: anti-Black language, African American English (AAE) dialect, and vulgarity. Our results show strong associations between annotator identity and beliefs and their ratings of toxicity. Notably, more conservative annotators and those who scored highly on our scale for racist beliefs were less likely to rate anti-Black language as toxic, but more likely to rate AAE as toxic. We additionally present a case study illustrating how a popular toxicity detection system’s ratings inherently reflect only specific beliefs and perspectives. Our findings call for contextualizing toxicity labels in social variables, which raises immense implications for toxic language annotation and detection.

## introduction determining whether a text is toxic (i.e., contains hate speech, abuse, or is offensive) is inherently a subjective task that requires a nuanced understanding of the pragmatic implications of language @xcite @xcite . without this nuance, both humans and machines are prone to biased judgments, such as over-relying on seemingly toxic keywords (e.g., expletives, swearwords; @xcite or backfiring against minorities @xcite . for example, racial biases have been uncovered in toxic language detection where text written in african american english (aae) is falsely flagged as toxic @xcite . the crux of the issue is that not all text is equally toxic for everyone @xcite ). yet, most previous research has treated this detection as a simple classification with one correct label, obtained by averaging judgments by a small set of human raters per post @xcite @xcite @xcite . such approaches ignore the variance in annotations @xcite @xcite based on who the annotators are, and what their beliefs are. in this work, we investigate the who, why, and what behind biases foot_0 in toxicity annotations, through online studies with demographically and politically diverse participants. we measure the effects of annotator identities (who annotates as toxic) and attitudes or beliefs (why they annotate as toxic) on toxicity perceptions, through the lens of social psychology research on hate speech, free speech, racist beliefs, altruism, political leaning, and more. we also analyze the effect of what is being rated, by considering three text characteristics: anti-black or racially prejudiced meaning, african american english (aae), and vulgar words. we seek to answer these questions via two online studies. in our breadth-of-workers controlled study, we collect ratings of toxicity for a set of 15 hand curated posts from 641 annotators of different races, attitudes, and political leanings. then, in our breadth-of-posts study, we simulate a typical toxic language annotation setting by collecting toxicity ratings for ∼600 posts, from a smaller but diverse pool of 173 annotators. 2 distilled in figure 1 , our most salient results across both studies show that annotators scoring higher on our racist beliefs scale were less likely to rate anti-black content as toxic ( §4). additionally, annotators' conservatism scores were associated with higher ratings of toxicity for aae ( §5), and conservative and traditionalist attitude scores with rating vulgar language as more toxic ( §6). we further provide a case study which shows that perspectiveapi, a popular toxicity detection system, mirrors ratings by annotators of certain attitudes and identities over others ( §7). for instance, for anti-black language, the system's scores better reflect ratings by annotators who score high on our scale for racist beliefs. our findings have immense implications for the design of toxic language annotation and automatic detection-we recommend contextualizing ratings in social variables and looking beyond aggregated discrete decisions ( §8). ## the who, why, and what of toxicity annotations we aim to investigate how annotators' ratings of the toxicity of text is influenced by their own identities (who they are; §2.1), and their beliefs 2 please contact the authors for the anonymized study data. ## data & study design we design two online studies to study the effect of annotator identities and attitudes on their toxicity ratings on posts with different characteristics. in either study, annotators are asked to rate how offensive and how racist they consider a post to be (see appendix b for the exact questions). 5 we specifically focus on readers' perceptions or opinions, instead of imposing prescriptive definitions of toxicity or hate speech which previous work has shown still suffers from large annotator disagreement @xcite . in the sections §4-6, we report only (near-)significant associations; see appendix e and f for all results. ## who finds anti-black posts toxic, and why? anti-black language denotes racially prejudiced or racist content-subtle @xcite or overt-which is often a desired target for toxic language detection research @xcite ; vid- anti-black posts rated as offensive rated as racist hᴀʀᴍoғhᴀᴛᴇsᴘᴇᴇᴄʜ 0.117 † 0.154 * fʀᴇᴇoғғsᴘᴇᴇᴄʜ n.s. -0.138 † rᴀᴄɪsᴛbᴇʟɪᴇғs -0.131 * -0.185 * ## who finds aae posts toxic, and why? african american english (aae) is a set of wellstudied varieties or dialects of u.s. english, common among, but not limited to, african-american or black speakers @xcite . this category has been shown to be considered "worse" english by non-aae speakers @xcite @xcite ; rosa and flo-aae posts rated as racist rᴀᴄɪsᴛbᴇʟɪᴇғs @xmath3.089 * politics (lib: 0, cons: 1) @xmath4.076 † based on prior work that correlates racial prejudice with negative attitudes towards aae @xcite , we hypothesize that annotators who are white and who score high in racistbeliefs will rate aae posts as more toxic. additionally, since aae can be considered non-canonical english @xcite , we hypothesize that annotators who are more conservative and who score higher in traditionalism and ling-purism will rate aae posts with higher toxicity. ## toxicity detection system case study: perspectiveapi our previous findings indicated that there is strong potential for annotator identities and beliefs to affect their toxicity ratings. we are additionally interested in how this influences the behavior of toxicity detection models trained on annotated data. we present a brief case study to answer this question with the perspectiveapi, 9 a widely used, commercial system for toxicity detection. appendix g provides a more in-depth description. we investigate whether perspectiveapi scores align with toxicity ratings from workers with specific identities or attitudes, using the 571 posts from our breadth-of-posts study. specifically, we compare the correlations between per-spectiveapi scores and ratings from annotators, broken down by annotators with different identities (e.g., men and women) or with higher or lower scores on attitude scales (split at the mean). see appendix g.1 for details about this methodology. overall, our findings indicate that perspec-tiveapi toxicity score predictions align with specific viewpoints or ideologies, depending on the text category. particularly, it seems that the api underestimates the toxicity of anti-black posts in a similar way to annotators who scored higher on the racistbeliefs scale, and aligns more with white annotator's perception of aae toxicity (vs. black annotators). this corroborate prior findings that show that toxicity detection models inherently encode a specific positionality (cambo, 2021) and replicate human biases @xcite . ## discussion & conclusion overall, our analyses showed that perceptions of toxicity are indeed affected by annotators' demographic identities and beliefs ( §2). we foundvia a breadth-of-workers study and a breadth-ofposts study ( §3)-several associations when isolating specific text characteristics: anti-black ( §4), aae ( §5), and vulgarity ( §6). finally, we showed that a popular toxicity detection system yields toxicity scores that are more aligned with raters with certain attitudes and identities than others ( §7). we discuss implications of our findings below. variation in toxicity ratings in hate speech datasets. in our study we deliberately sought rat-ing of perceptions of toxicity of posts by racially and politically diverse participants. however, many existing hate speech datasets instructed annotators to adhere to detailed definitions of toxicity @xcite , and some even selected crowdworkers for their liberal ideology @xcite @xcite . while those annotation setups and annotator homogeneity could cause less variation in toxicity annotations of anti-black, aae, and vulgar posts, there is still empirical evidence of anti-aae racial biases in some of these datasets @xcite . given the large variation in perceptions of toxicity that we showed and the implicit encoding of perspectives by toxicity models, we recommend researchers and dataset creators investigate and report annotator attitudes and demographics; researchers could collect attitude scores based on relevant social science research, perhaps in lightweight format as done in our breadth-ofposts study, and report those scores along with the dataset (e.g., in datasheets; @xcite . contextualize toxicity predictions in social variables. as shown in our results and previous studies (e.g., @xcite @xcite , determining what is toxic is subjective. however, given this subjectivity, the open question remains: whose perspective should be considered when using toxicity detection models? to try answering this question, we urge researchers and practitioners to consider all stakeholders and end users on which toxicity detection systems might be deployed (e.g., through humancentered design methods; @xcite @xcite . while currently, the decision of content moderation often solely lies in the hands of the platforms, we encourage the exploration of alternative solutions (e.g., community fact checkers, digital juries; @xcite . in general, we urge people to embrace that each design decision has socio-political implications @xcite , and encourage them to develop technologies to shift power to the targets of oppression @xcite @xcite . finally, given the increasingly essential role of online platforms in people's daily lives (rahman, 2017), we echo calls for policy regulating online spaces and toxicity detection algorithms @xcite @xcite . beyond toxicity classification: modeling distributions and generating explanations. our findings on the subjectivity of the toxicity detection tasks suggests that standard approaches of obtaining binary (or even n-ary) labels of toxicity and averaging them into a majority vote are inadequate. instead, researchers could consider modeling the distribution or variation in toxicity labels with respect to individual annotators @xcite @xcite or to specific identities or beliefs. but, perhaps more importantly, we encourage re-thinking the toxicity detection paradigm altogether. with the goal to assist human content moderators, foot_8 creating systems that explain biased implications of posts could be more helpful than opaque toxicity scores thus, we advocate for moving away from classification frameworks, and towards more nuanced, holistic, and explainable frameworks for inferring the desired concepts of toxicity and social biases (e.g., social bias frames; @xcite .