How Hate Speech Varies by Target Identity: A Computational Analysis (2022)

Abstract: This paper investigates how hate speech varies in systematic ways according to the identities it targets. Across multiple hate speech datasets annotated for targeted identities, we find that classifiers trained on hate speech targeting specific identity groups struggle to generalize to other targeted identities. This provides empirical evidence for differences in hate speech by target identity; we then investigate which patterns structure this variation. We find that the targeted demographic category (e.g. gender/sexuality or race/ethnicity) appears to have a greater effect on the language of hate speech than does the relative social power of the targeted identity group. We also find that words associated with hate speech targeting specific identities often relate to stereotypes, histories of oppression, current social movements, and other social contexts specific to identities. These experiments suggest the importance of considering targeted identity, as well as the social contexts associated with these identities, in automated hate speech classification

## introduction researchers working in natural language processing (nlp) often treat hate speech as a binary, unified, concept that can be detected from language alone. however, as a linguistic concept that relies heavily on social context, hate speech contains a variety of related phenomena @xcite . hate speech is characterized by variation in linguistic features (e.g. implicit vs. explicit), context (e.g. platforms, prior conversations), and communities (social histories and hierarchies). this paper focuses on a crucial aspect of this variation: how hate speech varies by the identity groups it targets. to study this variation, we analyze hate speech datasets that include annotations for which identity group is targeted. drawing from multiple of these datasets, we sample new corpora that target the same identity group. these identity groups vary according to several dimensions, including relevant demographic category (e.g. gender, religion) and relative social power (e.g. socially marginalized or dominant). we empirically test which dimensions most clearly separate different forms of hate speech by evaluating how well classifiers trained on one set of identities generalize to hate speech directed at different sets of identities. we find that hate speech varies most prominently by the targeted demographic category and less so by the social power of the targeted identity group. theorists working in philosophy and sociolinguistics have drawn attention to how hate speech directed at marginalized groups differs from hate directed toward socially dominant groups @xcite . however, we do not find that hate speech toward dominant groups is sufficiently different to consistently increase classification performance when removed from existing datasets. analyzing the most representative terms in hate speech directed toward different identities, we find that many words reflect identity-specific context such as histories of oppression or stereotypes. these results have implications for nlp researchers building generalizable hate speech classifiers, as well as for a more general understanding of variation in hate speech. ## data from surveys of hate speech datasets @xcite and the hate speech dataset catalogue foot_1 , we selected datasets with annotations for targeted identities. we only selected datasets that do not restrict target identities in order to minimize differences in other properties (e.g, domain, year) when comparing across targeted identities. this excludes hate speech datasets and shared tasks that focus on particular targeted identity groups, such as women or immigrants @xcite . we also did not consider hate speech datasets that label targeted demographic category, such as race or gender @xcite , but do not specify the identity group targeted. demographic category is just one of the dimensions of similarities and differences among identity groups that we wish to compare for their affect on hate speech. we included datasets from all domains, except those with synthetic data. since we only found one non-english dataset that contained unrestricted annotations for targeted identities @xcite , we focus on hate speech in english in this work. for generalization analyses, we sampled corpora specific to identity groups across datasets large enough to contain a minimum number of instances of hate speech against enough groups (described in section 4.1). these are the first 4 datasets noted in datasets are resampled to a 30/70 ratio of hate to non-hate to eliminate a source of variance among hate speech datasets known to affect generalization @xcite . non-hate instances are upsampled or downsampled to meet this ratio, which was chosen as typical of hate speech datasets @xcite ). if they do not already contain a binary hate speech label, dataset labels are binarized as described in appendix a. ## cross-identity generalization we examine variation among hate speech targeting different identities in a bottom-up, empirical fashion. in order to do this, we construct corpora of hate speech directed at the most commonly annotated target identities, grouped and normalized as described in section 3.1. we then trained hate speech classifiers on each target identity corpus and evaluated on corpora targeting other identities. along with practical implications for hate speech classification generalization, this analysis suggests which similarities and differences among identities are most relevant for differentiating hate speech. ## variation by demographic category poor generalization results across identity targets ( to test this, we manually assigned normalized and grouped identities to the categories referenced by the identity. for example, the identity of 'asian' references race/ethnicity, while 'asian women' references both race/ethnicity and gender/sexuality. in cases where target groups fit multiple categories (which is not common), we include instances in all corpora they reference. though targeted identities sometimes reference categories such as politics, interests, and age, the only categories that met a threshold of 900 hate speech instances uniformly sampled across datasets were race/ethnicity, religion, and gender/sexuality. details on corpora constructed by category can be found in we then train distilbert hate speech classification models on each corpus and test on all others to measure generalization performance in the same way as for identity generalization. results can be found in performance drops across identity categories, sometimes falling by almost half of the f1score. this suggests that for purposes of automatic classification, hate speech varies significantly by demographic category. classifiers generalize particularly poorly from race/ethnicity and religion to gender/sexuality, and less poorly between race/ethnicity and religion. this may be because of the blurred lines in hate speech targets between racial and religious categories, for example, by conflating muslims and arabs or targeting jews by both religious and racial characteristics. ## variation by power another significant dimension of variation among targeted identities is relative social power in the societies from which hate speech data has been drawn. work on hate speech detection in nlp is often motivated as an effort to fight sexism, racism, homophobia, and other oppressions of marginalized groups, and improve participation of these groups online @xcite . however, this work often frames hate speech as a property of language without considering social context. abstracting away from the particulars of targeted identities, datasets often include hate speech directed at any identity group, regardless of the social context of power or marginalization. such datasets thus include hate speech directed toward groups with relative social power, such as white people or men in english-speaking european and american contexts. calls are growing to consider the role of power and historical oppression in nlp work @xcite . moreover, some theorists of social meaning in language argue that hate speech is fundamentally different when directed at social groups with power @xcite . they note that such speech does not reference the same historical threat of possible violence and recurring oppression as does hate directed toward marginalized groups. from a lens of social dominance theory @xcite , hate speech serves either to perpetuate or challenge group hierarchies depending on its target. activists have called for social media platforms to incorporate this social context by treating hate speech toward marginalized groups as more serious than hate directed toward groups with relative social power @xcite . for these theoretical and practical reasons, we consider empirical differences in hate speech based on the social power of targeted identity groups. similar to previous experiments, we test the generalization of classifiers across identities with different levels of social power. we also test for effects on classification performance when removing hate directed toward socially dominant identity groups from hate speech datasets. if this type of hate is sufficiently different, including it could "muddy" the concept we are after and reduce the effectiveness of classifiers in identifying hate speech. removing it would more closely match commonly stated motivations of nlp work on hate speech. ## lexical variation across target identities to explore how hate speech varies by target identity, we examine the words most strongly associated with each target identity and grouping of identities. we use the sparse additive generative model (sage; @xcite to find words that are most representative of each hate speech corpus. sage finds representative words by learning a generative model that contrasts terms in documents in a section of a corpus with a background frequency distribution over the whole corpus. we run sage over 3 separate corpora: one where each section is an identity-specific split, another with category splits, and another with splits by relative social power. we run sage with a vocabulary size of the most frequent 3000 words and a smoothing rate of 50. larger vocabulary sizes and lower smoothing included less informative, specialty words that did not occur frequently in the corpus. the 10 most representative terms for each of these splits are shown in the difficulty in a binary distinction of dominance and marginalization can be seen through the identity top terms asian chinese, china, asian, ching, chong, asians, japanese, chinaman, ch*nk, japan black n*ggas, black, n*gga, n*gger, africa, blm, negro, ethiopian, blacks, african christians priest, catholic, jesus, priests, bible, christians, christianity, christian, church jews jewish, jews, holocaust, jew, israel, hitler, gas, oven, zionist, k*ke latinx latinos, latino, mexico, mexican, mexicans, beaner, sp*c, latin, hispanic, beaners lgbtq+ transgender, transgendered, transgenders, bisexual, queers, bathroom, f*g, gay men divorce, dudes, men, male, negative, movies, man, priests, soy, dad muslims, arabs islam, muslim, islamic, muslims, isis, terrorist, terrorists, iran, bomb, radical white redneck, white, supremacist, supremacy, mudshark, trash, fascist, shootings women hoes, sexist, woman, hoe, feminist, women, feminists, feminism, slut, bitches ## discussion our results demonstrate that hate speech varies considerably according to which identities are tar-geted. we show evidence that classifiers trained on hate toward one target identity generalize poorly to other target identities, especially across demographic categories such as race/ethnicity, religion and gender/sexuality. these results suggest that the designers of hate speech classifiers pay attention to the distribution of targeted identities in training data. many commonly used hate speech datasets do not specify this information. if the distribution skews toward a particular identity group (such as anti-black racism), then using such a classifier on data that has a different distribution (e.g., mostly antisemitic) would likely give poor performance. more generally, these results suggest a value in treating hate speech as a social and linguistic category with lots of internal variance. this variance depends in part on the social context around targeted identities. classifiers trained on hate speech toward dominant or marginalized groups suffered somewhat when tested on the opposite group. however, we did not find evidence that removing hate speech toward dominant groups clarifies the hate speech signal enough to consistently increase performance beyond what might be expected by removing a random set of targeted identities. this suggests that differences based on the social context of power do not affect the language of hate speech enough to be easily detectable by machine learning classifiers. differences in severity between hate speech targeting socially marginalized or powerful groups is more likely a matter of interpretation by those with social knowledge of power in a particular society. ## conclusion we present a meta-analysis of hate speech datasets annotated for identity group targets. this analysis shows that hate speech differs significantly by target identity, as classifiers trained on hate speech toward one identity do not generalize well to other identities. we then examine what factors of social context structure this variation by target identity. we find evidence for hate speech varying substantially by demographic category, and less so by the relative social power of targeted identities. these results reinforce the importance of variation by social context within hate speech and suggest that researchers pay attention to variation by target identity. future work may address improving generalization across target identities by strategically sampling training data or incorporating multiple identity-specific classifiers. similar analyses may also be conducted on multilingual hate speech datasets in future work.