Contextualizing Hate Speech Classifiers with Post-hoc Explanation (2020)

Abstract: Hate speech classifiers trained on imbalanced datasets struggle to determine if group identifiers like “gay” or “black” are used in offensive or prejudiced ways. Such biases manifest in false positives when these identifiers are present, due to models’ inability to learn the contexts which constitute a hateful usage of identifiers. We extract post-hoc explanations from fine-tuned BERT classifiers to detect bias towards identity terms. Then, we propose a novel regularization technique based on these explanations that encourages models to learn from the context of group identifiers in addition to the identifiers themselves. Our approach improved over baselines in limiting false positives on out-of-domain data while maintaining and in cases improving in-domain performance.

## introduction hate speech detection is part of the ongoing effort to limit the harm done by oppressive and abusive language @xcite @xcite . performance has improved with access to more data and more sophisticated algorithms (e.g., @xcite @xcite , but the relative sparsity of hate speech requires sampling using keywords (e.g., @xcite or sampling from environments with unusually high rates of hate speech (e.g., @xcite . modern text classifiers thus struggle to learn a model of hate speech that generalizes to real-world applications @xcite . a specific problem found in neural hate speech classifiers is their over-sensitivity to group identifiers like "muslim", "gay", and "black", which are only hate speech when combined with the right "[f]or many africans, the most threatening kind of ethnic hatred is black against black." -new york times "there is a great discrepancy between whites and blacks in sa. it is … [because] blacks will always be the most backward race in the world." anonymous user, gab.com context @xcite . in figure 1 we see two documents containing the word "black" that a finetuned bert model predicted to be hate speech, while only the second occurs in a hateful context. neural text classifiers achieve state-of-the-art performance in hate speech detection, but are uninterpretable and can break when presented with unexpected inputs @xcite . it is thus difficult to contextualize a model's treatment of identifier words. our approach to this problem is to use the sampling and occlusion (soc) explanation algorithm, which estimates model-agnostic, posthoc feature importance @xcite . we apply this approach to the gab hate corpus @xcite , a new corpus labeled for "hate-based rhetoric", and an annotated corpus from the stormfront white supremacist online forum @xcite . based on the explanations generated via soc, which showed models were biased towards group identifiers, we then propose a novel regularizationbased approach in order to increase model sensitivity to the context surrounding group identifiers. we apply regularization during training to the explanation-based importance of group identifiers, coercing models to consider the context surrounding them. we find that regularization reduces the attention given to group identifiers and heightens the importance of the more generalizable features of hate speech, such as dehumanizing and insulting language. in experiments on an out-of-domain test set of news articles containing group identifiers, which are heuristically assumed to be non-hate speech, we find that regularization greatly reduces the false positive rate, while in-domain, out-of-sample classification performance is either maintained or improved. ## related work our work is conceptually influenced by @xcite , who formulated hate speech detection as disambiguating the use of offensive words from abusive versus non-abusive contexts. more recent approaches applied to a wide typology of hate speech @xcite , build supervised models trained on annotated (e.g., @xcite or heuristically-labeled @xcite data. these models suffer from the highly skewed distributions of language in these datasets @xcite . research on bias in classification models also influences this work. @xcite measured and mitigated bias in toxicity classifiers towards social groups, avoiding undesirable predictions of toxicity towards innocuous sentences containing tokens like "gay". similarly, annotators' biases towards certain social groups were found to be magnified during classifier training mostafazadeh davani et al. @xcite . specifically within the domain of hate speech and abusive language, park et al. (2018) and @xcite have defined and studied genderand racial-bias, emphasizing issues of undetected dialect variation and imbalanced training data, respectively. techniques for bias reduction in these settings include data augmentation by training on less biased data, term swapping during training (i.e., swapping gender words), and using debiased word embeddings @xcite . complementing these works, we directly manipulate models' modeling of the context surrounding identifier terms by regularizing explanations of these terms. specifically, we use post-hoc explanation algorithms to interpret and modulate finetuned language models like bert @xcite , which achieve state of the art performance on many hate speech detection tasks @xcite . we focus on post-hoc explanation approaches, which interpret model predictions without elucidating the mechanisms by which the model works @xcite . these explanations reveal either word-level @xcite or phrase-level importance @xcite of inputs to predictions. ## data we selected two public corpora for our experiments which highlight the rhetorical aspects of hate speech, versus merely the usage of slurs and explicitly offensive language (see @xcite . the "gab hate corpus" (ghc; @xcite is a large, random sample (@xmath0,655) from the pushshift.io data dump of the gab network * , which we have annotated according to a typology of "hate-based rhetoric", a construct motivated by hate speech criminal codes outside the u.s. and social science research on prejudice and dehumanization. gab is a social network with a high rate of hate speech @xcite and populated by the "alt-right" @xcite . similarly with respect to domain and definitions, de @xcite sampled and annotated posts from the "stormfront" web domain @xcite and annotated at the sentence level according to a similar annotation guide as used in the ghc. train and test splits were randomly generated for stormfront sentences (80/20) with "hate" taken as a positive binary label, and a test set was compiled from the ghc by drawing a random stratified sample with respect to the "target population" tag (possible values including race/ethnicity target, gender, religious, etc.). a single "hate" label was created by taking the union of two main labels, "human degradation" and "calls for violence". training data for the ghc (ghc train ) included 24,353 posts with 2,027 labeled as hate, and test data for the ghc (ghc test ) included 1,586 posts with 372 labeled as hate. stormfront splits resulted in 7,896 (1,059 hate) training sentences, 979 (122) validation, and 1,998 (246) test. ## analyzing group identifier bias to establish and define our problem more quantitatively, we analyze hate speech models' bias towards group identifiers and how this leads to false positive errors during prediction. we analyze the top features of a linear model and use post-hoc explanations applied to a fine-tuned bert model in order to measure models' bias towards these terms. we then establish the effect of these tendencies on model predictions using an adversarial-like dataset of new york times articles. ## contextualizing hate speech models we have shown hate speech models to be oversensitive to group identifiers and unable to learn from the context surrounding these words during training. to address this problem in state-of-the-art models, we propose that models can be regularized to give no explained importance to identifier terms. we explain our approach as well as a naive baseline based on removing these terms. ## conclusion & future work regularizing soc explanations of group identifiers tunes hate speech classifiers to be more contextsensitive and less reliant on high-frequency words in imbalanced training sets. complementing prior work in bias detection and removal in the context of hate speech and in other settings, our method is directly integrated into transformer-based models and does not rely on data augmentation. as such, it is an encouraging technique towards directing models' internal representation of target phenomena via lexical anchors. future work includes direct extension and validation of this technique with other language models such as gpt-2 @xcite ; experimenting with other hate speech or offensive lan-guage datasets; and experimenting with these and other sets of identity terms. also motivated by the present work is the more general pursuit of integrating structure into neural models like bert. regularized hate speech classifiers increases sensitivity to the compositionality of hate speech, but the phenomena remain highly complex rhetorically and difficult to learn through supervision. for example, this post from the ghc requires background information and reasoning across sentences in order to classify as offensive or prejudiced: "donald trump received much criticism for referring to haiti, el salvador and africa as 'shitholes'. he was simply speaking the truth." the examples we presented (see appendix 4 and 5) show that regularization leads to models that are context-sensitive to a degree, but not to the extent of reasoning over sentences like those above. we hope that the present work can motivate more attempts to inject more structure into hate speech classification. explanation algorithms offer a window into complex predictive models, and regularization as performed in this work can improve models' internal representations of target phenomena. in this work, we effectively applied this technique to hate speech classifiers biased towards group identifiers; future work can determine the effectiveness and further potential for this technique in other tasks and contexts.