Demoting Racial Bias in Hate Speech Detection (2020)

Abstract: In the task of hate speech detection, there exists a high correlation between African American English (AAE) and annotators’ perceptions of toxicity in current datasets. This bias in annotated training data and the tendency of machine learning models to amplify it cause AAE text to often be mislabeled as abusive/offensive/hate speech (high false positive rate) by current hate speech classifiers. Here, we use adversarial training to mitigate this bias. Experimental results on one hate speech dataset and one AAE dataset suggest that our method is able to reduce the false positive rate for AAE text with only a minimal compromise on the performance of hate speech classification.

## introduction the prevalence of toxic comments on social media and the mental toll on human moderators has generated much interest in automated systems for detecting hate speech and abusive language @xcite , especially language that targets particular social groups @xcite @xcite . however, deploying these systems without careful consideration of social context can increase bias, marginalization, and exclusion @xcite . most datasets currently used to train hate speech classifiers were collected through crowdsourced annotations @xcite , despite the risk of annotator bias. @xcite show that non-experts are more likely to label text as abusive than expert annotators, and @xcite show how lack of social context in annotation tasks further increases the risk of annotator bias, which can in turn lead to the marginalization of racial minorities. more specifically, annotators are more likely to label comments as abusive if they are written in african american english (aae). these comments are assumed to be incorrectly labelled, as annotators do not mark them as abusive if they are properly primed with dialect and race information @xcite . these biases in annotations are absorbed and amplified by automated classifiers. classifiers trained on biased annotations are more likely to incorrectly label aae text as abusive than non-aae text: the false positive rate (fpr) is higher for aae text, which risks further suppressing an already marginalized community. more formally, the disparity in fpr between groups is a violation of the equality of opportunity criterion, a commonly used metric of algorithmic fairness whose violation indicates discrimination @xcite . according to @xcite , the false positive rate for hate speech/abusive language of the aae dialect can reach as high as 46%. thus, @xcite reveal two related issues in the task of hate speech classification: the first is biases in existing annotations, and the second is model tendencies to absorb and even amplify biases from spurious correlations present in datasets @xcite . while current datasets can be re-annotated, this process is timeconsuming and expensive. furthermore, even with perfect annotations, current hate speech detection models may still learn and amplify spurious correlations between aae and abusive language @xcite . in this work, we present an adversarial approach to mitigating the risk of racial bias in hate speech classifiers, even when there might be annotation bias in the underlying training data. in §2, we describe our methodology in general terms, as it can be useful in any text classification task that seeks to predict a target attribute (here, toxicity) without basing predictions on a protected attribute (here, aae). although we aim at preserving the utility of classification models, our primary goal is not to improve the raw performance over predicting the target attribute (hate speech detection), but rather to reduce the influence of the protected attribute. in §3 and §4, we evaluate how well our approach reduces the risk of racial bias in hate speech classification by measuring the fpr of aae text, i.e., how often the model incorrectly labels aae text as abusive. we evaluate our methodology using two types of data: (1) a dataset inferred to be aae using demographic information @xcite , and (2) datasets annotated for hate speech @xcite where we automatically infer aae dialect and then demote indicators of aae in corresponding hate speech classifiers. overall, our approach decreases the dialectal information encoded by the hate speech model, leading to a 2.2-3.2 percent reduction in fpr for aae text, without sacrificing the utility of hate speech classification. ## methodology our goal is to train a model that can predict a target attribute (abusive or not abusive language), but that does not base decisions off of confounds in data that result from protected attributes (e.g., aae dialect). in order to achieve this, we use an adversarial objective, which discourages the model from encoding information about the protected attribute. adversarial training is widely known for successfully adapting models to learn representations that are invariant to undesired attributes, such as demographics and topics, though they rarely disentangle attributes completely @xcite @xcite @xcite . model architecture our demotion model consists of three parts: 1) an encoder h that encodes the text into a high dimensional space; 2) a binary classifier c that predicts the target attribute from the input text; 3) an adversary d that predicts the protected attribute from the input text. we used a single-layer bidirectional lstm encoder with an attention mechanism. both classifiers are two-layer mlps with a tanh activation function. training procedure each data point in our training set is a triplet {(x i , y i , z i ); @xmath0. . . n }, where x i is the input text, y i is the label for the target attribute and z i is label of the protected attribute. the (x i , y i ) tuples are used to train the classifier c, and the (x i , z i ) tuple is used to train the adversary d. we adapt a two-phase training procedure from @xcite . we use this procedure because @xcite show that their model is more effective than alternatives in a setting similar to ours, where the lexical indicators of the target and protected attributes are closely connected (e.g., words that are common in non-abusive aae and are also common in abusive language datasets). in the first phase (pre-training), we use the standard supervised training objective to update encoder h and classifier c: after pre-training, the encoder should encode all relevant information that is useful for predicting the target attribute, including information predictive of the protected attribute. in the second phase, starting from the bestperforming checkpoint in the pre-training phase, we alternate training the adversary d with equation 2 and the other two models (h and c) with equation 3: unlike @xcite , we introduce a hyper-parameter α, which controls the balance between the two loss terms in equation 3. we find that α is crucial for correctly training the model (we detail this in §3). we first train the adversary to predict the protected attribute from the text representations outputted by the encoder. we then train the encoder to "fool" the adversary by generating representations that will cause the adversary to output random guesses, rather than accurate predictions. at the same time, we train the classifier to predict the target attribute from the encoder output. dataset example founta et al. (2018) i am hungry and i am dirty as hell bruh, need dat shower and dem calories blodgett et al. (2016) so much energy and time wasted hatin on someone when alla that coulda been put towards makin yourself better.... a. . . https://t.co/awcg1nct8t ## results and analysis notably, @xcite show that datasets that annotate text for hate speech without sufficient context-like dwmw17 and fdcl18-may suffer from inaccurate annotations, in that annotators fprs over the dwmw17 and fdcl18 test sets are actually lower-bounds, and the true fpr is could be even higher. furthermore, if we assume that the dwmw17 and fdcl18 training sets contain biased annotations, as suggested by @xcite , then a high fpr over the corresponding test sets suggests that the classification model amplifies bias in the training data, and labels non-toxic aae text as toxic even when annotators did not. to understand the poor performance of our model when trained and evaluated on dwmw17 data, we investigated the data distribution in the test set and found that the vast majority of tweets in trade-off between fpr and accuracy in order to better understand model performance, we explored the accuracy and fpr of our model throughout the entire training process. we evaluate the best checkpoint of the pre-trained model (0 th epoch) and checkpoints of each epoch during adversarial training and show the results in figure 1 . while the baseline model (0 th epoch, before any adversarial training) achieves high accuracy, it also has a high fpr rate, particularly over abusive language. after adversarial training, the fpr rate decreases with only minor changes in accuracy. however, checkpoints with lower fpr rates also often have lower accuracy. while tables 2 and 3 suggest that our model does achieve a balance between these metrics, figure 1 shows the difficulty of this task; that is, it is difficult to disentangle these attributes completely. ## related work preventing neural models from absorbing or even amplifying unwanted artifacts present in datasets is indispensable towards building machine learning systems without unwanted biases. one thread of work focuses on removing bias at the data level, through reducing annotator bias @xcite and augmenting imbalanced datasets @xcite . @xcite propose an unsupervised method based on balancing the training set and employing a proposed measurement for mitigating unintended bias in text classification models. @xcite present a gender-balanced dataset with ambiguous name-pair pronouns to provide diversity coverage for realworld data. in addition to annotator bias, sampling strategies also result in topic and author bias in datasets of abusive language detection, leading to decreased classification performance when testing in more realistic settings, necessitating the adoption of cross-domain evaluation for fairness @xcite . a related thread of work on debiasing focuses at the model level @xcite . adversarial training has been used to remove protected features from word embeddings @xcite and intermediate representations for both texts @xcite and images @xcite . though previous works have documented that adversarial training fails to obliterate protected features, @xcite show that using multiple adversaries more effectively forces the removal. along similar lines, multitask learning has been adopted for learning task-invariant representations. vaidya et al. (2019) show that multitask training on a related task e.g., identity prediction, allows the model to shift focus to toxic-related elements in hate speech detection. ## conclusion in this work, we use adversarial training to demote a protected attribute (aae dialect) when training a classifier to predict a target attribute (toxicity). while we focus on aae dialect and toxicity, our methodology readily generalizes to other settings, such as reducing bias related to age, gender, or income-level in any other text classification task. overall, our approach has the potential to improve fairness and reduce bias in nlp models.