Mitigating Biases in Toxic Language Detection through Invariant Rationalization (2021)

Abstract: Automatic detection of toxic language plays an essential role in protecting social media users, especially minority groups, from verbal abuse. However, biases toward some attributes, including gender, race, and dialect, exist in most training datasets for toxicity detection. The biases make the learned models unfair and can even exacerbate the marginalization of people. Considering that current debiasing methods for general natural language understanding tasks cannot effectively mitigate the biases in the toxicity detectors, we propose to use invariant rationalization (InvRat), a game-theoretic framework consisting of a rationale generator and a predictor, to rule out the spurious correlation of certain syntactic patterns (e.g., identity mentions, dialect) to toxicity labels. We empirically show that our method yields lower false positive rate in both lexical and dialectal attributes than previous debiasing methods.

## introduction as social media becomes more and more popular in recent years, many users, especially the minority groups, suffer from verbal abuse and assault. to protect these users from online harassment, it is necessary to develop a tool that can automatically detect the toxic language in social media. in fact, many toxic language detection (tld) systems have been proposed in these years based on different models, such as support vector machines (svm) @xcite , bi-directional long shortterm memory (bilstm) @xcite , logistic regression @xcite and fine-tuning @xcite . however, the existing tld systems exhibit some problematic and discriminatory behaviors @xcite . experiments show that the tweets containing certain surface markers, such as identity terms and expressions in african american english (aae), are more likely to be classified as hate speech by the current tld systems @xcite , although some of them are not actually hateful. such an issue is predominantly attributed to the biases in training datasets for the tld models; when the models are trained on the biased datasets, these biases are inherited by the models and further exacerbated during the learning process @xcite . the biases in tld systems can make the opinions from the members of minority groups more likely to be removed by the online platform, which may significantly hinder their experience as well as exacerbate the discrimination against them in real life. so far, many debiasing methods have been developed to mitigate biases in learned models, such as data re-balancing @xcite , residual fitting @xcite , adversarial training @xcite and data filtering approach @xcite . while most of these works are successful on other natural language processing (nlp) tasks, their performance on debasing the tld tasks are unsatisfactory @xcite . a possible reason is that the toxicity of language is more subjective and nuanced than general nlp tasks that often have unequivocally correct labels @xcite . as current debiasing techniques reduce the biased behaviors of models by correcting the training data or measuring the difficulty of modeling them, which prevents models from capturing spurious and nonlinguistic correlation between input texts and labels, the nuance of toxicity annotation can make such techniques insufficient for the tld task. in this paper, we address the challenge by combining the tld classifier with the selective rationalization method, which is widely used to inter-pret the predictions of complex neural networks. specifically, we use the framework of invariant rationalization (invrat) @xcite to rule out the syntactic and semantic patterns in input texts that are highly but spuriously correlated with the toxicity label, and mask such parts during inference. experimental results show that invrat successfully reduce the lexical and dialectal biases in the tld model with little compromise on overall performance. our method avoids superficial correlation at the level of syntax and semantics, and makes the toxicity detector learn to use generalizable features for prediction, thus effectively reducing the impact of dataset biases and yielding a fair tld model. ## previous works debiasing the tld task researchers have proposed a range of debiasing methods for the tld task. some of them try to mitigate the biases by processing the training dataset. for example, @xcite add additional non-toxic examples containing the identity terms highly correlated to toxicity to balance their distribution in the training dataset. @xcite use the combination of debiased word2vec and gender swap data augmentation to reduce the gender bias in tld task. @xcite apply the strategy of replacing the bias sensitive words (bsw) in training data based on multiple knowledge generalization. some researchers pay more attention to modifying the models and learning less biased features. @xcite use adversarial training to reduce the tendency of the tld system to misclassify the aae texts as toxic speech. @xcite propose a novel re-weighting mechanism to alleviate the racial bias in english tweets. @xcite implement a multi-task learning framework with an attention layer to prevent the model from picking up the spurious correlation between the certain trigger-words and toxicity labels. debiasing other nlp task there are many methods proposed to mitigate the biases in nlp tasks other than tld. @xcite train a robust classifier in an ensemble with a bias-only model to learn the more generalizable patterns in training dataset, which are difficult to be learned by the naive bias-only model. bras et al. (2020) develop aflite, an iterative greedy algorithm that can adversarially filter the biases from the training dataset, as well as the framework to support it. utama et al. (2020) introduce a novel approach of regularizing the confidence of models on the biased examples, which successfully makes the models perform well on both in-distribution and out-of-distribution data. ## conclusion in this paper, we propose to use invrat to reduce the biases in the tld models effectively. by separately using lexical and dialectal attributes as the environments in invrat framework, the rationale generator can learn to generate genuine linguistic clues and rule out spurious correlations. experimental results show that our method can better mitigate both lexical and dialectal biases without sacrificing much overall accuracy. furthermore, our method does not rely on complicated data filtering or relabeling process, so it can be applied to new datasets without much effort, showing the potential of being applied to practical scenarios.