On Gender Biases in Offensive Language Classification Models (2022)

Abstract: We explore whether neural Natural Language Processing models trained to identify offensive language in tweets contain gender biases. We add historically gendered and gender ambiguous American names to an existing offensive language evaluation set to determine whether models? predictions are sensitive or robust to gendered names. While we see some evidence that these models might be prone to biased stereotypes that men use more offensive language than women, our results indicate that these models? binary predictions might not greatly change based upon gendered names.

## introduction identifying offensive language in text is an increasingly important challenge that has sparked the release of datasets and advanced models focused on toxic language detection in multiple languages @xcite @xcite @xcite . for these models to be trustworthy when deployed in sensitive, real-world contexts, they must perform equally well for text written by male, female, or non-binary authors. however, based on known gender-based biases in nlp systems @xcite @xcite @xcite , especially among models trained to detect abusive language @xcite , we hypothesize that existing nlp systems that incorporate pre-trained word embeddings or transformer-based language models will perform differently given access to authors' names if those names are generally associated with a particular gender. 1 to test the hypothesis that offensive language identification models exhibit gender * work performed while at barnard college 1 in this paper we use an author's name assigned at birth as a proxy for their gender. while we acknowledge the limitations associated with inferring gender from an individual's name, in doing so we recreate real-world circumstances in biases, we adopt the perturbation sensitivity analysis framework @xcite . we perturb examples of an existing dataset by adding historically gendered or gender-ambiguous names to the original texts. we evaluate whether three classes of nlp models (bag of words, bilstm, and transformers) systematically change their predictions on our modified gendered examples. although we see statistically significant differences when comparing a bag of words model's and transformer model's predictions between male and female examples, we do not see convincingly strong evidence that the models' binary predictions for offensiveness consistently change with the addition of gendered names (figure 1 ). therefore, we compare how the model's predicted offensiveness probability changes for perturbed examples. we also explore if there are specific names for which the predicted class probability consistently changes. while we see some remnants of gendered which nlp systems would make gendered associations based upon a speaker's or author's name even when their gender is not explicitly mentioned. ## motivation & bias statement as user-generated content gradually dominates online spaces, offensive text has become more ubiquitous @xcite . unregulated inflammatory or hateful online discourse can have profound effects that extend beyond the web, from negative mental health impacts for targeted individuals to instigation of physical violence @xcite . hence, identifying and moderating toxic dialogue efficiently and accurately is a task that only grows more crucial, and developing automatic methods to detect and flag offensive language is critical. psychological studies spanning the past four decades conclude that, on average, "men use offensive language more than women" (although this gap has shrunk over time), likely as a result of how women are "socialized into subordinate roles and a less inflammatory manner of communicating" @xcite . moreover, these observed patterns of offensive or abusive content authorship translate to online communities like twitter @xcite . research into fairness in nlp indicates that systems trained on large corpora of human-written text tend to replicate existing stereotypes about gendered behavior @xcite . thus, offensive language detection classifiers based on social-media data risk inheriting these underlying assumptions that maleauthored tweets are more likely to utilize offensive language than text written by female individuals. as it becomes more common for social media platforms to rely on nlp systems to detect and remove profane or hateful content online, it be-comes increasingly vital that these classification models are robust to gender biases. while previous research has considered identity-based bias against a gendered subject in abusive language tasks @xcite and gender-based biases among annotations (excell and al moubayed, 2021), how the perceived gender of a speaker or author affects output model classification remains understudied. ## experimental setup our goal is to determine whether offensive language identification models are prone to gender biases. we train bag of word, bilstm, and transformer-based models on the offensive language identification dataset (olid; @xcite . olid is the official dataset used in the offenseval shared tasks @xcite , where tweets containing profanity, insults, threats, hate speech, etc, are labeled as offensive @xcite . olid contains 13,240 annotated english-language tweets (4400 offensive, 8840 not offensive) and 860 test examples (240 offensive, 620 not offensive). for model training, we split the original training set into 12,380 training and 860 dev examples. 2 ## further analysis since our results do not conclusively support our hypothesis that the models' binary predictions change for all considered models when explicitly adding gendered names to our test examples, we turn our attention towards exploring whether, and to what extent, the models' assigned probabilities change for our perturbed dataset. we also investigate whether these predicted probabilities consistently change for any specific names. ## conclusion we asked whether there exists a measurable genderbased asymmetry in models' performances for predicting offensiveness when a tweet explicitly states the speaker's name. our experimental results imply that a range of typical neural models might be robust to perceived author gender when classifying tweets as offensive though they might perceive male authored tweets to be slightly more offensive. our work supports recent findings that intrinsic biases in the word embedding space may not correlate to extrinsic measures of bias in downstream applications @xcite . while these findings on gender bias in offensive classification tasks are promising, we encourage further research to evaluate the extent to which these results generalize across more datasets and language phenomena as well as other social groups and intersectional identities, such as speaker race, age, and sexual orientation.