Towards Fairer NLP Models: Handling Gender Bias In Classification Tasks (2024)

Abstract: Measuring and mitigating gender bias in natural language processing (NLP) systems is crucial to ensure fair and ethical AI. However, a key challenge is the lack of explicit gender information in many textual datasets. This paper proposes two techniques, Identity Term Sampling (ITS) and Identity Term Pattern Extraction (ITPE), as alternatives to template-based approaches for measuring gender bias in text data. These approaches identify test data for measuring gender bias in the dataset itself and can be used to measure gender bias on any NLP classifier. We demonstrate the use of these approaches for measuring gender bias across various NLP classification tasks, including hate speech detection, fake news identification, and sentiment analysis. Additionally, we show how these techniques can benefit gender bias mitigation, proposing a variant of Counterfactual Data Augmentation (CDA), called Gender-Selective CDA (GS-CDA), which reduces the amount of data augmentation required in training data while effectively mitigating gender bias and maintaining overall classification performance.

## introduction in recent years, there has been a significant growth in research analyzing biases present in natural language processing (nlp) systems and models. this includes studies on biases present in embedding spaces, which are representations of words and sentences generated from large text data @xcite @xcite @xcite and in large language models @xcite . researchers have investigated how these biases manifest in nlp systems across a range of tasks, coreference resolution @xcite , machine translation @xcite @xcite , sentiment analysis @xcite , and hate speech/toxicity detection @xcite , among others. as nlp models are trained on human-generated text data, they can acquire and propagate societal biases present in that data when deployed in realworld applications, leading to concerns about discriminating outputs @xcite . machine learning models can be deliberately designed with a specific bias aligned with their intended purpose. for example, a toxic comment detector is meant to be biased toward giving higher scores to actual toxic comments over non-toxic ones. however, such models are not intended to discriminate based on attributes like gender that might be evident in comments. if a model exhibits this behavior by scoring comments differently due to gender references, it is considered an unintended and undesirable bias. while the bias towards accurately identifying toxic content is the intended goal, any bias that leads to unfair treatment or discrimination based on attributes such as gender is regarded as an unintended bias that needs to be addressed @xcite . biased algorithmic outcomes from ai systems can negatively impact users, creating a feedback loop that amplifies existing biases @xcite . these harmful effects can impact different groups based on the nature of the bias, such as women facing discrimination from gender biases, minorities affected by racial biases, or specific age groups impacted by age-related biases. evaluating and mitigating these unintended biases is crucial for developing trustworthy, fair, and ethical ai systems. bias statement.in textual classification tasks, gender bias refers to the presence of systematic errors or unfairness in predictions related to gender within the text data. our key concern is the potential allocational harms arising from such systematic gender biases in nlp models, where the systems may disproportionately misclassify or make inaccu-rate predictions for text associated with a particular gender group @xcite . for instance, a sentiment analysis model might demonstrate gender bias by associating certain emotions or sentiments more strongly with one gender, regardless of the context @xcite . hate speech detection models can also display gender biases towards specific identity terms due to factors like uneven distribution in datasets and excessive use of certain identity terms in hate speech sentences. for instance, terms such as "women" and "feminism" may often be associated with sexist comments in benchmark datasets, leading to incorrect generalisations by the model @xcite . this could lead to unfair censorship or moderation applied disproportionately to one gender. similarly, biased fake news detectors may struggle more to identify misinformation targeting or involving females versus males. such gender disparities in nlp system performance can propagate societal biases and enable discriminatory downstream impacts. our normative stance is that an ideal nlp system should perform equally well regardless of the gender mentioned or associated with the input text. significant differences in accuracy across genders in core classification tasks is an undesirable outcome that can enable allocational harms through unfair allocation of negative consequences like censorship, spread of misinformation, or mischaracterisation. a primary method for identifying gender bias in an nlp system is to measure whether the performance differs across genders. however, one of the main challenges in many textual corpora is the absence of explicit gender identification. gender bias evaluation testsets (gbets), named by @xcite have been employed to address this challenge. gbets facilitate gender identification by creating synthetic test sets that isolate specific groups of individuals. this enables the evaluation of bias across various natural language processing (nlp) tasks. there are three types of gbets @xcite , template-based datasets, natural languagebased datasets, and datasets generated for probing language models. the template approach involves creating sentence templates with words related to gender and the specific task being evaluated. from each template sentence individual sentences are generated, one for each gender. the performance of the nlp system is then compared across the groups of this synthetic test data, one group for each gender, allowing for the measurement of gender bias. this gender identity template approach has been used (across binary genders) for various nlp tasks, including abusive language detection @xcite , sentiment analysis @xcite , and coreference resolution @xcite . additionally, the gender identity template has been extended to include non-binary genders @xcite . while template-based approaches offer a way to create gender bias evaluation datasets, they face certain limitations. the artificially generated text may not accurately represent the true distribution and content of real-world data for the target task. additionally, the templates need to be carefully designed for each specific downstream task, lacking generalisability across different nlp applications. furthermore, studies have demonstrated that the performance of these synthetically generated test datasets on the intended downstream tasks is often poor. in this work, we propose two techniques to identify gender in natural language text to facilitate measuring gender bias in nlp systems, aiming to overcome the limitations of template-based approaches. the first technique, identity term sampling (its), is a knowledge-light approach built upon the work by (sobhani and delany, 2022) which we further extend in this study. the second technique, identity term pattern extraction (itpe), is a more knowledge-intensive alternative that we propose to address the shortcomings of its. both techniques involve selecting the test set used to measure gender bias in the nlp model from the main dataset itself, ensuring that the test data aligns with the training dataset for the target task and is not synthetically produced like template data. by leveraging the dataset itself, these techniques enable a more reliable and representative evaluation of gender bias within the nlp model's intended domain and data characteristics. we apply these new techniques, its and itpe, to measure gender bias across a diverse range of natural language processing classification tasks involving textual data about people. such tasks, including hate speech detection, fake news identification, and sentiment analysis, are more likely to exhibit gender bias due to the presence of personal references and mentions within the text. in addition, we use the itpe approach in a proposed variant of counterfactual data augmenta-tion (cda) @xcite , which we call gender-selective cda (gs-cda). this variant selectively applies cda only to the gender-identified instances in the training set, using our proposed itpe technique. we demonstrate that gs-cda effectively reduces gender bias gaps (in some cases more than cda itself) while maintaining overall classification performance with the significant benefit of reducing the computational overhead of augmenting the entire training data. ## approach to address the challenge of the lack of gender identification for evaluating gender bias in nlp models, we propose two distinct techniques: identity term sampling (its) which is a knowledgelight approach, and identity term pattern extraction (itpe), a more knowledge-intensive approach. these techniques aim to determine whether the natural language text is talking about a person and to identify the gender of that person by leveraging gender identity terms and associated patterns within the text. by applying these techniques to datasets that may be used to train models for downstream classification tasks, a section of the dataset, with gender identified, can be used as test data to measure the gender bias of the model built on that training data. identity term sampling (its) uses the frequency of gender identity terms in a data instance to identify the gender in a sample of text that could be about a person. its can assign gender to those data instances in a dataset that contains at least one gender identity term. in each data instance, the frequency of male and female identity terms listed in in cases where there are multiple occurrences of the matched pattern, the algorithm counts the frequency of male and female gender identity terms within the data instance. the gender with the higher cumulative frequency across these patterns is then assigned as the label for that instance. in cases where the data instance contains multiple sentences, the algorithm determines the overall gender label for that data instance by selecting the majority gender across all sentences. to illustrate how itpe and its operate in practice, we can examine the sentence: order pos pattern examples 1 subject he, she, my mother, that guy 2 pronoun-noun his cookbook, his name, her choice, her face 3 adjective-noun male oppression, stupid man, female announcer, female character 4 noun-noun boy scout, boy teams, women comedian, woman commentator 5 pronoun-verb he did, he thinks, she changed, she thought 6 proposition-pronoun to him, for him, about her, to her 7 verb-pronoun tell him, reassuring him, loves her, find her 8 determiner-noun the man, that boy, a girl, this woman 9 pronoun-adjective-noun his real name, her first mate itpe would first identify the subject "the pioneering female ceo". this matches the subject pattern (order 1 in ## measuring bias in an nlp task to evaluate the efficacy of the proposed itpe technique in measuring bias for various nlp tasks, we apply it to identify gender on several datasets that do not initially provide gender identification. we then use this to measure gender bias on a number of different types of nlp classification tasks, including hate speech detection, fake news identification, and sentiment analysis. we focus on using itpe as it generally can identify gender for a larger number of data instances in a dataset. we use three hate speech datasets, two fake news, and a sentiment analysis dataset. the hatespeech dataset (waseem and hovy, 2016) is a collection of almost 17k tweets consisting of 3,383 samples of sexist content, 1,972 samples of racist content, and 11,559 neutral samples. the dataset is transformed into a binary classification problem by labeling the sexist and racist samples as the "offensive" class and neutral samples as the "non-offensive" class. the hatespeech and offensive dataset (davidson et al., 2019) is a collection of almost 24k tweets. the majority of tweets are considered to be offensive language (77%), almost 17% are labeled as non-offensive and only almost 6% of the tweets are flagged as hate speech samples. by assigning the "offensive" class label to samples exhibiting hate speech and offensive, and the "non-offensive" label to non-offensive samples, we convert the dataset into a binary classification problem. the social bias inference corpus (sbic) dataset @xcite over 44k posts collected from various sources of potentially biased online content including twitter, reddit, and hate sites. each post is annotated by crowdsourcing workers on amazon mechanical turk, with different annotations per post. for classification in this study, we selected the data with offensive and non-offensive categories as the target labels. the word embedding over linguistic features for fake news detection (welfake) dataset @xcite ) consists of about 71k news articles with 35k real and about 37k fake news from popular news datasets. the dataset includes the title and body of the news, for the purpose of gender identification we only used the title. the second fakenews dataset is a kaggle dataset @xcite ) consisting of about 44k instances, each labeled as reliable or unreliable. each article in the dataset is provided with both a title and body text. however, for the purpose of gender bias evaluation and classification, we only use the title. the moji dataset (blodgett et al., 2016) contains over 2m tweets that are used for sentiment analysis, categorising them as either positive or negative. additionally, the dataset provides details regarding the type of english used in the tweets, which is a sensitive attribute in fairness-aware methods. this attribute distinguishes between african-american english (aae) and standard-american english(sae). for the classification tasks, we use a pre-trained bert model @xcite from the hugging face library @xcite . the datasets are split into stratified training and holdout testing splits, with an 80/20 ratio. the hyperparameters of the model are tuned on a 20% split of the training data for each dataset. the full holdout test split is used to measure the overall task performance (accuracy) of the models. to evaluate classification performance, we use average class accuracy (aca). measures for evaluating gender bias in nlp systems are often built upon the work of @xcite on equal opportunity and equalized odds. these measures utilize the gender distributions in the training data, rather than insisting on equal outcomes for both genders regardless of the ground truth prevalence (democratic parity). equality of opportunity considers where the predictions are independent of gender but conditional on the ground truth or positive outcome in the training data. in this work, we adapt the t p r gap measure used by @xcite , which measures the difference in the true positive rates across genders classification task, to a more general measure class gap to quantify disparities in a model's performance across genders. for a given class c, the class gap is defined as equation the gender-identified column in looking at the class gap results for hate speech in additionally, we compared our proposed approach using gender-identified instances from the original data (itpe approach) with a templatebased synthetic test set generation method. the template-based approach, following the work by @xcite , was applied specifically to the hate speech dataset, as it is more accessible for this type of dataset compared to others. for the hate speech dataset, the template-based approach generated 1480 synthetic test samples in total, with 740 pairs of male and female instances equally distributed across the "offensive" and "non-offensive" classes. the average class accuracy (aca) for the template-based test set is reported in the templatebased aca column of as the moji dataset had labels for the type of english, african american english (aae) and standard american english (sae), we had the opportunity to explore potential gender gap differences between a subset of aae and sae, to see if any disparities emerged when considering the racial characteristics present in language expression. we focused on the class gap within each subset. the results are presented in there is little bias in the aae subset with the class gap values showing a minimal difference between male and female instances. however, in the sae dataset, there is more bias shown with the positive sentiment class gap exhibiting a positive value, and the negative sentiment class gap with negative value. essentially, this suggests that for standard american english, the model tended to classify more male-written text as negative sentiment and female-written text as positive sentiment. in contrast, such distinctions were not observed in the african american english subset. in general, the results reveal more pronounced gender bias in the hate speech detection task compared to fake news identification and sentiment analysis which may not be surprising due to the nature of the task. hate speech models exhibit substantial class gender gaps, indicating biases in classifying offensive content based on gender mentions. in contrast, fake news detection models show relatively smaller gender gaps, while sentiment analysis exhibits negligible bias. however, upon examining individual groups of african american english and standard american english in the sentiment analysis task, gender bias is observed within the standard american english texts. ## using itpe in bias mitigation we have seen in the previous section that the models for hate speech detection exhibit gender bias. mitigating bias in machine learning models is a critical challenge to ensure fairness and prevent discrimination against protected groups. strategies employed for bias mitigation can be categorized into three main approaches: pre-processing, inprocessing (during training), and post-processing @xcite . preprocessing techniques adjust the training dataset prior to model training to achieve balanced representations across protected groups such as gender and race. a common approach is resampling the training set, such that the number of instances within each protected group is equal. one popular pre-processing technique for mitigating gender bias is counterfactual data augmentation (cda) @xcite . cda augments the training data with gender-swapped examples, building upon basic gender word swapping (e.g., "he" to "she") while addressing key limitations. it handles co-references to maintain grammatical consistency, swapping gendered words that co-refer to proper nouns (e.g., "queen elizabeth" to "king elizabeth"). cda offers a systematic approach to augmenting the data with counterfactual examples, providing a comprehensive solution to reduce gender bias encoding. in-processing or during-training approaches introduce constraints into the model optimization process. a widely adopted method is adversarial training, which jointly trains a discriminator to recover protected attributes from the model's representations and the main model to make accurate predictions while preventing the discriminator from determining the protected attributes @xcite . while adversarial training has been shown to reduce bias in machine learning models @xcite , one of its key limitations is that it requires having access to sensitive attribute labels (e.g. gender, race) during the training process. the need for annotated sensitive attribute data can be restrictive, as such labels may not always be available in the data. if we consider a task like hate speech identification and the datasets used in the previous section the sensitive attribute, gender, is not identified in to evaluate how useful gs-cda is in bias mitigation, we use the same approach discussed in section 2. the results of classification and gender bias after applying the cda and gs-cda to training data are shown in comparing the original classification and gender bias results on hate speech datasets in compared to the original models, applying cda during training data augmentation leads to a reduction in gender bias gaps. notably, cda lowers the offensive class gap from 0.093 to 0.072 on the hatespeech(w&h) dataset and the nonoffensive class gap from 0.109 to 0.068 on the sbic dataset. however, the classification accuracy (aca) remains almost the same. the gs-cda variant demonstrates even more promising results. gs-cda achieves further reductions in gender bias gaps, outperforming both the original models and the full cda approach. on the hatespeech(w& h) dataset, gs-cda lowers the offensive class gap to 0.039 and the non-offensive class gap to 0.060, while on sbic, the non-offensive gap is reduced to 0.032. remarkably, gs-cda maintains comparable or slightly improved aca compared to the original models. these findings suggest that se-lectively augmenting gender-identified instances is an effective strategy for mitigating bias while preserving overall classification performance. the tsize columns in the table show the number of training instances for the original datasets before any mitigation, as well as the training set size after applying the mitigation techniques. as can be observed, the training set size for cda is almost twice as large as the original dataset size. however, the training set size for gs-cda is significantly smaller than that of cda, adding only around 20% to the original dataset size. gs-cda offers an additional benefit over the full cda approach by avoiding the computational expense associated with doubling the training data size, as is the case with cda. ## conclusion this paper addresses the challenge of measuring and mitigating gender bias in nlp systems by proposing its and itpe as techniques for identifying gender information in textual data, which can be used as an alternative to template-based approaches for measuring gender bias. through the evaluation on multiple datasets, we demonstrate the techniques performance in accurately assigning gender labels. by applying itpe, we demonstrated measuring gender bias in various nlp classification tasks, including hate speech detection, fake news identification, and sentiment analysis. we showed that these techniques facilitate measuring gender bias in a wide variety of nlp classification tasks, which offers significant benefits over the existing template technique which has only been used for hate speech detection. however, it is important to acknowledge the limitations of our techniques. one limitation is the inability to recognize names. this is primarily because names vary significantly across different cultures and regions, and many libraries do not adequately support some names including irish, asian, and other ethnic groups. additionally, some names are unisex, making gender identification based on names alone tricky and often inaccurate. another important limitation is that this approach only considers binary gender, which excludes non-binary and other gender identities. in addition, we have used the itpe technique to mitigate observed gender bias by introducing gender-selective counterfactual data augmentation (gs-cda), a variant of the popular cda approach. gs-cda selectively augments only the gender-identified instances during training, leveraging itpe's capabilities. our results show that gs-cda effectively reduces gender bias gaps while maintaining overall classification performance, outperforming the conventional cda approach and using less augmented data. the proposed techniques, itpe and gs-cda, offer practical alternatives to template-based methods for measuring and mitigating gender bias in nlp systems. by addressing the limitations of template techniques and efficiently augmenting training data, these approaches pave the way for fairer and more ethical ai systems. as future work, these techniques will be extended to other protected attributes and applied to a broader range of nlp tasks to promote algorithmic fairness and responsible ai development. in addition, they will be extended to include non-binary and transgender individuals, emphasizing the importance of addressing the full spectrum of gender identities in nlp research. while our proposed methods have shown effectiveness in certain nlp tasks, it will be very intriguing to see how these methodologies generalize across different languages and cultures and perform in more diverse or complex datasets.