How Far Can It Go? On Intrinsic Gender Bias Mitigation for Text Classification (2023)

Abstract: To mitigate gender bias in contextualized language models, different intrinsic mitigation strategies have been proposed, alongside many bias metrics. Considering that the end use of these language models is for downstream tasks like text classification, it is important to understand how these intrinsic bias mitigation strategies actually translate to fairness in downstream tasks and the extent of this. In this work, we design a probe to investigate the effects that some of the major intrinsic gender bias mitigation strategies have on downstream text classification tasks. We discover that instead of resolving gender bias, intrinsic mitigation techniques and metrics are able to hide it in such a way that significant gender information is retained in the embeddings. Furthermore, we show that each mitigation technique is able to hide the bias from some of the intrinsic bias measures but not all, and each intrinsic bias measure can be fooled by some mitigation techniques, but not all. We confirm experimentally, that none of the intrinsic mitigation techniques used without any other fairness intervention is able to consistently impact extrinsic bias. We recommend that intrinsic bias mitigation techniques should be combined with other fairness interventions for downstream tasks.

## introduction the use of pretrained language models has seen a surge in popularity as a result of state-of-the-art performances that have been achieved with these models on various tasks. consequently, there has been a growing interest in exploring how gender bias pertains in these models @xcite . pretrained language models are used in two distinct phases: the pretraining phase and the finetuning phase. the pretraining phase typically involves training a model on a generic task such as masked language modeling on a diverse set of text corpora. in the finetuning phase, the pretrained model can be adapted for a specific task, such as sentiment analysis, by training on a domain-specific corpus. owing to the unique way of using pretrained models, bias generally manifests in two forms: intrinsic bias and extrinsic bias. intrinsic bias refers to bias that inherently exists in pretrained language models whereas extrinsic bias is used to refer to bias that exists in downstream models that are based on the pretrained model. since the success of downstream nlp tasks has mostly been dependent on pretrained models, it is intuitive to assume that bias in intrinsic models will compromise fairness in downstream tasks. only recently have more indepth examinations been done to investigate this assumption @xcite @xcite . however, conclusions have been inconsistent and the confounding effects of bias mitigation techniques remain unknown. the main focus of this work is to investigate the impact of intrinsic bias on extrinsic fairness and if techniques to mitigate intrinsic bias actually resolve bias or only mask it. we develop a probe to uncover intrinsic bias by determining the amount of gender information in a word embedding using a classifier. bearing in mind how abstractly and improperly intrinsic bias has been defined @xcite , coupled with the discovery that results from different metrics for intrinsic bias in many cases do not correlate @xcite , we find this probe effective as an extra step in evaluating the efficacy of these mitigation strategies. we realize from this study that how intrinsic bias has been measured and the choice of bias mitigation strategies explored by some existing works have not been ideal. we further use this probe to investigate if some proposed mitigation strategies superficially conceal bias. in this work, when we refer to bias in a language model, we mean stereotyping bias as defined by @xcite as "the undesired variation of the @xcite of certain words in that language model according to certain prior words in a given domain". we focus our experiments on gender bias for two primary reasons: its intuitive nature making it easy to analyze and discuss, and the accessibility of resources and datasets regarding gender. for the same reason, we narrow our definition of bias in our experiments and analysis to binary gender bias foot_0 . this paper considers the primary goal of mitigating intrinsic bias to ensure fairness in downstream tasks. we consider one case of binary classification and one multiclass classification case, all on english language corpora using the bert-large @xcite and the albert-large (lan et al., 2019) pretrained models for each task. in summary, we develop an extensive probe foot_1 to uncover intrinsic bias in pertained contextualized language models, and seek to answer three key research questions: rq1: do different intrinsic bias metrics respond differently to different bias mitigation techniques? ( § 4.1). rq2: can intrinsic bias mitigation techniques hide bias instead of resolving it? ( § 5.1). rq3: do intrinsic bias mitigation techniques in language models improve fairness in downstream tasks? ( § 5.2) 2 measuring and mitigating bias since many techniques for measuring and mitigating gender bias have been introduced for both intrinsic and extrinsic bias, we only discuss techniques we use in experiments in this paper. bias mitigation techniques can be applied to pretrained or finetuned language models, or both. figure 1 illustrates these interactions with both training settings and this section will discuss both intrinsic ( § 2.1) and extrinsic ( § 2.3) mitigation techniques. additionally, this section will provide a brief overview of bias measures foot_2 . ## probing for gender information we design a probe to investigate how much gender information is retained after mitigating bias in pretrained language models. our goal is to measure the amount of gender information in the resulting embeddings of language models after mitigation techniques from § 2.1 have been applied to them. defining attribute and stereotype terms. we first identify two sets of attribute terms that define females and males respectively. let w @xmath0, f emale, woman, ...} and w @xmath1, male, man, ...} be the respective sets of female and male gender-defining words. the set @xmath2. secondly, we define two sets of stereotype terms which are composed of target words associated with females and males respectively foot_7 . let x @xmath3, nurse, sof tball, ...} and x @xmath4, baseball, engineer, ...} be the respective sets of female and male stereotype words such that @xmath5. extracting word embeddings. we extract all sentences from a text corpus containing words in each set of words. especially for stereotype terms, we ensure that the sentences do not also include words in the attribute wordlist since attribute words in the same context can introduce gender information in the embedding produced. let s @xmath6, ..., s wn } be the set of all sentences containing a word w. we use the pretrained language model to extract embeddings for w such that @xmath7, ..., e wn }, where e w @xmath8. if w appears in s w i multiple times, then |e w | > n. training the classifier. we split w into a train and test sets w t and w i respectively such that w t ∩ w @xmath9. this is to verify that the trained model is not merely learning to identify words that appear in the train set rather than words containing gender information in general. we reserve the set of stereotypes x solely for evaluating bias (inference) in the language models. taking e @xmath10, e f 2 , ...} and e @xmath11, e m2 , ...} to be the set of all embeddings for female and male words respectively, given @xmath12, we train a classifier c : @xmath13, m} to predict whether an embedding e belongs to e f or e m . evaluating bias. to determine the conformity of a language model l to gender stereotypes x, we introduce two bias criteria (or notions of bias): bias accuracy: we compute the accuracy of c to correctly predict the gender association of all stereotype words in x based on the embeddings produced by l. a high accuracy depicts the presence of pro-stereotype information in the embeddings produced by the language model l, even though the predictions could be of low confidence i.e. just enough gender information to correctly predict the gender association. mean bias confidence: we also consider the mean of the individual softmax scores for each prediction. in the case of binary gender, a softmax value of 0.5 means the embedding is perfectly neutral, hence, we define mean bias confidence as 1 n n i |bias(e i ) -0.5|. high mean confidence depicts an enormous amount of gender information contained in the embedding, although this information may not necessarily be pro-stereotype. randomization test. to test the efficacy of our method, we carry out a randomization test by iteratively splitting @xmath14. we use x ′ to define the random pair generated in the ith iteration. we repeat this randomized split over 100 iterations; x ′(i) ∈ {x ′(1) , ..., x ′( 100 ) }. by computing a p-value based on a one-sample t-test, we compute if the mean of accuracy scores from the random samples x ′ is significantly different from the score from{x f , x m }. ## investigating intrinsic bias in mitigated models with existing metrics we first use the two existing metrics defined in § 2.2 (seat and lpbs) to quantify bias in bertlarge and albert-large pretrained models. using the same metrics, we then explore how intrinsic bias changes in mitigated versions of these models using the mitigation techniques outlined in § 2.1: context-debias, sent-debias, and cda pretraining. this is important to evaluate the efficacy of these mitigation strategies, and to make relevant postulations. 4.1 rq1: do intrinsic bias measures respond differently to bias mitigation techniques? in figure 2 , we plot the intrinsic bias results foot_8 of seat and lpbs on bert-large. the first observation we point out, particularly for context-debias and sent-debias, is the contradicting nature of bias scores obtained from seat and lpbs. whereas seat shows a drastic reduction in intrinsic bias for context-debias, lpbs indicates worsening bias. the converse is true for sent-debias where lpbs shows a significant reduction in intrinsic bias but seat shows worse scores. this confirms our suspicion that different mitigation strategies respond differently to different metrics. the use of some metrics can be problematic for evaluating some mitigation techniques for intrinsic bias. we find that some metrics are not ideal to use with some mitigation techniques due to differences in how various mitigation techniques interact with a language model. consider a metric like lpbs which measures the association between a stereotype term and a gender p (@xmath15. this association is captured at the inner layers of the model, but since post-hoc approaches like sentdebias do not alter the internal representation of the model, lpbs in its original form becomes ineffective -figure 7 . we refer to this original form of lpbs as attribute-lpbs. sent-debias ends up debiasing the attribute terms rather than the stereotype terms, thus, constantly producing low probability difference scores. even so, we find lpbs being used in this form with sent-debias in works such as @xcite . using stereotype terms instead of attribute terms p (@xmath16, which we refer to as target-lpbs, is an option to solve this discrepancy, but given that many stereotype terms are usually out of vocabulary, these models will resort to wordpiece tokenization @xcite which will be more challenging to handle. nonetheless, in figure 3 , when the two versions of lpbs are compared, sent-debias continues to have the best intrinsic fairness scores in both cases foot_9 . ## probing further to uncover bias we use the methodology in section 3 to probe for bias in pretrained language models and their bias-mitigated versions to see if the intrinsic bias mitigation techniques actually mitigate biases or superficially conceal them from commonplace metrics like seat and lpbs. @xcite discovered that, for non-contextualized word embeddings, the bias mitigation techniques proposed by @xcite were hiding bias instead of resolving bias. we use our probe to see if this is equally the case for intrinsic bias mitigation in contextualized embeddings. the consequence of mitigation techniques superficially hiding bias is that downstream classifiers can learn to pick up residual traces of gender information. we validate our results by carrying out a randomization test, as described in section 3, to test for statistical significance of our results. we conduct the test under the null hypothesis h 0 that the prediction accuracy of our original gender-stereotype sets will not differ significantly from the random split. we show in in figure 4 , we show the results foot_10 of our probe for bert-large. we highlight some key observations. ## related work park et al. ( 2018 ) study the effects of three techniques to mitigate gender bias in abusive language detection; debiased word embeddings (based on the approach by @xcite ), gender swap (counterfactual augmentation of classification data), bias finetuning using non-contextualized word embeddings: word2vec @xcite and fasttext @xcite . they found that a combination of debiasing and gender swapping produced the best fairness results when implemented with a gated recurrent unit (gru). @xcite examine three debiasing techniques for mitigating gender bias in downstream text classification using bias-in-bios (de-arteaga et al., 2019b), with non-contextualized embeddings. specifically, they examine scrubbing (deletion of identity terms) and approaches that focus on non-contextualized embeddings namely: debiasing (based on neutralization and equalization by bolukbasi et al. ( 2016 )), and strong debiasing @xcite . experimenting with glove embeddings @xcite , they show that strong debiased produces the best fairness results whilst maintaining a good accuracy score (second only to the unmitigated model). they again show that standard debiasing @xcite can be counter-productive in terms of fairness; it can rather reduce fairness in downstream text classification. @xcite meticulously investigate the effect of intrinsic bias in downstream text classification in roberta. they also conclude that intrinsic/upstream bias does not significantly contribute to downstream bias. however, as we have previously discussed, using the default form of log probability bias score (attribute-lpbs) @xcite which they adopt to measure intrinsic bias may not be an effective technique to use with sentdebias. orgad et al. ( 2022 ), the closest and concurrent with our work, investigate the relationship between intrinsic and extrinsic bias and also develop a probe based on a classification-based technique minimum description length @xcite to detect bias in roberta. they reach similar conclusions indicating how intrinsic metrics, like ceat in their case, could conceal bias. ## conclusion in this paper, we develop a probe to investigate intrinsic bias in two language models, bert-large and albert-large on two text classification tasks. we use this probe to answer key research questions outlined for this research. we find that intrinsic bias metrics can be sensitive to certain intrinsic bias mitigation techniques. we also show that intrinsic bias mitigation techniques and metrics are capable of concealing bias instead of resolving it. we discover that intrinsic bias mitigation techniques we considered do not significantly improve fairness in text classification when used without other fairness interventions like data pre-processing. we recommend that intrinsic bias mitigation should ideally be combined with other fairness interventions.