How Gender Debiasing Affects Internal Model Representations, and Why It Matters (2022)

Abstract: Common studies of gender bias in NLP focus either on extrinsic bias measured by model performance on a downstream task or on intrinsic bias found in models’ internal representations. However, the relationship between extrinsic and intrinsic bias is relatively unknown. In this work, we illuminate this relationship by measuring both quantities together: we debias a model during downstream fine-tuning, which reduces extrinsic bias, and measure the effect on intrinsic bias, which is operationalized as bias extractability with information-theoretic probing. Through experiments on two tasks and multiple bias metrics, we show that our intrinsic bias metric is a better indicator of debiasing than (a contextual adaptation of) the standard WEAT metric, and can also expose cases of superficial debiasing. Our framework provides a comprehensive perspective on bias in NLP models, which can be applied to deploy NLP systems in a more informed manner. Our code and model checkpoints are publicly available.

## introduction efforts to identify and mitigate gender bias in natural language processing (nlp) systems typically target one of two notions of bias. extrinsic evaluation methods and debiasing techniques focus on the bias reflected in a downstream task @xcite , while intrinsic methods focus on a model's internal representations, such as word or sentence embedding geometry @xcite @xcite . despite an abundance of evidence pointing towards gender bias in pretrained language models (lms), the extent of harm caused by these biases is not clear when it is not reflected in a specific downstream task (barocas train another model on the same task using a debiased dataset, and finally (c) measure intrinsic bias in both models and compare. @xcite @xcite . for instance, while the word embedding proximity of "doctor" to "man" and "nurse" to "woman" is intuitively normatively wrong, it is not clear when such phenomena would lead to downstream predictions manifesting in social biases. recently, @xcite have shown that debiasing static embeddings intrinsically is not correlated with extrinsic gender bias measures, but the nature of the reverse relationship is unknown: how are extrinsic interventions reflected in intrinsic representations? furthermore, @xcite demonstrated that a number of intrinsic debiasing methods applied to static embeddings only partially remove the bias and that most of it is still hidden within the embed-ding. complementing their view, we examine extrinsic debiasing methods, as well as demonstrate the possible harm this could cause. contrary to their conclusion, we do not claim that these debiasing methods should not be trusted, as long as they are utilized with care. our goal is to gain a better understanding of the relationship between a model's internal representations and its extrinsic gender bias by examining the effects of various debiasing methods on the model's representations. specifically, we fine-tune models with and without gender debiasing strategies, evaluate their external bias using various bias metrics, and measure intrinsic bias in the representations. we operationalize intrinsic bias via two metrics: first, we use ceat @xcite , a contextual adaptation of the widely used intrinsic bias metric @xcite . second, we propose to use an information-theoretic probe to quantify the degree to which gender can be extracted from the internal model representations. then, we examine how these intrinsic metrics correlate with a variety of extrinsic bias metrics that we measure on the model's downstream performance. our approach is visualised in figure 1 . we perform extensive experiments on two downstream tasks (occupation prediction and coreference resolution); several debiasing strategies that involve alterations to the training dataset (such as removing names and gender indicators, or balancing the data by oversampling or downsampling); and a multitude of extrinsic bias metrics. our analysis reveals new insights into the way language models encode and use information on gender: the coreference task exhibits a high correlation. the occupation prediction task exhibits a lower correlation, but it increases after retraining (a case of superficial debiasing). gender extractability shows higher correlations with extrinsic metrics than ceat, increasing the confidence in this metric as a reliable measure for gender bias in nlp models. ## methodology in this study, we investigate the relationship between extrinsic bias metrics of a task and a model's internal representations, under various debiasing conditions, for two datasets in english. we perform extrinsic debiasing, evaluate various extrinsic and intrinsic bias metrics before and after debiasing, and examine correlations. dataset. let @xmath0, y , z} be a dataset consisting of input data x, labels y and protected attributes z. 2 this work focuses on gender as the protected attribute z. in all definitions, f and m indicate female and male gender, respectively, as the value of the protected attribute z. trained model. the model is optimized to solve the downstream task posed by the dataset. it can be formalized as f • g : @xmath1, where g(•) is the feature extractor, implemented by a language model, e.g., roberta @xcite , f (•) is the classification function, and y is the set of the possible labels for the task. ## experiments in each experiment, we fine-tune a model for a downstream task. for training, we use either the original dataset or a dataset debiased with one of the methods from section 2.2. figure 2 presents examples of debiasing methods for the two downstream tasks. we measure two intrinsic metrics by probing that model's inner representations for gender extractability (as measured by mdl) and by ceat, and test various extrinsic metrics. the relation between one intrinsic and one extrinsic metric becomes one data point, and we repeat over many random seeds (for both the model and the probe). further implementation details are in appendix a. ## results tables 1a and 1b present intrinsic and extrinsic metrics for roberta models on the occupation prediction and coreference resolution tasks, respectively. we present a representative subset of the measured metrics that demonstrate the observed phenomena; full results are found in appendix b. the deberta model results are consistent with the roberta model trends. ## related work there are few studies that examine both intrinsic and extrinsic metrics. previous work by @xcite showed that debiasing static embeddings intrinsically is not correlated with extrinsic bias, challenging the assumption that intrinsic metrics are predictive of bias. we examine the other direction, exploring how extrinsic debiasing affects intrinsic metrics. we also extend beyond their work to contextualized embeddings, a wider range of extrinsic metrics, and a new, more effective intrinsic metric based on information-theoretic probing. a contemporary work by cao et al. (2022) measured the correlations between intrinsic and extrinsic metrics in contextualized settings across different language models. in contrast, our work examines the correlations across different versions of the same language model by fine-tuning it using various debiasing techniques. studies that inspect extrinsic metrics include either a challenge dataset curated to expose differences in model behavior by gender, or a test dataset labelled by gender. among these datasets are winobias @xcite , winogender @xcite and gap @xcite for coreference resolution, winomt @xcite for machine translation, eec @xcite for sentiment analysis, bold @xcite for language generation, gendered nli @xcite ) for natural language inference and bias in bios (de-arteaga et al., 2019) for occupation prediction. studies that measure gender bias intrinsically in static word or sentence embeddings measure characteristics of the geometry, such as the prox-imity between female-and male-related words to stereotypical words, or how embeddings cluster or relate to a gender subspace @xcite @xcite . however, metrics and debiasing methods for static embeddings do not apply directly to contextualized ones. several studies use sentence templates to adapt to contextual embeddings @xcite @xcite . this templated approach is difficult to scale, and lacks the range of representations that a contextual embedding offers. other work extracts embedding representations of words from natural corpora @xcite @xcite . these studies often adapt the weat method @xcite , which measures embedding geometry. none measure the effect of the presumably found "bias" on a downstream task. there is a growing conversation in the field @xcite @xcite about the importance of articulating the harms of measured bias. in general, extrinsic metrics have clear, interpretable impacts for which harm can be defined. intrinsic metrics have an unclear effect. without evidence from a concrete downstream task, a found intrinsic bias is only theoretically harmful. our work is a step towards understanding whether intrinsic metrics provide valuable insights about bias in a model. ## discussion and conclusions this study examined whether bias in internal representations is related to extrinsic bias. we designed a new framework in which we debias a model on a downstream task, and measure its intrinsic bias. we found that gender extractability from internal representations, measured by compression rate via mdl probing, reflects bias in a model. compression was much more reliable than an alternative intrinsic metric for contextualised representations, ceat. compression correlated well-to varying degrees-with many extrinsic metrics. we thus encourage nlp practitioners to use compression as an intrinsic indicator for gender bias in nlp models. when comparing two alternative models, a lower compression rate provides confidence in a model's superiority in terms of gender bias. the relative success of compression over ceat may be because the compression rate was calculated on the same dataset as the extrinsic metrics, whereas ceat was measured on a different dataset not necessarily aligned with a specific downstream task. the use of a non-task-aligned dataset is a common strategy among other intrinsic metrics @xcite @xcite ). another possible explanation is that compression rate measures a more focused concept, namely the gender information within the internal representations. ceat measures proximity among embeddings of general terms that may include other social contexts that do not directly relate to gender (e.g. a female term like 'lady' or 'sarah' contains information about not just gender but class, culture, formality, etc, and it can be hard to isolate just one of these from the rest). our results show that when a debiasing method reduces extrinsic metrics but not compression, it indicates that the language model remains biased. when such superficial debiasing occurs, the debiased language model may be reapplied to another task, as in @xcite , resulting in unexpected biases and nullifying the supposed debiasing. our findings suggest that practitioners of nlp should take special care when adopting previously debiased models and inspect them carefully, perhaps using our framework. our results differ from those of @xcite , who found that the debiasing increases bias extractability as measured by compression rate. however, they studied different, non-social biases, that arise from spurious or unintended correlations in training datasets (often called dataset biases). in our case, some debiasing strategies increase intrinsic bias while others decrease it. future work could investigate why debiasing affects extractability differently for these two types of biases. our work also highlighted the importance of the classification layer. using a debiased objective, such as a balanced dataset, the classification layer can provide significant debiasing. this holds even if the internal representations are biased and the classifier is a single linear layer, as shown in the occupation prediction task. bias stems in part from internal lm bias and in part from classification bias. practitioners should focus their efforts on both parts when attempting to debias a model. we used a broader set of extrinsic metrics than is typically used, and found that the bias metrics behaved differently: some decreased more than others after debiasing, and they correlated differently with compression rate. debiasing efforts may not be fully understood by testing only a few extrinsic metrics. however, compression as an intrinsic bias metric can indicate meaningful debiasing of internal model representations even when not all metrics are easily measurable, since it correlates well with many extrinsic metrics. a major limitation of this study is the use of gender as a binary variable, which is trans-exclusive. cao and daumé iii (2020) made the first steps towards inclusive gender bias evaluation in nlp, revealing that coreference systems fail on genderinclusive text. further work is required to adjust our framework to non-binary genders, potentially revealing insights about the poor performance of nlp systems in that area.