Original: How Gender Debiasing Affects Internal Model Representations, and Why It Matters
Revised: How Gender Debiasing Affects Internal Model Representations, and Why It Matters

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: Common studies of gender bias in NLP focuseither on extrinsic bias measured by model performance on a downstream task or on intrinsicbias found in models’ internal representations.
Revised: Common studies of gender bias in NLP focus either on extrinsic bias measured by model performance on a downstream task or on intrinsic bias found in models’ internal representations.

Original: However, the relationship between extrinsicand intrinsic bias is relatively unknown.
Revised: However, the relationship between extrinsic and intrinsic bias is relatively unknown.

Original: Inthis work, we illuminate this relationship bymeasuring both quantities together: we debiasa model during downstream fine-tuning, whichreduces extrinsic bias, and measure the effecton intrinsic bias, which is operationalized asbias extractability with information-theoreticprobing.
Revised: In this work, we illuminate this relationship by measuring both quantities together: we debias a model during downstream fine-tuning, which reduces extrinsic bias, and measure the effect on intrinsic bias, which is operationalized as bias extractability with information-theoretic probing.

Original: Through experiments on two tasksand multiple bias metrics, we show that ourintrinsic bias metric is a better indicator of debiasing than (a contextual adaptation of) thestandard WEAT metric, and can also exposecases of superficial debiasing.
Revised: Through experiments on two tasks and multiple bias metrics, we show that our intrinsic bias metric is a better indicator of debiasing than (a contextual adaptation of) the standard WEAT metric, and can also expose cases of superficial debiasing.

Original: Our frameworkprovides a comprehensive perspective on biasin NLP models, which can be applied to deploy
Revised: Our framework provides a comprehensive perspective on bias in NLP models, which can be applied to deploy

Original: NLP systems in a more informed manner.
Revised: NLP systems in a more informed manner.

Original: Ourcode will be made publicly available.

Original:  Introduction
Revised:  Introduction

Original: Efforts to identify and mitigate gender bias in Natural Language Processing (NLP) systems typicallytarget one of two notions of bias.
Revised: Efforts to identify and mitigate gender bias in Natural Language Processing (NLP) systems typically target one of two notions of bias.

Original: Extrinsic evaluation methods and debiasing techniques focus on thebias reflected in a downstream task (De-Arteagaet al., 2019; Zhao et al., 2018), while intrinsicmethods focus on a model’s internal representations, such as word or sentence embedding geometry (Caliskan et al., 2017; Bolukbasi et al., 2016;
Revised: Extrinsic evaluation methods and debiasing techniques focus on the bias reflected in a downstream task (De-Arteaga et al., 2019; Zhao et al., 2018), while intrinsic methods focus on a model’s internal representations, such as word or sentence embedding geometry (Caliskan et al., 2017; Bolukbasi et al., 2016;

Original: Guo and Caliskan, 2021).
Revised: Guo and Caliskan, 2021).

Original: Despite an abundanceof evidence pointing towards gender bias in pretrained language models (LMs), the extent of harmcaused by these biases is not clear when it is notreflected in a specific downstream task (Barocaset al., 2017; Kate Crawford, 2017; Blodgett et al.
Revised: Despite an abundance of evidence pointing towards gender bias in pretrained language models (LMs), the extent of harm caused by these biases is not clear when it is not reflected in a specific downstream task (Barocas RoBERTa

Original: ,2020; Bommasani et al., 2021).
Revised: [Table]

Original: For instance, whilethe word embedding proximity of “doctor” to “man”and “nurse” to “woman” is intuitively normatively
Revised: RoBERTa

Original: [Figure]
Revised: [Table]

Revised: RoBERTa

Original: [Table] 
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: et al., 2017; Kate Crawford, 2017; Blodgett et al., 2020; Bommasani et al., 2021).

Original:  wrong, it is not clear when such phenomena would lead to downstream predictions manifesting in social biases.
Revised: For instance, while the word embedding proximity of “doctor” to “man” and “nurse” to “woman” is intuitively normatively wrong, it is not clear when such phenomena would lead to downstream predictions manifesting in social biases.

Original: Recently, Goldfarb-Tarrant et al.
Revised: Recently, Goldfarb-Tarrant et al.

Original: (2021) have shown that debiasing static embeddings intrinsically is not correlated with extrinsic gender bias measures, but the nature of the reverse relationship is unknown: how are extrinsic interventions reflected in intrinsic representations?
Revised: (2021) have shown that debiasing static embeddings intrinsically is not correlated with extrinsic gender bias measures, but the nature of the reverse relationship is unknown: how are extrinsic interventions reflected in intrinsic representations?

Original: Furthermore,
Revised: Furthermore,

Original: Gonen and Goldberg (2019a) demonstrated that a number of intrinsic debiasing methods applied to static embeddings only partially remove the bias and that most of it is still hidden within the embedding.
Revised: Gonen and Goldberg (2019a) demonstrated that a number of intrinsic debiasing methods applied to static embeddings only partially remove the bias and that most of it is still hidden within the embed- ding.

Original: Complementing their view, we examine extrinsic debiasing methods, as well as demonstrate the possible harm this could cause.
Revised: Complementing their view, we examine extrinsic debiasing methods, as well as demonstrate the possible harm this could cause.

Original: Contrary to their conclusion, we do not claim that these debias-  ing methods should not be trusted, as long as theyare utilized with care .
Revised: Contrary to their conclusion, we do not claim that these debiasing methods should not be trusted, as long as they are utilized with care .

Original: Our goal is to gain a better understanding of therelationship between a model’s internal representations and its extrinsic gender bias by examiningthe effects of various debiasing methods on themodel’s representations.
Revised: Our goal is to gain a better understanding of the relationship between a model’s internal representations and its extrinsic gender bias by examining the effects of various debiasing methods on the model’s representations.

Original: Specifically, we fine-tunemodels with and without gender debiasing strategies, evaluate their external bias using various biasmetrics, and measure intrinsic bias in the representations.
Revised: Specifically, we fine-tune models with and without gender debiasing strategies, evaluate their external bias using various bias metrics, and measure intrinsic bias in the representations.

Original: We operationalize intrinsic bias via twometrics: First, we use CEAT (Guo and Caliskan,2021), a contextual adaptation of the widely usedintrinsic bias metric WEAT (Caliskan et al., 2017).
Revised: We operationalize intrinsic bias via two metrics: First, we use CEAT (Guo and Caliskan, 2021), a contextual adaptation of the widely used intrinsic bias metric WEAT (Caliskan et al., 2017).

Original: Second, we propose to use an information-theoreticprobe to quantify the degree to which gender can beextracted from the internal model representations.
Revised: Second, we propose to use an information-theoretic probe to quantify the degree to which gender can be extracted from the internal model representations.

Original: Then, we examine how these intrinsic metrics correlate with a variety of extrinsic bias metrics that wemeasure on the model’s downstream performance.
Revised: Then, we examine how these intrinsic metrics correlate with a variety of extrinsic bias metrics that we measure on the model’s downstream performance.

Original: Our approach is visualised in Figure 1.
Revised: Our approach is visualised in Figure 1.

Original: We perform extensive experiments on two downstream tasks (occupation prediction and coreference resolution); several debiasing strategies thatinvolve alterations to the training dataset (such asremoving names and gender indicators, or balancing the data by oversampling or downsampling);and a multitude of extrinsic bias metrics.
Revised: We perform extensive experiments on two downstream tasks (occupation prediction and coreference resolution); several debiasing strategies that involve alterations to the training dataset (such as removing names and gender indicators, or balancing the data by oversampling or downsampling); and a multitude of extrinsic bias metrics.

Original: Our analysis reveals new insights into the way languagemodels encode and use information on gender:
Revised: Our analysis reveals new insights into the way language models encode and use information on gender:

Original: • The effect of debiasing on internal represen- tations is reflected in gender extractability, while not always in CEAT.
Revised: • The effect of debiasing on internal representations is reflected in gender extractability, while not always in CEAT.

Original: • In cases of high gender extractability but low
Revised: Thus, gender extractability is a more reliable indicator of gender bias in NLP models.

Original:  extrinsic bias metrics, the debiasing is super- ficial, and the internal representations are a good indicator for this: The bias is still present in internal representations and can be restored 
Revised: • In cases of high gender extractability but low extrinsic bias metrics, the debiasing is superficial, and the internal representations are a good indicator for this: The bias is still present in internal representations and can be restored by retraining the classification layer.

Original: by retraining the classification layer.
Revised: Therefore, our proposed measuring method can help in detecting such cases before deploying the model.

Original: • The two tasks show different patterns of cor- relation between intrinsic and extrinsic bias.
Revised: • The two tasks show different patterns of correlation between intrinsic and extrinsic bias.

Original: The coreference task exhibits a high correla- tion.
Revised: The coreference task exhibits a high correlation.

Original: The occupation prediction task exhibits a lower correlation, but it increases after retrain- ing (a case of superficial debiasing).
Revised: The occupation prediction task exhibits a lower correlation, but it increases after retraining (a case of superficial debiasing).

Original: Gender extractability shows higher correlations to ex- trinsic metrics than CEAT.
Revised: Gender extractability shows higher correlations with extrinsic metrics than CEAT, increasing the confidence in this metric as a reliable measure for gender bias in NLP models

Original:  Methodology
Revised:  Methodology

Original: In this study, we investigate the relationship between extrinsic bias metrics of a task and a model’s internal representations, under various debiasing conditions, for two datasets in English.
Revised: In this study, we investigate the relationship between extrinsic bias metrics of a task and a model’s internal representations, under various debiasing conditions, for two datasets in English.

Original: We perform extrinsic debiasing, evaluate various extrinsic and intrinsic bias metrics before and after debiasing, and examine correlations.
Revised: We perform extrinsic debiasing, evaluate various extrinsic and intrinsic bias metrics before and after debiasing, and examine correlations.

Original: Dataset.
Revised: Dataset.

Original: Let D =
Revised: Let D =

Original: {X , Y , Z} be a dataset consisting of input data X , labels Y and protected attributes Z . 

Revised: { X , Y , Z } be a dataset consisting of input data X , labels Y and protected attributes Z .

Original: This work focuses on gender as the protected attribute Z .
Revised: This work focuses on gender as the protected attribute z .

Original: In all definitions, F and M indicate female and male gender, respectively, as the value of the protected attribute Z .
Revised: In all definitions, F and M indicate female and male gender, respectively, as the value of the protected attribute z .

Original: Trained Model.
Revised: Trained Model.

Original: The model is optimized to solve the downstream task posed by the dataset.
Revised: The model is optimized to solve the downstream task posed by the dataset.

Original: It can be formalized as f ( g ( x )) (cid:55)→ R |Y| , where g ( · ) is the feature extractor, implemented by a language model, e.g., RoBERTa (Liu et al., 2019),and f ( · ) is the classification function.
Revised: It can be formalized as f ◦ g : X → R |Y| , where g ( · ) is the feature extractor, implemented by a language model, e.g., RoBERTa (Liu et al., 2019), f ( · ) is the classification function, and Y is the set of the possible labels for the task.

Original: 2. Bias Metrics
Revised: 2. Bias Metrics

Original: Each bias evaluation method described in the literature can be categorized as extrinsic or intrinsic.
Revised: Each bias evaluation method described in the literature can be categorized as extrinsic or intrinsic.

Original: In all definitions, R indicates the model’s predictions.
Revised: In all definitions, r indicates the model’s output probabilities.

Original: 2.1. Extrinsic Metrics
Revised: 2.1. Extrinsic Metrics

Original: Extrinsic methods involve measuring the bias of a model solving a downstream problem.
Revised: Extrinsic methods involve measuring the bias of a model solving a downstream problem.

Original: The extrinsic metric is a mapping:
Revised: The extrinsic metric is a function:

Original: [Equation]
Revised: [Equation]

Original: The output represents the quantity of bias measured; the further from 0 the number is, the larger the bias is.
Revised: The output represents the quantity of bias measured; the further from 0 the number is, the larger the bias is.

Original: Our analysis comprises a wide range of extrinsic metrics, including some that have been measured in the past on the analyzed tasks (Zhao et al., 2018; De-Arteaga et al., 2019; Ravfogel et al., 2020; Goldfarb-Tarrant et al., 2021) and some that have never been measured before, and shows our results apply to many of them.
Revised: Our analysis comprises a wide range of extrinsic metrics, including some that have been measured in the past on the analyzed tasks (Zhao et al., 2018; De-Arteaga et al., 2019; Ravfogel et al., 2020; Goldfarb-Tarrant et al., 2021) and some that have never been measured before, and shows our results apply to many of them.

Original: For illustration, we will consider occupation prediction, a common task in research on gender bias (De-Arteaga et al., 2019;
Revised: For illustration, we will consider occupation prediction, a common task in research on gender bias (De-Arteaga et al., 2019;

Original: Ravfogel et al., 2020; Romanov et al., 2019).
Revised: Ravfogel et al., 2020; Romanov et al., 2019).

Original: The input X is a biography and the prediction Y is the profession of the person described in it.
Revised: The input x is a biography and the prediction y is the profession of the person described in it.

Original: The protected attribute Z is the gender of that person.
Revised: The protected attribute z is the gender of that person.

Original: Performance gap.
Revised: Performance gap.

Original: This is the difference in per- formance metric for two different groups, for in- stance two groups of binary genders, or a group of pro-stereotypical and a group of anti-stereotypical 
Revised: This is the difference in performance metric for two different groups, for instance two groups of binary genders, or a group of pro-stereotypical and a group of anti-stereotypical examples.

Original: examples.

Original: We measure the following metrics:
Revised: We measure the following metrics:

Original: True
Revised: True

Original: Positive Rate (TPR), False Positive Rate (FPR), and
Revised: Positive Rate (TPR), False Positive Rate (FPR), and

Original: Precision.
Revised: Precision.

Original: In occupation prediction, for instance, the TPR gap for each profession y expresses the difference in the percentage of women and men whose profession is y and are correctly classified 
Revised: In occupation prediction, for instance, the TPR gap for each profession y expresses the difference in the percentage of women and men whose profession is y and are correctly classified as such.

Original: as such. We also measure F1 of three standard clustering metrics for coreference resolution.
Revised: We also measure F1 of three standard clustering metrics for coreference resolution.

Original: We compute two types of performance gap met-
Revised: Each such performance gap captures a different facet of gender bias, and one might be more interested in one of the metrics depending on the application.

Original: rics: (1) the sum of absolute gap values over all classes; (2) the Pearson correlation between the performance gap for a class and the percentage of women in that class.
Revised: We compute two types of performance gap metrics: (1) the sum of absolute gap values over all classes; (2) the Pearson correlation between the performance gap for a class and the percentage of women in that class.

Original: For instance, if y is a pro- fession, we measure the correlation between per- formance gaps and percentages of women in each profession.
Revised: For instance, if y is a profession, we measure the correlation between performance gaps and percentages of women in each profession.

Original: The two metrics are closely related but answer slightly different questions: the sum quanti- fies how a model behaves differently on different genders, and the correlation shows the relation of 
Revised: The two metrics are closely related but answer slightly different questions: the sum quantifies how a model behaves differently on different genders, and the correlation shows the relation of model behaviour to social biases (in the world or the data) without regard to actual gap size.

Original: model behaviour to social biases (in the world or

Original: the data) without regard to actual gap size.

Original: Statistical metrics.
Revised: Statistical metrics.

Original: For breadth of analysis, we examine three additional statistical metrics(Baro-
Revised: For breadth of analysis, we examine three additional statistical metrics

Original: cas et al., 2019), which correspond to different no- tions of bias.
Revised: (Barocas et al., 2019), which correspond to different notions of bias.

Original: All three are measured as differences ( d ) between two probability distributions, and we then obtain a single bias quantity per metric by summing all computed distances.
Revised: All three are measured as differences ( d ) between two probability distributions, and we then obtain a single bias quantity per metric by summing all computed distances.

Original: • Independence : d (cid:0) P ( R | Z = z ) , P ( R ) (cid:1) ∀ z ∈
Revised: • Independence : [Equation]

Original: { F, M } .
Revised: { F, M } .

Original: For instance, we measure the difference between the distribution of model’s predictions on women and the distribution of all predictions.
Revised: For instance, we measure the difference between the distribution of model’s predictions on women and the distribution of all predictions.

Revised: Independence is stronger as the prediction r is less correlated with the protected attribute z .

Revised: This is measured with no relation to the gold labels.

Original: • Separation : d (cid:0)P ( R | Y
Revised: • Separation : d (cid:0)

Original: = y, Z = z ) , P ( R | Y =
Revised: [Equation]

Original: y ) (cid:1) ∀ y ∈ Y , z ∈ { F, M } .
Revised: ∀ y ∈ Y , z ∈ { F, M } .

Original: For instance, we mea- sure the difference between the distribution of a model’s predictions on women who are teachers and the distribution of predictions on all teachers.
Revised: For instance, we measure the difference between the distribution of a model’s predictions on women who are teachers and the distribution of predictions on all teachers.

Original:  • Sufficiency :d (cid:0) P ( Y | R = r, Z = z ) , P ( Y | R =
Revised: This encapsulates the TPR and FPR equality discussed previously, and can be seen as a more general metric. • Sufficiency :

Original: r ) (cid:1) .
Revised: [Equation]

Original: For instance, we measure the difference be- tween the distribution of gold labels on women 
Revised: For instance, we measure the difference between the distribution of gold labels on women classi- fied

Original: classified as teachers by the model and the distri-
Revised: as teachers by the model and the distribu tion

Original: bution of gold labels on all individuals classified 
Revised: of gold labels on all individuals classified as teachers by the model

Revised: Sufficiency

Original: as teachers by the model.
Revised: relates to the concept of calibration in classification.

Revised: A

Revised: difference

Revised: in

Revised: the classifier’s scores

Revised: for men and for

Revised: women

Revised: indicates

Revised: that

Revised: it might

Revised: be

Revised: penalizing or over-promoting

Revised: one of the genders

Original: 2.1. Intrinsic Metrics
Revised: 2.1. Intrinsic Metrics

Original: Intrinsic methods are applied to the representation obtained from the feature extractor.
Revised: Intrinsic methods are applied to the representation obtained from the feature extractor.

Original: These methods are independent of any downstream task.
Revised: These methods are independent of any downstream task.

Original: The intrinsic metric is a mapping:
Revised: The intrinsic metric is a function:

Original: I ( g ( X ) , Z ) (cid:55)→ R
Revised: [Equation]

Original: Compression.
Revised: Compression.

Original: Our main intrinsic metric is the compression of gender information evaluated by a minimum description length (MDL) probing classifier (Voita and Titov, 2020), trained to predict gender from the model’s representations.
Revised: Our main intrinsic metric is the compression of gender information evaluated by a minimum description length (MDL) probing classifier (Voita and Titov, 2020), trained to predict gender from the model’s representations.

Original: Probing classifiers are widely used for predicting various properties of interest from frozen model representations (Belinkov and Glass, 2019).
Revised: Probing classifiers are widely used for predicting various properties of interest from frozen model representations (Belinkov and Glass, 2019).

Original: MDL probes were proposed because a probe’s accuracy may be misleading due to memorization and other issues (Hewitt and Liang, 2019; Belinkov, 2021).
Revised: MDL probes were proposed because a probe’s accuracy may be misleading due to memorization and other issues (Hewitt and Liang, 2019; Belinkov, 2021).

Original: We use the MDL online code, where the probe is trained in timesteps, on increasing subsets of the training set, then evaluated against the rest of it.
Revised: We use the MDL online code, where the probe is trained in timesteps, on increasing subsets of the training set, then evaluated against the rest of it.

Original: Higher compression indicates greater gender extractability.
Revised: Higher compression indicates greater gender extractability.

Original: CEAT.
Revised: CEAT.

Original: We also measure CEAT (Guo and
Revised: We also measure CEAT (Guo and

Original: Caliskan, 2021), which is a contextualized version of WEAT (Caliskan et al., 2017), a widely used bias metric for static word embeddings.
Revised: Caliskan, 2021), which is a contextualized version of WEAT (Caliskan et al., 2017), a widely used bias metric for static word embeddings.

Original: WEAT defines sets X and Y of target words, and sets A and B of attribute words.
Revised: WEAT defines sets X and Y of target words, and sets A and B of attribute words.

Original: For instance, A , B contain males and females names, while X , Y contain career and family related words, respectively.
Revised: For instance, A and B contain males and females names, while X and Y contain career and family related words, respectively.

Original: The bias is operationalized as the geometric proximity between the target and attribute word embeddings, and is quantified in CEAT by the Combined Effect Size (CES) and a p-value for the null hypothesis of having no biased associations.
Revised: The bias is operationalized as the geometric proximity between the target and attribute word embeddings, and is quantified in CEAT by the Combined Effect Size (CES) and a p-value for the null hypothesis of having no biased associations.

Original: For more information on CEAT refer to Appendix A.4.3.
Revised: For more information on CEAT refer to Appendix A.4.3.

Original: 2. Debiasing Techniques
Revised: 2. Debiasing Techniques

Original: We debias models by modifying the downstream task’s training data before fine-tuning.
Revised: We debias models by modifying the downstream task’s training data before fine-tuning.

Original: Scrubbing (De-Arteaga et al., 2019) removes first names and gender-specific terms (“he”, “she”, “husband”, “wife”, “Mr”, “Mrs”, etc.).
Revised: Scrubbing (De-Arteaga et al., 2019) removes first names and gender-specific terms (“he”, “she”, “husband”, “wife”, “Mr”, “Mrs”, etc.).

Original: Balancing subsamples or oversamples examples such that each gender is equally represented in the resulting dataset w.r.t each label.
Revised: Balancing subsamples or oversamples examples such that each gender is equally represented in the resulting dataset w.r.t each label.

Original: Anonymization (Zhao et al., 2018) removes named entities.
Revised: Anonymization (Zhao et al., 2018) removes named entities.

Original: Counterfactual Augmentation (Zhao et al., 2018) involves replacing male entities in an example with female entities, and adding the modified example to the training set.
Revised: Counterfactual Augmentation (Zhao et al., 2018) involves replacing male entities in an example with female entities, and adding the modified example to the training set.

Original: As some of these are dataset/task-specific, we givemore details in the following section.
Revised: As some of these are dataset/task-specific, we give more details in the following section.

Original:  Experiments
Revised:  Experiments

Original: In each experiment, we fine-tune a model for adownstream task.
Revised: In each experiment, we fine-tune a model for a downstream task.

Original: For training, we use either theoriginal dataset or a dataset debiased with one ofthe methods from Section 2.2.
Revised: For training, we use either the original dataset or a dataset debiased with one of the methods from Section 2.2.

Original: Figure 2 presentsexamples of debiasing methods for the two downstream tasks.
Revised: Figure 2 presents examples of debiasing methods for the two downstream tasks.

Original: We measure two intrinsic metrics byprobing that model’s inner representations for gender extractability (as measured by MDL) and by
Revised: We measure two intrinsic metrics by probing that model’s inner representations for gender extractability (as measured by MDL) and by

Original: CEAT, and test various extrinsic metrics.
Revised: CEAT, and test various extrinsic metrics.

Original: The relation between one intrinsic and one extrinsic metricbecomes one data point, and we repeat over manyrandom seeds (for both the model and the probe).
Revised: The relation between one intrinsic and one extrinsic metric becomes one data point, and we repeat over many random seeds (for both the model and the probe).

Original: Further implementation details are in appendix A.
Revised: Further implementation details are in appendix A.

Original: 3. Occupation Prediction
Revised: 3. Occupation Prediction

Original: The task of occupation prediction is to predict aperson’s occupations (from a closed set), based ontheir biography.
Revised: The task of occupation prediction is to predict a person’s occupations (from a closed set), based on their biography.

Original: We use the Bias in Bios dataset(De-Arteaga et al., 2019).
Revised: We use the Bias in Bios dataset (De-Arteaga et al., 2019).

Original: Regardless of the training method, the test set is subsampled such thateach profession has equal gender representation. 
Revised: Regardless of the training method, the test set is subsampled such that each profession has equal gender representation. [Table]

Revised: [Figure]

Original: Model.
Revised: Model.

Original: Our model is a RoBERTa model (Liuet al., 2019) topped with a linear classifier, whichreceives the [CLS] token embedding as input andgenerates a probability distribution over the professions.
Revised: Our model is a RoBERTa model (Liu et al., 2019) topped with a linear classifier, which receives the [CLS] token embedding as input and generates a probability distribution over the professions.

Original: In addition, we train a baseline classifierlayer on top of a frozen, non-finetuned RoBERTa.
Revised: In addition, we train a baseline classifier layer on top of a frozen, non-finetuned RoBERTa.

Revised: We also experimented with a DeBERTa model (He et al., 2020), results are in the appendix (B).

Original: Debiasing Techniques.
Revised: Debiasing Techniques.

Original: Following De-Arteagaet al.
Revised: Following De-Arteaga et al.

Original: (2019) we experiment with scrubbing thetraining dataset.
Revised: we experiment with scrubbing the training dataset.

Original: Figure 2 shows an example biography snippet and its scrubbed version.
Revised: Figure 2 shows an example biography snippet and its scrubbed version.

Original: We alsoconduct balancing (per profession, subsamplingand oversampling to ensure an equal number ofmales and females per profession), which has notpreviously been used on this dataset and task.
Revised: We also conduct balancing (per profession, subsampling and oversampling to ensure an equal number of males and females per profession), which has not previously been used on this dataset and task.

Original: Metrics.
Revised: Metrics.

Original: We measure all bias metrics from Section 2.1 except for F1.
Revised: We measure all bias metrics from Section 2.1 except for F1.

Original: Probing.
Revised: Probing.

Original: The probing dataset for this task is thetest set, and the gender label of a single biographyis the gender of the person described in it.
Revised: The probing dataset for this task is the test set, and the gender label of a single biography is the gender of the person described in it.

Original: We probethe [CLS] token representation of the biography.
Revised: We probe the [CLS] token representation of the biography.

Original: Inaddition to the models described above, we measure baseline extractability of gender informationfrom a randomly initialized RoBERTa model.
Revised: In addition to the models described above, we measure baseline extractability of gender information from a randomly initialized RoBERTa model.

Original: 3. Coreference Resolution
Revised: 3. Coreference Resolution

Original: The task of coreference resolution is to find all textual expressions referring to the same real-world Britney currently [Table] [Table]
Revised: The task of coreference resolution is to find all textual expressions referring to the same real-world entities.

Original: has

Original: [Table]

Original: [Table]

Original: Scrubbing

Original: [Table]

Original: Counterfactual augmentation

Original: Occupation Classification

Original: Coreference Resolution

Original: [Figure] entities. We train on Ontonotes 5.0 (Weischedel et al., 2013) and test on the Winobias challenge dataset (Zhao et al., 2018).
Revised: We train on Ontonotes 5.0 (Weischedel et al., 2013) and test on the Winobias challenge dataset (Zhao et al., 2018).

Original: Winobias consists of sentence pairs, pro- and anti-stereotypical variants, with individuals referred to by their profession.
Revised: Winobias consists of sentence pairs, pro- and anti-stereotypical variants, with individuals referred to by their profession.

Original: For example, “The physician hired the secretary because he/
Revised: For example, “The physician hired the secretary because he/

Original: she was busy.”
Revised: she was busy.”

Revised: is pro/anti-stereotypical,

Revised: Extrinsic

Revised: [Table]

Original: is pro/anti-stereotypical, based on US labor statistics.
Revised: [Table] [Table] [Table] based on US labor statistics.

Original: 3 A coreference system is measured by the performance gap between the pro- and anti-stereotypical subsets.
Revised: 4 A coreference system is measured by the performance gap between the pro- and anti-stereotypical subsets.

Original: Model.
Revised: Model.

Original: We use the model presented in Lee et al.
Revised: We use the model presented in Lee et al.

Original: (2018a) with RoBERTa as a feature extractor.
Revised: (2018a) with RoBERTa as a feature extractor.

Original: Debiasing Techniques.
Revised: Debiasing Techniques.

Original: Following Zhao et al.
Revised: Following Zhao et al.

Original: (2018), we apply anonymization (denoted as Anon) and counterfactual augmentation (CA) on the training set.
Revised: (2018), we apply anonymization (denoted as Anon) and counterfactual augmentation (CA) on the training set.

Original: These techniques were used jointly in previous work; we examine each individually as well.
Revised: These techniques were used jointly in previous work; we examine each individually as well.

Original: Metrics.
Revised: Metrics.

Original: Following Zhao et al.
Revised: Following Zhao et al.

Original: (2018), we measure the F1 difference between anti- and prostereotypical examples.
Revised: (2018), we measure the F1 difference between anti- and prostereotypical examples.

Original: 4 We also interpret the task as a classification problem, and measure all metrics from Section 2.1.
Revised: 5 We also interpret the task as a classification problem, and measure all metrics from Section 2.1.

Original: For more details refer to
Revised: For more details refer to

Original: Appendix A.4.2.
Revised: Appendix A.4.2.

Original: Probing.
Revised: Probing

Original: We probe the representation of a profession word as extracted from Winobias sentences, after masking out the pronouns.
Revised: We probe the representation of a profession word as extracted from Winobias sentences, after masking out the pronouns.

Original: We define a profession’s gender as the stereotypical gender for this profession.
Revised: We define a profession’s gender as the stereotypical gender for this profession.

Original: To prevent memorization by the probe— given the small number of professions—the dataset is sorted so that professions are gradually added to the training set, so a success on the validation set is on previously unseen professions.[Table] [Table] [Table]
Revised: To prevent memorization by the probe— given the small number of professions—the dataset is sorted so that professions are gradually added to the training set, so a success on the validation set is on previously unseen professions.

Original: [Table]

Original: [Table]

Original: CA

Original: [Table]

Original: Anon + CA

Original: [Table]

Original:  Results
Revised:  Results

Original: Tables 1a and 1b present intrinsic and extrinsicmetrics on the occupation prediction and coreference resolution tasks, respectively.
Revised: Tables 1a and 1b present intrinsic and extrinsic metrics on the occupation prediction and coreference resolution tasks, respectively.

Original: We present arepresentative subset of the measured metrics thatdemonstrate the observed phenomena; full resultsare found in Appendix B.
Revised: We present a representative subset of the measured metrics that demonstrate the observed phenomena; full results are found in Appendix B.

Original: 4. Compression Reflects Debiasing Effects
Revised: 4. Compression Reflects Debiasing Effects

Original: As shown in the tables, compression captures differences in models that were debiased differently.
Revised: As shown in the tables, compression captures differences in models that were debiased differently.

Original: CEAT, however, cannot differentiate between occupation prediction models.
Revised: CEAT, however, cannot differentiate between occupation prediction models.

Original: For example, in occupation prediction (Table 1a) the compression ratevaries significantly between a non-debiased and adebiased model via scrubbing and oversampling,while CEAT detects no difference between the models.
Revised: For example, in occupation prediction (Table 1a) the compression rate varies significantly between a non-debiased and a debiased model via scrubbing and oversampling, while CEAT detects no difference between the models.

Original: In coreference resolution (Table 1b), both compression and CEAT are able to identify differencesbetween the non-debiased model and the others,such as CA, which has both a lower compressionand CEAT effect.
Revised: In coreference resolution (Table 1b), both compression and CEAT are able to identify differences between the non-debiased model and the others, such as CA, which has both a lower compression and CEAT effect.

Original: But the CEAT effect sizes aresmall (below 0.5), which implies no bias in contrastto the extrinsic metrics.
Revised: But the CEAT effect sizes are small (below 0.5), which implies no bias in contrast to the extrinsic metrics.

Original: 4. High Gender Extractability Implies
Revised: High Gender Extractability Implies

Original: Superficial Debiasing
Revised: Superficial Debiasing

Original:  and intrinsic effects of debiasing
Revised: Extrinsic and intrinsic effects of debiasing.

Original: In occupation classification (Table 1a), somewhat surprisingly, subsampling the training data has the strongest effect on extrinsic metrics, but not on compression rate.
Revised: In occupation classification (Table 1a), somewhat surprisingly, subsampling the training data has the strongest effect on extrinsic metrics, but not on compression rate.

Original: Scrubbing reduces both intrinsic and extrinsic metrics, although its effect on extrinsic metrics is limited compared to subsampling.
Revised: Scrubbing reduces both intrinsic and extrinsic metrics, although its effect on extrinsic metrics is limited compared to subsampling.

Original: Training with oversampling caused less reduction in extrinsic bias metrics.
Revised: Training with oversampling caused less reduction in extrinsic bias metrics.

Original: A consequence of oversampling is that some metrics are less biased, but compression rates are increased, so gender information is more accessible.
Revised: A consequence of oversampling is that some metrics are less biased, but compression rates are increased, so gender information is more accessible.

Original: The effectiveness of subsampling over other metrics is further discussed in appendix C.
Revised: The effectiveness of subsampling over other metrics is further discussed in appendix C.

Original: In coreference resolution (Table 1b), while both CA and CA with anonymization reduced gender extractability as well as external bias metrics, anonymization alone increased intrinsic bias without affecting external bias metrics significantly.
Revised: In coreference resolution (Table 1b), while both CA and CA with anonymization reduced gender extractability as well as external bias metrics, anonymization alone increased intrinsic bias without affecting external bias metrics significantly.

Original: Debiasing without fine-tuning.
Revised: Debiasing without fine-tuning.

Original: As the effect on extrinsic bias did not match the effect on intrinsic bias in several cases, we examined the role of the classification layer.
Revised: As the effect on extrinsic bias did not match the effect on intrinsic bias in several cases, we examined the role of the classification layer.

Original: We trained a model for occupa- R [Table] Compression
Revised: We trained a model for occupation prediction without fine-tuning the underlying

Original: R

Original: [Table]

Original: CEAT

Original: R

Original: [Table]

Original: Compression

Original: R

Original: [Table]

Original: CEAT

Original: Metric

Original: Before

Original: After

Original: Before

Original: After

Original: Before

Original: After

Original: Before

Original: After

Original: F

Original: [Table]

Original: anti ) - - - - 0.

Original: TPR gap (P) 0.

Original: TPR gap (S) 0.

Original: FPR gap (P) 0.

Original: FPR gap (S) 0.

Original: Precision gap (P) 0.

Original: Precision gap (S) 0.

Original: Independence gap (S) 0.

Original: Separation gap (S) 0.

Original: Sufficiency gap (S) 0.

Original: = Pearson; S =

Original: Sum. tion prediction without fine-tuning the underlying

Original: RoBERTa model.
Revised: RoBERTa model.

Original: Training on a subsampled datasetalso reduced the extrinsic metrics (0.15, 0.03, 0.20,and 0.31, respectively, on TPR gaps Pearson, FPRgaps sum, separation sum, and sufficiency sum).
Revised: Training on a subsampled dataset also reduced the extrinsic metrics (0.15, 0.03, 0.20, and 0.31, respectively, on TPR gaps Pearson, FPR gaps sum, separation sum, and sufficiency sum).

Original: Detailed results of this experiment can be found in
Revised: Detailed results of this experiment can be found in

Original: Appendix B. Since no updates were made to the
Revised: Appendix B. Since no updates were made to the

Original: LM, the internal representations could not be debiased, thus the debiasing observed in this model canonly be superficial.
Revised: LM, the internal representations could not be debiased, thus the debiasing observed in this model can only be superficial.

Original: Retraining the classification layer.
Revised: Retraining the classification layer.

Original: Fine-tuningof both tasks revealed that lower extrinsic metricsdid not always lead to lower compression.
Revised: Fine-tuning of both tasks revealed that lower extrinsic metrics did not always lead to lower compression.

Original: Doesthis indicate cases where the debiasing process isonly superficial, and the internal representationsremain biased?
Revised: Does this indicate cases where the debiasing process is only superficial, and the internal representations remain biased?

Original: To test this hypothesis, we froze thepreviously fine-tuned LM’s weights, and retrainedthe classification layer.
Revised: To test this hypothesis, we froze the previously fine-tuned LM’s weights, and retrained the classification layer.

Original: We used the original (nondebiased) training set for retraining.
Revised: We used the original (nondebiased) training set for retraining.

Original: Tables 1a and 1b also compare extrinsic metricsbefore and after retraining.
Revised: Tables 1a and 1b also compare extrinsic metrics before and after retraining.

Original: All models show biasrestoration, due to the classification layer beingtrained on the biased dataset.
Revised: All models show bias restoration, due to the classification layer being trained on the biased dataset.

Original: The amount of biasrestored varies between models in a way that ispredictable by the compression metric.
Revised: The amount of bias restored varies between models in a way that is predictable by the compression metric.

Original: In the occupation prediction task, comparing Before and After numbers in Table 1a, the modelfine-tuned using a scrubbed dataset—which has the lowest compression rate—displays the least bias restoration, confirming that the LM absorbed the process of debiasing.
Revised: In the occupation prediction task, comparing Before and After numbers in Table 1a, the model fine-tuned using a scrubbed dataset—which has the lowest compression rate—displays the least bias restoration, confirming that the LM absorbed the process of debiasing.

Original: The model fine-tuned on subsampled data has higher extrinsic bias after retraining.
Revised: The model fine-tuned on subsampled data has higher extrinsic bias after retraining.

Original: Hence, the debiasing was primarily cosmetic, and the representations within the LM were not debiased.
Revised: Hence, the debiasing was primarily cosmetic, and the representations within the LM were not debiased.

Original: The model fine-tuned on oversampled data—which has the highest compression—has the highest extrinsic bias (except for FPR), even though this was not true before retraining.
Revised: The model fine-tuned on oversampled data—which has the highest compression—has the highest extrinsic bias (except for FPR), even though this was not true before retraining.

Original: In coreference resolution, comparing Before and
Revised: In coreference resolution, comparing Before and

Original: After numbers in Table 1b, models with the least extrinsic bias (CA and CA+Anon) are also least biased after retraining.
Revised: After numbers in Table 1b, models with the least extrinsic bias (CA and CA+Anon) are also least biased after retraining.

Original: Compression rate predicted this; these models also had lower compression rates than non-debiased models.
Revised: Compression rate predicted this; these models also had lower compression rates than non-debiased models.

Original: Interestingly, the model fine-tuned with an anonymized dataset is the most biased after retraining, consistent with its high compression rate relative to the other models.
Revised: Interestingly, the model fine-tuned with an anonymized dataset is the most biased after retraining, consistent with its high compression rate relative to the other models.

Original: As with subsampling and oversampling in occupation prediction, anonymization’s (lack of) effect on extrinsic metrics was cosmetic (compare None and Anon in Before block, Table 1b).
Revised: As with subsampling and oversampling in occupation prediction, anonymization’s (lack of) effect on extrinsic metrics was cosmetic (compare None and Anon in Before block, Table 1b).

Original: Anonymization actually had a biasing effect on the LM, which was realized after retraining.
Revised: Anonymization actually had a biasing effect on the LM, which was realized after retraining.

Original: We conclude that compression rate is a useful indicator of superficial debiasing.
Revised: We conclude that compression rate is a useful indicator of superficial debiasing, and can potentially be used to verify and gain confidence in attempts to debias an NLP model, especially when there is little or no testing data.

Original: 4. Correlation between Extrinsic and 
Revised: 4. Correlation between Extrinsic and Intrinsic Metrics

Original: Intrinsic Metrics

Original: Table 2 shows correlations between compression rate and various extrinsic metrics before and after retraining.
Revised: Table 2 shows correlations between compression rate and various extrinsic metrics before and after R [Table] Compression

Revised: R

Revised: [Table]

Original: In occupation prediction, certain extrinsic metrics have a weak correlation with compres-
Revised: CEAT R [Table] Compression

Revised: R

Revised: [Table]

Revised: CEAT

Revised: Metric

Revised: Before

Revised: After

Revised: Before

Revised: After

Revised: Before

Revised: After

Revised: Before

Revised: After

Revised: F

Revised: [Table]

Revised: anti ) - - - - 0.

Revised: TPR gap (P) 0.

Revised: TPR gap (S) 0.

Revised: FPR gap (P) 0.

Revised: FPR gap (S) 0.

Original: [Figure]
Revised: Precision gap (P) 0. Precision gap (S) 0. Independence gap (S) 0. Separation gap (S) 0.

Revised: Sufficiency gap (S) 0.

Original: [Figure]
Revised: [Figure]

Original: sion rate, while others do not.
Revised: retraining. In occupation prediction, certain extrinsic metrics have a weak correlation with compression rate, while others do not.

Original: Except one metric(FPR gap sum), the compression rate and the extrinsic metric correlate more after retraining.
Revised: Except one metric (FPR gap sum), the compression rate and the extrinsic metric correlate more after retraining.

Original: Figure 3illustrates this for TPR-gap (Pearson).
Revised: Figureillustrates this for TPR-gap (Pearson).

Original: The increaseis due to superficial debiasing, especially by subsampling data, which prior to retraining had lowextrinsic metrics and relatively high intrinsic metrics.
Revised: The increase is due to superficial debiasing, especially by subsampling data, which prior to retraining had low extrinsic metrics and relatively high intrinsic metrics.

Original: This shows that correlation between extrinsicmetrics and compression rate for certain metricsis stronger than it appeared before retraining.
Revised: This shows that correlation between extrinsic metrics and compression rate for certain metrics is stronger than it appeared before retraining.

Original: It isunsurprising that CEAT does not correlate with anyextrinsic metrics, since CEAT could not distinguishbetween different models.
Revised: It is unsurprising that CEAT does not correlate with any extrinsic metrics, since CEAT could not distinguish between different models.

Original: Coreference resolution shows stronger correlations between compression rate and extrinsic metrics, but low correlations between Pearson metrics.
Revised: Coreference resolution shows stronger correlations between compression rate and extrinsic metrics, but low correlations between Pearson metrics.

Original: We further discuss cases of no correlation in appendix D. Correlations decrease after retraining,but metrics that were highly correlated remain so( > 0 .
Revised: We further discuss cases of no correlation in appendix D. Correlations decrease after retraining, but metrics that were highly correlated remain so ( > 0 .

Original: 7 after retraining).
Revised: 7 after retraining).

Original: The correlations are visualized for F1 difference metrics in Figure 4.
Revised: The correlations are visualized for F1 difference metrics in Figure 4.

Original: CEAT and extrinsic metrics correlate much less than compression rate (Table 2).
Revised: CEAT and extrinsic metrics correlate much less than compression rate (Table 2).

Original: Our results are in line with those of Goldfarb-Tarrant et al.
Revised: Our results are in line with those of Goldfarb-Tarrant et al.

Original: (2021), who found a lack of correlation between extrinsic metrics and
Revised: (2021), who found a lack of correlation between extrinsic metrics and

Original: WEAT, the static-embedded version of CEAT.
Revised: WEAT, the static-embedded version of CEAT.

Revised: Given that recent work (Goldfarb-Tarrant et al., 2021; Cao et al., 2022) questions the validity of intrinsic metrics as a reliable indicator for gender bias, the compression rate provides a reliable alternative to current intrinsic metrics, by offering correlation to many extrinsic bias metrics.

Revised: [Figure]

Original:  Related Work
Revised:  Related Work

Original: There are few studies that examine both intrinsic and extrinsic metrics.
Revised: There are few studies that examine both intrinsic and extrinsic metrics.

Original: Previous work by GoldfarbTarrant et al.
Revised: Previous work by GoldfarbTarrant et al.

Original: (2021) showed that debiasing static embeddings intrinsically is not correlated with extrinsic bias, challenging the assumption that intrinsic metrics are predictive of bias.
Revised: (2021) showed that debiasing static embeddings intrinsically is not correlated with extrinsic bias, challenging the assumption that intrinsic metrics are predictive of bias.

Original: We examine the other direction, exploring how extrinsic debiasing affects intrinsic metrics.
Revised: We examine the other direction, exploring how extrinsic debiasing affects intrinsic metrics.

Original: We also extend  their work to contextualized embeddings, a wider range of extrinsic metrics, and a new, more effective intrinsic metric based on information-theoretic probing.
Revised: We also extend beyond their work to contextualized embeddings, a wider range of extrinsic metrics, and a new, more effective intrinsic metric based on information-theoretic probing.

Revised: A contemporary work by Cao et al.

Revised: (2022) measured the correlations between intrinsic and extrinsic metrics in contextualized settings across different language models.

Revised: In contrast, our work examines the correlations across different versions of the same language model by fine-tuning it using various debiasing techniques.

Original: Studies that inspect extrinsic metrics include either a challenge dataset curated to expose differences inmodel behavior by gender, or a test dataset labelledby gender.
Revised: Studies that inspect extrinsic metrics include either a challenge dataset curated to expose differences in model behavior by gender, or a test dataset labelled by gender.

Original: Among these datasets are Winobias(Zhao et al., 2018), Winogender (Rudinger et al.,2018) and GAP (Webster et al., 2018) for coreference resolution, WinoMT (Stanovsky et al., 2019)for machine translation, EEC (Kiritchenko and 
Revised: Among these datasets are Winobias (Zhao et al., 2018), Winogender (Rudinger et al., 2018) and GAP (Webster et al., 2018) for coreference resolution, WinoMT (Stanovsky et al., 2019) for machine translation, EEC (Kiritchenko and Mohammad, 2018) for sentiment analysis,

Original: Mohammad, 2018) for sentiment analysis, BOLD(Dhamala et al., 2021) for language generation,gendered NLI (Sharma et al., 2020) for natural language inference and Bias in Bios (De-Arteaga et al. for occupation prediction.
Revised: BOLD (Dhamala et al., 2021) for language generation, gendered NLI (Sharma et al., 2020) for natural language inference and Bias in Bios (DeArteaga et al., 2019) for occupation prediction.

Original: Studies that measure gender bias intrinsicallyin static word or sentence embeddings measurecharacteristics of the geometry, such as the proximity between female- and male-related words tostereotypical words, or how embeddings clusteror relate to a gender subspace (Bolukbasi et al. ,2016; Caliskan et al., 2017; Gonen and Goldberg,2019b; Ethayarajh et al., 2019).
Revised: Studies that measure gender bias intrinsically in static word or sentence embeddings measure characteristics of the geometry, such as the proximity between female- and male-related words to stereotypical words, or how embeddings cluster or relate to a gender subspace (Bolukbasi et al., 2016; Caliskan et al., 2017; Gonen and Goldberg, 2019b; Ethayarajh et al., 2019).

Original: However, metricsand debiasing methods for static embeddings donot apply directly to contextualized ones.
Revised: However, metrics and debiasing methods for static embeddings do not apply directly to contextualized ones.

Original: Severalstudies use sentence templates to adapt to contextual embeddings (May et al., 2019; Kurita et al. ,2019; Tan and Celis, 2019).
Revised: Several studies use sentence templates to adapt to contextual embeddings (May et al., 2019; Kurita et al., 2019; Tan and Celis, 2019).

Original: This templated approach is difficult to scale, and lacks the range ofrepresentations that a contextual embedding offers.
Revised: This templated approach is difficult to scale, and lacks the range of representations that a contextual embedding offers.

Original: Other work extracts embedding representations ofwords from natural corpora (Zhao et al., 2019; Guoand Caliskan, 2021; Basta et al., 2019).
Revised: Other work extracts embedding representations of words from natural corpora (Zhao et al., 2019; Guo and Caliskan, 2021; Basta et al., 2019).

Original: Thesestudies often adapt the WEAT method (Caliskanet al., 2017), which measures embedding geometry.
Revised: These studies often adapt the WEAT method (Caliskan et al., 2017), which measures embedding geometry.

Original: None measure the effect of the presumably found“bias” on a downstream task.
Revised: None measure the effect of the presumably found “bias” on a downstream task.

Original: There is a growing conversation in the field(Barocas et al., 2017; Kate Crawford, 2017; Blodgett et al., 2020; Bommasani et al., 2021) about theimportance of articulating the harms of measuredbias.
Revised: There is a growing conversation in the field (Barocas et al., 2017; Kate Crawford, 2017; Blodgett et al., 2020; Bommasani et al., 2021) about the importance of articulating the harms of measured bias.

Original: In general, extrinsic metrics have clear, interpretable impacts for which harm can be defined.
Revised: In general, extrinsic metrics have clear, interpretable impacts for which harm can be defined.

Original: Intrinsic metrics have an unclear effect.
Revised: Intrinsic metrics have an unclear effect.

Original: Withoutevidence from a concrete downstream task, a foundintrinsic bias is only theoretically harmful.
Revised: Without evidence from a concrete downstream task, a found intrinsic bias is only theoretically harmful.

Original: Ourwork is a step towards understanding whether intrinsic metrics provide valuable insights about biasin a model.
Revised: Our work is a step towards understanding whether intrinsic metrics provide valuable insights about bias in a model.

Original:  Discussion and Conclusions
Revised:  Discussion and Conclusions

Original: This study examined whether bias in internal representations is related to extrinsic bias.
Revised: This study examined whether bias in internal representations is related to extrinsic bias.

Original: We designeda new framework in which we debias a model ona downstream task, and measure its intrinsic bias.
Revised: We designed a new framework in which we debias a model on a downstream task, and measure its intrinsic bias.

Original: We found that gender extractability from internal representations, measured by compression rate via
Revised: We found that gender extractability from internal representations, measured by compression rate via

Original: MDL probing, reflects bias in a model.
Revised: MDL probing, reflects bias in a model.

Original: Compression was much more reliable than an alternative intrinsic metric for contextualised representations,
Revised: Compression was much more reliable than an alternative intrinsic metric for contextualised representations,

Original: CEAT.
Revised: CEAT.

Original: Compression correlated well—to varying degrees—with many extrinsic metrics.
Revised: Compression correlated well—to varying degrees—with many extrinsic metrics.

Original: Our results show that when a debiasing method reduces extrinsic metrics but not compression, it indicates that the language model remains biased.
Revised: We thus encourage NLP practitioners to use compression as an intrinsic indicator for gender bias in NLP models. When comparing two alternative models, a lower compression rate provides confidence in a model’s superiority in terms of gender bias. The relative success of compression over CEAT may be explained by the fact that the compression rate was calculated on the same dataset as the extrinsic metrics. CEAT, on the other hand, was measured on a general dataset which is not necessarily related to a speicifc downstream task.

Revised: The use of a general dataset is a common strategy among other intrinsic metrics (May et al., 2019; Kurita et al., 2019; Basta et al., 2021).

Revised: Another explanation is that the compression rate measures a more focused concept, namely the gender information within the internal representations.

Revised: CEAT measures proximity in general terms that may include other social contexts that do not directly relate to gender.

Revised: Our results show that when a debiasing method reduces extrinsic metrics but not compression, it indicates that the language model remains biased.

Original: When such superficial debiasing occurs, the debiased language model may be reapplied to another task, as in Jin et al.
Revised: When such superficial debiasing occurs, the debiased language model may be reapplied to another task, as in Jin et al.

Original: (2021), resulting in unexpected biases and nullifying the supposed debiasing.
Revised: (2021), resulting in unexpected biases and nullifying the supposed debiasing.

Original: Our findings suggest that practitioners of NLP should take special care when adopting previously debiased models and inspect them carefully, perhaps using our framework.
Revised: Our findings suggest that practitioners of NLP should take special care when adopting previously debiased models and inspect them carefully, perhaps using our framework.

Revised: In another study (Mendelson and Belinkov, 2021a) it was found – for a different type of bias which is not gender bias – that as the model is less biased, the biased features are more extractable as shown by the compression rate on various probing tasks.

Revised: Analysis of our results showed that the change in internal representation depends on the choice of debiasing strategy - some strategies increase intrinsic bias and others decrease it.

Original: Our work also highlighted the importance of the classification layer.
Revised: Our work also highlighted the importance of the classification layer.

Original: Using a debiased objective, such as a balanced dataset, the classification layer can provide significant debiasing.
Revised: Using a debiased objective, such as a balanced dataset, the classification layer can provide significant debiasing.

Original: This holds even if the internal representations are biased and the classifier is a single linear layer, as shown in the occupation prediction task.
Revised: This holds even if the internal representations are biased and the classifier is a single linear layer, as shown in the occupation prediction task.

Original: Bias stems in part from internal LM bias and in part from classification bias.
Revised: Bias stems in part from internal LM bias and in part from classification bias.

Original: Practitioners should focus their efforts on both parts when attempting to debias a model.
Revised: Practitioners should focus their efforts on both parts when attempting to debias a model.

Original: We used a broader set of extrinsic metrics than is typically used, and found that the bias metrics behaved differently: some decreased more than others after debiasing, and they correlated differently with compression rate.
Revised: We used a broader set of extrinsic metrics than is typically used, and found that the bias metrics behaved differently: some decreased more than others after debiasing, and they correlated differently with compression rate.

Original: Debiasing efforts may not be fully understood by testing only a few extrinsic metrics.
Revised: Debiasing efforts may not be fully understood by testing only a few extrinsic metrics.

Original: MDL probing can indicate meaningful debiasing of internal model representations even when not all metrics are easily measurable, since it correlates well with many extrinsic metrics.
Revised: However, compression as an intrinsic bias metric can indicate meaningful debiasing of internal model representations even when not all metrics are easily measurable, since it correlates well with many extrinsic metrics.

Original: A major limitation of this study is the use of gender as a binary variable, which is trans-exclusive.
Revised: A major limitation of this study is the use of gender as a binary variable, which is trans-exclusive.

Original: Cao and Daumé III (2020) made the first steps towards inclusive gender bias evaluation in NLP, revealing that coreference systems fail on genderinclusive text.
Revised: Cao and Daumé III (2020) made the first steps towards inclusive gender bias evaluation in NLP, revealing that coreference systems fail on genderinclusive text.

Original: Further work is required to adjust our framework to non-binary genders, potentially revealing insights about the poor performance of
Revised: Further work is required to adjust our framework to non-binary genders, potentially revealing insights about the poor performance of

Original: NLP systems in that area.
Revised: NLP systems in that area.

Revised: Acknowledgements

Revised: This research was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 448/20) and by an Azrieli Foundation Early Career Faculty Fellowship.

Revised: We also thank Kate McCurdy and Andreas

Revised: Grivas for comments on early drafts, the members of the Technion CS NLP group for their valuable feedback, and the anonymous reviewers for their useful suggestions.

Original: A Implementation Details
Revised: A Implementation Details

Original: We used RoBERTa in all models which has 120Mparameters.
Revised: We used RoBERTa in all models (base size, 120M parameters).

Original: We use following random seeds in allrepeated experiments: 0, 5, 11, 26, 42, 46, 50, 63,83, 90.
Revised: We use following random seeds in all repeated experiments: 0, 5, 11, 26, 42, 46, 50, 63, 83, 90.

Revised: Our code was implemented mainly using the Python libraries Pytorch (Paszke et al., 2019),

Revised: Transformers (Wolf et al., 2020), Sklearn (Pedregosa et al., 2011), and the experiments were logged using Wandb (Biewald, 2020).

Original: A. Occupation Clasification
Revised: A. Occupation Classification

Original: We fine-tuned a RoBERTa-base model with a linear classification layer on top.
Revised: We fine-tuned a RoBERTa-base model with a linear classification layer on top.

Original: Training was donefor 10 epochs at a learning rate of 5e-5, batch sizeof 64.
Revised: Training was done for 10 epochs at a learning rate of 5e-5, batch size of 64.

Original: The input to RoBERTa was the biographytokens, which is limited to the first 128 tokens.
Revised: The input to RoBERTa was the biography tokens, which is limited to the first 128 tokens.

Original: Theresulting [CLS] token embedding is fed to the classifier to predict the occupation.
Revised: The resulting [CLS] token embedding is fed to the classifier to predict the occupation.

Original: The probing taskinvolves using the same [CLS] token and trainingthe probing classifier to predict the gender of theperson in the biography.
Revised: The probing task involves using the same [CLS] token and training the probing classifier to predict the gender of the person in the biography.

Original: The experiments withoutfine-tuning included either a pre-trained or a previously fine-tuned RoBERTa.
Revised: The experiments without fine-tuning included either a pre-trained or a previously fine-tuned RoBERTa.

Original: We first extractedthe pre-trained RoBERTa’s embeddings of tokensfrom the [CLS] and then trained a linear classifier on them.
Revised: We first extracted the pre-trained RoBERTa’s embeddings of tokens from the [CLS] and then trained a linear classifier on them.

Original: The learning rate was 0.001 and thebatch size was 64.
Revised: The learning rate was 0.001 and the batch size was 64.

Original: We trained the classificationlayer with pre-trained RoBERTa on 300 epochs,but with fine-tuned RoBERTa, 10 epochs were sufficient.
Revised: We trained the classification layer with pre-trained RoBERTa on 300 epochs, but with fine-tuned RoBERTa, 10 epochs were sufficient.

Original: For all training processes, the epoch withthe greatest validation accuracy was saved.
Revised: For all training processes, the epoch with the greatest validation accuracy was saved.

Original: Finetuning took 7 hours on  GeForce RTX 2080{+
+} Ti GPU.
Revised: Finetuning took 7 hours on a GeForce RTX {+
+} Ti GPU.

Original: Bias in Bios contains almost 400k biographies, andwe obtain validation (10%) and test set (25%) bysplitting with Scikit-learn’s (Pedregosa et al., 2011)test_train_split with our random seeds.
Revised: Bias in Bios contains almost 400k biographies, and we obtain validation (10%) and test set (25%) by splitting with Scikit-learn’s (Pedregosa et al., 2011) test_train_split with our random seeds.

Original: A. Coreference Resolution
Revised: A. Coreference Resolution

Original: We use the implementation of Xu and Choi (2020),a model that was introduced by Lee et al.
Revised: We use the implementation of Xu and Choi (2020), a model that was introduced by Lee et al.

Original: (2018b)and has been adopted by many coreference resolution models.
Revised: (2018b) and has been adopted by many coreference resolution models.

Original: Coreference resolution is the processof clustering different mentions in a text that referto the same real-world entities.
Revised: Coreference resolution is the process of clustering different mentions in a text that refer to the same real-world entities.

Original: The task is solvedby detecting mentions through text spans and thenpredicting for each pair of spans if they representthe same entity.
Revised: The task is solved by detecting mentions through text spans and then predicting for each pair of spans if they represent the same entity.

Original: The span representations wereextracted with a RoBERTa model, which is finetuned throughout the training process, except in theretraining experiment.
Revised: The span representations were extracted with a RoBERTa model, which is fine-tuned throughout the training process, except in the retraining experiment.

Original: Fine-tuning took 3 hourson NVIDIA RTX A6000 GPU.
Revised: Fine-tuning took 3 hours on an NVIDIA RTX A6000 GPU.

Original: Ontonotes 5.0 has625k sentences and we use the standard validationand test splits.
Revised: Ontonotes 5.0 has 625k sentences and we use the standard validation and test splits.

Original: A. Probing Classifier
Revised: A. Probing Classifier

Original: We use the MDL probe (Voita and Titov, 2020) implementation by Mendelson and Belinkov (2021).
Revised: We use the MDL probe (Voita and Titov, 2020) implementation by Mendelson and Belinkov (2021b).

Original: In all experiments, we use a linear probe and train it with a batch size of 16 and a learning rate of 1e-3.
Revised: In all experiments, we use a linear probe and train it with a batch size of 16 and a learning rate of 1e-3.

Original: The timestamps used, meaning the accumulating fractions of data that the probe is trained on, are 2.0%, 3.0%, 4.4%, 6.5%, 9.5%, 14.0%, 21.0%, 31.0%, 45.7%, 67.6%, 100%.
Revised: The timestamps used, meaning the accumulating fractions of data that the probe is trained on, are 2.0%, 3.0%, 4.4%, 6.5%, 9.5%, 14.0%, 21.0%, 31.0%, 45.7%, 67.6%, 100%.

Original: A . Metrics A.4. Fairness-Based Metrics Implementation
Revised: A. Metrics A.4. Fairness-Based Metrics Implementation

Original: All three statistical fairness metrics measure the difference between two probability distributions, where this difference describes a notion of bias.
Revised: All three statistical fairness metrics measure the difference between two probability distributions, where this difference describes a notion of bias.

Original: We calculate Independence and Separation via
Revised: We calculate Independence and Separation via

Original: Kullback–Leibler (KL) divergence, using the AllenNLP implementation ( https://github.com/ allenai/allennlp ).
Revised: Kullback–Leibler (KL) divergence, using the AllenNLP implementation ( https://github.com/ allenai/allennlp ).

Original: We calculate Sufficiency via
Revised: We calculate Sufficiency via

Original: Wasserstein distance instead, which is motivated by Kwegyir-Aggrey et al.
Revised: Wasserstein distance instead, which is motivated by Kwegyir-Aggrey et al.

Original: In this case, we cannot use KL divergence, since there are some classes that do not occur in model predictions for both male and female genders.
Revised: In this case, we cannot use KL divergence, since there are some classes that do not occur in model predictions for both male and female genders.

Original: This causes the probability distributions to not have the same support, and KL divergence is unbounded.
Revised: This causes the probability distributions to not have the same support, and KL divergence is unbounded.

Original: Wasserstein distance lacks the requirement for equal support.
Revised: Wasserstein distance lacks the requirement for equal support

Original:  A.4. Classification Metrics Interpretation in 
Revised: . A.4. Classification Metrics Interpretation in Winobias

Original: Winobias

Original: Winobias datasets contain pairs of stereotypical and anti-stereotypical sentences.
Revised: Winobias datasets contain pairs of stereotypical and anti-stereotypical sentences.

Original: The stereotypes are derived from the US labor statistics (for instance, a profession with a majority of males is stereotypically male).
Revised: The stereotypes are derived from the US labor statistics (for instance, a profession with a majority of males is stereotypically male).

Original: Since coreference resolution is viewed as a clustering problem, it is usually measured via clustering evaluation metrics.
Revised: Since coreference resolution is viewed as a clustering problem, it is usually measured via clustering evaluation metrics.

Original: Coreference resolution is commonly measured as the average F1 score of these, and the same is true for Winobias.
Revised: Coreference resolution is commonly measured as the average F1 score of these, and the same is true for Winobias.

Original: Nevertheless, coreference resolution is accomplished by making a prediction for each pair of mentions, so it can be seen as a classification task.
Revised: Nevertheless, coreference resolution is accomplished by making a prediction for each pair of mentions, so it can be seen as a classification task.

Original: Winobias can be viewed as a simpler task than general coreference resolution, as it contains exactly two mentions of professions and one pronoun, which refers to exactly one profession.
Revised: Winobias can be viewed as a simpler task than general coreference resolution, as it contains exactly two mentions of professions and one pronoun, which refers to exactly one profession.

Original: Therefore, we reframe it as a classification problem.
Revised: Therefore, we reframe it as a classification problem.

Original: In a Winobias sentence with two professions x and y , as well as a pronoun p , where p is referring to x , a true positive would be to cluster x and p together, while a false positive would be to cluster y and p together.
Revised: In a Winobias sentence with two professions x and y , as well as a pronoun p , where p is referring to x , a true positive would be to cluster x and p together, while a false positive would be to cluster y and p together.

Original: Our classification metrics are derived based on thesedefinitions.
Revised: Our classification metrics are derived based on these definitions.

Original: For instance, the TPR gap for profession “teacher”, which is a stereotypical femaleoccupation, is the TPR rate on pro-stereotypicalsentences (with a female pronoun) minus the TPRrate on anti-stereotypical sentences (with a malepronoun).
Revised: For instance, the TPR gap for profession “teacher”, which is a stereotypical female occupation, is the TPR rate on pro-stereotypical sentences (with a female pronoun) minus the TPR rate on anti-stereotypical sentences (with a male pronoun).

Original: A.4. CEAT
Revised: A.4. CEAT

Original: The Word Embedding Association Test (WEAT)developed by (Caliskan et al., 2017) is a methodfor evaluating bias in static word embeddings.
Revised: The Word Embedding Association Test (WEAT) developed by (Caliskan et al., 2017) is a method for evaluating bias in static word embeddings.

Original: Thetest is defined as follows: given two sets of targetwords X, Y (e.g., ’executive’, ’management’, ’professional’ and ’home’, ’parents’, ’children’) andtwo sets of attribute words (e.g., male names andfemale names), and using ⃗w to represent the wordembedding for word w , the effect size is: 
Revised: The test is defined as follows: given two sets of target words X , Y (e.g., ’executive’, ’management’, ’professional’ and ’home’, ’parents’, ’children’) and two sets of attribute words (e.g., male names and female names), and using ⃗w to represent the word embedding for word w , the effect size is: [Equation]

Original: ES
Revised: where s

Original: = mean x ∈ X s ( x, A, B ) − mean
Revised: [Equation] x [Equation]

Original: y ∈ Y s ( y, A, B )wheres ( x, A, B ) =
Revised: A

Original: [Equation]
Revised: [Equation]

Original: In essence, the effect size measures how different are the distances between the embedding vectors of each target group and the attribute groups.
Revised: In essence, the effect size measures how different are the distances between the embedding vectors of each target group and the attribute groups.

Original: Specifically, if s ( x, A, B ) > 0 , ⃗x is more similar to attribute words B and vice versa.
Revised: Specifically, if s ( x, A , B ) > 0 , ⃗ x is more similar to attribute words B and vice versa.

Original: For instance, a larger effect size is observed if targetwords X are more similar to attribute words A andtarget words Y are more similar to attribute words
Revised: For instance, a larger effect size is observed if target words X are more similar to attribute words A and target words Y are more similar to attribute words

Original: B .
Revised: B .

Original: | ES | > 0 .
Revised: | ES | > 0 .

Original: 5 and | ES | > 0 .
Revised: 5 and | ES | > 0 .

Original: 8 are consideredmedium and large effect sizes, respectively (Riceand Harris, 2005).
Revised: 8 are considered medium and large effect sizes, respectively (Rice and Harris, 2005).

Original: The null hypothesis holds thatthere is no difference between the two sets of targetwords in terms of their relative similarity to thetwo sets of attribute words, indicating that there areno biased associations.
Revised: The null hypothesis holds that there is no difference between the two sets of target words in terms of their relative similarity to the two sets of attribute words, indicating that there are no biased associations.

Original: Statistical significance isdefined by the p-value of WEAT, which reflects theprobability of observing the effect size under thenull hypothesis.
Revised: Statistical significance is defined by the p-value of WEAT, which reflects the probability of observing the effect size under the null hypothesis.

Original: Since a word can take on a great variety of vectorrepresentations in a contextual setting, ES variesaccording to the sentences used to extract wordrepresentation.
Revised: Since a word can take on a great variety of vector representations in a contextual setting, ES varies according to the sentences used to extract word representation.

Original: Thus, to adopt WEAT to contextualized representations, the Combined Effect Size(CES) (Guo and Caliskan, 2021) is derived as thedistribution of WEAT effect sizes over many possible contextual word representations:
Revised: Thus, to adopt WEAT to contextualized representations, the Combined Effect Size (CES) (Guo and Caliskan, 2021) is derived as the distribution of WEAT effect sizes over many possible contextual word representations:

Original: [Equation]
Revised: [Equation]

Original: where ES i denotes the WEAT effect size of the i ’th choice of word representations from a large corpus, and v i is the inverse of the sum of in-sample variance V i and between-sample variance in the distribution of random-effects.
Revised: where ES i denotes the WEAT effect size of the i ’th choice of word representations from a large corpus, and v i is the inverse of the sum of in-sample variance V i and between-sample variance in the distribution of random-effects.

Original: As in Guo and Caliskan (2021), the representation for each word is derived from 10,000 random sentences extracted from a corpus of Reddit comments.
Revised: As in Guo and Caliskan (2021), the representation for each word is derived from 10,000 random sentences extracted from a corpus of Reddit comments.

Original: The combined effect size of each of the models is examined on WEAT stimulus 6, which contains target words of career/family and attribute words of male/female names.
Revised: The combined effect size of each of the models is examined on WEAT stimulus 6, which contains target words of career/family and attribute words of male/female names.

Original: This was the only one that detected bias on a pre-trained RoBERTa (CES close to 0.5 and p < 0 . 05 ).
Revised: This was the only one that detected bias on a pre-trained RoBERTa (CES close to 0.5 and p < 0 . 05 ).

Original: The points that we kept in our analysis are those where p < 0 .
Revised: The points that we kept in our analysis are those where p < 0 .

Original: 05 , which make up 90% of the points in occupation prediction and 95% of the points in coreference resolution.
Revised: 05 , which make up 90% of the points in occupation prediction and 95% of the points in coreference resolution.

Original: B Full Results
Revised: B Full Results

Original: In this section we provide the full results of a
Revised: In this section we provide the full results of a

Original: RoBERTa model trained on the downstream task.
Revised: RoBERTa model trained on the downstream task.

Original: The results for the occupation prediction task after fine-tuning are presented in Table 3 and Table 4 presents the retrained model results.
Revised: The results for the occupation prediction task after fine-tuning are presented in Table 3 and Table 4 presents the retrained model results.

Original: Figure illustrates the correlations between extrinsic metrics and compression rate before and after retraining.
Revised: Figureillustrates the correlations between extrinsic metrics and compression rate before and after retraining.

Original: Table 5 presents the complete results of the model trained without fine-tuning, meaning that the RoBERTa model is the pretrained version from
Revised: Table 5 presents the complete results of the model trained without fine-tuning, meaning that the RoBERTa model is the pretrained version from

Original: Liu et al.
Revised: Liu et al.

Original: and only the classification layer was updated.
Revised: and only the classification layer was updated.

Original: Subsampling the dataset has significant debiasing effects, which suggests that this debiasing method can achieve low extrinsic bias even when internal bias exists.
Revised: Subsampling the dataset has significant debiasing effects, which suggests that this debiasing method can achieve low extrinsic bias even when internal bias exists.

Revised: Table 6 presents the results after we performed the experiments on a

Revised: DeBERTa (He et al., 2020) model, with occupation classification task.

Revised: The trends are similar to those of RoBERTa, with the same metrics showing an increase, no change, or decrease in correlation after re-training, suggesting a general trend in the behavior of these metrics in relation to internal model representations.

Original: Regarding the coreference resolution task, Table 6 displays the results on a finetuned model and
Revised: Regarding the coreference resolution task, Table 7 displays the results on a finetuned model and

Original: Table 7 displays the retraining results.
Revised: Table 8 displays the retraining results.

Original: Figure shows the correlations between compression rate and extrinsic metrics before and after the retraining.
Revised: Figureshows the correlations between compression rate and extrinsic metrics before and after the retraining.

Original: Debiasing Strategy
Revised: Debiasing Strategy

Original: [Table]
Revised: [Table]

Original: Debiasing
Revised: Debiasing

Original: [Table] 
Revised: [Table] [Table]

Revised: R 2 Compression

Revised: Metric

Revised: Before

Revised: After

Revised: TPR gap (P) 0.

Revised: TPR gap (S) 0.

Revised: FPR gap (P) 0.

Revised: FPR gap (S)

Revised: [Table]

Revised: Precision gap (P)

Revised: [Table]

Revised: Precision gap (S)

Original: [Table]
Revised: [Table]

Revised: Independence gap (S) 0.

Revised: Separation gap (S) 0.

Revised: Sufficiency gap (S) 0.

Original: Debiasing Strategy
Revised: Debiasing Strategy

Original: [Table]
Revised: [Table]

Original: Independence
Revised: Independence

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Separation gap
Revised: Separation gap

Original: [Table]
Revised: [Table]

Original: 1.247 ±
Revised: 1.247 ±

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 344 ±
Revised: 344 ±

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Sufficiency gap
Revised: Sufficiency gap

Original: [Table]
Revised: [Table]

Original: S)
Revised: S)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Compression
Revised: Compression

Original: [Table]
Revised: [Table]

Original: F1 (Ontonotes test)
Revised: F1 (Ontonotes test)

Original: [Table]
Revised: [Table]

Original: F1 diff ( pro − anti ) 6.072 ± 0.
Revised: F1 diff ( pro − anti ) 6.072 ± 0.

Original: 7.417* ± 1.
Revised: 7.417* ± 1.

Original: 3.674* ±0.
Revised: 3.674* ±

Original: [Table]
Revised: [Table]

Original: ± 0. TPR gap (P) 0.635 ± 0.
Revised: [Table] TPR gap (P) 0.635 ± 0.

Original: 0.688* ± 0.
Revised: 0.688* ± 0.

Original: 0.679* ± 0.
Revised: 0.679* ± 0.

Original: 0.654 ± 0.
Revised: 0.654 ± 0.

Original: TPR gap (S) 4.561 ± 0.
Revised: TPR gap (S) 4.561 ± 0.

Original: 5.143* ± 0.
Revised: 5.143* ± 0.

Original: 2.590* ± 0.
Revised: 2.590* ± 0.

Original: 2.178 * ± 0.
Revised: 2.178 * ± 0.

Original: FPR gap (P) 0.579 ± 0.
Revised: FPR gap (P) 0.579 ± 0.

Original: 0.637* ± 0.
Revised: 0.637* ± 0.

Original: 0.620* ± 0.
Revised: 0.620* ± 0.

Original: 0.692* ± 0.
Revised: 0.692* ± 0.

Original: FPR gap (S) 0.113 ± 0.
Revised: FPR gap (S) 0.113 ± 0.

Original: 0.126* ± 0.
Revised: 0.126* ± 0.

Original: 0.063* ± 0.
Revised: 0.063* ± 0.

Original: 0.052 * ± 0.
Revised: 0.052 * ± 0.

Original: Precision gap (P) 0.
Revised: Precision gap (P) 0.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C Why is scrubbing not as effective assubsampling?
Revised: C Why is scrubbing not as effective as subsampling?

Original: The debiasing method of subsampling significantlyreduced external biases in the occupation prediction task.
Revised: The debiasing method of subsampling significantly reduced external biases in the occupation prediction task.

Original: Although compression rates show thatscrubbing reduced more gender information, subsampling outperforms it as a debiasing method.
Revised: Although compression rates show that scrubbing reduced more gender information, subsampling outperforms it as a debiasing method.

Original: Wefind that in spite of the scrubbing, a probe is ableto correctly identify the gender from an internalrepresentation with 68.8% accuracy compared to90.7% on the original, non-scrubbed data.
Revised: We find that in spite of the scrubbing, a probe is able to correctly identify the gender from an internal representation with 68.8% accuracy compared to 90.7% on the original, non-scrubbed data.

Original: Thismeans that although the scrubbing process reducesextrinsic bias significantly, gender information isstill embedded in the [CLS] token embeddings.
Revised: This means that although the scrubbing process reduces extrinsic bias significantly, gender information is still embedded in the [CLS] token embeddings.

Original: To investigate the source of gender informationafter scrubbing, we use logistic regression (LR)model to predict the gender from the Bag-of-Words of the scrubbed biographies.
Revised: To investigate the source of gender information after scrubbing, we use logistic regression (LR) model to predict the gender from the Bag-of-Words of the scrubbed biographies.

Original: We perform an iterative process for automatic extra scrubbing: in each iteration we (1) train a LR model for gender prediction (2) scrub the n most significant words for each gender according to the LR weights.
Revised: We perform an iterative process for automatic extra scrubbing: in each iteration we (1) train a LR model for gender prediction (2) scrub the n most significant words for each gender according to the LR weights.

Original: The most relevant words among 5 seeds of training with n= words scrubbed per iteration are displayed in Table 8.
Revised: The most relevant words among 5 seeds of training with n= words scrubbed per iteration are displayed in Table 9.

Original: The model learns indirect correlations to gender in the absence of explicit gendered words.
Revised: The model learns indirect correlations to gender in the absence of explicit gendered words.

Original: Because the significant words are related to male- or femaledominated professions, we conducted the process on a specific profession.
Revised: Because the significant words are related to male- or femaledominated professions, we conducted the process on a specific profession.

Original: Table 9 presents the most significant words for biographies of nurses.
Revised: Table 10 presents the most significant words for biographies of nurses.

Original: There are differences in wording even between females and males in the same profession.
Revised: There are differences in wording even between females and males in the same profession.

Original: The results of this study are in line with the results of other studies that have been conducted on the way biographies are written for men and women (Wagner et al., 2016; Sun and Peng, 2021).
Revised: The results of this study are in line with the results of other studies that have been conducted on the way biographies are written for men and women (Wagner et al., 2016; Sun and Peng, 2021).

Original: Subsampling is therefore more effective even when gender information is present since it prevents the model from learning correlations between gender information and a profession whereas scrubbing only attempts to remove gender indicators without removing correlations.
Revised: Subsampling is therefore more effective even when gender information is present since it prevents the model from learning correlations between gender information and a profession whereas scrubbing only attempts to remove gender indicators without removing correlations.

Original: On the other hand, it is possible that oversampling is less effective for debiasing since seeing more non-unique examples an unrepresented group encourages learning correlations.
Revised: On the other hand, it is possible that oversampling is less effective for debiasing since seeing more non-unique examples an unrepresented group encourages learning correlations.

Original: D A closer look into no-correlation cases 
Revised: D A closer look into no-correlation cases D. Occupation Prediction

Original: D. Occupation Prediction

Original: Although compression has the ability to identify bias in most cases, some metrics still show little or no correlation with compression rate.
Revised: Although compression has the ability to identify bias in most cases, some metrics still show little or no correlation with compression rate.

Original: These results suggest that gender information comprises only one facet of embedded bias in the representations.
Revised: These results suggest that gender information comprises only one facet of embedded bias in the representations.

Original: Other factors that may influence these metrics are not considered or measured, such as the connection between a name and a profession.
Revised: Other factors that may influence these metrics are not considered or measured, such as the connection between a name and a profession.

Original: For example, as can be see in Tables 3 and 4,
Revised: For example, as can be see in Tables 3 and 4,

Original: LMs finetuned on subsampled data have the largest
Revised: LMs finetuned on subsampled data have the largest

Original: FPR gaps after retraining, despite being the least biased before retraining, while those finetuned on oversampled data have the next-to-lowest FPR gaps after retraining.
Revised: FPR gaps after retraining, despite being the least biased before retraining, while those finetuned on oversampled data have the next-to-lowest FPR gaps after retraining.

Original: The information encoded in the internal representations may have been encoded in a manner that allowed the classification layer to exhibit a smaller FPR gap when trained on a balanced dataset.
Revised: The information encoded in the internal representations may have been encoded in a manner that allowed the classification layer to exhibit a smaller FPR gap when trained on a balanced dataset.

Original: However, when the classification layer was retrained on biased training data, it usedthe same features to make biased predictions.
Revised: However, when the classification layer was retrained on biased training data, it used the same features to make biased predictions.

Original: D. Coreference Resolution
Revised: D. Coreference Resolution

Original: The cases where there is no correlation betweenour intrinsic metric and an extrinsic metric are thecases where the metric is based on Pearson correlation.
Revised: The cases where there is no correlation between our intrinsic metric and an extrinsic metric are the cases where the metric is based on Pearson correlation.

Original: Unlike occupation prediction, coreferenceresolution seems to exhibit no correlation betweenthose metrics and compression rate.
Revised: Unlike occupation prediction, coreference resolution seems to exhibit no correlation between those metrics and compression rate.

Original: These metricsare computed as the Pearson correlation between aperformance gap for a specific profession and thepercentage of women in that profession, howeverthe percentages are computed differently in eachtask: in occupation prediction, the percentages arecomputed from the train set, focusing on the representation each gender has in the data.
Revised: These metrics are computed as the Pearson correlation between a performance gap for a specific profession and the percentage of women in that profession, however the percentages are computed differently in each task: in occupation prediction, the percentages are computed from the train set, focusing on the representation each gender has in the data.

Original: In Winobias, the percentages are taken from the US laborstatistics, and are unrelated to the training datasetstatistics.
Revised: In Winobias, the percentages are taken from the US labor statistics, and are unrelated to the training dataset statistics.

Original: We note that the two statistics can be different - the real-world representation of women in aprofession does not have to be equal to their representation in written text (Suresh and Guttag, 2021).
Revised: We note that the two statistics can be different - the real-world representation of women in a profession does not have to be equal to their representation in written text (Suresh and Guttag, 2021).

Original: We thus decided to test what happens if we changethe statistics used in Winobias to dataset statistics,but Ontonotes 5.0 has very little representation toeach profession and the statistics extracted fromit would not be reliable.
Revised: We thus decided to test what happens if we change the statistics used in Winobias to dataset statistics, but Ontonotes 5.0 has very little representation to each profession and the statistics extracted from it would not be reliable.

Original: We thus took a differentapproach and computed the Pearson correlationsfor occupation prediction with real world statisticsinstead of dataset statistics.
Revised: We thus took a different approach and computed the Pearson correlations for occupation prediction with real world statistics instead of dataset statistics.

Original: To do this, we mappedthe professions appearing in this dataset to professions from the US labor statistics, and droppedthose who could no be mapped (6 out of 29 of theprofessions which is 21.4%).
Revised: To do this, we mapped the professions appearing in this dataset to professions from the US labor statistics, and dropped those who could no be mapped (6 out of 29 of the professions which is 21.4%).

Original: We then repeatedall experiments on the Pearson metrics using thesestatistics.
Revised: We then repeated all experiments on the Pearson metrics using these statistics.

Original: Figure 7 shows the results.
Revised: Figure 7 shows the results.

Original: Correlationsare very different when computed with respect toreal-world statistics.
Revised: Correlations are very different when computed with respect to real-world statistics.

Original: TPR-gap has no correlation atall although it had with training data statistics, thecorrelation for FPR-gap after retraining exists butis negative, and the correlation with precision-gapdoes not exist after retraining.
Revised: TPR-gap has no correlation at all although it had with training data statistics, the correlation for FPR-gap after retraining exists but is negative, and the correlation with precision-gap does not exist after retraining.

Original: We thus concludethat the Pearson metrics are less reliable as they areheavily dependent on the statistics with respect towhich they are calculated.
Revised: We thus conclude that the Pearson metrics are less reliable as they are heavily dependent on the statistics with respect to which they are calculated.