Original: How Gender Debiasing Affects Internal Model Representations, and Why It Matters
Revised: How Gender Debiasing Affects Internal Model Representations, and Why It Matters

Original: Abstract
Revised: Abstract

Original: Common studies of gender bias in NLP focus either on extrinsic bias measured by model performance on a downstream task or on intrinsic bias found in models’ internal representations.
Revised: Common studies of gender bias in NLP focus either on extrinsic bias measured by model performance on a downstream task or on intrinsic bias found in models’ internal representations.

Original: However, the relationship between extrinsic and intrinsic bias is relatively unknown.
Revised: However, the relationship between extrinsic and intrinsic bias is relatively unknown.

Original: In this work, we illuminate this relationship by measuring both quantities together: we debias a model during downstream fine-tuning, which reduces extrinsic bias, and measure the effect on intrinsic bias, which is operationalized as bias extractability with information-theoretic probing.
Revised: In this work, we illuminate this relationship by measuring both quantities together: we debias a model during downstream fine-tuning, which reduces extrinsic bias, and measure the effect on intrinsic bias, which is operationalized as bias extractability with information-theoretic probing.

Original: Through experiments on two tasks and multiple bias metrics, we show that our intrinsic bias metric is a better indicator of debiasing than (a contextual adaptation of) the standard WEAT metric, and can also expose cases of superficial debiasing.
Revised: Through experiments on two tasks and multiple bias metrics, we show that our intrinsic bias metric is a better indicator of debiasing than (a contextual adaptation of) the standard WEAT metric, and can also expose cases of superficial debiasing.

Original: Our framework provides a comprehensive perspective on bias in NLP models, which can be applied to deploy
Revised: Our framework provides a comprehensive perspective on bias in NLP models, which can be applied to deploy

Original: NLP systems in a more informed manner.
Revised: NLP systems in a more informed manner.

Original:  Introduction
Revised:  Introduction

Original: Efforts to identify and mitigate gender bias in Natural Language Processing (NLP) systems typically target one of two notions of bias.
Revised: Efforts to identify and mitigate gender bias in Natural Language Processing (NLP) systems typically target one of two notions of bias.

Original: Extrinsic evaluation methods and debiasing techniques focus on the bias reflected in a downstream task (De-Arteaga et al., 2019;
Revised: Extrinsic evaluation methods and debiasing techniques focus on the bias reflected in a downstream task (De-Arteaga et al., 2019;

Original: Zhao et al., 2018), while intrinsic methods focus on a model’s internal representations, such as word or sentence embedding geometry (Caliskan et al., 2017; Bolukbasi et al., 2016;
Revised: Zhao et al., 2018), while intrinsic methods focus on a model’s internal representations, such as word or sentence embedding geometry (Caliskan et al., 2017; Bolukbasi et al., 2016;

Original: Guo and Caliskan, 2021).
Revised: Guo and Caliskan, 2021).

Original: Despite an abundance of evidence pointing towards gender bias in pretrained language models (LMs), the extent of harm caused by these biases is not clear when it is not reflected in a specific downstream task (Barocas
Revised: Despite an abundance of evidence pointing towards gender bias in pretrained language models (LMs), the extent of harm caused by these biases is not clear when it is not reflected in a specific downstream task (Barocas

Original: RoBERTa
Revised: RoBERTa

Original: [Table]
Revised: [Table]

Original: RoBERTa
Revised: RoBERTa

Original: [Table]
Revised: [Table]

Original: RoBERTa
Revised: RoBERTa

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: et al., 2017; Kate Crawford, 2017; Blodgett et al., 2020; Bommasani et al., 2021).
Revised: et al., 2017; Kate Crawford, 2017; Blodgett et al., 2020; Bommasani et al., 2021).

Original: For instance, while the word embedding proximity of “doctor” to “man” and “nurse” to “woman” is intuitively normatively wrong, it is not clear when such phenomena would lead to downstream predictions manifesting in social biases.
Revised: For instance, while the word embedding proximity of “doctor” to “man” and “nurse” to “woman” is intuitively normatively wrong, it is not clear when such phenomena would lead to downstream predictions manifesting in social biases.

Original: Recently, Goldfarb-Tarrant et al.
Revised: Recently, Goldfarb-Tarrant et al.

Original: (2021) have shown that debiasing static embeddings intrinsically is not correlated with extrinsic gender bias measures, but the nature of the reverse relationship is unknown: how are extrinsic interventions reflected in intrinsic representations?
Revised: (2021) have shown that debiasing static embeddings intrinsically is not correlated with extrinsic gender bias measures, but the nature of the reverse relationship is unknown: how are extrinsic interventions reflected in intrinsic representations?

Original: Furthermore,
Revised: Furthermore,

Original: Gonen and Goldberg (2019a) demonstrated that a number of intrinsic debiasing methods applied to static embeddings only partially remove the bias and that most of it is still hidden within the embed- ding.
Revised: Gonen and Goldberg (2019a) demonstrated that a number of intrinsic debiasing methods applied to static embeddings only partially remove the bias and that most of it is still hidden within the embed- ding.

Original: Complementing their view, we examine extrinsic debiasing methods, as well as demonstrate the possible harm this could cause.
Revised: Complementing their view, we examine extrinsic debiasing methods, as well as demonstrate the possible harm this could cause.

Original: Contrary to their conclusion, we do not claim that these debiasing methods should not be trusted, as long as they are utilized with care .
Revised: Contrary to their conclusion, we do not claim that these debiasing methods should not be trusted, as long as they are utilized with care .

Original: Our goal is to gain a better understanding of the relationship between a model’s internal representations and its extrinsic gender bias by examining the effects of various debiasing methods on the model’s representations.
Revised: Our goal is to gain a better understanding of the relationship between a model’s internal representations and its extrinsic gender bias by examining the effects of various debiasing methods on the model’s representations.

Original: Specifically, we fine-tune models with and without gender debiasing strategies, evaluate their external bias using various bias metrics, and measure intrinsic bias in the representations.
Revised: Specifically, we fine-tune models with and without gender debiasing strategies, evaluate their external bias using various bias metrics, and measure intrinsic bias in the representations.

Original: We operationalize intrinsic bias via two metrics: First, we use CEAT (Guo and Caliskan, 2021), a contextual adaptation of the widely used intrinsic bias metric WEAT (Caliskan et al., 2017).
Revised: We operationalize intrinsic bias via two metrics: First, we use CEAT (Guo and Caliskan, 2021), a contextual adaptation of the widely used intrinsic bias metric WEAT (Caliskan et al., 2017).

Original: Second, we propose to use an information-theoretic probe to quantify the degree to which gender can be extracted from the internal model representations.
Revised: Second, we propose to use an information-theoretic probe to quantify the degree to which gender can be extracted from the internal model representations.

Original: Then, we examine how these intrinsic metrics correlate with a variety of extrinsic bias metrics that we measure on the model’s downstream performance.
Revised: Then, we examine how these intrinsic metrics correlate with a variety of extrinsic bias metrics that we measure on the model’s downstream performance.

Original: Our approach is visualised in Figure 1.
Revised: Our approach is visualised in Figure 1.

Original: We perform extensive experiments on two downstream tasks (occupation prediction and coreference resolution); several debiasing strategies that involve alterations to the training dataset (such as removing names and gender indicators, or balancing the data by oversampling or downsampling); and a multitude of extrinsic bias metrics.
Revised: We perform extensive experiments on two downstream tasks (occupation prediction and coreference resolution); several debiasing strategies that involve alterations to the training dataset (such as removing names and gender indicators, or balancing the data by oversampling or downsampling); and a multitude of extrinsic bias metrics.

Original: Our analysis reveals new insights into the way language models encode and use information on gender:
Revised: Our analysis reveals new insights into the way language models encode and use information on gender:

Original: • The effect of debiasing on internal representations is reflected in gender extractability, while not always in CEAT.
Revised: • The effect of debiasing on internal representations is reflected in gender extractability, while not always in CEAT.

Original: Thus, gender extractability is a more reliable indicator of gender bias in NLP models.
Revised: Thus, gender extractability is a more reliable indicator of gender bias in NLP models.

Original: • In cases of high gender extractability but low extrinsic bias metrics, the debiasing is superficial, and the internal representations are a good indicator for this: The bias is still present in internal representations and can be restored by retraining the classification layer.
Revised: • In cases of high gender extractability but low extrinsic bias metrics, the debiasing is superficial, and the internal representations are a good indicator for this: The bias is still present in internal representations and can be restored by retraining the classification layer.

Original: Therefore, our proposed measuring method can help in detecting such cases before deploying the model.
Revised: Therefore, our proposed measuring method can help in detecting such cases before deploying the model.

Original: • The two tasks show different patterns of correlation between intrinsic and extrinsic bias.
Revised: • The two tasks show different patterns of correlation between intrinsic and extrinsic bias.

Original: The coreference task exhibits a high correlation.
Revised: The coreference task exhibits a high correlation.

Original: The occupation prediction task exhibits a lower correlation, but it increases after retraining (a case of superficial debiasing).
Revised: The occupation prediction task exhibits a lower correlation, but it increases after retraining (a case of superficial debiasing).

Original: Gender extractability shows higher correlations with extrinsic metrics than CEAT, increasing the confidence in this metric as a reliable measure for gender bias in NLP models
Revised: Gender extractability shows higher correlations with extrinsic metrics than CEAT, increasing the confidence in this metric as a reliable measure for gender bias in NLP models

Original:  Methodology
Revised:  Methodology

Original: In this study, we investigate the relationship between extrinsic bias metrics of a task and a model’s internal representations, under various debiasing conditions, for two datasets in English.
Revised: In this study, we investigate the relationship between extrinsic bias metrics of a task and a model’s internal representations, under various debiasing conditions, for two datasets in English.

Original: We perform extrinsic debiasing, evaluate various extrinsic and intrinsic bias metrics before and after debiasing, and examine correlations.
Revised: We perform extrinsic debiasing, evaluate various extrinsic and intrinsic bias metrics before and after debiasing, and examine correlations.

Original: Dataset.
Revised: Dataset.

Original: Let D =
Revised: Let D =

Original: { X , Y , Z } be a dataset consisting of input data X , labels Y and protected attributes Z . 

Revised: { X , Y , Z } be a dataset consisting of input data X , labels Y and protected attributes Z .

Original: This work focuses on gender as the protected attribute z .
Revised: This work focuses on gender as the protected attribute z .

Original: In all definitions, F and M indicate female and male gender, respectively, as the value of the protected attribute z .
Revised: In all definitions, F and M indicate female and male gender, respectively, as the value of the protected attribute z .

Original: Trained Model.
Revised: Trained Model.

Original: The model is optimized to solve the downstream task posed by the dataset.
Revised: The model is optimized to solve the downstream task posed by the dataset.

Original: It can be formalized as f ◦ g : X → R |Y| , where g ( · ) is the feature extractor, implemented by a language model, e.g., RoBERTa (Liu et al., 2019), f ( · ) is the classification function, and Y is the set of the possible labels for the task.
Revised: It can be formalized as f ◦ g : X → R |Y| , where g ( · ) is the feature extractor, implemented by a language model, e.g., RoBERTa (Liu et al., 2019), f ( · ) is the classification function, and Y is the set of the possible labels for the task.

Original: 2. Bias Metrics
Revised: 2. Bias Metrics

Original: Each bias evaluation method described in the literature can be categorized as extrinsic or intrinsic.
Revised: Each bias evaluation method described in the literature can be categorized as extrinsic or intrinsic.

Original: In all definitions, r indicates the model’s output probabilities.
Revised: In all definitions, r indicates the model’s output probabilities.

Original: 2.1. Extrinsic Metrics
Revised: 2.1. Extrinsic Metrics

Original: Extrinsic methods involve measuring the bias of a model solving a downstream problem.
Revised: Extrinsic methods involve measuring the bias of a model solving a downstream problem.

Original: The extrinsic metric is a function:
Revised: The extrinsic metric is a function:

Original: [Equation]
Revised: [Equation]

Original: The output represents the quantity of bias measured; the further from 0 the number is, the larger the bias is.
Revised: The output represents the quantity of bias measured; the further from 0 the number is, the larger the bias is.

Original: Our analysis comprises a wide range of extrinsic metrics, including some that have been measured in the past on the analyzed tasks (Zhao et al., 2018; De-Arteaga et al., 2019; Ravfogel et al., 2020; Goldfarb-Tarrant et al., 2021) and some that have never been measured before, and shows our results apply to many of them.
Revised: Our analysis comprises a wide range of extrinsic metrics, including some that have been measured in the past on the analyzed tasks (Zhao et al., 2018; De-Arteaga et al., 2019; Ravfogel et al., 2020; Goldfarb-Tarrant et al., 2021) and some that have never been measured before, and shows our results apply to many of them.

Original: For illustration, we will consider occupation prediction, a common task in research on gender bias (De-Arteaga et al., 2019;
Revised: For illustration, we will consider occupation prediction, a common task in research on gender bias (De-Arteaga et al., 2019;

Original: Ravfogel et al., 2020; Romanov et al., 2019).
Revised: Ravfogel et al., 2020; Romanov et al., 2019).

Original: The input x is a biography and the prediction y is the profession of the person described in it.
Revised: The input x is a biography and the prediction y is the profession of the person described in it.

Original: The protected attribute z is the gender of that person.
Revised: The protected attribute z is the gender of that person.

Original: Performance gap.
Revised: Performance gap.

Original: This is the difference in performance metric for two different groups, for instance two groups of binary genders, or a group of pro-stereotypical and a group of anti-stereotypical examples.
Revised: This is the difference in performance metric for two different groups, for instance two groups of binary genders, or a group of pro-stereotypical and a group of anti-stereotypical examples.

Original: We measure the following metrics:
Revised: We measure the following metrics:

Original: True
Revised: True

Original: Positive Rate (TPR), False Positive Rate (FPR), and
Revised: Positive Rate (TPR), False Positive Rate (FPR), and

Original: Precision.
Revised: Precision.

Original: In occupation prediction, for instance, the TPR gap for each profession y expresses the difference in the percentage of women and men whose profession is y and are correctly classified as such.
Revised: In occupation prediction, for instance, the TPR gap for each profession y expresses the difference in the percentage of women and men whose profession is y and are correctly classified as such.

Original: We also measure F1 of three standard clustering metrics for coreference resolution.
Revised: We also measure F1 of three standard clustering metrics for coreference resolution.

Original: Each such performance gap captures a different facet of gender bias, and one might be more interested in one of the metrics depending on the application.
Revised: Each such performance gap captures a different facet of gender bias, and one might be more interested in one of the metrics depending on the application.

Original: We compute two types of performance gap metrics: (1) the sum of absolute gap values over all classes; (2) the Pearson correlation between the performance gap for a class and the percentage of women in that class.
Revised: We compute two types of performance gap metrics: (1) the sum of absolute gap values over all classes; (2) the Pearson correlation between the performance gap for a class and the percentage of women in that class.

Original: For instance, if y is a profession, we measure the correlation between performance gaps and percentages of women in each profession.
Revised: For instance, if y is a profession, we measure the correlation between performance gaps and percentages of women in each profession.

Original: The two metrics are closely related but answer slightly different questions: the sum quantifies how a model behaves differently on different genders, and the correlation shows the relation of model behaviour to social biases (in the world or the data) without regard to actual gap size.
Revised: The two metrics are closely related but answer slightly different questions: the sum quantifies how a model behaves differently on different genders, and the correlation shows the relation of model behaviour to social biases (in the world or the data) without regard to actual gap size.

Original: Statistical metrics.
Revised: Statistical metrics.

Original: For breadth of analysis, we examine three additional statistical metrics
Revised: For breadth of analysis, we examine three additional statistical metrics

Original: (Barocas et al., 2019), which correspond to different notions of bias.
Revised: (Barocas et al., 2019), which correspond to different notions of bias.

Original: All three are measured as differences ( d ) between two probability distributions, and we then obtain a single bias quantity per metric by summing all computed distances.
Revised: All three are measured as differences ( d ) between two probability distributions, and we then obtain a single bias quantity per metric by summing all computed distances.

Original: • Independence :
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: { F, M } .
Revised: { F, M } .

Original: For instance, we measure the difference between the distribution of model’s predictions on women and the distribution of all predictions.
Revised: For instance, we measure the difference between the distribution of model’s predictions on women and the distribution of all predictions.

Original: Independence is stronger as the prediction r is less correlated with the protected attribute z .
Revised: Independence is stronger as the prediction r is less correlated with the protected attribute z .

Original: This is measured with no relation to the gold labels.
Revised: It is measured with no relation to the gold labels.

Original: • Separation : d (cid:0)
Revised: • Separation : d (cid:0)

Original: [Equation]
Revised: [Equation]

Original: ∀ y ∈ Y , z ∈ { F, M } .
Revised: ∀ y ∈ Y , z ∈ { F, M } .

Original: For instance, we measure the difference between the distribution of a model’s predictions on women who are teachers and the distribution of predictions on all teachers.
Revised: For instance, we measure the difference between the distribution of a model’s predictions on women who are teachers and the distribution of predictions on all teachers.

Original: This encapsulates the TPR and FPR equality discussed previously, and can be seen as a more general metric.
Revised: It encapsulates the TPR and FPR gaps discussed previously, and can be seen as a more general metric.

Original: • Sufficiency:
Revised: • Sufficiency

Original: [Equation]
Revised: [Equation]

Original: For instance, we measure the difference between the distribution of gold labels on women classi- fied as teachers by the model 
Revised: For instance, we measure the difference between the distribution of gold labels on women classified as teachers by the model and the distribution of gold labels on all individuals classified as teachers by the model.

Original: and

Original: the distribu tion

Original: of gold labels on all individuals classified as teachers by the

Original: model

Original: Sufficiency

Original:  relates to the concept of calibration in classification.
Revised: Sufficiency relates to the concept of calibration in classification.

Original: A

Original: difference

Original: in

Original: the classifier’s scores

Original: for men and for

Original: women

Original: indicates

Original: that

Original:  it might be penalizing or over-promoting one of the genders
Revised: A difference in the classifier’s scores for men and for women indicates that it might be penalizing or over-promoting one of the genders.

Original: 2.1. Intrinsic Metrics
Revised: 2.1. Intrinsic Metrics

Original: Intrinsic methods are applied to the representation obtained from the feature extractor.
Revised: Intrinsic methods are applied to the representation obtained from the feature extractor.

Original: These methods are independent of any downstream task.
Revised: These methods are independent of any downstream task.

Original: The intrinsic metric is a function:
Revised: The intrinsic metric is a function:

Original: [Equation]
Revised: I ( g ( X ) , Z ) ∈ R

Original: Compression.
Revised: Compression.

Original: Our main intrinsic metric is the compression of gender information evaluated by a minimum description length (MDL) probing classifier (Voita and Titov, 2020), trained to predict gender from the model’s representations.
Revised: Our main intrinsic metric is the compression of gender information evaluated by a minimum description length (MDL) probing classifier (Voita and Titov, 2020), trained to predict gender from the model’s representations.

Original: Probing classifiers are widely used for predicting various properties of interest from frozen model representations (Belinkov and Glass, 2019).
Revised: Probing classifiers are widely used for predicting various properties of interest from frozen model representations (Belinkov and Glass, 2019).

Original: MDL probes were proposed because a probe’s accuracy may be misleading due to memorization and other issues (Hewitt and Liang, 2019; Belinkov, 2021).
Revised: MDL probes were proposed because a probe’s accuracy may be misleading due to memorization and other issues (Hewitt and Liang, 2019; Belinkov, 2021).

Original: We use the MDL online code, where the probe is trained in timesteps, on increasing subsets of the training set, then evaluated against the rest of it.
Revised: We use the MDL online code, where the probe is trained in timesteps, on increasing subsets of the training set, then evaluated against the rest of it.

Original: Higher compression indicates greater gender extractability.
Revised: Higher compression indicates greater gender extractability.

Original: CEAT.
Revised: CEAT.

Original: We also measure CEAT (Guo and
Revised: We also measure CEAT (Guo and

Original: Caliskan, 2021), which is a contextualized version of WEAT (Caliskan et al., 2017), a widely used bias metric for static word embeddings.
Revised: Caliskan, 2021), which is a contextualized version of WEAT (Caliskan et al., 2017), a widely used bias metric for static word embeddings.

Original: WEAT defines sets X and Y of target words, and sets A and
Revised: WEAT defines sets X and Y of target words, and sets A and

Original: B of attribute words.
Revised: B of attribute words.

Original: For instance, A and B contain males and females names, while X and Y contain career and family related words, respectively.
Revised: For instance, A and B contain males and females names, while X and Y contain career and family related words, respectively.

Original: The bias is operationalized as the geometric proximity between the target and attribute word embeddings, and is quantified in CEAT by the Combined Effect
Revised: The bias is operationalized as the geometric proximity between the target and attribute word embeddings, and is quantified in CEAT by the Combined Effect

Original: Size (CES) and a p-value for the null hypothesis of having no biased associations.
Revised: Size (CES) and a p-value for the null hypothesis of having no biased associations.

Original: For more information on CEAT refer to Appendix A.4.3.
Revised: For more information on CEAT refer to Appendix A.4.3.

Original: 2. Debiasing Techniques
Revised: 2. Debiasing Techniques

Original: We debias models by modifying the downstream task’s training data before fine-tuning.
Revised: We debias models by modifying the downstream task’s training data before fine-tuning.

Original: Scrubbing (De-Arteaga et al., 2019) removes first names and gender-specific terms (“he”, “she”, “husband”, “wife”, “Mr”, “Mrs”, etc.).
Revised: Scrubbing (De-Arteaga et al., 2019) removes first names and gender-specific terms (“he”, “she”, “husband”, “wife”, “Mr”, “Mrs”, etc.).

Original: Balancing subsamples or oversamples examples such that each gender is equally represented in the resulting dataset w.r.t each label.
Revised: Balancing subsamples or oversamples examples such that each gender is equally represented in the resulting dataset w.r.t each label.

Original: Anonymization (Zhao et al., 2018) removes named entities.
Revised: Anonymization (Zhao et al., 2018) removes named entities.

Original: Counterfactual Augmentation (Zhao et al., 2018) involves replacing male entities in an example with female entities, and adding the modified example to the training set.
Revised: Counterfactual Augmentation (Zhao et al., 2018) involves replacing male entities in an example with female entities, and adding the modified example to the training set.

Original: As some of these are dataset/task-specific, we give more details in the following section.
Revised: As some of these are dataset/task-specific, we give more details in the following section.

Original:  Experiments
Revised:  Experiments

Original: In each experiment, we fine-tune a model for a downstream task.
Revised: In each experiment, we fine-tune a model for a downstream task.

Original: For training, we use either the original dataset or a dataset debiased with one of the methods from Section 2.2.
Revised: For training, we use either the original dataset or a dataset debiased with one of the methods from Section 2.2.

Original: Figure 2 presents examples of debiasing methods for the two downstream tasks.
Revised: Figure 2 presents examples of debiasing methods for the two downstream tasks.

Original: We measure two intrinsic metrics by probing that model’s inner representations for gender extractability (as measured by MDL) and by
Revised: We measure two intrinsic metrics by probing that model’s inner representations for gender extractability (as measured by MDL) and by

Original: CEAT, and test various extrinsic metrics.
Revised: CEAT, and test various extrinsic metrics.

Original: The relation between one intrinsic and one extrinsic metric becomes one data point, and we repeat over many random seeds (for both the model and the probe).
Revised: The relation between one intrinsic and one extrinsic metric becomes one data point, and we repeat over many random seeds (for both the model and the probe).

Original: Further implementation details are in appendix A.
Revised: Further implementation details are in appendix A.

Original: 3. Occupation Prediction
Revised: 3. Occupation Prediction

Original: The task of occupation prediction is to predict a person’s occupations (from a closed set), based on their biography.
Revised: The task of occupation prediction is to predict a person’s occupations (from a closed set), based on their biography.

Original: We use the Bias in Bios dataset (De-Arteaga et al., 2019).
Revised: We use the Bias in Bios dataset (De-Arteaga et al., 2019).

Original: Regardless of the training method, the test set is subsampled such that each profession has equal gender representation.
Revised: Regardless of the training method, the test set is subsampled such that each profession has equal gender representation.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Model.
Revised: Model.

Original: Our  model is a RoBERTa model (Liu et al., 2019) topped with a linear classifier, which receives the [CLS] token embedding as input and generates a probability distribution over the professions.
Revised: Our main model is a RoBERTa model (Liu et al., 2019) topped with a linear classifier, which receives the [CLS] token embedding as input and generates a probability distribution over the professions.

Original: In addition, we train a baseline classifier layer on top of a frozen, non-finetuned RoBERTa.
Revised: In addition, we experiment with training a baseline classifier layer on top of a frozen, non-finetuned RoBERTa.

Original: We also experimented with a DeBERTa model (He et al., 2020),  results are in the appendix (B).
Revised: We also replicate our RoBERTa experiments with a DeBERTa model (He et al., 2020), to verify that our results are are not model specific and hold more broadly.

Original: Debiasing Techniques.
Revised: Debiasing Techniques.

Original: Following De-Arteaga et al.
Revised: Following De-Arteaga et al.

Original: we experiment with scrubbing the training dataset.
Revised: we experiment with scrubbing the training dataset.

Original: Figure 2 shows an example biography snippet and its scrubbed version.
Revised: Figure 2 shows an example biography snippet and its scrubbed version.

Original: We also conduct balancing (per profession, subsampling and oversampling to ensure an equal number of males and females per profession), which has not previously been used on this dataset and task.
Revised: We also conduct balancing (per profession, subsampling and oversampling to ensure an equal number of males and females per profession), which has not previously been used on this dataset and task.

Original: Metrics.
Revised: Metrics.

Original: We measure all bias metrics from Section 2.1 except for F1.
Revised: We measure all bias metrics from Section 2.1 except for F1.

Original: Probing.
Revised: Probing.

Original: The probing dataset for this task is the test set, and the gender label of a single biography is the gender of the person described in it.
Revised: The probing dataset for this task is the test set, and the gender label of a single biography is the gender of the person described in it.

Original: We probe the [CLS] token representation of the biography.
Revised: We probe the [CLS] token representation of the biography.

Original: In addition to the models described above, we measure baseline extractability of gender information from a randomly initialized RoBERTa model.
Revised: In addition to the models described above, we measure baseline extractability of gender information from a randomly initialized RoBERTa model.

Original: 3. Coreference Resolution
Revised: 3. Coreference Resolution

Original: The task of coreference resolution is to find all textual expressions referring to the same real-world entities.
Revised: The task of coreference resolution is to find all textual expressions referring to the same real-world entities.

Original: We train on Ontonotes 5.0 (Weischedel et al., 2013) and test on the Winobias challenge dataset (Zhao et al., 2018).
Revised: We train on Ontonotes 5.0 (Weischedel et al., 2013) and test on the Winobias challenge dataset (Zhao et al., 2018).

Original: Winobias consists of sentence pairs, pro- and anti-stereotypical variants, with individuals referred to by their profession.
Revised: Winobias consists of sentence pairs, pro- and anti-stereotypical variants, with individuals referred to by their profession.

Original: For example, “The physician hired the secretary because he/
Revised: For example, “The physician hired the secretary be-

Original: she was busy.”

Original: is pro/anti-stereotypical,

Original: Extrinsic
Revised: Extrinsic

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  based on US labor statistics.
Revised: cause he/she was busy.” is pro/anti-stereotypical, based on US labor statistics.

Original: 4 A coreference system is measured by the performance gap between the pro- and anti-stereotypical subsets.
Revised: 4 A coreference system is measured by the performance gap between the pro- and anti-stereotypical subsets.

Original: Model.
Revised: Model.

Original: We use the model presented in Lee et al.
Revised: We use the model presented in Lee et al.

Original: (2018a) with RoBERTa as a feature extractor.
Revised: (2018a) with RoBERTa as a feature extractor.

Original: Debiasing Techniques.
Revised: Debiasing Techniques.

Original: Following Zhao et al.
Revised: Following Zhao et al.

Original: (2018), we apply anonymization (denoted as Anon) and counterfactual augmentation (CA) on the training set.
Revised: (2018), we apply anonymization (denoted as Anon) and counterfactual augmentation (CA) on the training set.

Original: These techniques were used jointly in previous work; we examine each individually as well.
Revised: These techniques were used jointly in previous work; we examine each individually as well.

Original: Metrics.
Revised: Metrics.

Original: Following Zhao et al.
Revised: Following Zhao et al.

Original: (2018), we measure the F1 difference between anti- and prostereotypical examples.
Revised: (2018), we measure the F1 difference between anti- and prostereotypical examples.

Original: 5 We also interpret the task as a classification problem, and measure all metrics from Section 2.1.
Revised: 5 We also interpret the task as a classification problem, and measure all metrics from Section 2.1.

Original: For more details refer to
Revised: For more details refer to

Original: Appendix A.4.2.
Revised: Appendix A

Original: Probing
Revised: Probing.

Original: We probe the representation of a profession word as extracted from Winobias sentences, after masking out the pronouns.
Revised: We probe the representation of a profession word as extracted from Winobias sentences, after masking out the pronouns.

Original: We define a profession’s gender as the stereotypical gender for this profession.
Revised: We define a profession’s gender as the stereotypical gender for this profession.

Original: To prevent memorization by the probe— given the small number of professions—the dataset is sorted so that professions are gradually added to the training set, so a success on the validation set is on previously unseen professions.
Revised: To prevent memorization by the probe— given the small number of professions—the dataset is sorted so that professions are gradually added to the training set, so a success on the validation set is on previously unseen professions.

Original:  Results
Revised:  Results

Original: Tables 1a and 1b present intrinsic and extrinsic metrics  on the occupation prediction and coreference resolution tasks, respectively.
Revised: Tables 1a and 1b present intrinsic and extrinsic metrics for RoBERTa models on the occupation prediction and coreference resolution tasks, respectively.

Original: We present a representative subset of the measured metrics that demonstrate the observed phenomena; full results are found in Appendix B.
Revised: We present a representative subset of the measured metrics that demonstrate the observed phenomena; full results for both RoBERTa and

Revised: DeBERTa models across all metrics are found in

Revised: Appendix B. The DeBERTa model results are consistent with the RoBERTa model trends.

Original: 4. Compression Reflects Debiasing Effects
Revised: 4. Compression Reflects Debiasing Effects

Original: As shown in the tables, compression captures differences in models that were debiased differently.
Revised: As shown in the tables, compression captures differences in models that were debiased differently.

Original: CEAT, however, cannot differentiate between occupation prediction models.
Revised: CEAT, however, cannot differentiate between occupation prediction models.

Original: For example, in occupation prediction (Table 1a) the compression rate varies significantly between a non-debiased and a debiased model via scrubbing and oversampling, while CEAT detects no difference between the models.
Revised: For example, in occu- pation prediction (Table 1a) the compression rate varies significantly between a non-debiased and a debiased model via scrubbing and oversampling, while CEAT detects no difference between the models.

Original: In coreference resolution (Table 1b), both compression and CEAT are able to identify differences between the non-debiased model and the others, such as CA, which has both a lower compression and CEAT effect.
Revised: In coreference resolution (Table 1b), both compression and CEAT are able to identify differences between the non-debiased model and the others, such as CA, which has both a lower compression and CEAT effect.

Original: But the CEAT effect sizes are small (below 0.5), which implies no bias in contrast to the extrinsic metrics.
Revised: But the CEAT effect sizes are small (below 0.5), which implies no bias, in contrast to the extrinsic metrics.

Original: High Gender Extractability Implies
Revised: High Gender Extractability Implies

Original: Superficial Debiasing
Revised: Superficial Debiasing

Original: Extrinsic and intrinsic effects of debiasing.
Revised: Extrinsic and intrinsic effects of debiasing.

Original: In occupation classification (Table 1a), somewhat surprisingly, subsampling the training data has the strongest effect on extrinsic metrics, but not on compression rate.
Revised: In occupation classification (Table 1a), somewhat surprisingly, subsampling the training data has the strongest effect on extrinsic metrics, but not on compression rate.

Original: Scrubbing reduces both intrinsic and extrinsic metrics, although its effect on extrinsic metrics is limited compared to subsampling.
Revised: Scrubbing reduces both intrinsic and extrinsic metrics, although its effect on extrinsic metrics is limited compared to subsampling.

Original: Training with oversampling caused less reduction in extrinsic bias metrics.
Revised: Training with oversampling caused less reduction in extrinsic bias metrics.

Original: A consequence of oversampling is that some metrics are less biased, but compression rates are increased, so gender information is more accessible.
Revised: A consequence of oversampling is that some metrics are less biased, but compression rates are increased, so gender information is more accessible.

Original: The effectiveness of subsampling over other metrics is further discussed in appendix C.
Revised: The effectiveness of subsampling over other metrics is further discussed in appendix C.

Original: In coreference resolution (Table 1b), while both CA and CA with anonymization reduced gender extractability as well as external bias metrics, anonymization alone increased intrinsic bias without affecting external bias metrics significantly.
Revised: In coreference resolution (Table 1b), while both CA and CA with anonymization reduced gender extractability as well as external bias metrics, anonymization alone increased intrinsic bias without affecting external bias metrics significantly.

Original: Debiasing without fine-tuning.
Revised: Debiasing without fine-tuning.

Original: As the effect on extrinsic bias did not match the effect on intrinsic bias in several cases, we examined the role of the classification layer.
Revised: As the effect on extrinsic bias did not match the effect on intrinsic bias in several cases, we examined the role of the classification layer.

Original: We trained a model for occupation prediction without fine-tuning the underlying
Revised: We trained a model for occupation prediction without fine-tuning the underlying

Original: RoBERTa model.
Revised: RoBERTa model.

Original: Training on a subsampled dataset also reduced the extrinsic metrics (0.15, 0.03, 0.20, and 0.31, respectively, on TPR gaps Pearson, FPR gaps sum, separation sum, and sufficiency sum).
Revised: Training on a subsampled dataset also reduced the extrinsic metrics (0.15, 0.03, 0.20, and 0.31, respectively, on TPR gaps Pearson, FPR gaps sum, separation sum, and sufficiency sum).

Original: Detailed results of this experiment can be found in
Revised: Detailed results of this experiment can be found in

Original: Appendix B. Since no updates were made to the
Revised: Appendix B. Since no updates were made to the

Original: LM, the internal representations could not be debiased, thus the debiasing observed in this model can only be superficial.
Revised: LM, the internal representations could not be debiased, thus the debiasing observed in this model can only be superficial.

Original: Retraining the classification layer.
Revised: Retraining the classification layer.

Original: Fine-tuning of both tasks revealed that lower extrinsic metrics did not always lead to lower compression.
Revised: Fine-tuning of both tasks revealed that lower extrinsic metrics did not always lead to lower compression.

Original: Does this indicate cases where the debiasing process is only superficial, and the internal representations remain biased?
Revised: Does this indicate cases where the debiasing process is only superficial, and the internal representations remain biased?

Original: To test this hypothesis, we froze the previously fine-tuned LM’s weights, and retrained the classification layer.
Revised: To test this hypothesis, we froze the previously fine-tuned LM’s weights, and retrained the classification layer.

Original: We used the original (nondebiased) training set for retraining.
Revised: We used the original (nondebiased) training set for retraining.

Original: Tables 1a and 1b also compare extrinsic metrics before and after retraining.
Revised: Tables 1a and 1b also compare extrinsic metrics before and after retraining.

Original: All models show bias restoration, due to the classification layer being trained on the biased dataset.
Revised: All models show bias restoration, due to the classification layer being trained on the biased dataset.

Original: The amount of bias restored varies between models in a way that is predictable by the compression metric.
Revised: The amount of bias restored varies between models in a way that is predictable by the compression metric.

Original: In the occupation prediction task, comparing Before and After numbers in Table 1a, the model fine-tuned using a scrubbed dataset—which has the lowest compression rate—displays the least bias restoration, confirming that the LM absorbed the process of debiasing.
Revised: In the occupation prediction task, comparing Before and After numbers in Table 1a, the model fine-tuned using a scrubbed dataset—which has the lowest compression rate—displays the least bias restoration, confirming that the LM absorbed the process of debiasing.

Original: The model fine-tuned on subsampled data has higher extrinsic bias after retraining.
Revised: The model fine-tuned on subsampled data has higher extrinsic bias after retraining.

Original: Hence, the debiasing was primarily cosmetic, and the representations within the LM were not debiased.
Revised: Hence, the debiasing was primarily cosmetic, and the representations within the LM were not debiased.

Original: The model fine-tuned on oversampled data—which has the highest compression—has the highest extrinsic bias (except for FPR), even though this was not true before retraining.
Revised: The model fine-tuned on oversampled data—which has the highest compression—has the highest extrinsic bias (except for FPR), even though this was not true before retraining.

Original: In coreference resolution, comparing Before and
Revised: In coreference resolution, comparing Before and

Original: After numbers in Table 1b, models with the least extrinsic bias (CA and CA+Anon) are also least biased after retraining.
Revised: After numbers in Table 1b, models with the least extrinsic bias (CA and CA+Anon) are also least biased after retraining.

Original: Compression rate predicted this; these models also had lower compression rates than non-debiased models.
Revised: Compression rate predicted this; these models also had lower compression rates than non-debiased models.

Original: Interestingly, the model fine-tuned with an anonymized dataset is the most biased after retraining, consistent with its high compression rate relative to the other models.
Revised: Interestingly, the model fine-tuned with an anonymized dataset is the most biased after retraining, consistent with its high compression rate relative to the other models.

Original: As with subsampling and oversampling in occupation prediction, anonymization’s (lack of) effect on extrinsic metrics was cosmetic (compare None and Anon in Before block, Table 1b).
Revised: As with subsampling and oversampling in occupation prediction, anonymization’s (lack of) effect on extrinsic metrics was cosmetic (compare None and Anon in Before block, Table 1b).

Original: Anonymization actually had a biasing effect on the LM, which was realized after retraining.
Revised: Anonymization actually had a biasing effect on the LM, which was realized after retraining.

Original: We conclude that compression rate is a useful indicator of superficial debiasing, and can potentially be used to verify and gain confidence in attempts to debias an NLP model, especially when there is little or no testing data.
Revised: We conclude that compression rate is a useful indicator of superficial debiasing, and can potentially be used to verify and gain confidence in attempts to debias an NLP model, especially when there is little or no testing data.

Original: 4. Correlation between Extrinsic and Intrinsic Metrics

Original: Table 2 shows correlations between compression rate and various extrinsic metrics before and after

Original: R
Revised: R

Original: [Table]
Revised: [Table]

Original: Compression
Revised: Compression

Original: R
Revised: R

Original: [Table]
Revised: [Table]

Original: CEAT
Revised: CEAT

Original: R
Revised: R

Original: [Table]
Revised: [Table]

Original: Compression
Revised: Compression

Original: R
Revised: R

Original: [Table]
Revised: [Table]

Original: CEAT
Revised: CEAT

Original: Metric
Revised: Metric

Original: Before
Revised: Before

Original: After
Revised: After

Original: Before
Revised: Before

Original: After
Revised: After

Original: Before
Revised: Before

Original: After
Revised: After

Original: Before
Revised: Before

Original: After
Revised: After

Original: F
Revised: F

Original: [Table]
Revised: [Table]

Original: anti ) - - - - 0.
Revised: anti ) - - - - 0.

Original: TPR gap (P) 0.
Revised: TPR gap (P) 0.

Original: TPR gap (S) 0.
Revised: TPR gap (S) 0.

Original: FPR gap (P) 0.
Revised: FPR gap (P) 0.

Original: FPR gap (S) 0.
Revised: FPR gap (S) 0.

Original: Precision gap (P) 0.
Revised: Precision gap (P) 0.

Original: Precision gap (S) 0.
Revised: Precision gap (S) 0.

Original: Independence gap (S) 0. Separation gap (S) 0.
Revised: Independence gap (S) 0. Separation gap (S) 0.

Original: Sufficiency gap (S) 0.
Revised: Sufficiency gap (S) 0.

Original: [Figure]
Revised: [Figure]

Original:  retraining.
Revised: Table 2 shows correlations between compression rate and various extrinsic metrics before and after retraining.

Original: In occupation prediction, certain extrinsic metrics have a weak correlation with compression rate, while others do not.
Revised: In occupation prediction, certain extrinsic metrics have a weak correlation with compression rate, while others do not.

Original: Except one metric (FPR gap sum), the compression rate and the extrinsic metric correlate more after retraining.
Revised: Except one metric (FPR gap sum), the compression rate and the extrinsic metric correlate more after retraining.

Original: Figureillustrates this for TPR-gap (Pearson).
Revised: Figureillustrates this for TPR-gap (Pearson).

Original: The increase is due to superficial debiasing, especially by subsampling data, which prior to retraining had low extrinsic metrics and relatively high intrinsic metrics.
Revised: The increase is due to superficial debiasing, especially by subsampling data, which prior to retraining had low extrinsic metrics and relatively high intrinsic metrics.

Original: This shows that correlation between extrinsic metrics and compression rate for certain metrics is stronger than it appeared before retraining.
Revised: This shows that correlation between extrinsic metrics and compression rate for certain metrics is stronger than it appeared before retraining.

Original: It is unsurprising that CEAT does not correlate with any extrinsic metrics, since CEAT could not distinguish between different models.
Revised: It is unsurprising that CEAT does not correlate with any extrinsic metrics, since CEAT could not distinguish between different models.

Original: Coreference resolution shows stronger correlations between compression rate and extrinsic metrics, but low correlations between Pearson metrics.
Revised: Coreference resolution shows stronger correlations between compression rate and extrinsic metrics, but low correlations between Pearson metrics.

Original: We further discuss cases of no correlation in appendix D. Correlations decrease after retraining, but metrics that were highly correlated remain so ( > 0 .
Revised: We further discuss cases of no correlation in appendix D. Correlations decrease after retraining, but metrics that were highly correlated remain so ( > 0 .

Original: 7 after retraining).
Revised: 7 after retraining).

Original: The correlations are visualized for F1 difference metrics in Figure 4.
Revised: The correlations are visualized for F1 difference metrics in Figure 4.

Original: CEAT and extrinsic metrics correlate much less than compression rate (Table 2).
Revised: CEAT and extrinsic metrics correlate much less than compression rate (Table 2).

Original: Our results are in line with those of Goldfarb-Tarrant et al.
Revised: Our results are in line with those of Goldfarb-Tarrant et al.

Original: (2021), who found a lack of correlation between extrinsic metrics and
Revised: (2021), who found a lack of correlation between extrinsic metrics and

Original: WEAT, the static-embedded version of CEAT.
Revised: WEAT, the static-embedded version of CEAT.

Original: Given that recent work (Goldfarb-Tarrant et al., 2021; Cao et al., 2022) questions the validity ofintrinsic metrics as a reliable indicator for gender bias, the compression rate provides a reliable alternative to current intrinsic metrics, by offering correlation to many extrinsic bias metrics.
Revised: Given that recent work (Goldfarb-Tarrant et al., 2021; Cao et al., 2022) questions the validity of

Original: [Figure]
Revised: [Figure]

Revised: intrinsic metrics as a reliable indicator for gender bias, the compression rate provides a reliable alternative to current intrinsic metrics, by offering correlation to many extrinsic bias metrics.

Original:  Related Work
Revised:  Related Work

Original: There are few studies that examine both intrinsic and extrinsic metrics.
Revised: There are few studies that examine both intrinsic and extrinsic metrics.

Original: Previous work by GoldfarbTarrant et al.
Revised: Previous work by GoldfarbTarrant et al.

Original: (2021) showed that debiasing static embeddings intrinsically is not correlated with extrinsic bias, challenging the assumption that intrinsic metrics are predictive of bias.
Revised: (2021) showed that debiasing static embeddings intrinsically is not correlated with extrinsic bias, challenging the assumption that intrinsic metrics are predictive of bias.

Original: We examine the other direction, exploring how extrinsic debiasing affects intrinsic metrics.
Revised: We examine the other direction, exploring how extrinsic debiasing affects intrinsic metrics.

Original: We also extend beyond their work to contextualized embeddings, a wider range of extrinsic metrics, and a new, more effective intrinsic metric based on information-theoretic probing.
Revised: We also extend beyond their work to contextualized embeddings, a wider range of extrinsic metrics, and a new, more effective intrinsic metric based on information-theoretic probing.

Original: A contemporary work by Cao et al.
Revised: A contemporary work by Cao et al.

Original: (2022) measured the correlations between intrinsic and extrinsic metrics in contextualized settings across different language models.
Revised: (2022) measured the correlations between intrinsic and extrinsic metrics in contextualized settings across different language models.

Original: In contrast, our work examines the correlations across different versions of the same language model by fine-tuning it using various debiasing techniques.
Revised: In contrast, our work examines the correlations across different versions of the same language model by fine-tuning it using various debiasing techniques.

Original: Studies that inspect extrinsic metrics include either a challenge dataset curated to expose differences in model behavior by gender, or a test dataset labelled by gender.
Revised: Studies that inspect extrinsic metrics include either a challenge dataset curated to expose differences in model behavior by gender, or a test dataset labelled by gender.

Original: Among these datasets are Winobias (Zhao et al., 2018), Winogender (Rudinger et al., 2018) and GAP (Webster et al., 2018) for coreference resolution, WinoMT (Stanovsky et al., 2019) for machine translation, EEC (Kiritchenko and Mohammad, 2018) for sentiment analysis,
Revised: Among these datasets are Winobias (Zhao et al., 2018), Winogender (Rudinger et al., 2018) and GAP (Webster et al., 2018) for coreference resolution, WinoMT (Stanovsky et al., 2019) for machine translation, EEC (Kiritchenko and Mohammad, 2018) for sentiment analysis,

Original: BOLD (Dhamala et al., 2021) for language generation, gendered NLI (Sharma et al., 2020) for natural language inference and Bias in Bios (DeArteaga et al., 2019) for occupation prediction.
Revised: BOLD (Dhamala et al., 2021) for language generation, gendered NLI (Sharma et al., 2020) for natural language inference and Bias in Bios (DeArteaga et al., 2019) for occupation prediction.

Original: Studies that measure gender bias intrinsically in static word or sentence embeddings measure characteristics of the geometry, such as the proximity between female- and male-related words to stereotypical words, or how embeddings cluster or relate to a gender subspace (Bolukbasi et al., 2016; Caliskan et al., 2017; Gonen and Goldberg, 2019b; Ethayarajh et al., 2019).
Revised: Studies that measure gender bias intrinsically in static word or sentence embeddings measure characteristics of the geometry, such as the proximity between female- and male-related words to stereotypical words, or how embeddings cluster or relate to a gender subspace (Bolukbasi et al., 2016; Caliskan et al., 2017; Gonen and Goldberg, 2019b; Ethayarajh et al., 2019).

Original: However, metrics and debiasing methods for static embeddings do not apply directly to contextualized ones.
Revised: However, metrics and debiasing methods for static embeddings do not apply directly to contextualized ones.

Original: Several studies use sentence templates to adapt to contextual embeddings (May et al., 2019; Kurita et al., 2019; Tan and Celis, 2019).
Revised: Several studies use sentence templates to adapt to contextual embeddings (May et al., 2019; Kurita et al., 2019; Tan and Celis, 2019).

Original: This templated approach is difficult to scale, and lacks the range of representations that a contextual embedding offers.
Revised: This templated approach is difficult to scale, and lacks the range of representations that a contextual embedding offers.

Original: Other work extracts embedding representations of words from natural corpora (Zhao et al., 2019; Guo and Caliskan, 2021; Basta et al., 2019).
Revised: Other work extracts embedding representations of words from natural corpora (Zhao et al., 2019; Guo and Caliskan, 2021; Basta et al., 2019).

Original: These studies often adapt the WEAT method (Caliskan et al., 2017), which measures embedding geometry.
Revised: These studies often adapt the WEAT method (Caliskan et al., 2017), which measures embedding geometry.

Original: None measure the effect of the presumably found “bias” on a downstream task.
Revised: None measure the effect of the presumably found “bias” on a downstream task.

Original: There is a growing conversation in the field (Barocas et al., 2017; Kate Crawford, 2017; Blodgett et al., 2020; Bommasani et al., 2021) about the importance of articulating the harms of measured bias.
Revised: There is a growing conversation in the field (Barocas et al., 2017; Kate Crawford, 2017; Blodgett et al., 2020; Bommasani et al., 2021) about the importance of articulating the harms of measured bias.

Original: In general, extrinsic metrics have clear, interpretable impacts for which harm can be defined.
Revised: In general, extrinsic metrics have clear, interpretable impacts for which harm can be defined.

Original: Intrinsic metrics have an unclear effect.
Revised: Intrinsic metrics have an unclear effect.

Original: Without evidence from a concrete downstream task, a found intrinsic bias is only theoretically harmful.
Revised: Without evidence from a concrete downstream task, a found intrinsic bias is only theoretically harmful.

Original: Our work is a step towards understanding whether intrinsic metrics provide valuable insights about bias in a model.
Revised: Our work is a step towards understanding whether intrinsic metrics provide valuable insights about bias in a model.

Original:  Discussion and Conclusions
Revised:  Discussion and Conclusions

Original: This study examined whether bias in internal representations is related to extrinsic bias.
Revised: This study examined whether bias in internal representations is related to extrinsic bias.

Original: We designed a new framework in which we debias a model on a downstream task, and measure its intrinsic bias.
Revised: We designed a new framework in which we debias a model on a downstream task, and measure its intrinsic bias.

Original: We found that gender extractability from internal representations, measured by compression rate via
Revised: We found that gender extractability from internal representations, measured by compression rate via

Original: MDL probing, reflects bias in a model.
Revised: MDL probing, reflects bias in a model.

Original: Compression was much more reliable than an alternative intrinsic metric for contextualised representations,
Revised: Compression was much more reliable than an alternative intrinsic metric for contextualised representations,

Original: CEAT.
Revised: CEAT.

Original: Compression correlated well—to varying degrees—with many extrinsic metrics.
Revised: Compression correlated well—to varying degrees—with many extrinsic metrics.

Original: We thus encourage NLP practitioners to use compression as an intrinsic indicator for gender bias in NLP models.
Revised: We thus encourage NLP practitioners to use compression as an intrinsic indicator for gender bias in NLP models.

Original: When comparing two alternative models, a lower compression rate provides confidence in a model’s superiority in terms of gender bias.
Revised: When comparing two alternative models, a lower compression rate provides confidence in a model’s superiority in terms of gender bias.

Original: The relative success of compression over CEAT may be explained by the fact that the compression rate was calculated on the same dataset as the extrinsic metrics.
Revised: The relative success of compression over CEAT may be because the compression rate was calculated on the same dataset as the extrinsic metrics, whereas

Original: CEAT, on the other hand, was measured on a general datasetwhich is not necessarily related to a speicifc downstream task.
Revised: CEAT was measured on a different dataset not necessarily aligned with a specific downstream task.

Original: The use of a general dataset is a common strategy among other intrinsic metrics (May et al., 2019; Kurita et al., 2019; Basta et al., 2021).
Revised: The use of a non-task-aligned dataset is a common strategy among other intrinsic metrics (May et al., 2019; Kurita et al., 2019; Basta et al., 2021).

Original: Another  explanation is thatthe compression rate measures a more focused concept, namely the gender information within the internal representations.
Revised: Another possible explanation is that compression rate measures a more focused concept, namely the gender information within the internal representations.

Original: CEAT measures proximity in general terms that may include other social contexts that do not directly relate to gender.
Revised: CEAT measures proximity among embeddings of general terms that may include other social contexts that do not directly relate to gender (e.g. a female term like ‘lady’ or ‘Sarah’ contains information about not just gender but class, culture, formality, etc, and it can be hard to isolate just one of these from the rest).

Original: Our results show that when a debiasing method reduces extrinsic metrics but not compression, it indicates that the language model remains biased.
Revised: Our results show that when a debiasing method reduces extrinsic metrics but not compression, it indicates that the language model remains biased.

Original: When such superficial debiasing occurs, the debiased language model may be reapplied to another task, as in Jin et al.
Revised: When such superficial debiasing occurs, the debiased language model may be reapplied to another task, as in Jin et al.

Original: (2021), resulting in unexpected biases and nullifying the supposed debiasing.
Revised: (2021), resulting in unexpected biases and nullifying the supposed debiasing.

Original: Our findings suggest that practitioners of NLP should take special care when adopting previously debiased models and inspect them carefully, perhaps using our framework.
Revised: Our findings suggest that practitioners of NLP should take special care when adopting previously debiased models and inspect them carefully, perhaps using our framework.

Original: In another study (Mendelson and Belinkov, 2021a) it was found– for a different type of bias which is not gender bias – thatas the model is less biased, the biased features are more extractable as shown bythe compression rate on various probing tasks.
Revised: Our results differ from those of Mendelson and Belinkov (2021a), who found that the debiasing increases bias extractability as measured by compression rate. However, they studied different, non-social biases, that arise from spurious or unintended correlations in training datasets (often called dataset biases).

Original: Analysis of our results showed that the change in internal representation depends on the choice of debiasing strategy - some  strategies increase intrinsic bias and others decrease it. 
Revised: In our case, some debiasing strategies increase intrinsic bias while others decrease it. Future work could investigate why debiasing affects extractability differently for these two types of biases.

Original: Our work also highlighted the importance of the classification layer.
Revised: Our work also highlighted the importance of the classification layer.

Original: Using a debiased objective, such as a balanced dataset, the classification layer can provide significant debiasing.
Revised: Using a debiased objective, such as a balanced dataset, the classification layer can provide significant debiasing.

Original: This holds even if the internal representations are biased and the classifier is a single linear layer, as shown in the occupation prediction task.
Revised: This holds even if the internal representations are biased and the classifier is a single linear layer, as shown in the occupation prediction task.

Original: Bias stems in part from internal LM bias and in part from classification bias.
Revised: Bias stems in part from internal LM bias and in part from classification bias.

Original: Practitioners should focus their efforts on both parts when attempting to debias a model.
Revised: Practitioners should focus their efforts on both parts when attempting to debias a model.

Original: We used a broader set of extrinsic metrics than is typically used, and found that the bias metrics behaved differently: some decreased more than others after debiasing, and they correlated differently with compression rate.
Revised: We used a broader set of extrinsic metrics than is typically used, and found that the bias metrics behaved differently: some decreased more than others after debiasing, and they correlated differently with compression rate.

Original: Debiasing efforts may not be fully understood by testing only a few extrinsic metrics.
Revised: Debiasing efforts may not be fully understood by testing only a few extrinsic metrics.

Original: However, compression as an intrinsic bias metric can indicate meaningful debiasing of internal model representations even when not all metrics are easily measurable, since it correlates well with many extrinsic metrics.
Revised: However, compression as an intrinsic bias metric can indicate meaningful debiasing of internal model representations even when not all metrics are easily measurable, since it correlates well with many extrinsic metrics.

Original: A major limitation of this study is the use of gender as a binary variable, which is trans-exclusive.
Revised: A major limitation of this study is the use of gender as a binary variable, which is trans-exclusive.

Original: Cao and Daumé III (2020) made the first steps towards inclusive gender bias evaluation in NLP, revealing that coreference systems fail on genderinclusive text.
Revised: Cao and Daumé III (2020) made the first steps towards inclusive gender bias evaluation in NLP, revealing that coreference systems fail on genderinclusive text.

Original: Further work is required to adjust our framework to non-binary genders, potentially revealing insights about the poor performance of
Revised: Further work is required to adjust our framework to non-binary genders, potentially revealing insights about the poor performance of

Original: NLP systems in that area.
Revised: NLP systems in that area.

Original: Acknowledgements
Revised: Acknowledgements

Original: This research was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 448/20) and by an Azrieli Foundation Early Career Faculty Fellowship.
Revised: This research was supported by the ISRAEL SCIENCE FOUNDATION (grant No. 448/20) and by an Azrieli Foundation Early Career Faculty Fellowship.

Original: We also thank Kate McCurdy and Andreas
Revised: We also thank Kate McCurdy and Andreas

Original: Grivas for comments on early drafts, the members of the Technion CS NLP group for their valuable feedback, and the anonymous reviewers for their useful suggestions.
Revised: Grivas for comments on early drafts, the members of the Technion CS NLP group for their valuable feedback, and the anonymous reviewers for their useful suggestions.

Original: A Implementation Details
Revised: A Implementation Details

Original: We used RoBERTa in all models (base size, 120M parameters).
Revised: We used RoBERTa in all models (base size, 120M parameters).

Original: We use following random seeds in all repeated experiments: 0, 5, 11, 26, 42, 46, 50, 63, 83, 90.
Revised: We use following random seeds in all repeated experiments: 0, 5, 11, 26, 42, 46, 50, 63, 83, 90.

Original: Our code was implemented mainly using the Python libraries Pytorch (Paszke et al., 2019),
Revised: Our code was implemented mainly using the Python libraries Pytorch (Paszke et al., 2019),

Original: Transformers (Wolf et al., 2020), Sklearn (Pedregosa et al., 2011), and the experiments were logged using Wandb (Biewald, 2020).
Revised: Transformers (Wolf et al., 2020), Sklearn (Pedregosa et al., 2011), and the experiments were logged using Wandb (Biewald, 2020).

Original: A. Occupation Classification
Revised: A. Occupation Classification

Original: We fine-tuned a RoBERTa-base model with a linear classification layer on top.
Revised: We fine-tuned a RoBERTa-base model with a linear classification layer on top.

Original: Training was done for 10 epochs at a learning rate of 5e-5, batch size of 64.
Revised: Training was done for 10 epochs at a learning rate of 5e-5, batch size of 64.

Original: The input to RoBERTa was the biography tokens, which is limited to the first 128 tokens.
Revised: The input to RoBERTa was the biography tokens, which is limited to the first 128 tokens.

Original: The resulting [CLS] token embedding is fed to the classifier to predict the occupation.
Revised: The resulting [CLS] token embedding is fed to the classifier to predict the occupation.

Original: The probing task involves using the same [CLS] token and training the probing classifier to predict the gender of the person in the biography.
Revised: The probing task involves using the same [CLS] token and training the probing classifier to predict the gender of the person in the biography.

Original: The experiments without fine-tuning included either a pre-trained or a previously fine-tuned RoBERTa.
Revised: The experiments without fine-tuning included either a pre-trained or a previously fine-tuned RoBERTa.

Original: We first extracted the pre-trained RoBERTa’s embeddings of tokens from the [CLS] and then trained a linear classifier on them.
Revised: We first extracted the pre-trained RoBERTa’s embeddings of tokens from the [CLS] and then trained a linear classifier on them.

Original: The learning rate was 0.001 and the batch size was 64.
Revised: The learning rate was 0.001 and the batch size was 64.

Original: We trained the classification layer with pre-trained RoBERTa on 300 epochs, but with fine-tuned RoBERTa, 10 epochs were sufficient.
Revised: We trained the classification layer with pre-trained RoBERTa on 300 epochs, but with fine-tuned RoBERTa, 10 epochs were sufficient.

Original: For all training processes, the epoch with the greatest validation accuracy was saved.
Revised: For all training processes, the epoch with the greatest validation accuracy was saved.

Original: Finetuning took 7 hours on a GeForce RTX 

Revised: Finetuning took 7 hours on a GeForce RTX

Original: Ti
Revised: Ti

Original: GPU.
Revised: GPU.

Original: Bias in Bios contains almost 400k biographies, and we obtain validation (10%) and test set (25%) by splitting with Scikit-learn’s (Pedregosa et al., 2011) test_train_split with our random seeds.
Revised: Bias in Bios contains almost 400k biographies, and we obtain validation (10%) and test set (25%) by splitting with Scikit-learn’s (Pedregosa et al., 2011) test_train_split with our random seeds.

Original: A. Coreference Resolution
Revised: A. Coreference Resolution

Original: We use the implementation of Xu and Choi (2020), a model that was introduced by Lee et al.
Revised: We use the implementation of Xu and Choi (2020), a model that was introduced by Lee et al.

Original: (2018b)
Revised: (2018b)

Original: and has been adopted by many coreference resolution models.
Revised: and has been adopted by many coreference resolution models.

Original: Coreference resolution is the process of clustering different mentions in a text that refer to the same real-world entities.
Revised: Coreference resolution is the process of clustering different mentions in a text that refer to the same real-world entities.

Original: The task is solved by detecting mentions through text spans and then predicting for each pair of spans if they represent the same entity.
Revised: The task is solved by detecting mentions through text spans and then predicting for each pair of spans if they represent the same entity.

Original: The span representations were extracted with a RoBERTa model, which is fine-tuned throughout the training process, except in the retraining experiment.
Revised: The span representations were extracted with a RoBERTa model, which is fine-tuned throughout the training process, except in the retraining experiment.

Original: Fine-tuning took 3 hours on an NVIDIA RTX A6000 GPU.
Revised: Fine-tuning took 3 hours on an NVIDIA RTX A6000 GPU.

Original: Ontonotes 5.0 has 625k sentences and we use the standard validation and test splits.
Revised: Ontonotes 5.0 has 625k sentences and we use the standard validation and test splits.

Original: A. Probing Classifier
Revised: A. Probing Classifier

Original: We use the MDL probe (Voita and Titov, 2020) implementation by Mendelson and Belinkov (2021b).
Revised: We use the MDL probe (Voita and Titov, 2020) implementation by Mendelson and Belinkov (2021b).

Original: In all experiments, we use a linear probe and train it with a batch size of 16 and a learning rate of 1e-3.
Revised: In all experiments, we use a linear probe and train it with a batch size of 16 and a learning rate of 1e-3.

Original: The timestamps used, meaning the accumulating fractions of data that the probe is trained on, are 2.0%, 3.0%, 4.4%, 6.5%, 9.5%, 14.0%, 21.0%, 31.0%, 45.7%, 67.6%, 100%.
Revised: The timestamps used, meaning the accumulating fractions of data that the probe is trained on, are 2.0%, 3.0%, 4.4%, 6.5%, 9.5%, 14.0%, 21.0%, 31.0%, 45.7%, 67.6%, 100%.

Original: A. Metrics A.4. Fairness-Based Metrics Implementation
Revised: A. Metrics A.4. Fairness-Based Metrics Implementation

Original: All three statistical fairness metrics measure the difference between two probability distributions, where this difference describes a notion of bias.
Revised: All three statistical fairness metrics measure the difference between two probability distributions, where this difference describes a notion of bias.

Original: We calculate Independence and Separation via
Revised: We calculate Independence and Separation via

Original: Kullback–Leibler (KL) divergence, using the AllenNLP implementation ( https://github.com/ allenai/allennlp ).
Revised: Kullback–Leibler (KL) divergence, using the AllenNLP implementation ( https://github.com/ allenai/allennlp ).

Original: We calculate Sufficiency via
Revised: We calculate Sufficiency via

Original: Wasserstein distance instead, which is motivated by Kwegyir-Aggrey et al.
Revised: Wasserstein distance instead, which is motivated by Kwegyir-Aggrey et al.

Original: In this case, we cannot use KL divergence, since there are some classes that do not occur in model predictions for both male and female genders.
Revised: In this case, we cannot use KL divergence, since there are some classes that do not occur in model predictions for both male and female genders.

Original: This causes the probability distributions to not have the same support, and KL divergence is unbounded.
Revised: This causes the probability distributions to not have the same support, and KL divergence is unbounded.

Original: Wasserstein distance lacks the requirement for equal support
Revised: Wasserstein distance lacks the requirement for equal support

Original: . A.4. Classification Metrics Interpretation in Winobias
Revised: . A.4. Classification Metrics Interpretation in Winobias

Original: Winobias datasets contain pairs of stereotypical and anti-stereotypical sentences.
Revised: Winobias datasets contain pairs of stereotypical and anti-stereotypical sentences.

Original: The stereotypes are derived from the US labor statistics (for instance, a profession with a majority of males is stereotypically male).
Revised: The stereotypes are derived from the US labor statistics (for instance, a profession with a majority of males is stereotypically male).

Original: Since coreference resolution is viewed as a clustering problem, it is usually measured via clustering evaluation metrics.
Revised: Since coreference resolution is viewed as a clustering problem, it is usually measured via clustering evaluation metrics.

Original: Coreference resolution is commonly measured as the average F1 score of these, and the same is true for Winobias.
Revised: Coreference resolution is commonly measured as the average F1 score of these, and the same is true for Winobias.

Original: Nevertheless, coreference resolution is accomplished by making a prediction for each pair of mentions, so it can be seen as a classification task.
Revised: Nevertheless, coreference resolution is accomplished by making a prediction for each pair of mentions, so it can be seen as a classification task.

Original: Winobias can be viewed as a simpler task than general coreference resolution, as it contains exactly two mentions of professions and one pronoun, which refers to exactly one profession.
Revised: Winobias can be viewed as a simpler task than general coreference resolution, as it contains exactly two mentions of professions and one pronoun, which refers to exactly one profession.

Original: Therefore, we reframe it as a classification problem.
Revised: Therefore, we reframe it as a classification problem.

Original: In a Winobias sentence with two professions x and y , as well as a pronoun p , where p is referring to x , a true positive would be to cluster x and p together, while a false positive would be to cluster y and p together.
Revised: In a Winobias sentence with two professions x and y , as well as a pronoun p , where p is referring to x , a true positive would be to cluster x and p together, while a false positive would be to cluster y and p together.

Original: Our classification metrics are derived based on these definitions.
Revised: Our classification metrics are derived based on these definitions.

Original: For instance, the TPR gap for profession “teacher”, which is a stereotypical female occupation, is the TPR rate on pro-stereotypical sentences (with a female pronoun) minus the TPR rate on anti-stereotypical sentences (with a male pronoun).
Revised: For instance, the TPR gap for profession “teacher”, which is a stereotypical female occupation, is the TPR rate on pro-stereotypical sentences (with a female pronoun) minus the TPR rate on anti-stereotypical sentences (with a male pronoun).

Original: A.4. CEAT
Revised: A.4. CEAT

Original: The Word Embedding Association Test (WEAT) developed by (Caliskan et al., 2017) is a method for evaluating bias in static word embeddings.
Revised: The Word Embedding Association Test (WEAT) developed by (Caliskan et al., 2017) is a method for evaluating bias in static word embeddings.

Original: The test is defined as follows: given two sets of target words X , Y (e.g., ’executive’, ’management’, ’professional’ and ’home’, ’parents’, ’children’) and two sets of attribute words (e.g., male names and female names), and using ⃗w to represent the word embedding for word w , the effect size is:
Revised: The test is defined as follows: given two sets of target words X , Y (e.g., ’executive’, ’management’, ’professional’ and ’home’, ’parents’, ’children’) and two sets of attribute words (e.g., male names and female names), and using ⃗w to represent the word embedding for word w , the effect size is:

Original: [Equation]
Revised: [Equation]

Original: where s
Revised: where s

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: In essence, the effect size measures how different are the distances between the embedding vectors of each target group and the attribute groups.
Revised: In essence, the effect size measures how different are the distances between the embedding vectors of each target group and the attribute groups.

Original: Specifically, if s ( x, A , B ) > 0 , ⃗ x is more similar to attribute words B and vice versa.
Revised: Specifically, if s ( x, A , B ) > 0 , ⃗ x is more similar to attribute words B and vice versa.

Original: For instance, a larger effect size is observed if target words X are more similar to attribute words A and target words Y are more similar to attribute words
Revised: For instance, a larger effect size is observed if target words X are more similar to attribute words A and target words Y are more similar to attribute words

Original: B .
Revised: B .

Original: | ES | > 0 .
Revised: | ES | > 0 .

Original: 5 and | ES | > 0 .
Revised: 5 and | ES | > 0 .

Original: 8 are considered medium and large effect sizes, respectively (Rice and Harris, 2005).
Revised: 8 are considered medium and large effect sizes, respectively (Rice and Harris, 2005).

Original: The null hypothesis holds that there is no difference between the two sets of target words in terms of their relative similarity to the two sets of attribute words, indicating that there are no biased associations.
Revised: The null hypothesis holds that there is no difference between the two sets of target words in terms of their relative similarity to the two sets of attribute words, indicating that there are no biased associations.

Original: Statistical significance is defined by the p-value of WEAT, which reflects the probability of observing the effect size under the null hypothesis.
Revised: Statistical significance is defined by the p-value of WEAT, which reflects the probability of observing the effect size under the null hypothesis.

Original: Since a word can take on a great variety of vector representations in a contextual setting, ES varies according to the sentences used to extract word representation.
Revised: Since a word can take on a great variety of vector representations in a contextual setting, ES varies according to the sentences used to extract word representation.

Original: Thus, to adopt WEAT to contextualized representations, the Combined Effect Size (CES) (Guo and Caliskan, 2021) is derived as the distribution of WEAT effect sizes over many possible contextual word representations:
Revised: Thus, to adopt WEAT to contextualized representations, the Combined Effect Size (CES) (Guo and Caliskan, 2021) is derived as the distribution of WEAT effect sizes over many possible contextual word representations:

Original: [Equation]
Revised: [Equation]

Original: where ES i denotes the WEAT effect size of the i ’th choice of word representations from a large corpus, and v i is the inverse of the sum of in-sample variance V i and between-sample variance in the distribution of random-effects.
Revised: where ES i denotes the WEAT effect size of the i ’th choice of word representations from a large corpus, and v i is the inverse of the sum of in-sample variance V i and between-sample variance in the distribution of random-effects.

Original: As in Guo and Caliskan (2021), the representation for each word is derived from 10,000 random sentences extracted from a corpus of Reddit comments.
Revised: As in Guo and Caliskan (2021), the representation for each word is derived from 10,000 random sentences extracted from a corpus of Reddit comments.

Original: The combined effect size of each of the models is examined on WEAT stimulus 6, which contains target words of career/family and attribute words of male/female names.
Revised: The combined effect size of each of the models is examined on WEAT stimulus 6, which contains target words of career/family and attribute words of male/female names.

Original: This was the only one that detected bias on a pre-trained RoBERTa (CES close to 0.5 and p < 0 . 05 ).
Revised: This was the only one that detected bias on a pre-trained RoBERTa (CES close to 0.5 and p < 0 . 05 ).

Original: The points that we kept in our analysis are those where p < 0 .
Revised: The points that we kept in our analysis are those where p < 0 .

Original: 05 , which make up 90% of the points in occupation prediction and 95% of the points in coreference resolution.
Revised: 05 , which make up 90% of the points in occupation prediction and 95% of the points in coreference resolution.

Original: B Full Results
Revised: B Full Results

Original: In this section we provide the full results of a
Revised: In this section we provide the full results of a

Original: RoBERTa model trained on the downstream task.
Revised: RoBERTa model trained on the downstream task.

Original: The results for the occupation prediction task after fine-tuning are presented in Table 3 and Table 4 presents the retrained model results.
Revised: The results for the occupation prediction task after fine-tuning are presented in Table 3 and Table 4 presents the retrained model results.

Original: Figureillustrates the correlations between extrinsic metrics and compression rate before and after retraining.
Revised: Figureillustrates the correlations between extrinsic metrics and compression rate before and after retraining.

Original: Table 5 presents the complete results of the model trained without fine-tuning, meaning that the RoBERTa model is the pretrained version from Liu et al.
Revised: Table 5 presents the complete results of the model trained without fine-tuning, meaning that the RoBERTa model is the pretrained version from Liu et al.

Original: and only the classification layer was updated.
Revised: and only the classification layer was updated.

Original: Subsampling the dataset has significant debiasing effects, which suggests that this debiasing method can achieve low extrinsic bias even when internal bias exists.
Revised: Subsampling the dataset has significant debiasing effects, which suggests that this debiasing method can achieve low extrinsic bias even when internal bias exists.

Original: Table 6 presents the results after we performed the experiments on a DeBERTa  (He et al., 2020) model, with occupation classification task.
Revised: Table 6 presents the results using a DeBERTa model (He et al., 2020), for the occupation classification task.

Original: The trends are similar to those of RoBERTa, with the same metrics showing an increase, no change, or decrease in correlation after re-training, suggesting a general trend in the behavior of these metrics in relation to internal model representations.
Revised: The trends are similar to those of RoBERTa, with the same metrics showing an increase, no change, or decrease in correlation after re-training, suggesting a general trend in the behavior of these metrics in relation to internal model representations.

Original: Regarding the coreference resolution task, Table 7 displays the results on a finetuned model and
Revised: Regarding the coreference resolution task, Table 7 displays the results on a finetuned model and

Original: Table 8 displays the retraining results.
Revised: Table 8 displays the retraining results.

Original: Figureshows the correlations between compression rate and extrinsic metrics before and after the retraining.
Revised: Figureshows the correlations between compression rate and extrinsic metrics before and after the retraining.

Original: Debiasing Strategy
Revised: Debiasing Strategy

Original: [Table]
Revised: [Table]

Original: Debiasing
Revised: Debiasing

Original: [Table]

Original: [Table]
Revised: [Table]

Original: R 2 Compression

Original: Metric

Original: Before

Original: After

Original: TPR gap (P) 0. TPR gap (S) 0. FPR gap (P) 0. FPR gap (S)
Revised: Debiasing Strategy

Original: [Table]
Revised: [Table]

Original: Precision gap (P)

Original: [Table]
Revised: [Table]

Original: Precision gap (S)
Revised: After

Original: [Table]
Revised: [Table]

Original: Independence gap (S) 0.

Original: Separation gap (S) 0.

Original: Sufficiency gap (S) 0.

Original: Debiasing Strategy
Revised: Debiasing Strategy

Original: [Table]
Revised: [Table]

Original: Independence
Revised: Independence

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Separation gap
Revised: Separation gap

Original: [Table]
Revised: [Table]

Original: 1.247 ±
Revised: 1.247 ±

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 344 ±
Revised: 344 ±

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Sufficiency gap
Revised: Sufficiency gap

Original: [Table]
Revised: [Table]

Original: S)
Revised: S)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Compression
Revised: Compression

Original: [Table]
Revised: [Table]

Original: F1 (Ontonotes test)
Revised: F1 (Ontonotes test)

Original: [Table]
Revised: [Table]

Original: F1 diff ( pro − anti ) 6.072 ± 0.
Revised: F1 diff ( pro − anti ) 6.072 ± 0.

Original: 7.417* ± 1.
Revised: 7.417* ± 1.

Original: 3.674* ±
Revised: 3.674* ±

Original: [Table] [Table]
Revised: [Table] [Table]

Original: TPR gap (P) 0.635 ± 0.
Revised: TPR gap (P) 0.635 ± 0.

Original: 0.688* ± 0.
Revised: 0.688* ± 0.

Original: 0.679* ± 0.
Revised: 0.679* ± 0.

Original: 0.654 ± 0.
Revised: 0.654 ± 0.

Original: TPR gap (S) 4.561 ± 0.
Revised: TPR gap (S) 4.561 ± 0.

Original: 5.143* ± 0.
Revised: 5.143* ± 0.

Original: 2.590* ± 0.
Revised: 2.590* ± 0.

Original: 2.178 * ± 0.
Revised: 2.178 * ± 0.

Original: FPR gap (P) 0.579 ± 0.
Revised: FPR gap (P) 0.579 ± 0.

Original: 0.637* ± 0.
Revised: 0.637* ± 0.

Original: 0.620* ± 0.
Revised: 0.620* ± 0.

Original: 0.692* ± 0.
Revised: 0.692* ± 0.

Original: FPR gap (S) 0.113 ± 0.
Revised: FPR gap (S) 0.113 ± 0.

Original: 0.126* ± 0.
Revised: 0.126* ± 0.

Original: 0.063* ± 0.
Revised: 0.063* ± 0.

Original: 0.052 * ± 0.
Revised: 0.052 * ± 0.

Original: Precision gap (P) 0.
Revised: Precision gap (P) 0.

Original: [Table]
Revised: [Table]

Revised: of

Revised: Pearson are discussed in

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C Why is scrubbing not as effective as subsampling?
Revised: C Why is scrubbing not as effective as subsampling?

Original: The debiasing method of subsampling significantly reduced external biases in the occupation prediction task.
Revised: The debiasing method of subsampling significantly reduced external biases in the occupation prediction task.

Original: Although compression rates show that scrubbing reduced more gender information, subsampling outperforms it as a debiasing method.
Revised: Although compression rates show that scrubbing reduced more gender information, subsampling outperforms it as a debiasing method.

Original: We find that in spite of the scrubbing, a probe is able to correctly identify the gender from an internal representation with 68.8% accuracy compared to 90.7% on the original, non-scrubbed data.
Revised: We find that in spite of the scrubbing, a probe is able to correctly identify the gender from an internal representation with 68.8% accuracy compared to 90.7% on the original, non-scrubbed data.

Original: This means that although the scrubbing process reduces extrinsic bias significantly, gender information is still embedded in the [CLS] token embeddings.
Revised: This means that although the scrubbing process reduces extrinsic bias significantly, gender information is still embedded in the [CLS] token embeddings.

Original: To investigate the source of gender information after scrubbing, we use logistic regression (LR) model to predict the gender from the Bag-of-Words of the scrubbed biographies.
Revised: To investigate the source of gender information after scrubbing, we use logistic regression (LR) model to predict the gender from the Bag-of-Words of the scrubbed biographies.

Original: We perform an iterative process for automatic extra scrubbing: in each iteration we (1) train a LR model for gender prediction (2) scrub the n most significant words for each gender according to the LR weights.
Revised: We perform an iterative process for automatic extra scrubbing: in each iteration we (1) train a LR model for gender prediction (2) scrub the n most significant words for each gender according to the LR weights.

Original: The most relevant words among 5 seeds of training with n= words scrubbed per iteration are displayed in Table 9.
Revised: The most relevant words among 5 seeds of training with n= words scrubbed per iteration are displayed in Table 9.

Original: The model learns indirect correlations to gender in the absence of explicit gendered words.
Revised: The model learns indirect correlations to gender in the absence of explicit gendered words.

Original: Because the significant words are related to male- or femaledominated professions, we conducted the process on a specific profession.
Revised: Because the significant words are related to male- or femaledominated professions, we conducted the process on a specific profession.

Original: Table 10 presents the most significant words for biographies of nurses.
Revised: Table 10 presents the most significant words for biographies of nurses.

Original: There are differences in wording even between females and males in the same profession.
Revised: There are differences in wording even between females and males in the same profession.

Original: The results of this study are in line with the results of other studies that have been conducted on the way biographies are written for men and women (Wagner et al., 2016; Sun and Peng, 2021).
Revised: The results of this study are in line with the results of other studies that have been conducted on the way biographies are written for men and women (Wagner et al., 2016; Sun and Peng, 2021).

Original: Subsampling is therefore more effective even when gender information is present since it prevents the model from learning correlations between gender information and a profession whereas scrubbing only attempts to remove gender indicators without removing correlations.
Revised: Subsampling is therefore more effective even when gender information is present since it prevents the model from learning correlations between gender information and a profession whereas scrubbing only attempts to remove gender indicators without removing correlations.

Original: On the other hand, it is possible that oversampling is less effective for debiasing since seeing more non-unique examples an unrepresented group encourages learning correlations.
Revised: On the other hand, it is possible that oversampling is less effective for debiasing since seeing more non-unique examples an unrepresented group encourages learning correlations.

Original: D A closer look into no-correlation cases D. Occupation Prediction
Revised: D A closer look into no-correlation cases D. Occupation Prediction

Original: Although compression has the ability to identify bias in most cases, some metrics still show little or no correlation with compression rate.
Revised: Although compression has the ability to identify bias in most cases, some metrics still show little or no correlation with compression rate.

Original: These results suggest that gender information comprises only one facet of embedded bias in the representations.
Revised: These results suggest that gender information comprises only one facet of embedded bias in the representations.

Original: Other factors that may influence these metrics are not considered or measured, such as the connection between a name and a profession.
Revised: Other factors that may influence these metrics are not considered or measured, such as the connection between a name and a profession.

Original: For example, as can be see in Tables 3 and 4,
Revised: For example, as can be see in Tables 3 and 4,

Original: LMs finetuned on subsampled data have the largest
Revised: LMs finetuned on subsampled data have the largest

Original: FPR gaps after retraining, despite being the least biased before retraining, while those finetuned on oversampled data have the next-to-lowest FPR gaps after retraining.
Revised: FPR gaps after retraining, despite being the least biased before retraining, while those finetuned on oversampled data have the next-to-lowest FPR gaps after retraining.

Original: The information encoded in the internal representations may have been encoded in a manner that allowed the classification layer to exhibit a smaller FPR gap when trained on a balanced dataset.
Revised: The information encoded in the internal representations may have been encoded in a manner that allowed the classification layer to exhibit a smaller FPR gap when trained on a balanced dataset.

Original: However, when the classification layer was retrained on biased training data, it used the same features to make biased predictions.
Revised: However, when the classification layer was retrained on biased training data, it used the same features to make biased predictions.

Original: D. Coreference Resolution
Revised: D. Coreference Resolution

Original: The cases where there is no correlation between our intrinsic metric and an extrinsic metric are the cases where the metric is based on Pearson correlation.
Revised: The cases where there is no correlation between our intrinsic metric and an extrinsic metric are the cases where the metric is based on Pearson correlation.

Original: Unlike occupation prediction, coreference resolution seems to exhibit no correlation between those metrics and compression rate.
Revised: Unlike occupation prediction, coreference resolution seems to exhibit no correlation between those metrics and compression rate.

Original: These metrics are computed as the Pearson correlation between a performance gap for a specific profession and the percentage of women in that profession, however the percentages are computed differently in each task: in occupation prediction, the percentages are computed from the train set, focusing on the representation each gender has in the data.
Revised: These metrics are computed as the Pearson correlation between a performance gap for a specific profession and the percentage of women in that profession, however the percentages are computed differently in each task: in occupation prediction, the percentages are computed from the train set, focusing on the representation each gender has in the data.

Original: In Winobias, the percentages are taken from the US labor statistics, and are unrelated to the training dataset statistics.
Revised: In Winobias, the percentages are taken from the US labor statistics, and are unrelated to the training dataset statistics.

Original: We note that the two statistics can be different - the real-world representation of women in a profession does not have to be equal to their representation in written text (Suresh and Guttag, 2021).
Revised: We note that the two statistics can be different - the real-world representation of women in a profession does not have to be equal to their representation in written text (Suresh and Guttag, 2021).

Original: We thus decided to test what happens if we change the statistics used in Winobias to dataset statistics, but Ontonotes 5.0 has very little representation to each profession and the statistics extracted from it would not be reliable.
Revised: We thus decided to test what happens if we change the statistics used in Winobias to dataset statistics, but Ontonotes 5.0 has very little representation to each profession and the statistics extracted from it would not be reliable.

Original: We thus took a different approach and computed the Pearson correlations for occupation prediction with real world statistics instead of dataset statistics.
Revised: We thus took a different approach and computed the Pearson correlations for occupation prediction with real world statistics instead of dataset statistics.

Original: To do this, we mapped the professions appearing in this dataset to professions from the US labor statistics, and dropped those who could no be mapped (6 out of 29 of the professions which is 21.4%).
Revised: To do this, we mapped the professions appearing in this dataset to professions from the US labor statistics, and dropped those who could no be mapped (6 out of 29 of the professions which is 21.4%).

Original: We then repeated all experiments on the Pearson metrics using these statistics.
Revised: We then repeated all experiments on the Pearson metrics using these statistics.

Original: Figure 7 shows the results.
Revised: Figure 7 shows the results.

Original: Correlations are very different when computed with respect to real-world statistics.
Revised: Correlations are very different when computed with respect to real-world statistics.

Original: TPR-gap has no correlation at all although it had with training data statistics, the correlation for FPR-gap after retraining exists but is negative, and the correlation with precision-gap does not exist after retraining.
Revised: TPR-gap has no correlation at all although it had with training data statistics, the correlation for FPR-gap after retraining exists but is negative, and the correlation with precision-gap does not exist after retraining.

Original: We thus conclude that the Pearson metrics are less reliable as they are heavily dependent on the statistics with respect to which they are calculated.
Revised: We thus conclude that the Pearson metrics are less reliable as they are heavily dependent on the statistics with respect to which they are calculated.