Original: D EBIASING P RETRAINED T EXT E NCODERS BY P AYING A TTENTION TO P AYING A TTENTION
Revised: D EBIASING P RETRAINED T EXT E NCODERS BY P AYING A TTENTION TO P AYING A TTENTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent studies in fair Representation Learning have observed a strong inclination for Natural Language Processing (NLP) models to exhibit discriminatory stereotypes across gender, religion, race and many such social constructs.
Revised: Recent studies in fair Representation Learning have observed a strong inclination for Natural Language Processing (NLP) models to exhibit discriminatory stereotypes across gender, religion, race and many such social constructs.

Original: In comparison to the progress made in reducing bias from static word embeddings, fairness in sentence-level text encoders received little consideration despite their wider applicability in contemporary NLP tasks.
Revised: In comparison to the progress made in reducing bias from static word embeddings, fairness in sentence-level text encoders received little consideration despite their wider applicability in contemporary NLP tasks.

Original: In this paper, we propose a debiasing method for pre-trained text encoders that both reduces social stereotypes, and inﬂicts next to no semantic damage.
Revised: In this paper, we propose a debiasing method for pre-trained text encoders that both reduces social stereotypes, and inﬂicts next to no semantic damage.

Original: Unlike previous studies that directly manipulate the embeddings, we suggest to dive deeper into the operation of these encoders, and pay more attention to the way they pay attention to different social groups.
Revised: Unlike previous studies that directly manipulate the embeddings, we suggest to dive deeper into the operation of these encoders, and pay more attention to the way they pay attention to different social groups.

Original: We ﬁnd that most stereotypes are also encoded in the attention layer.
Revised: We ﬁnd that most stereotypes are also encoded in the attention layer.

Original: Then, we work on model debiasing by redistributing the attention scores of a text encoder such that it forgets any preference to historically advantaged groups, and attends to all social classes with the same intensity.
Revised: Then, we work on model debiasing by redistributing the attention scores of a text encoder such that it forgets any preference to historically advantaged groups, and attends to all social classes with the same intensity.

Original: Our experiments conﬁrm that we successfully reduce bias with little damage to semantic representation.
Revised: Our experiments conﬁrm that we successfully reduce bias with little damage to semantic representation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Natural Language Processing (NLP) is increasingly penetrating real-world operations such as recruitment (Hansen et al., 2015), legal systems (Dale, 2019), healthcare (Velupillai et al., 2018) and
Revised: Natural Language Processing (NLP) is increasingly penetrating real-world operations such as recruitment (Hansen et al., 2015), legal systems (Dale, 2019), healthcare (Velupillai et al., 2018) and

Original: Web Search (Nalisnick et al., 2016).
Revised: Web Search (Nalisnick et al., 2016).

Original: Part of this success is attributed to the underlying embedding layer which encodes sophisticated semantic representations of language (Camacho-Collados & Pilehvar, 2018).
Revised: Part of this success is attributed to the underlying embedding layer which encodes sophisticated semantic representations of language (Camacho-Collados & Pilehvar, 2018).

Original: The wide adoption of modern NLP models in critical domains has also inﬂicted a more thorough scrutiny.
Revised: The wide adoption of modern NLP models in critical domains has also inﬂicted a more thorough scrutiny.

Original: Recent research has uncovered some propensities of NLP models to replicate discriminatory social biases (Bolukbasi et al., 2016; Caliskan et al., 2017;
Revised: Recent research has uncovered some propensities of NLP models to replicate discriminatory social biases (Bolukbasi et al., 2016; Caliskan et al., 2017;

Original: May et al., 2019) which may cause unintended and undesired model behaviors with respect to social groups.
Revised: May et al., 2019) which may cause unintended and undesired model behaviors with respect to social groups.

Original: Social bias in
Revised: Social bias in

Original: NLP is mainly caused by unbalanced mentions of attributes near advantaged groups in training data (Zhao et al., 2018a).
Revised: NLP is mainly caused by unbalanced mentions of attributes near advantaged groups in training data (Zhao et al., 2018a).

Original: For example, in most existing text corpora, very few cooks are referred to by male pronouns (e.g. he, him, himself) (Zhao et al., 2017).
Revised: For example, in most existing text corpora, very few cooks are referred to by male pronouns (e.g. he, him, himself) (Zhao et al., 2017).

Original: Accordingly, text encoders or language models trained on such data may use this shortcut to inadvertently disassociate cooks from men, and learn that cooking is a female attribute.
Revised: Accordingly, text encoders or language models trained on such data may use this shortcut to inadvertently disassociate cooks from men, and learn that cooking is a female attribute.

Original: Worse, NLP models may even amplify social biases if left unchecked (Bommasani et al., 2021).
Revised: Worse, NLP models may even amplify social biases if left unchecked (Bommasani et al., 2021).

Original: Methods to debias static word embeddings such as Word2vec (Mikolov et al., 2013) or GloVe (Pennington et al., 2014) have been applied for gender, race and religion (Bolukbasi et al., 2016; Zhao et al., 2018b; Kaneko & Bollegala, 2019; Ravfogel et al., 2020).
Revised: Methods to debias static word embeddings such as Word2vec (Mikolov et al., 2013) or GloVe (Pennington et al., 2014) have been applied for gender, race and religion (Bolukbasi et al., 2016; Zhao et al., 2018b; Kaneko & Bollegala, 2019; Ravfogel et al., 2020).

Original: However, by the time NLP practitioners started casting more attention to the fairness problem of their models, they had already switched to the more powerful sentence-level transformers in the likes of BERT (Devlin et al., 2018), GPT3 (Brown et al., 2020) or T5 (Raffel et al., 2020) which owe their success to the novel self-attention mechanism (Vaswani et al., 2017).
Revised: However, by the time NLP practitioners started casting more attention to the fairness problem of their models, they had already switched to the more powerful sentence-level transformers in the likes of BERT (Devlin et al., 2018), GPT3 (Brown et al., 2020) or T5 (Raffel et al., 2020) which owe their success to the novel self-attention mechanism (Vaswani et al., 2017).

Original: This leap in accuracy in several NLP tasks does not extend to fairness since research discovered social stereotypes in modern text encoders (May et al., 2019; Nadeem et al., 2020; Nangia et al.).
Revised: This leap in accuracy in several NLP tasks does not extend to fairness since research discovered social stereotypes in modern text encoders (May et al., 2019; Nadeem et al., 2020; Nangia et al.).

Original: To date, debiasing them remains comparatively under-explored.
Revised: To date, debiasing them remains comparatively under-explored.

Original: Mitigating biases in text encoders is difﬁcult for four reasons: (1) They are expensive to retrain, so conventional methods based on Counterfactual Data Augmentation (CDA) to rebalance groupattribute mentions (Zhao et al., 2018a; Webster et al., 2020) become prohibitive as they generate more training data, and all debiasing attempts might be limited to either ﬁnetuning or adapting (Houlsby et al., 2019; Lauscher et al., 2021).
Revised: Mitigating biases in text encoders is difﬁcult for four reasons: (1) They are expensive to retrain, so conventional methods based on Counterfactual Data Augmentation (CDA) to rebalance groupattribute mentions (Zhao et al., 2018a; Webster et al., 2020) become prohibitive as they generate more training data, and all debiasing attempts might be limited to either ﬁnetuning or adapting (Houlsby et al., 2019; Lauscher et al., 2021).

Original: (2) Static embeddings encode words whereas text encoders need context 1 .
Revised: (2) Static embeddings encode words whereas text encoders need context 1 .

Original: Thus, it is not straightforward to use existing debiasing techniques for static embeddings off-the-shelf as it is not clear how to generate context for single words.
Revised: Thus, it is not straightforward to use existing debiasing techniques for static embeddings off-the-shelf as it is not clear how to generate context for single words.

Original: Previous work tackled this problem by either designing bleached sentence templates (May et al., 2019; Kurita et al., 2019) where they ﬁll in the blanks with words of interest, or sampling sentences from large corpora where the words are mentioned (Liang et al., 2020a; Cheng et al., 2020), thus creating context.
Revised: Previous work tackled this problem by either designing bleached sentence templates (May et al., 2019; Kurita et al., 2019) where they ﬁll in the blanks with words of interest, or sampling sentences from large corpora where the words are mentioned (Liang et al., 2020a; Cheng et al., 2020), thus creating context.

Original: The former betrays the expressiveness of natural language while the latter suffers from sampling and preprocessing bias (Liang et al., 2020a).
Revised: The former betrays the expressiveness of natural language while the latter suffers from sampling and preprocessing bias (Liang et al., 2020a).

Original: The input space of text encoders is the set of all possible sentences, so we cannot debias every single input as it is done with static embeddings.
Revised: The input space of text encoders is the set of all possible sentences, so we cannot debias every single input as it is done with static embeddings.

Original: (4) Text encoders are larger in capacity and complexity.
Revised: (4) Text encoders are larger in capacity and complexity.

Original: This suggests that they can accommodate subtler and more sophisticated forms of stereotype, especially in their attention component, which renders bias imperceptible to existing detection methods as they are not designed to operate on attention.
Revised: This suggests that they can accommodate subtler and more sophisticated forms of stereotype, especially in their attention component, which renders bias imperceptible to existing detection methods as they are not designed to operate on attention.

Original: Despite these difﬁculties, previous works (Liang et al., 2020a; Webster et al., 2020; Liang et al., 2020b; Cheng et al., 2020; Kaneko & Bollegala, 2021; Lauscher et al., 2021) addressed the problem of reducing bias from modern text encoders with different techniques.
Revised: Despite these difﬁculties, previous works (Liang et al., 2020a; Webster et al., 2020; Liang et al., 2020b; Cheng et al., 2020; Kaneko & Bollegala, 2021; Lauscher et al., 2021) addressed the problem of reducing bias from modern text encoders with different techniques.

Original: However, most of them make strong assumptions about the linearity of bias.
Revised: However, most of them make strong assumptions about the linearity of bias.

Original: Moreover, they operate on the embeddings produced by text encoders, and leave their most important block - attention - largely unrectiﬁed.
Revised: Moreover, they operate on the embeddings produced by text encoders, and leave their most important block - attention - largely unrectiﬁed.

Original: In this paper we explore attention-based debiasing.
Revised: In this paper we explore attention-based debiasing.

Original: This approach stems from our observation that attention exhibits a great deal of social biases.
Revised: This approach stems from our observation that attention exhibits a great deal of social biases.

Original: We empirically show that this is the case, propose a novel method to reduce stereotypes from attention blocks, and demonstrate that it is effective in mitigating biases from sentence representations as a whole.
Revised: We empirically show that this is the case, propose a novel method to reduce stereotypes from attention blocks, and demonstrate that it is effective in mitigating biases from sentence representations as a whole.

Original: Given an input sentence, our method compels the text encoder of interest to redistribute its internal attention scores such that each word in the input allocates the same attention for different social groups.
Revised: Given an input sentence, our method compels the text encoder of interest to redistribute its internal attention scores such that each word in the input allocates the same attention for different social groups.

Original: Thus, it learns to forget previously encoded preferences, and generate fair representations, free of stereotypical inﬂuence.
Revised: Thus, it learns to forget previously encoded preferences, and generate fair representations, free of stereotypical inﬂuence.

Original: We also keep semantic information loss at a minimum while debiasing by distilling knowledge (Hinton et al., 2015; Gou et al., 2021) from an unaltered teacher text encoder.
Revised: We also keep semantic information loss at a minimum while debiasing by distilling knowledge (Hinton et al., 2015; Gou et al., 2021) from an unaltered teacher text encoder.

Original: In this setting, we encourage the debiased model to copy the original attention from its teacher to minimize semantic offset.
Revised: In this setting, we encourage the debiased model to copy the original attention from its teacher to minimize semantic offset.

Original: Unlike most previous work which focus only on gender, we address ﬁve bias types in our experiments (gender, race, religion, age and sexual orientation).
Revised: Unlike most previous work which focus only on gender, we address ﬁve bias types in our experiments (gender, race, religion, age and sexual orientation).

Original: We conduct likelihood- and inference-based evaluations to measure the intensity of bias in our ﬁnal debiased models.
Revised: We conduct likelihood- and inference-based evaluations to measure the intensity of bias in our ﬁnal debiased models.

Original: Experiments demonstrate that the technique we propose effectively reduces bias, and outperforms existing debiasing methods.
Revised: Experiments demonstrate that the technique we propose effectively reduces bias, and outperforms existing debiasing methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we discuss related work about debiasing static word embeddings and sentence-level text encoders.
Revised: In this section, we discuss related work about debiasing static word embeddings and sentence-level text encoders.

Original: Then, we shed some light on work done on the attention mechanism in general.
Revised: Then, we shed some light on work done on the attention mechanism in general.

Original: It should be noted that bias at data level (Pryzant et al., 2020; Cryan et al., 2020) and in language generation tasks (Sheng et al., 2020; Sap et al., 2020; Dhamala et al., 2021) are also active and complementary areas of research.
Revised: It should be noted that bias at data level (Pryzant et al., 2020; Cryan et al., 2020) and in language generation tasks (Sheng et al., 2020; Sap et al., 2020; Dhamala et al., 2021) are also active and complementary areas of research.

Original: However, due to space limitations, they will not be discussed in this paper.
Revised: However, due to space limitations, they will not be discussed in this paper.

Original: 2. B IAS IN S TATIC WORD EMBEDDINGS
Revised: 2. B IAS IN S TATIC WORD EMBEDDINGS

Original: The work of Bolukbasi et al.
Revised: The work of Bolukbasi et al.

Original: (2016) pioneered bias research in NLP by discovering that static word embeddings such as Word2Vec (Mikolov et al., 2013) or GloVe (Pennington et al., 2014) encode signiﬁcant amounts of binary gender bias.
Revised: (2016) pioneered bias research in NLP by discovering that static word embeddings such as Word2Vec (Mikolov et al., 2013) or GloVe (Pennington et al., 2014) encode signiﬁcant amounts of binary gender bias.

Original: They proposed Hard-Debias : a simple method to remove biases by projecting gender-neutral word embeddings onto a gender-free direction.
Revised: They proposed Hard-Debias : a simple method to remove biases by projecting gender-neutral word embeddings onto a gender-free direction.

Original: Manzini et al.
Revised: Manzini et al.

Original: (2019) extended Hard-Debias to the multiclass setting where they also treat racial and religious stereotypes.
Revised: (2019) extended Hard-Debias to the multiclass setting where they also treat racial and religious stereotypes.

Original: In both works, the bias direction is deﬁned by a manually pre-compiled list of stereotyped words.
Revised: In both works, the bias direction is deﬁned by a manually pre-compiled list of stereotyped words.

Original: In contrast, Ravfogel et al.
Revised: In contrast, Ravfogel et al.

Original: (2020) suggest a data-driven approach to learn bias directions with a linear classiﬁer.
Revised: (2020) suggest a data-driven approach to learn bias directions with a linear classiﬁer.

Original: Debiasing is then conducted by iteratively projecting word embeddings on the null space of the classiﬁer’s matrix.
Revised: Debiasing is then conducted by iteratively projecting word embeddings on the null space of the classiﬁer’s matrix.

Original: On the other hand, ﬁnetuning is the debiasing approach that attracted the widest adoption, either by using an autoencoder (Kaneko & Bollegala, 2019), attraction-repulsion mechanism (Kumar et al., 2020), or adversarial attacks (Xie et al., 2017;
Revised: On the other hand, ﬁnetuning is the debiasing approach that attracted the widest adoption, either by using an autoencoder (Kaneko & Bollegala, 2019), attraction-repulsion mechanism (Kumar et al., 2020), or adversarial attacks (Xie et al., 2017;

Original: Li et al., 2018; Elazar & Goldberg, 2018).
Revised: Li et al., 2018; Elazar & Goldberg, 2018).

Original: Unlike these post-processing methods, Zhao et al.
Revised: Unlike these post-processing methods, Zhao et al.

Original: (2018b) added a new fairness constraint to GloVe loss function, and retrained their fair word embeddings from scratch.
Revised: (2018b) added a new fairness constraint to GloVe loss function, and retrained their fair word embeddings from scratch.

Original: 2. B IAS IN T EXT ENCODERS
Revised: 2. B IAS IN T EXT ENCODERS

Original: Research on biases in sentence representations is dominated by detection rather than correction and mitigation.
Revised: Research on biases in sentence representations is dominated by detection rather than correction and mitigation.

Original: To date, there are three main approaches to detect stereotypes in text encoders: (1) representation-based : where vector relationships between different types of inputs are measured.
Revised: To date, there are three main approaches to detect stereotypes in text encoders: (1) representation-based : where vector relationships between different types of inputs are measured.

Original: For example, May et al.
Revised: For example, May et al.

Original: (2019) extended the WEAT test (Caliskan et al., 2017) into sentence vector space (SEAT), and compared the cosine similarity between representations of two sets of targets and two sets of attributes.
Revised: (2019) extended the WEAT test (Caliskan et al., 2017) into sentence vector space (SEAT), and compared the cosine similarity between representations of two sets of targets and two sets of attributes.

Original: All sentences in SEAT follow a predeﬁned template.
Revised: All sentences in SEAT follow a predeﬁned template.

Original: (2) likelihood-based :
Revised: (2) likelihood-based :

Original: These approaches examine how often text encoders prefer stereotypes over anti-stereotypes.
Revised: These approaches examine how often text encoders prefer stereotypes over anti-stereotypes.

Original: Preferences in this case are deﬁned in terms of higher likelihoods as produced by language models using embeddings of the text encoders under study.
Revised: Preferences in this case are deﬁned in terms of higher likelihoods as produced by language models using embeddings of the text encoders under study.

Original: Two benchmarks are widely used for measuring bias:
Revised: Two benchmarks are widely used for measuring bias:

Original: StereoSet (Nadeem et al., 2020) and Crows-Pairs (Nangia et al.).
Revised: StereoSet (Nadeem et al., 2020) and Crows-Pairs (Nangia et al.).

Original: Both datasets are organized in pairs or triples of minimally-distant sentences which differ only in the word(s) carrying a stereotypical connotation.
Revised: Both datasets are organized in pairs or triples of minimally-distant sentences which differ only in the word(s) carrying a stereotypical connotation.

Original: (3) inference-based : These methods employ text encoders in downstream NLP tasks (Blodgett et al., 2020) such as natural language inference (Dev et al., 2020), sentiment analysis (D ´ ıaz et al., 2018) or language generation (Sap et al., 2020; Sheng et al., 2020).
Revised: (3) inference-based : These methods employ text encoders in downstream NLP tasks (Blodgett et al., 2020) such as natural language inference (Dev et al., 2020), sentiment analysis (D ´ ıaz et al., 2018) or language generation (Sap et al., 2020; Sheng et al., 2020).

Original: Bias in such settings is declared as the difference in outcome when the models are tested with the same input sentence, differing only in social groups.
Revised: Bias in such settings is declared as the difference in outcome when the models are tested with the same input sentence, differing only in social groups.

Original: Bias mitigation approaches are mostly inspired by debiasing static embeddings.
Revised: Bias mitigation approaches are mostly inspired by debiasing static embeddings.

Original: In projection-based methods, Liang et al.
Revised: In projection-based methods, Liang et al.

Original: (2020a) contextualize words into sentences by sampling them from existing corpora before applying Hard-Debias.
Revised: (2020a) contextualize words into sentences by sampling them from existing corpora before applying Hard-Debias.

Original: Kaneko & Bollegala (2021) minimize the projection of sentence representations on a learned bias subspace, while Qian et al.
Revised: Kaneko & Bollegala (2021) minimize the projection of sentence representations on a learned bias subspace, while Qian et al.

Original: ; Bordia & Bowman (2019); Liang et al.
Revised: ; Bordia & Bowman (2019); Liang et al.

Original: (2020b) add bias-reduction objectives to their loss functions.
Revised: (2020b) add bias-reduction objectives to their loss functions.

Original: Another line of research uses CDA (Webster et al., 2020) to balance gender correlations in training data, while
Revised: Another line of research uses CDA (Webster et al., 2020) to balance gender correlations in training data, while

Original: Lauscher et al.
Revised: Lauscher et al.

Original: (2021) use adapters to reduce the large training time that CDA incurs.
Revised: (2021) use adapters to reduce the large training time that CDA incurs.

Original: Finally, Cheng et al.
Revised: Finally, Cheng et al.

Original: (2020) use contrastive learning, and add a fair ﬁlter that minimizes mutual information between stereotypes and anti-stereotypes.
Revised: (2020) use contrastive learning, and add a fair ﬁlter that minimizes mutual information between stereotypes and anti-stereotypes.

Original: In our work, rather than extending approaches from static embeddings, we focus on the self-attention mechanism which is characteristic of many text encoders, and show that fair attention leads to fair representations.
Revised: In our work, rather than extending approaches from static embeddings, we focus on the self-attention mechanism which is characteristic of many text encoders, and show that fair attention leads to fair representations.

Original: 2. A TTENTION ANALYSIS IN TEXT ENCODERS
Revised: 2. A TTENTION ANALYSIS IN TEXT ENCODERS

Original: Clark et al.
Revised: Clark et al.

Original: (2019) analyzed BERT’s attention heads and found that some of them correspond remarkably well to linguistic patterns of coreference and syntax without additional training.
Revised: (2019) analyzed BERT’s attention heads and found that some of them correspond remarkably well to linguistic patterns of coreference and syntax without additional training.

Original: Michel et al.
Revised: Michel et al.

Original: (2019) observe that not all attention heads within a model are made equal.
Revised: (2019) observe that not all attention heads within a model are made equal.

Original: They also propose a pruning algorithm to reduce the energy footprint of these models by eliminating the least important heads without much attenuation to the overall performance.
Revised: They also propose a pruning algorithm to reduce the energy footprint of these models by eliminating the least important heads without much attenuation to the overall performance.

Original: Given the convenient interpretabilty of attention, it has also been used in a myriad of visualization works (Vig, 2019; Hoover et al., 2020;
Revised: Given the convenient interpretabilty of attention, it has also been used in a myriad of visualization works (Vig, 2019; Hoover et al., 2020;

Original: Tenney et al., 2020; Bastings & Filippova, 2020) in an attempt to dissect and explain the inner functioning of text encoders.
Revised: Tenney et al., 2020; Bastings & Filippova, 2020) in an attempt to dissect and explain the inner functioning of text encoders.

Original: Most attention studies in text encoders are designed for analysis purposes.
Revised: Most attention studies in text encoders are designed for analysis purposes.

Original: In contrast, we are the ﬁrst to leverage the attention mechanism in order to make text encoders fairer and less stereotyped.
Revised: In contrast, we are the ﬁrst to leverage the attention mechanism in order to make text encoders fairer and less stereotyped.

Original:  D EBIASING M ETHOD 3. M OTIVATING EXAMPLE & I NTUITION
Revised:  D EBIASING M ETHOD 3. M OTIVATING EXAMPLE & I NTUITION

Original: Despite the applicability of our work on any model that is built upon self-attention, we focus in this paper on models based on the encoder side of the transformer architecture, such as BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), or ALBERT (Lan et al., 2019).
Revised: Despite the applicability of our work on any model that is built upon self-attention, we focus in this paper on models based on the encoder side of the transformer architecture, such as BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), or ALBERT (Lan et al., 2019).

Original: This owes to the decoder side being usually used in auto-regression tasks, and less often to encode text.
Revised: This owes to the decoder side being usually used in auto-regression tasks, and less often to encode text.

Original: Transformers consist of multiple layers, each composed of a self-attention block followed by a feed-forward block to make embeddings.
Revised: Transformers consist of multiple layers, each composed of a self-attention block followed by a feed-forward block to make embeddings.

Original: A self-attention block contains multiple heads.
Revised: A self-attention block contains multiple heads.

Original: Each head transforms the input into
Revised: Each head transforms the input into

Original: [Figure]
Revised: [Figure]

Revised: ), such each token learns to attend to its most related tokens, hence the prevalence of attention in deﬁning the understanding of natural language.

Original: the we assume that undesired  stereotypes are primarily encoded in the 
Revised: In this work, we assume that undesired social stereotypes are primarily encoded in the self-attention block.

Original: In order to verify this hypothesis, we show and analyze some attention maps of BERT in Figure 1 2 .
Revised: In order to verify this hypothesis, we show and analyze some attention maps of BERT in Figure 1 2 .

Original: Consider the following sentence ”The doctor asked the nurse a question.”
Revised: Consider the following sentence ”The doctor asked the nurse a question.”

Original: Aiming to analyze how every word representation 3 relates to different social groups, we add a dummy second input consisting of words representing two distinct genders ( he and she after the [SEP] token).
Revised: Aiming to analyze how every word representation 3 relates to different social groups, we add a dummy second input consisting of words representing two distinct genders ( he and she after the [SEP] token).

Original: Figure 1(a) illustrates that the token doctor pays much more attention to he than to she 4 .
Revised: Figure 1(a) illustrates that the token doctor pays much more attention to he than to she 4 .

Original: Figure 1(b) reveals that nurse attends to she .
Revised: Figure 1(b) reveals that nurse attends to she .

Original: This ﬁnding suggests that gender stereotypes are deeply encoded in  weights.
Revised: This ﬁnding suggests that gender stereotypes are deeply encoded in attention weights.

Original: Likewise, in Figure 1( the token  is more related  than to white or black  conforming to the famous racial 
Revised: Likewise, in Figure 1(c), the token math is more related to asian than to white or black , conforming to the famous racial stereotype casting asians as really good mathematicians.

Original: These examples align  are ﬁrst and  in  before they  to  downstream 
Revised: These examples align with our intuition that social stereotypes are ﬁrst and foremost encoded in the self-attention block of text encoders before they propagate to their embeddings or predictions in downstream NLP tasks.

Original: Consequently we propose 
Revised: Consequently, we propose Att-D , a ﬁnetuning method for reducing undesired biases of text encoders from their attention component.

Original: The intuition of our debiasing strategy is as follows:
Revised: The intuition of our debiasing strategy is as follows:

Original: Given that attention weights conform with undesired biases (e.g., doctor attending to he , and nurse to she in Figure 1), we aim to equalize the attention scores of every word in the input sentence with respect to social groups.
Revised: Given that attention weights conform with undesired biases (e.g., doctor attending to he , and nurse to she in Figure 1), we aim to equalize the attention scores of every word in the input sentence with respect to social groups.

Original: Following the examples of Figure 1, Att-D redistributes the attention scores of doctor such that it attends to he and she with the same intensity, thus eliminating any preference toward one of the groups.
Revised: Following the examples of Figure 1, Att-D redistributes the attention scores of doctor such that it attends to he and she with the same intensity, thus eliminating any preference toward one of the groups.

Original: However, alterations to the attention of doctor on the remaining words of the input sentence must be kept to a minimum in order not to corrupt the semantic understanding of the original text encoder.
Revised: However, alterations to the attention of doctor on the remaining words of the input sentence must be kept to a minimum in order not to corrupt the semantic understanding of the original text encoder.

Original: 3. D EBIASING W ORKFLOW
Revised: 3. D EBIASING W ORKFLOW

Original: Att-D consists of three steps: First, for each input sentence s in the training corpus S , we make an artiﬁcial second input s g consisting of words related to social groups (similar to the examples on
Revised: Att-D consists of three steps: First, for each input sentence s in the training corpus S , we make an artiﬁcial second input s g consisting of words related to social groups (similar to the examples on

Original: Figure 1).
Revised: Figure 1).

Original: Pretrained text encoders expect either one or two inputs before they produce attentions and embeddings.
Revised: Pretrained text encoders expect either one or two inputs before they produce attentions and embeddings.

Original: In this work, we use both s (as ﬁrst input) and s g (as second input) separated by [SEP] token.
Revised: In this work, we use both s (as ﬁrst input) and s g (as second input) separated by [SEP] token.

Original: Consequently, the resulting self-attention includes both s and s g .
Revised: Consequently, the resulting self-attention includes both s and s g .

Original: The second step of
Revised: The second step of

Original: Att-D equalizes the attention weights of all heads in all the layers of the text encoder of interest such that each token in s pays the same amount of attention to tokens of s g , thus eliminating preferences and stereotypes.
Revised: Att-D equalizes the attention weights of all heads in all the layers of the text encoder of interest such that each token in s pays the same amount of attention to tokens of s g , thus eliminating preferences and stereotypes.

Original: Finally, we minimize semantic loss by compelling our model to learn the original semantics from an unaltered teacher model by copying its internal attention in a knowledge distillation setting (Hinton et al., 2015).
Revised: Finally, we minimize semantic loss by compelling our model to learn the original semantics from an unaltered teacher model by copying its internal attention in a knowledge distillation setting (Hinton et al., 2015).

Original: We schematize the operation of Att-D in Figure 2.
Revised: We schematize the operation of Att-D in Figure 2.

Original: Gr 1 , Gr 2 and Gr n in the ﬁgure correspond to the tokens of s g .
Revised: Gr 1 , Gr 2 and Gr n in the ﬁgure correspond to the tokens of s g .

Original: Both matrices represent one attention head of the text encoder before (left) and after (right) debiasing.
Revised: Both matrices represent one attention head of the text encoder before (left) and after (right) debiasing.

Original: The matrices should be read in rows.
Revised: The matrices should be read in rows.

Original: Each row depicts the attention weights of the corresponding token on all the other tokens of the input ( s + s g ).
Revised: Each row depicts the attention weights of the corresponding token on all the other tokens of the input ( s + s g ) 5 .

Original: [Figure]
Revised: [Figure]

Original: The matrices are conceptually split in four blocks: (1) attentions of s on s , (2) attentions of s on s g , (3) attentions of s g on s , and (4) attentions of s g on s g .
Revised: The matrices are conceptually split in four blocks: (1) attentions of s on s , (2) attentions of s on s g , (3) attentions of s g on s , and (4) attentions of s g on s g .

Original: Debiasing consists in making the columns of block 2 equal.
Revised: Debiasing consists in making the columns of block 2 equal.

Original: In other words, each token in s pays the same amount of attention to all the groups as indicated in the right side of Figure 2.
Revised: In other words, each token in s pays the same amount of attention to all the groups as indicated in the right side of Figure 2.

Original: Ideally, debiasing should also preserve the semantics of the original text encoder.
Revised: Ideally, debiasing should also preserve the semantics of the original text encoder.

Original: That is why block 1 of Figure 2 should be kept unchanged.
Revised: That is why block 1 of Figure 2 should be kept unchanged.

Original: Both blocks 3 and 4 are irrelevant to the results, since they denote attentions of our artiﬁcially inserted second input s g .
Revised: Both blocks 3 and 4 are irrelevant to the results, since they denote attentions of our artiﬁcially inserted second input s g .

Original: Besides, they do not participate in deﬁning neither fairness nor representativeness of the text encoder.
Revised: Besides, they do not participate in deﬁning neither fairness nor representativeness of the text encoder.

Original: So, we do not impose any restrictions on them.
Revised: So, we do not impose any restrictions on them.

Original: In the following, we describe each step of
Revised: In the following, we describe each step of

Original: Att-D in detail.
Revised: Att-D in detail.

Original: 3.2. G ENERATING AUGMENTED INPUTS
Revised: 3.2. G ENERATING AUGMENTED INPUTS

Original: [Table]
Revised: [Table]

Original: The ﬁrst step involves identifying bias types that we want to mitigate from pretrained text encoders, such as gender, race, religion.
Revised: The ﬁrst step involves identifying bias types that we want to mitigate from pretrained text encoders, such as gender, race, religion.

Original: This is achieved by deﬁning a set of tuples G such that G =
Revised: This is achieved by deﬁning a set of tuples G such that G =

Original: {T 1 , T 2 , ..., T k } where each T i describes social groups of a given bias type, or their attributes.
Revised: {T 1 , T 2 , ..., T k } where each T i describes social groups of a given bias type, or their attributes.

Original: In the case of binary gender,
Revised: In the case of binary gender,

Original: T i = { ”male” , ”female” } or T i = { ”he” , ”she” } are both possible deﬁnitions (and many others are possible for non-binary cases).
Revised: T i = { ”male” , ”female” } or T i = { ”he” , ”she” } are both possible deﬁnitions (and many others are possible for non-binary cases).

Original: Table 1 shows some tuples that we use in Att-D. Unlike previous work (Liang et al., 2020a; Kaneko & Bollegala, 2021), we do not aim to learn bias subspaces.
Revised: Table 1 shows some tuples that we use in Att-D. Unlike previous work (Liang et al., 2020a; Kaneko & Bollegala, 2021), we do not aim to learn bias subspaces.

Original: Therefore, Att-D charges next to no constraint in the deﬁnition of G , contrary to existing methods which need group deﬁnitions to be formulated in a certain way, and given in enough quantities.
Revised: Therefore, Att-D charges next to no constraint in the deﬁnition of G , contrary to existing methods which need group deﬁnitions to be formulated in a certain way, and given in enough quantities.

Original: On the other hand, providing a single tuple per bias type is sufﬁcient for Att-D to function, even though more contributes to better precision.
Revised: On the other hand, providing a single tuple per bias type is sufﬁcient for Att-D to function, even though more contributes to better precision.

Original: During debiasing, we pick a tuple T i from G randomly and construct s g , a bleached sentence formed by words of T i .
Revised: During debiasing, we pick a tuple T i from G randomly and construct s g , a bleached sentence formed by words of T i .

Original: For example, given Table 1, s g can be ”man, woman” or ”old, young” .
Revised: For example, given Table 1, s g can be ”man, woman” or ”old, young” .

Original: The input to the text encoder is both the original input sentence s and the artiﬁcial one s g .
Revised: The input to the text encoder is both the original input sentence s and the artiﬁcial one s g .

Original: Pretrained text encoders separate the two halves of the input with a special token [SEP] as illustrated in Figure 1, and compute attention maps for the entire sequence ( s + s g ).
Revised: Pretrained text encoders separate the two halves of the input with a special token [SEP] as illustrated in Figure 1, and compute attention maps for the entire sequence ( s + s g ).

Original: 3.2. E QUALIZING ATTENTIONS ON SOCIAL GROUPS
Revised: 3.2. E QUALIZING ATTENTIONS ON SOCIAL GROUPS

Original: After obtaining attention maps of the augmented input from Section 3.2.1, which is produced by the pretrained text encoder of interest E , we make each token of s pay equal amounts of attention to the tokens of s g which deﬁne social groups.
Revised: After obtaining attention maps of the augmented input from Section 3.2.1, which is produced by the pretrained text encoder of interest E , we make each token of s pay equal amounts of attention to the tokens of s g which deﬁne social groups.

Original: The rationale is to eliminate any inclination for E to prefer a social group to the detriment of others.
Revised: The rationale is to eliminate any inclination for E to prefer a social group to the detriment of others.

Original: Suppose A l,h,s,s g
Revised: Suppose A l,h,s,s g

Original: =
Revised: =

Original: Attn ( s, s g ; l, h ) is the attention matrix at layer l , head h of the encoder E , computed from the input s
Revised: Attn ( s, s g ; l, h ) is the attention matrix at layer l , head h of the encoder E , computed from the input s

Original: + s g .
Revised: + s g .

Original: Here, we make the reasonable assumption that s g contains at least two social groups.
Revised: Here, we make the reasonable assumption that s g contains at least two social groups.

Original: In this spirit, equalizing attention vectors of block 2 (as deﬁned in Figure 2) is equivalent to making them equal to a pivot vector.
Revised: In this spirit, equalizing attention vectors of block 2 (as deﬁned in Figure 2) is equivalent to making them equal to a pivot vector.

Original: In our method, we consider the attention vector of s on the ﬁrst social group as the pivot (ﬁrst column in block 2 of
Revised: In our method, we consider the attention vector of s on the ﬁrst social group as the pivot (ﬁrst column in block 2 of

Original: Figure 2), and minimize the mean square error between the pivot and the attention vectors of s on the other groups, one at a time.
Revised: Figure 2), and minimize the mean square error between the pivot and the attention vectors of s on the other groups, one at a time.

Original: The equalization loss is given by Equation 1.
Revised: The equalization loss is given by Equation 1.

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where L is the number of layers of the text encoder, H the number of heads,
Revised: where L is the number of layers of the text encoder, H the number of heads,

Original: || s g
Revised: || s g

Original: ||
Revised: ||

Original: the number of social groups in s g and σ is the position of the special token [SEP] that marks the end of s and the beginning of s g .
Revised: the number of social groups in s g and σ is the position of the special token [SEP] that marks the end of s and the beginning of s g .

Original: As can be seen, A l,h,s,s g : σ,σ +1 is the pivot vector containing attention scores of s on the ﬁrst social group token (whose position is directly after [SEP], i.e., σ + 1 ).
Revised: As can be seen, A l,h,s,s g : σ,σ +1 is the pivot vector containing attention scores of s on the ﬁrst social group token (whose position is directly after [SEP], i.e., σ + 1 ).

Original: Equation 1 forces attention scores on subsequent social groups to be the same as on the ﬁrst one, thus making them all equal.
Revised: Equation 1 forces attention scores on subsequent social groups to be the same as on the ﬁrst one, thus making them all equal.

Original: 3.2. P RESERVING SEMANTIC INFORMATION
Revised: 3.2. P RESERVING SEMANTIC INFORMATION

Original: Text encoders must preserve their ability to represent natural language and keep the same performance on downstream NLP tasks.
Revised: Text encoders must preserve their ability to represent natural language and keep the same performance on downstream NLP tasks.

Original: For this reason, debiasing must ensure that useful information is preserved as much as possible.
Revised: For this reason, debiasing must ensure that useful information is preserved as much as possible.

Original: We minimize semantic information loss in a knowledge distillation setting (Hinton et al., 2015; Gou et al., 2021).
Revised: We minimize semantic information loss in a knowledge distillation setting (Hinton et al., 2015; Gou et al., 2021).

Original: We cast the text encoder that we want to debias as the student model, and recruit another model to play the role of the teacher .
Revised: We cast the text encoder that we want to debias as the student model, and recruit another model to play the role of the teacher .

Original: We do not apply our debiasing strategy on the teacher since it provides a reference to the original unaltered language representations.
Revised: We do not apply our debiasing strategy on the teacher since it provides a reference to the original unaltered language representations.

Original: We distill semantic information in the form of attention maps from the teacher and instill it in the student.
Revised: We distill semantic information in the form of attention maps from the teacher and instill it in the student.

Original: Stated differently, we compel the student to learn from the teacher and reproduce its attention scores for every input sentence in the training corpus S .
Revised: Stated differently, we compel the student to learn from the teacher and reproduce its attention scores for every input sentence in the training corpus S .

Original: In this work, the student and the teacher models must share the exact same architecture in order to make knowledge distillation ﬂow correctly from source to target, unlike other works in knowledge distillation (Gou et al., 2021) wherein the student is much smaller and should be more compact than the teacher for model compression purposes.
Revised: In this work, the student and the teacher models must share the exact same architecture in order to make knowledge distillation ﬂow correctly from source to target, unlike other works in knowledge distillation (Gou et al., 2021) wherein the student is much smaller and should be more compact than the teacher for model compression purposes.

Original: As in Section 3.2.2, let A l,h,s,s g be the attention of the student model at layer l , head h with s and s g as input.
Revised: As in Section 3.2.2, let A l,h,s,s g be the attention of the student model at layer l , head h with s and s g as input.

Original: Likewise, let O l,h,s,s g deﬁne the same attention matrix, but for the original teacher model.
Revised: Likewise, let O l,h,s,s g deﬁne the same attention matrix, but for the original teacher model.

Original: We formalize the preservation of semantic information as a regularizer where we minimize the squared l 2 distance between the student’s and the teacher’s attention scores according to the following:
Revised: We formalize the preservation of semantic information as a regularizer where we minimize the squared l 2 distance between the student’s and the teacher’s attention scores according to the following:

Original: [Equation]
Revised: [Equation]

Original: where L is the number of layers, H is the number of heads, and σ is the position of the [SEP] token.
Revised: where L is the number of layers, H is the number of heads, and σ is the position of the [SEP] token.

Original: As can be seen from Equation 2, the student learns only to replicate block 1 (as in Figure 2) of the attention matrices.
Revised: As can be seen from Equation 2, the student learns only to replicate block 1 (as in Figure 2) of the attention matrices.

Original: This is because block 1 contains attention scores of the original input sentence s on itself, thus encoding an important aspect of semantics.
Revised: This is because block 1 contains attention scores of the original input sentence s on itself, thus encoding an important aspect of semantics.

Original: We force the student not to reproduce the attention distribution on social groups (block 2) from the teacher since these are fundamentally biased, and are left to the care of our debiasing objective.
Revised: We force the student not to reproduce the attention distribution on social groups (block 2) from the teacher since these are fundamentally biased, and are left to the care of our debiasing objective.

Original: We describe the overall training objective as a linear combination of the previously deﬁned losses, with λ as a hyperparameter to control the weight of debiasing over semantic preservation.
Revised: We describe the overall training objective as a linear combination of the previously deﬁned losses, with λ as a hyperparameter to control the weight of debiasing over semantic preservation.

Original: [Equation]
Revised: [Equation]

Original: 3. N EGATIVE S AMPLING & L AYER S ELECTION
Revised: 3. N EGATIVE S AMPLING & L AYER S ELECTION

Original: The strict application of Att-D as discussed so far may accidentally lead to some undesired spurious phenomena.
Revised: The strict application of Att-D as discussed so far may accidentally lead to some undesired spurious phenomena.

Original: While learning to equalize attention on social groups that constitute the second half of the input, the text encoder might potentially bear the risk of distributing its attention uniformly on any second half of the input, no matter what it is.
Revised: While learning to equalize attention on social groups that constitute the second half of the input, the text encoder might potentially bear the risk of distributing its attention uniformly on any second half of the input, no matter what it is.

Original: This is particularly alarming when the text encoder is subsequently employed in double-sentence tasks (Wang et al., 2018) such as semantic textual similarity, paraphrase detection or sentence entailment.
Revised: This is particularly alarming when the text encoder is subsequently employed in double-sentence tasks (Wang et al., 2018) such as semantic textual similarity, paraphrase detection or sentence entailment.

Original: To overcome the above obstacle, we introduce negative sampling.
Revised: To overcome the above obstacle, we introduce negative sampling.

Original: Instead of using words related to social groups in order to generate the artiﬁcial second input s g , we randomly sample words (negative examples) from the vocabulary.
Revised: Instead of using words related to social groups in order to generate the artiﬁcial second input s g , we randomly sample words (negative examples) from the vocabulary.

Original: In this case, we do not equalize the attentions but compel the student to copy its teacher even for blocks 2, 3 and 4.
Revised: In this case, we do not equalize the attentions but compel the student to copy its teacher even for blocks 2, 3 and 4.

Original: We do this in order to prevent the text encoder from learning to assign the same attention weight to all tokens of the second input when these do not deﬁne social groups.
Revised: We do this in order to prevent the text encoder from learning to assign the same attention weight to all tokens of the second input when these do not deﬁne social groups.

Original: We control the ratio of negative examples with a hyperparameter η .
Revised: We control the ratio of negative examples with a hyperparameter η .

Original: Another concern when debiasing text encoders is that their layers do not necessarily encode the same information.
Revised: Another concern when debiasing text encoders is that their layers do not necessarily encode the same information.

Original: Bhardwaj et al.
Revised: Bhardwaj et al.

Original: (2021) found that BERT layers display widely different reactions when probed with a gender-detection classiﬁer.
Revised: (2021) found that BERT layers display widely different reactions when probed with a gender-detection classiﬁer.

Original: This means that they do not encode gender stereotype identically.
Revised: This means that they do not encode gender stereotype identically.

Original: Therefore, it is not clear which layers and/or attention heads are best for debiasing.
Revised: Therefore, it is not clear which layers and/or attention heads are best for debiasing.

Original: To investigate this issue, we consider seven settings: debiasing all layers, ﬁrst 6 , ﬁrst 3 , last 6 , last 3 , and alternating layers with strides of one 6 or two 7 .
Revised: To investigate this issue, we consider seven settings: debiasing all layers, ﬁrst 6 , ﬁrst 3 , last 6 , last 3 , and alternating layers with strides of one 7 or two 8 .

Original: We ﬁnd that working with all layers yields the best results.
Revised: We ﬁnd that working with all layers yields the best results.

Original: So, unless otherwise speciﬁed in this paper, debiasing concerns all the layers.
Revised: So, unless otherwise speciﬁed in this paper, debiasing concerns all the layers.

Original:  E VALUATION
Revised:  E VALUATION

Original: In this section, we ﬁrst describe our experimental setup, then evaluate Att-D from two viewpoints: fairness and representativeness .
Revised: In this section, we ﬁrst describe our experimental setup, then evaluate Att-D from two viewpoints: fairness and representativeness .

Original: Unlike previous work in debiasing text encoders (Liang et al., 2020a; Cheng et al., 2020; Kaneko & Bollegala, 2021), we do not conduct representation-based evaluations - SEAT (May et al., 2019) and WEAT (Caliskan et al., 2017) - in this paper to measure bias since their effectiveness for such a purpose has been questioned (Silva et al., 2021).
Revised: Unlike previous work in debiasing text encoders (Liang et al., 2020a; Cheng et al., 2020; Kaneko & Bollegala, 2021), we do not conduct representation-based evaluations - SEAT (May et al., 2019) and WEAT (Caliskan et al., 2017) - in this paper to measure bias since their effectiveness for such a purpose has been questioned (Silva et al., 2021).

Original: On the other hand, we evaluate fairness with both likelihood- and inference-based methods.
Revised: On the other hand, we evaluate fairness with both likelihood- and inference-based methods.

Original: Other quantitative and qualitative evaluations can be found in the appendix 8 .
Revised: Other quantitative and qualitative evaluations can be found in the appendix 9 .

Original: 4. D EBIASING SETUP
Revised: 4. D EBIASING SETUP

Original: In the experimental evaluation we focus on ﬁve different bias types which are commonly addressed by the bias literature and debiasing approaches.
Revised: In the experimental evaluation we focus on ﬁve different bias types which are commonly addressed by the bias literature and debiasing approaches.

Original: To facilitate comparison, we also follow existing literature in deﬁning the social groups for each type of bias, although the approach presented here is not restricted to that, and can be leveraged for both other kinds of biases and for a more inclusive deﬁnition of the groups.
Revised: To facilitate comparison, we also follow existing literature in deﬁning the social groups for each type of bias, although the approach presented here is not restricted to that, and can be leveraged for both other kinds of biases and for a more inclusive deﬁnition of the groups.

Original: In the experiments we show results of debiasing based on (binary) gender ( male , female ), race ( white , black , asian , hispanic ), religion ( muslim , christian , jewish , buddhist ), age ( young , old ) and sexual orientation ( heterosexual , non-heterosexual ).
Revised: In the experiments we show results of debiasing based on (binary) gender ( male , female ), race ( white , black , asian , hispanic ), religion ( muslim , christian , jewish , buddhist ), age ( young , old ) and sexual orientation ( heterosexual , non-heterosexual ).

Original: We leverage the deﬁnition words from previous work (Liang et al., 2020a) and augment them with our own tuples.
Revised: We leverage the deﬁnition words from previous work (Liang et al., 2020a) and augment them with our own tuples.

Original: The full list can be found in the appendix.
Revised: The full list can be found in the appendix.

Original: We apply our debiasing method on BERT 9 (Devlin et al., 2018), and use the News-commentary-v15 corpus 10 as training data.
Revised: We apply our debiasing method on BERT 10 (Devlin et al., 2018), and use the News-commentary-v15 corpus 11 as training data.

Original: It contains 223,153 sentences of which we use 80% for training and 20% for development.
Revised: It contains 223,153 sentences of which we use 80% for training and 20% for development.

Original: 4. E VALUATIONS OF F AIRNESS 4.2. L IKELIHOOD - BASED E VALUATION
Revised: 4. E VALUATIONS OF F AIRNESS 4.2. L IKELIHOOD - BASED E VALUATION

Original: We use the publicly available subsets of two stereotype benchmarks: StereoSet (Nadeem et al., 2020) and Crows-Pairs (Nangia et al.).
Revised: We use the publicly available subsets of two stereotype benchmarks: StereoSet (Nadeem et al., 2020) and Crows-Pairs (Nangia et al.).

Original: Both provide likelihood-based diagnostics to measure how often stereotypes are considered more likely than anti-stereotypes by the text encoder of interest 11 .
Revised: Both provide likelihood-based diagnostics to measure how often stereotypes are considered more likely than anti-stereotypes by the text encoder of interest 12 .

Original: CrowsPairs is organized in pairs of minimally distant sentences, i.e. sentences that differ only in one word, usually the one describing a social group.
Revised: CrowsPairs is organized in pairs of minimally distant sentences, i.e. sentences that differ only in one word, usually the one describing a social group.

Original: We pick the following example from the dataset: Jews are such greedy people (Stereotype) versus Christians are such greedy people (Anti-stereotype).
Revised: We pick the following example from the dataset: Jews are such greedy people (Stereotype) versus Christians are such greedy people (Anti-stereotype).

Original: CrowsPairs uses word likelihoods produced by the text encoder to measure how often stereotypes are ranked higher than anti-stereotypes.
Revised: CrowsPairs uses word likelihoods produced by the text encoder to measure how often stereotypes are ranked higher than anti-stereotypes.

Original: An ideal unbiased text encoder should score 50% in the CrowsPairs challenge, meaning that it prefers neither stereotypes nor anti-stereotypes.
Revised: An ideal unbiased text encoder should score 50% in the CrowsPairs challenge, meaning that it prefers neither stereotypes nor anti-stereotypes.

Original: 100% and 0% are both undesired extreme outcomes.
Revised: 100% and 0% are both undesired extreme outcomes.

Original: In contrast, StereoSet adds a third absurd sentence to capture the language modeling capabilities of the text encoder in addition to measuring bias.
Revised: In contrast, StereoSet adds a third absurd sentence to capture the language modeling capabilities of the text encoder in addition to measuring bias.

Original: The triple of sentences in StereoSet are also minimally distant but differ in the attribute word, not in the group.
Revised: The triple of sentences in StereoSet are also minimally distant but differ in the attribute word, not in the group.

Original: For example, in Girls tend to be more [MASK] than boys , StereoSet generates the evaluation sentences by replacing the mask with the following attributes: soft to denote stereotype, determined for antistereotyping, and ﬁsh to make an unrelated sentence.
Revised: For example, in Girls tend to be more [MASK] than boys , StereoSet generates the evaluation sentences by replacing the mask with the following attributes: soft to denote stereotype, determined for antistereotyping, and ﬁsh to make an unrelated sentence.

Original: As in Crows-Pairs, StereoSet quantiﬁes how
Revised: As in Crows-Pairs, StereoSet quantiﬁes how

Original: [Table]
Revised: [Table]

Original: much the model prefers stereotypes over anti-stereotypes.
Revised: much the model prefers stereotypes over anti-stereotypes.

Original: Moreover, it also measures how often meaningful sentences rank higher than unrelated sentences in order to assess the quality of the text encoder in terms of representativeness.
Revised: Moreover, it also measures how often meaningful sentences rank higher than unrelated sentences in order to assess the quality of the text encoder in terms of representativeness.

Original: An ideal model should have a stereotype score ( ss ) of 50% and a language modeling ( lm ) of 100%.
Revised: An ideal model should have a stereotype score ( ss ) of 50% and a language modeling ( lm ) of 100%.

Original: We compare our method against the original BERT base model to see the effect of debiasing 12 .
Revised: We compare our method against the original BERT base model to see the effect of debiasing 13 .

Original: Also, for fair comparisons against previous work, we decided to include the baselines whose ﬁnal debiased models have been published in order to avoid errors of training and/or tuning hyperparameters.
Revised: Also, for fair comparisons against previous work, we decided to include the baselines whose ﬁnal debiased models have been published in order to avoid errors of training and/or tuning hyperparameters.

Original: Thus, we compare Att-D against Sent-D (Liang et al., 2020a) and the debiasing procedure proposed by
Revised: Thus, we compare Att-D against Sent-D (Liang et al., 2020a) and the debiasing procedure proposed by

Original: Kaneko & Bollegala (2021).
Revised: Kaneko & Bollegala (2021).

Original: Sent-D currently holds state-of-the-art performance in debiasing.
Revised: Sent-D currently holds state-of-the-art performance in debiasing.

Original: We also conduct a simple ablation study by training without negative examples ( Att-D - ).
Revised: We also conduct a simple ablation study by training without negative examples ( Att-D - ).

Original: We ﬁnetune
Revised: We ﬁnetune

Original: Att-D and the baselines on language modeling to be able to conduct likelihood-based experiments.
Revised: Att-D and the baselines on language modeling to be able to conduct likelihood-based experiments.

Original: Tables 2 and 3 report the evaluation results on StereoSet and Crows-Pairs respectively.
Revised: Tables 2 and 3 report the evaluation results on StereoSet and Crows-Pairs respectively.

Original: [Table]
Revised: [Table]

Original: We see that the original BERT contains significant levels of biases (56.04 in StereoSet and 60.48 in Crows-Pairs).
Revised: We see that the original BERT contains significant levels of biases (56.04 in StereoSet and 60.48 in Crows-Pairs).

Original: It is important to note that Kaneko & Bollegala (2021) only focused on gender bias, which is clear in Table 2 where only gender stereotype has been reduced.
Revised: It is important to note that Kaneko & Bollegala (2021) only focused on gender bias, which is clear in Table 2 where only gender stereotype has been reduced.

Original: We observe that focusing on one bias type can make text representations even more biased for the other dimensions as can be seen in Kaneko for race and religion (Table 2)
Revised: We observe that focusing on one bias type can make text representations even more biased for the other dimensions as can be seen in Kaneko for race and religion (Table 2)

Original: [Table]
Revised: [Table]

Original: and for sexual orientation and disability (Table 3).
Revised: and for sexual orientation and disability (Table 3).

Original: In contrast,
Revised: In contrast,

Original: Att-D always reduces the intensity of stereotyping in BERT (up to 9.53%)
Revised: Att-D always reduces the intensity of stereotyping in BERT (up to 9.53%)

Original: [Table]
Revised: [Table]

Original: and yields the best results overall 13 .
Revised: and yields the best results overall 14 .

Original: We also notice that we manage to reduce biases linked to dimensions we did not include in our design such as profession, nationality and disability.
Revised: We also notice that we manage to reduce biases linked to dimensions we did not include in our design such as profession, nationality and disability.

Original: We speculate that these bias types are connected to those we worked on mitigating.
Revised: We speculate that these bias types are connected to those we worked on mitigating.

Original: Therefore, we conjecture that reducing multiple biases at the same time meets better success in mitigating unforeseen stereotypes than working on every bias type separately.
Revised: Therefore, we conjecture that reducing multiple biases at the same time meets better success in mitigating unforeseen stereotypes than working on every bias type separately.

Original: 4.2. I NFERENCE - BASED E VALUATION
Revised: 4.2. I NFERENCE - BASED E VALUATION

Original: This approach of measuring bias builds on the intuition of Dev et al.
Revised: This approach of measuring bias builds on the intuition of Dev et al.

Original: (2020) stating that biased representations lead to invalid inferences, whose ratio quantiﬁes bias.
Revised: (2020) stating that biased representations lead to invalid inferences, whose ratio quantiﬁes bias.

Original: They construct a challenge benchmark for the natural language inference task where every hypothesis should be neutral to its premise.
Revised: They construct a challenge benchmark for the natural language inference task where every hypothesis should be neutral to its premise.

Original: For example, suppose that the premise is The driver owns a van and the hypothesis is The man owns a van .
Revised: For example, suppose that the premise is The driver owns a van and the hypothesis is The man owns a van .

Original: The hypothesis neither entails nor contradicts the premise.
Revised: The hypothesis neither entails nor contradicts the premise.

Original: If the predictions of a classiﬁer deviate from neutrality, the underlying text encoder is doomed as biased.
Revised: If the predictions of a classiﬁer deviate from neutrality, the underlying text encoder is doomed as biased.

Original: All the data points in Dev et al.
Revised: All the data points in Dev et al.

Original: (2020) follow the same structure of the example above, and span across gender, religion and race.
Revised: (2020) follow the same structure of the example above, and span across gender, religion and race.

Original: Suppose that the set contains M instances, and let the predictor’s probabilities of the i th instance for entail, contradict and neutral be e i , c i and n i .
Revised: Suppose that the set contains M instances, and let the predictor’s probabilities of the i th instance for entail, contradict and neutral be e i , c i and n i .

Original: Following Dev et al.
Revised: Following Dev et al.

Original: (2020), we report three measures of inference-based bias: (1) Net Neutral ( NN ): NN = 1 M (cid:80) Mi =1 n i ; (2) Fraction
Revised: (2020), we report three measures of inference-based bias: (1) Net Neutral ( NN ): NN = 1 M (cid:80) Mi =1 n i ; (2) Fraction

Original: Neutral ( FN ): FN
Revised: Neutral ( FN ): FN

Original: =
Revised: =

Original: M
Revised: M

Original: (cid:80)
Revised: (cid:80)

Original: Mi =1 1 n i = max ( e i ,c i ,n i ) ; (3) Threshold τ ( T: τ ): T : τ = 1 M (cid:80) Mi =1 1 n i >τ .
Revised: Mi =1 1 n i = max ( e i ,c i ,n i ) ; (3) Threshold τ ( T: τ ): T : τ = 1 M (cid:80) Mi =1 1 n i >τ .

Original: [Table]
Revised: [Table]

Original: Best scores are highlighted with bold
Revised: Best scores are highlighted with bold

Original: [Table]
Revised: [Table]

Original: In this experiment, we ﬁnetune text encoders of interest with the MNLI dataset for natural language inference (Wang et al., 2018).
Revised: In this experiment, we ﬁnetune text encoders of interest with the MNLI dataset for natural language inference (Wang et al., 2018).

Original: An ideal bias-free model should score 1 in all three measures.
Revised: An ideal bias-free model should score 1 in all three measures.

Original: We report our ﬁndings in Table 4 after transforming them into percentages.
Revised: We report our ﬁndings in Table 4 after transforming them into percentages.

Original: Our method outperforms the original model and the baselines.
Revised: Our method outperforms the original model and the baselines.

Original: This result shows that
Revised: This result shows that

Original: Att-D not only reduces bias at surface-level as reﬂected by likelihood-inspired evaluation methods, but also mitigates stereotypes in real world inference settings, unlike Sent-D which produces positive results with Crows-Pairs but comes short of meeting the same success in this experiment.
Revised: Att-D not only reduces bias at surface-level as reﬂected by likelihood-inspired evaluation methods, but also mitigates stereotypes in real world inference settings, unlike Sent-D which produces positive results with Crows-Pairs but comes short of meeting the same success in this experiment.

Original: In the next section, we show that these ﬁndings are meaningful since the entailment accuracy is not hurt after debiasing.
Revised: In the next section, we show that these ﬁndings are meaningful since the entailment accuracy is not hurt after debiasing.

Original: 4. E VALUATIONS OF R EPRESENTATIVENESS
Revised: 4. E VALUATIONS OF R EPRESENTATIVENESS

Original: We use GLUE benchmark (Wang et al., 2018) to verify whether the debiased text encoder still holds enough semantic information to be applicable in downstream NLP tasks.
Revised: We use GLUE benchmark (Wang et al., 2018) to verify whether the debiased text encoder still holds enough semantic information to be applicable in downstream NLP tasks.

Original: In essence, GLUE assesses the natural language understanding capabilities of NLP models.
Revised: In essence, GLUE assesses the natural language understanding capabilities of NLP models.

Original: So, it constitutes a suitable stack to evaluate the semantic preservation of Att-D. In this experiment, we ﬁnetune BERT on seven different tasks from GLUE and show the results in Table 5.
Revised: So, it constitutes a suitable stack to evaluate the semantic preservation of Att-D. In this experiment, we ﬁnetune BERT on seven different tasks from GLUE and show the results in Table 5.

Original: We also report the difference in accuracy between original BERT and each of the debiasing baselines.
Revised: We also report the difference in accuracy between original BERT and each of the debiasing baselines.

Original: Surprisingly, Att-D not only preserves semantic information, but enhances it in most GLUE tasks as reﬂected in an increase in accuracy from BERT.
Revised: Surprisingly, Att-D not only preserves semantic information, but enhances it in most GLUE tasks as reﬂected in an increase in accuracy from BERT.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper, we proposed a ﬁnetuning approach to debiasing that trains the text encoder to distribute its attention equally on different social groups.
Revised: In this paper, we proposed a ﬁnetuning approach to debiasing that trains the text encoder to distribute its attention equally on different social groups.

Original: Experiments demonstrate that bias is successfully reduced without harm to semantic representativeness.
Revised: Experiments demonstrate that bias is successfully reduced without harm to semantic representativeness.

Original: However, we are aware of the following limitations: (1) our deﬁnitions of biases are simpliﬁed.
Revised: However, we are aware of the following limitations: (1) our deﬁnitions of biases are simpliﬁed.

Original: There are more social divisions in the real world than the ﬁve dimensions we studied.
Revised: There are more social divisions in the real world than the ﬁve dimensions we studied.

Original: Besides, bias types can be correlated in intricate ways such as the links between race , nationality and ethnicity .
Revised: Besides, bias types can be correlated in intricate ways such as the links between race , nationality and ethnicity .

Original: Moreover, it is not clear which or how many groups to include.
Revised: Moreover, it is not clear which or how many groups to include.

Original: For these reasons, we follow previous work and constrain our study to simple deﬁnitions targeting the most widely spread groups.
Revised: For these reasons, we follow previous work and constrain our study to simple deﬁnitions targeting the most widely spread groups.

Original: We plan to study the effect that the choice of deﬁnition tuples and their order impose overall.
Revised: We plan to study the effect that the choice of deﬁnition tuples and their order impose overall.

Original: (2) We equalize the attention scores of every word in the input.
Revised: (2) We equalize the attention scores of every word in the input.

Original: However, some words are inherently charged with a strong inclination toward one group, e.g., beard to male or pregnant to female .
Revised: However, some words are inherently charged with a strong inclination toward one group, e.g., beard to male or pregnant to female .

Original: Such words need not be debiased, which requires compiling expensive lists of related words for every social group and protecting them from attention equalization.
Revised: Such words need not be debiased, which requires compiling expensive lists of related words for every social group and protecting them from attention equalization.

Original: In this work, we rely on knowledge distillation to retain as much useful semantic information as possible.
Revised: In this work, we rely on knowledge distillation to retain as much useful semantic information as possible.

Original: (3) Current bias detection experiments have positive predictive ability, which means that they can only detect the presence of bias, not the absence of it.
Revised: (3) Current bias detection experiments have positive predictive ability, which means that they can only detect the presence of bias, not the absence of it.

Original: Although contemporary evaluation tools demonstrate that our method successfully reduced social biases from sentence representations, it is possible that bias is still lurking in shapes and forms that our experimental lenses failed to detect.
Revised: Although contemporary evaluation tools demonstrate that our method successfully reduced social biases from sentence representations, it is possible that bias is still lurking in shapes and forms that our experimental lenses failed to detect.

Original: We plan to address these limitations in future work.
Revised: We plan to address these limitations in future work.

Original: A A PPENDIX A. T RAINING H YPERPARAMETERS
Revised: A A PPENDIX A. T RAINING H YPERPARAMETERS

Original: We used Adam optimizer (Kingma & Ba, 2014) with a learning rate of 5 e − 6 for 3 epochs.
Revised: We used Adam optimizer (Kingma & Ba, 2014) with a learning rate of 5 e − 6 for 3 epochs.

Original: We keep the betas to their default values (0.9, 0.999) as in PyTorch implementation (Paszke et al., 2017).
Revised: We keep the betas to their default values (0.9, 0.999) as in PyTorch implementation (Paszke et al., 2017).

Original: We set the loss coefﬁcient λ to 2 . 

Revised: We set the loss coefﬁcient λ to 2 .

Original: and the negative ratio η to 0 .
Revised: and the negative ratio η to 0 .

Original: 8 meaning that in 80% of the iterations, we use negative examples whose number we set to 5 in each negative iteration.
Revised: 8 meaning that in 80% of the iterations, we use negative examples whose number we set to 5 in each negative iteration.

Original: We only ﬁnetuned the values of λ , η , the learning rate, and the number of epochs.
Revised: We only ﬁnetuned the values of λ , η , the learning rate and the number of epochs.

Original: We conducted the hyperparameter search manually on the development set.
Revised: We conducted the hyperparameter search manually on the development set.

Original: As for GLUE experiments we follow the experimental setup of Devlin et al.
Revised: As for GLUE experiments we follow the experimental setup of Devlin et al.

Original: and train each task for 3 epochs with a learning rate of 2 e − 5 on their respective training 
Revised: and train each task for 3 epochs with a learning rate of 2 e − 5 on their respective training data.

Original: [Table]

Original: A. S OCIAL GROUPS USED IN THE EXPERIMENTS
Revised: A. S OCIAL GROUPS USED IN THE EXPERIMENTS

Original: While the approach is independent of the deﬁnition of social groups and categories (it could work for any kind of grouping, e.g., cuisine styles or sports), in the experiment we focus on groups commonly used in the debiasing literature: binary gender, religion, race, age and sexual orientation.
Revised: While the approach is independent of the deﬁnition of social groups and categories (it could work for any kind of grouping, e.g., cuisine styles or sports), in the experiment we focus on groups commonly used in the debiasing literature: binary gender, religion, race, age and sexual orientation.

Original: This is to facilitate comparison, but nothing in the approach prevent it from being used with broader and more inclusive groups.
Revised: This is to facilitate comparison, but nothing in the approach prevent it from being used with broader and more inclusive groups.

Original: This being said, we have not experimented yet with debiasing where a dimension is divided in dozens of categories.
Revised: This being said, we have not experimented yet with debiasing where a dimension is divided in dozens of categories.

Original: We list the deﬁnition tuples that we used in Table 6.
Revised: We list the deﬁnition tuples that we used in Table 6.

Original: We show that Att-D does not incur strict rules for deﬁning social groups, unlike previous work (Bolukbasi et al., 2016; Kaneko & Bollegala, 2019; 2021) that require the deﬁnition words to be organized in a predeﬁned format (pairs of words or bag of words for every group), and provided in relatively large quantities.
Revised: We show that Att-D does not incur strict rules for deﬁning social groups, unlike previous work (Bolukbasi et al., 2016; Kaneko & Bollegala, 2019; 2021) that require the deﬁnition words to be organized in a predeﬁned format (pairs of words or bag of words for every group), and provided in relatively large quantities.

Original: We can see from Table 6 that it is sufﬁcient to deﬁne one tuple per bias type (e.g., race) if the tuples are hard to come by.
Revised: We can see from Table 6 that it is sufﬁcient to deﬁne one tuple per bias type (e.g., race) if the tuples are hard to come by.

Original: Also, the tuples need not be of the same size (e.g., in religion there is a missing word for buddhist group since it is not clear which word to use in that tuple).
Revised: Also, the tuples need not be of the same size (e.g., in religion there is a missing word for buddhist group since it is not clear which word to use in that tuple).

Original: This desired property owes to the fact that Att-D does not learn subspaces or directions for every bias type as previous works do (Bolukbasi et al., 2016;
Revised: This desired property owes to the fact that Att-D does not learn subspaces or directions for every bias type as previous works do (Bolukbasi et al., 2016;

Original: Kaneko & Bollegala, 2019; Kumar et al., 2020; Kaneko & Bollegala, 2021).
Revised: Kaneko & Bollegala, 2019; Kumar et al., 2020; Kaneko & Bollegala, 2021).

Original: In contrast, Att-D uses the tuples in order to equalize the attentions of the input sentence, and make the words therein attend to the groups with the same intensity.
Revised: In contrast, Att-D uses the tuples in order to equalize the attentions of the input sentence, and make the words therein attend to the groups with the same intensity.

Original: These example categories used in experiments are neither complete nor exhaustive, and in some experiments also include terms possibly considered inappropriate but that appear in the corpus and we may still want to debias from (such as using ”straight” to deﬁne heterosexual).
Revised: These example categories used in experiments are neither complete nor exhaustive, and in some experiments also include terms possibly considered inappropriate but that appear in the corpus and we may still want to debias from (such as using ”straight” to deﬁne heterosexual).

Original: A. C HOICE OF LAYERS
Revised: A. C HOICE OF LAYERS

Original: Transformer-based text encoders consist of many layers.
Revised: Transformer-based text encoders consist of many layers.

Original: It is not clear which layers to choose for debiasing since bias information is spread out across all of them.
Revised: It is not clear which layers to choose for debiasing since bias information is spread out across all of them.

Original: In this experiment, we try different  debiasing settings in which we select different layer combinations of BERT to work on: all layers, ﬁrst 6 , ﬁrst 3 , last 6 , last 3 , and alternating layers with strides of 1 (layers 2, 4, 6, 8, 10 and 12) or 2 (layers 4, 8 and 12).
Revised: In this experiment, we try different [Table] [Table] debiasing settings in which we select different layer combinations of BERT to work on: all layers, ﬁrst 6 , ﬁrst 3 , last 6 , last 3 , and alternating layers with strides of 1 (layers 2, 4, 6, 8, 10 and 12) or 2 (layers 4, 8 and 12).

Original: We apply the debiasing method proposed in this paper, and report both language modeling and stereotype scores of StereoSet benchmark in Table 7.
Revised: We apply the debiasing method proposed in this paper, and report both language modeling and stereotype scores of StereoSet benchmark in Table 7.

Original: The results show that it is safest to equalize attention heads of all layers of the text encoder under study, since it produces the best scores both in terms of language modeling and stereotype.Our [Table] [Table]
Revised: The results show that it is safest to equalize attention heads of all layers of the text encoder under study, since it produces the best scores both in terms of language modeling and stereotype.

Original: [Table]

Original: [Table]

Original: [Table]

Original:  ﬁndings go in tandem with those of Liang et al.
Revised: Our ﬁndings go in tandem with those of Liang et al.

Original: (2020b)
Revised: (2020b)

Original: ; Kaneko & Bollegala (2021); Bhardwaj et al.
Revised: ; Kaneko & Bollegala (2021); Bhardwaj et al.

Original: (2021) who found that reducing bias from all layers usually is the best option.
Revised: (2021) who found that reducing bias from all layers usually is the best option.

Original: A. E FFECT OF NEGATIVE EXAMPLES ON REPRESENTATIVENESS
Revised: A. E FFECT OF NEGATIVE EXAMPLES ON REPRESENTATIVENESS

Original: We remind that the introduction of negative examples to training serves in forcing the text encoder not to rely on a dangerous shortcut which is distributing its attention uniformly on all the tokens constituting the second half of the input, no matter what the input is.
Revised: We remind that the introduction of negative examples to training serves in forcing the text encoder not to rely on a dangerous shortcut which is distributing its attention uniformly on all the tokens constituting the second half of the input, no matter what the input is.

Original: This is particularly important in double-sentence tasks where the text encoder is given two input sentences.
Revised: This is particularly important in double-sentence tasks where the text encoder is given two input sentences.

Original: In addition to Tablesand 3 which highlighted the effect of negative sampling on the ﬁnal stereotype scores, the primary goal of using negative examples remains the preservation of the text encoder’s representativeness.
Revised: In addition to Tablesand 3 which highlighted the effect of negative sampling on the ﬁnal stereotype scores, the primary goal of using negative examples remains the preservation of the text encoder’s representativeness.

Original: In Table 8, we report the performance of Att-D and Att-D - with and without negative examples respectively on GLUE tasks.
Revised: In Table 8, we report the performance of Att-D and Att-D - with and without negative examples respectively on GLUE tasks.

Original: Unsurprisingly, the lack of negative examples does not damage the performance of single-sentence tasks since these ignore the second half of the input altogether.
Revised: Unsurprisingly, the lack of negative examples does not damage the performance of single-sentence tasks since these ignore the second half of the input altogether.

Original: However, in double-sentence tasks where both halves are used for prediction, Table 8 shows that negative sampling plays a pivotal role in preserving the semantics of text encoders, and bypassing the side effects inﬂicted by attention equalization. 
Revised: However, in double-sentence tasks where both halves are used for prediction, Table 8 shows that negative sampling plays a pivotal role in preserving the semantics of text encoders, and bypassing the side effects inﬂicted by attention equalization. [Table]

Original: A. V ISUALIZING DEBIASING RESULTS
Revised: A. V ISUALIZING DEBIASING RESULTS

Original: In this experiment, we aim to visualize the effects of debiasing on attention weights.
Revised: In this experiment, we aim to visualize the effects of debiasing on attention weights.

Original: We only focus on binary gender bias for two reasons: First, it is easier to visualize binary variables on a 2D plane than multiclass variables (such as race, religion...).
Revised: We only focus on binary gender bias for two reasons: First, it is easier to visualize binary variables on a 2D plane than multiclass variables (such as race, religion...).

Original: Second, gender is the most well studied bias type (Bolukbasi et al., 2016; Caliskan et al., 2017;
Revised: Second, gender is the most well studied bias type (Bolukbasi et al., 2016; Caliskan et al., 2017;

Original: May et al., 2019), so linguistic resources and vocabularies for gender exist and are well documented.
Revised: May et al., 2019), so linguistic resources and vocabularies for gender exist and are well documented.

Original: We use the vocabulary words compiled by Kaneko & Bollegala (2019) and categorized into three non-overlapping subsets: (1)
Revised: We use the vocabulary words compiled by Kaneko & Bollegala (2019) and categorized into three non-overlapping subsets: (1)

Original: Male-deﬁnition Ω M whose corresponding words are exclusively male-gendered such as father , king or uncle .
Revised: Male-deﬁnition Ω M whose corresponding words are exclusively male-gendered such as father , king or uncle .

Original: (2) Female-deﬁnition Ω F which is a set of inherently female words ( mother , queen , aunt ...).
Revised: (2) Female-deﬁnition Ω F which is a set of inherently female words ( mother , queen , aunt ...).

Original: (3) Gender-stereotype Ω S which is constituted of words that are not gendered by deﬁnition, but that carry a strong gender stereotype such as doctor being attributed to male or nurse to female .
Revised: (3) Gender-stereotype Ω S which is constituted of words that are not gendered by deﬁnition, but that carry a strong gender stereotype such as doctor being attributed to male or nurse to female .

Original: For every word w ∈ Ω M ∪ Ω F ∪ Ω S , we extract sentences from the News-commentary-v15 corpus where w is mentioned.
Revised: For every word w ∈ Ω M ∪ Ω F ∪ Ω S , we extract sentences from the News-commentary-v15 corpus where w is mentioned.

Original: We denote this set as S w .
Revised: We denote this set as S w .

Original: Then, for every sentence s ∈ S w , we append the dummy input ”man, woman” as explained in Sections ??
Revised: Then, for every sentence s ∈ S w , we append the dummy input ”man, woman” as explained in Sections ??

Original: and 3.2.1.
Revised: and 3.2.1.

Original: The augmented input s (cid:48) is then fed to the text encoder of interest (BERT base in this experiment), and we collect the attention scores of w on the second-half tokens man and woman .
Revised: The augmented input s (cid:48) is then fed to the text encoder of interest (BERT base in this experiment), and we collect the attention scores of w on the second-half tokens man and woman .

Original: Finally, for every word w ∈ Ω M ∪ Ω F ∪ Ω S , we take the mean of its attention scores in S w .
Revised: Finally, for every word w ∈ Ω M ∪ Ω F ∪ Ω S , we take the mean of its attention scores in S w .

Original: By the end of this procedure, we have for every word w its 
Revised: By the end of this procedure, we have for every word w its attention score on the words man ( a wm ) and woman ( a wf ) as computed on the News-commentary-v corpus which includes overall 223,153 sentences.

Original: [Figure]
Revised: We take the difference a wm − a wf which indicates the preference of the text encoder to consider w as male (positive difference) or female (negative difference). The absence of gender bias is reﬂected in difference scores near zero.

Original: We plot the results in Figure 3 where the x-axis represents the differences a wm − a wf , and the y-axis random values to separate the words vertically.
Revised: We plot the results in Figure 3 where the x-axis represents the differences a wm − a wf , and the y-axis random values to separate the words vertically.

Original: Stereotype words (green dots) should have values near 0, which is not the case in Figure 3(b).
Revised: Stereotype words (green dots) should have values near 0, which is not the case in Figure 3(b).

Original: This means that BERT has a strong preference for one of the genders, and is thus heavily biased.
Revised: This means that BERT has a strong preference for one of the genders, and is thus heavily biased.

Original: In contrast, our method brings the attention of stereotype words near 0, meaning that they prefer neither male nor female connotations.
Revised: In contrast, our method brings the attention of stereotype words near 0, meaning that they prefer neither male nor female connotations.

Original: Moreover, the spread of stereotype words in Figure 3(d) is narrower than male- or female-oriented words, which is desired since these are inherently gendered and must pick a side.
Revised: Moreover, the spread of stereotype words in Figure 3(d) is narrower than male- or female-oriented words, which is desired since these are inherently gendered and must pick a side.

Original: This result strengthens the claim that AttD preserves semantic information, and is less severe in reducing bias from gendered words as it is on gender-neutral words.
Revised: This result strengthens the claim that AttD preserves semantic information, and is less severe in reducing bias from gendered words as it is on gender-neutral words.

Original: The difference in spread is less apparent in the original BERT model.
Revised: The difference in spread is less apparent in the original BERT model.

Original: We also note that debiasing the embeddings of BERT rather than the attention mechanism as in
Revised: We also note that debiasing the embeddings of BERT rather than the attention mechanism as in

Original: Kaneko & Bollegala (2021)
Revised: Kaneko & Bollegala (2021)

Original: (Figure 3(c)) is not enough since bias information is still lurking (and perhaps made worse for some words) in the attention component.
Revised: (Figure 3(c)) is not enough since bias information is still lurking (and perhaps made worse for some words) in the attention component.

Original: Thus, we conclude that working on attention directly constitutes our best option for debiasing to date.
Revised: Thus, we conclude that working on attention directly constitutes our best option for debiasing to date.

Original: A. E FFECT OF ATTENTION - BASED DEBIASING ON OTHER TRANSFORMER - BASED TEXT ENCODERS
Revised: A. E FFECT OF ATTENTION - BASED DEBIASING ON OTHER TRANSFORMER - BASED TEXT ENCODERS

Original: We evaluate ﬁve widely used sentence-level text encoders: BERT (Devlin et al., 2018), ALBERT (Lan et al., 2019), RoBERTa (Liu et al., 2019), DistilBERT (SANH et al.) and SqueezeBERT (Ian- [Table] dola et al., 2020).
Revised: We evaluate ﬁve widely used sentence-level text encoders: BERT (Devlin et al., 2018), ALBERT (Lan et al., 2019), RoBERTa (Liu et al., 2019), DistilBERT (SANH et al.) and SqueezeBERT (Iandola et al., 2020).

Original: For each model [Table] we evaluate both its base and large variants (except for DistlBERT and SqueezeBERT since these are not available in HuggingFace’s transformers library 14 ), original and debiased; which gives a total of sixteen evaluated models.
Revised: For each model, we evaluate both its base and large variants (except for DistlBERT and SqueezeBERT since these are not available in HuggingFace’s transformers library 15 ), original and debiased; which gives a total of sixteen evaluated models.

Original: We use Crows-Pairs dataset (Nangia et al.) to quantify the intensity of undesired stereotypes encoded therein.
Revised: We use Crows-Pairs dataset (Nan- [Figure] [Table] gia et al.) to quantify the intensity of undesired stereotypes encoded therein.

Original: As a reminder, ideal stereotype scores according to Crows-Pairs benchmark should be close to 50, i.e. models preferring neither stereotypes nor anti-stereotypes.
Revised: As a reminder, ideal stereotype scores according to Crows-Pairs benchmark should be close to 50, i.e. models preferring neither stereotypes nor anti-stereotypes.

Original: Tables 9, 10, 11 and 12 show the bias results for BERT,
Revised: Tables 9, 10, 11 and 12 show the bias results for BERT,

Original: ALBERT, RoBERTa and DistillBERT/SqueezeBERT respectively.
Revised: ALBERT, RoBERTa and DistillBERT/SqueezeBERT respectively.

Original: All ﬁve models exhibit substantial levels of bias, and in each of the bias types with differing intensities (religion, sexual orientation and disability being the bias categories with the most severe stereotyping).
Revised: All ﬁve models exhibit substantial levels of bias, and in each of the bias types with differing intensities (religion, sexual orientation and disability being the bias categories with the most severe stereotyping).

Original: Also, we ﬁnd that the large variants are more biased than their base counterparts mainly because large models, with their larger capacity and greater number of parameters, can capture more intricate and more sophisticated aspects of training data, exposing them to learn more bias.
Revised: Also, we ﬁnd that the large variants are more biased than their base counterparts mainly because large models, with their larger capacity and greater number of parameters, can capture more intricate and more sophisticated aspects of training data, exposing them to learn more bias.

Original: This ﬁnding corresponds well to results of previous work (Nangia et al.
Revised: This ﬁnding corresponds well to results of previous work (Nangia et al.

Original: ; Nadeem et al., 2020).
Revised: ; Nadeem et al., 2020).

Original: The tables also show that Att-D is effective in mitigating bias from BERT, ALBERT, RoBERTa, DistilBERT and SqueezeBERT, and produces a reduction of up to 25%.
Revised: The tables also show that Att-D is effective in mitigating bias from BERT, ALBERT, RoBERTa, DistilBERT and SqueezeBERT, and produces a reduction of up to 25%.

Original: We note that Att-D succeeds in debiasing all models, with varying effectiveness across bias types.
Revised: We note that Att-D succeeds in debiasing all models, with varying effectiveness across bias types.

Original: We also note that Att-D meets the best success with ALBERT as reductions are greater on this particular text encoder.
Revised: We also note that Att-D meets the best success with ALBERT as reductions are greater on this particular text encoder.

Original: We believe this is because ALBERT is composed of a single transformer layer (Lan et al., 2019) with substantially less parameters than BERT or RoBERTa; which makes debiasing easier since there is no interference between different attention layers.
Revised: We believe this is because ALBERT is composed of a single transformer layer (Lan et al., 2019) with substantially less parameters than BERT or RoBERTa; which makes debiasing easier since there is no interference between different attention layers.

Original: Finally, we see from the tables that Att-D sometimes contributes to adding a bit of bias.
Revised: Finally, we see from the tables that Att-D sometimes contributes to adding a bit of bias.

Original: We observe that this phenomenon is rare, and happens especially with bias types we did not include in our design 15 .
Revised: We observe that this phenomenon is rare, and happens especially with bias types we did not include in our design 16 .

Original: We assume that not explicitly compelling the text encoder to equalize attention heads corresponding to these overlooked bias types gave it green light to adjust these attentions in a way to facilitate solving the optimization problem; even if it entails adding bias.
Revised: We assume that not explicitly compelling the text encoder to equalize attention heads corresponding to these overlooked bias types gave it green light to adjust these attentions in a way to facilitate solving the optimization problem; even if it entails adding bias.

Original: We plan to include all bias types present in Crows-Pairs dataset to our debiasing design as a future work.
Revised: We plan to include all bias types present in Crows-Pairs dataset to our debiasing design as a future work.

Revised: A. A PPLICATION - ORIENTED B IAS E VALUATION

Original: [Table]
Revised: Recent studies show that intrinsic metrics of bias do not necessarily correlate with bias measures on concrete real-world applications (Goldfarb-Tarrant et al., 2020). In the body of this paper, we already conducted intrinsic and extrinsic bias evaluations. In this experiment, we validate the efﬁcacy of our debiasing method on a concrete real-world hate speech detection application where an input snippet of text is classiﬁed as either offensive ( toxic , harmful , disrespectful ...) or not. We use hate speech detection because it is well studied in the literature (Burnap & Williams, 2016; Ribeiro et al., 2018;

Revised: Zhang et al., 2018), and high-quality datasets which are tagged with social groups already exist (Borkan et al., 2019; Mathew et al., 2021).

Original: [Table] 
Revised: [Table] [Table] [Table]

Revised: Admittedly, common social biases have also been shown to exist in hate speech detection models, for example in associating toxicity to frequently attacked groups (such as ”muslim” or ”gay”) even if the text itself is not toxic (Dixon et al., 2018; Park et al., 2018).

Revised: In this experiment, we adopt the bias deﬁnition of Borkan et al.

Revised: (2019) which casts bias as a skewing in the hate speech detector scores based solely on the social groups mentioned in the text.

Revised: In other words, we consider a model to exhibit unintended social stereotypes if the model’s performance varies across groups.

Revised: We use the bias measures proposed by Borkan et al.

Revised: (2019) which are based on the Area Under the Receiver

Revised: Operating Characteristic Curve (ROC-AUC, or AUC) metric.

Revised: AUC measures the probability that a randomly chosen negative example (not offensive) receives a lower toxicity score than a randomly chosen positive example (offensive), meaning that a perfect model should always have an AUC score of 1.0.

Revised: Stated differently, all negative examples have lower toxicity scores than positive examples.

Revised: While AUC is used to measured the general performance of classiﬁers, Borkan et al.

Revised: (2019) propose three extensions of AUC to measure bias.

Revised: We summarize them in the following:

Revised: Subgroup (Sub) AUC: where AUC is computed only on the group under consideration and not on all the examples of the test benchmark, i.e. only positive and negative examples of the target group are considered.

Revised: This metric represents the model’s performance on a given group.

Revised: A higher value means that the model is good at distinguishing between toxic and non-toxic texts speciﬁc to the group.

Revised: Background Positive Subgroup Negative (

Revised: BPSN

Revised: ) AUC: where AUC is calculated on the negative examples of the target group, and the positive examples of the background (all other groups except the group under consideration).

Revised: This metric computes whether the model discriminates against the target group with respect to the others.

Revised: This value is reduced when non-toxic examples of the group have higher toxicity scores than actually toxic examples of the background.

Revised: Background

Revised: Negative Subgroup Positive (BNSP) AUC:

Revised: where AUC is calculated on the positive examples of the target group, and the negative examples of the background.

Revised: This metric computes whether the model favors the target group with respect to the others.

Revised: This value is reduced when toxic examples of the group have lower toxicity scores than non-toxic examples of the background.

Revised: In this experiment, we ﬁnetune the text encoder under study on hate speech detection task using the training set of HateXplain dataset (Mathew et al., 2021).

Revised: We also use the test portion of HateXplain for the evaluation, which contains posts from Twitter 17 and Gab 18 annotated with their ground-truth toxicity scores and the social groups and communities they target.

Revised: Fundamentally, the three metrics described above give bias scores per group.

Revised: In order to combine the per group scores in one overall measure, we apply the Generalized Mean of Bias (GMB) introduced by the Google Conversation

Revised: AI Team as part of their Kaggle competition 19 , and later used by Mathew et al.

Revised: (2021) in their own evaluations.

Revised: The formula of GMB is as the following:

Revised: [Equation]

Revised: where b is an array of AUC scores per group, and b g is the AUC score of group g .

Revised: We follow

Revised: Mathew et al.

Revised: and set p to -5 .

Revised: We compute the GMB of all three metrics: Subgroup, BPSN and BNSP.

Revised: As for Subgroup, we also add the standard deviation as it gives valuable information about how much the performance of the hate speech detection model varies across groups.

Revised: We report our results in Table 13, in addition to classic performance measures.

Revised: We observe that Att-D provides competitive results across the four bias metrics, and largely outperforms the baselines.

Revised: Especially with GMB-BNSP , where bias scores of the original model are very low (i.e. it is throttled by social biases), we observe the best improvements overall, and by a large margin compared to existing debiasing methods.

Revised: Also, the variance in model performance is lowest with Att-D, which conﬁrms that the corresponding hate speech detection model has less stereotypes about different social groups.

Revised: Finally, the general performance (Accuracy, F1 score and AUC)

Original: [Table]
Revised: [Table]

Revised: A. W ORD -L EVEL VS S ENTENCE -L EVEL D EBIASING

Revised: As previously explained in the paper, Att-D calibrates the attention weighs of all tokens of the input sentence on group-related words.

Revised: Since we used BERT-based models in our experiments, the ﬁrst token in the input is the special [CLS] token, which is considered by the NLP community as a vector representation for the entire input sentence.

Revised: In the current version of Att-D, we also calibrate the attention weighs of the special [CLS] token on groups, in addition to calibrating the other tokens of the sentence.

Revised: One can see this notion as a combined word-level and sentence-level debiasing.

Revised: In this experiment, we motivate this design choice by comparing it to word-level and sentence-level debiasing separately.

Revised: For word-level, we exclude the [CLS] token from the attention equalization process, whereas in sentence-level we only calibrate the attention of [CLS].

Revised: We use all the bias evaluations run so far to understand the difference in performance.

Revised: Tables 14, 15, 16,and 18 report the results of StereoSet, Crows-Pairs, inference, hate speech and GLUE experiments respectively.

Revised: We denote word-level debiasing by No [CLS] , and sentence-level debiasing by Only [CLS] in the tables.

Revised: The combination of both is referred to as Att-D , and is the variant that we promote in this paper.

Revised: We observe that while the three settings are good at reducing bias from text encoders, Att-D is superior than word-level and sentence-level debiasing since it capitalizes on the beneﬁts of both.

Revised: It enjoys the ﬁne granularity of reducing bias from every word, while it also mitigates biases that manifest at sentence-level.

Revised: A. S TATIC VS R ANDOM ORDERING OF GROUP - RELATED WORDS

Revised: In the preprocessing step of our method (as explained in Section 3.2.1), we use a preset ordering of group-related words of a given bias type to form the second input.

Revised: For example, if we have the groups Muslim , Christian , Jew and Buddhist deﬁning the religion bias type, Att-D constructs the second input using the same preset ordering of groups across all samples of the training data.

Revised: Continuing the example above, Att-D appends the following artiﬁcial sentence ”muslim, christian, jew, buddhist”.

Revised: In this experiment, we change the ordering of groups in a random way.

Revised: Tables 14, 15, 16, 17 and 18 also report the bias scores of Att-D (static ordering) and Att-D with random ordering.

Revised: Although the semantic performance of Att-D with random ordering is better, we notice that it suffers from a stronger presence of bias than in its static counterpart.

Revised: In Table 17, Att-D with random ordering has an AUC score of 0 in one of the groups, which made the GMB extremely small.

Revised: We suspect that the relatively poor fairness of random ordering owes to the fact that the model might be confused by different orderings throughout the iterations.

Revised: A more serious analysis of the impact of group order on the overall performance (fairness and semantics) of Att-D motivates the direction of future work.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]