Original: D ETECTING AND M ITIGATING I NDIRECT S TEREOTYPES IN W ORD E MBEDDINGS
Revised: D ETECTING AND M ITIGATING I NDIRECT S TEREOTYPES IN W ORD E MBEDDINGS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Societal biases in the usage of words, including harmful stereotypes, are frequently learned by common word embedding methods.
Revised: Societal biases in the usage of words, including harmful stereotypes, are frequently learned by common word embedding methods.

Original: These biases manifest not only between a word and an explicit marker of its stereotype, but also between words that share related stereotypes.
Revised: These biases manifest not only between a word and an explicit marker of its stereotype, but also between words that share related stereotypes.

Original: This latter phenomenon, sometimes called “indirect bias,” has resisted prior attempts at debiasing.
Revised: This latter phenomenon, sometimes called “indirect bias,” has resisted prior attempts at debiasing.

Original: In this paper, we propose a novel method to mitigate indirect bias in distributional word embeddings by modifying biased relationships between words before embeddings are learned.
Revised: In this paper, we propose a novel method to mitigate indirect bias in distributional word embeddings by modifying biased relationships between words before embeddings are learned.

Original: This is done by considering how the co-occurrence probability of a given pair of words changes in the presence of words marking an attribute of bias, and using this to average out the effect of a bias attribute.
Revised: This is done by considering how the co-occurrence probability of a given pair of words changes in the presence of words marking an attribute of bias, and using this to average out the effect of a bias attribute.

Original: To evaluate this method, we perform a series of common tests and demonstrate thatthe semantic quality of the word embeddings is retained while measures of bias in the  embeddings are reduced.
Revised: To evaluate this method, we perform a series of common tests and demonstrate that measures of bias in the word embeddings are reduced in exchange for some reduction in the semantic quality of the embeddings.

Original: In addition, we conduct novel tests for measuring indirect stereotypes by extending the Word Embedding Association Test (WEAT) with new test sets for indirect binary gender stereotypes.
Revised: In addition, we conduct novel tests for measuring indirect stereoFIX types by extending the Word Embedding Association Test (WEAT) with new test sets for indirect binary gender stereotypes.

Original: With these tests, we demonstratethat this method can reduce the presence of more subtle stereotypes notproperly addressed by previous work. 
Revised: With these tests, we demonstrate the presence of more subtle stereotypes not addressed by previous work. The proposed method is able to reduce the presence of some of these new stereotypes, serving as a crucial next step towards non-stereotyped word embeddings. FIX

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Distributional word embeddings, such as Word2Vec (Mikolov et al., 2013a) and GloVe (Pennington et al., 2014), are computer representations of words as vectors in semantic space.
Revised: Distributional word embeddings, such as Word2Vec (Mikolov et al., 2013a) and GloVe (Pennington et al., 2014), are computer representations of words as vectors in semantic space.

Original: These embeddings are popular because the geometry of the vectors corresponds to semantic and syntactic structure (Mikolov et al., 2013b).
Revised: These embeddings are popular because the geometry of the vectors corresponds to semantic and syntactic structure (Mikolov et al., 2013b).

Original: Unfortunately, societal stereotypes, such as those pertaining to race, gender, national origin, or sexuality, are typically reflected in word embeddings (Bolukbasi et al., 2016;
Revised: Unfortunately, societal stereotypes, such as those pertaining to race, gender, national origin, or sexuality, are typically reflected in word embeddings (Bolukbasi et al., 2016;

Original: Caliskan et al., 2017; Garg et al., 2018; Papakyriakopoulos et al., 2020).
Revised: Caliskan et al., 2017; Garg et al., 2018; Papakyriakopoulos et al., 2020).

Original: These stereotypes are so pervasive that they have proved resistant to many existing debiasing techniques (Gonen &
Revised: These stereotypes are so pervasive that they have proved resistant to many existing debiasing techniques (Gonen &

Original: Goldberg, 2019).
Revised: Goldberg, 2019).

Original: Techniques attempting to remove or mitigate bias in word vectors are common in the literature.
Revised: Techniques attempting to remove or mitigate bias in word vectors are common in the literature.

Original: The typical case study for bias mitigation methods in the literature is binary 1 gender.
Revised: The typical case study for bias mitigation methods in the literature is binary 1 gender.

Original: Subspace methods, such as hard debiasing from Bolukbasi et al.
Revised: Subspace methods, such as hard debiasing from Bolukbasi et al.

Original: (2016) and GN-GloVe from Zhao et al.
Revised: (2016) and GN-GloVe from Zhao et al.

Original: (2018b), attempt to identify or create a vector subspace of gender-related information (typically a “gender direction”) and drop this subspace.
Revised: (2018b), attempt to identify or create a vector subspace of gender-related information (typically a “gender direction”) and drop this subspace.

Original: Counterfactual Data Substitution from Maudslay et al.
Revised: Counterfactual Data Substitution from Maudslay et al.

Original: (2019), based on Counterfactual Data Augmentation from Lu et al.
Revised: (2019), based on Counterfactual Data Augmentation from Lu et al.

Original: (2020), swaps explicitly gendered words to counter stereotyped associations.
Revised: (2020), swaps explicitly gendered words to counter stereotyped associations.

Original: James & Alvarez-Melis (2019) and Qian et al.
Revised: James & Alvarez-Melis (2019) and Qian et al.

Original: both propose methods to reduce bias towards binary gender by encouraging learned conditional probabilities of words appearing with “he” and with “she” to be equal.
Revised: both propose methods to reduce bias towards binary gender by encouraging learned conditional probabilities of words appearing with “he” and with “she” to be equal.

Original: Gonen & Goldberg (2019) showed that common “debiasing” methods failed to meaningfully reduce bias in word embeddings.
Revised: Gonen & Goldberg (2019) showed that common “debiasing” methods failed to meaningfully reduce bias in word embeddings.

Original: They describe how bias can manifest not only as undesirable association between stereotyped words and words marking a bias attribute 2 , but also between stereotyped words themselves.
Revised: They describe how bias can manifest not only as undesirable association between stereotyped words and words marking a bias attribute 2 , but also between stereotyped words themselves.

Original: These manifestations are sometimes called direct bias and indirect bias 3 , using the terminology introduced by Bolukbasi et al.
Revised: These manifestations are sometimes called direct bias and indirect bias 3 , using the terminology introduced by Bolukbasi et al.

Original: An example of this second manifestation of bias is that the word “doctor” might be associated more strongly with stereotypically masculinewords than with stereotypically feminine words.
Revised: An example of this second manifestation of bias is that the word “doctor” might be associated more strongly with stereotypically masculinewords than with stereotypically feminine words.

Original: At the time, bias mitigation algorithms commonly attempted to address direct bias while leaving indirect bias mostly present.
Revised: At the time, bias mitigation algorithms commonly attempted to address direct bias while leaving indirect bias mostly present.

Original: A common trend in the study of the indirect bias is the departure from stereotypes as the object of study in favor of clustering .
Revised: A common trend in the study of the indirect bias is the departure from stereotypes as the object of study in favor of clustering .

Original: While the measures introduced by Bolukbasi et al.
Revised: While the measures introduced by Bolukbasi et al.

Original: (2016); Caliskan et al.
Revised: (2016); Caliskan et al.

Original: (2017) attempt to quantify the existence of commonly understood stereotypes, work on indirect bias typically uses the measures introduced by Gonen & Goldberg (2019) which merely attempt to measure how well proposed bias mitigation methods disperse words with similar relationships to the bias attribute in the embedding space.
Revised: (2017) attempt to quantify the existence of commonly understood stereotypes, work on indirect bias typically uses the measures introduced by Gonen & Goldberg (2019) which merely attempt to measure how well proposed bias mitigation methods disperse words with similar relationships to the bias attribute in the embedding space.

Original: These clustering measures, while useful at capturing some forms of indirect bias, are limited.
Revised: These clustering measures, while useful at capturing some forms of indirect bias, are limited.

Original: In particular, it is unclear how dispersed stereotyped words should be in the embedding space, given that the stereotype of a word is not entirely arbitrary and can potentially be estimated based on its semantic, non-stereotypical, meaning.
Revised: In particular, it is unclear how dispersed stereotyped words should be in the embedding space, given that the stereotype of a word is not entirely arbitrary and can potentially be estimated based on its semantic, non-stereotypical, meaning.

Original: These new bias measures have inspired countless new bias mitigation methods.
Revised: These new bias measures have inspired countless new bias mitigation methods.

Original: Nearest neighbor bias mitigation from James & Alvarez-Melis (2019) attempts to equalize each word’s association with its masculine (defined by the original undebiased embeddings) neighbors and its feminine neighbors.
Revised: Nearest neighbor bias mitigation from James & Alvarez-Melis (2019) attempts to equalize each word’s association with its masculine (defined by the original undebiased embeddings) neighbors and its feminine neighbors.

Original: Double hard debias from Wang et al.
Revised: Double hard debias from Wang et al.

Original: (2020) projects off the direction defined by the “most gender-biased words” (again, based on alignment in the original embedding’s “gender direction”) in addition to the standard gender-related subspace.
Revised: (2020) projects off the direction defined by the “most gender-biased words” (again, based on alignment in the original embedding’s “gender direction”) in addition to the standard gender-related subspace.

Original: Bordia & Bowman (2019) modify the loss function when learning word embeddings to penalize neutral words having large components in the gender-related subspace which can then be dropped off.
Revised: Bordia & Bowman (2019) modify the loss function when learning word embeddings to penalize neutral words having large components in the gender-related subspace which can then be dropped off.

Original: Kumar et al.
Revised: Kumar et al.

Original: (2020) propose RAN-Debias which attempts to disperse words in the embedding space that share similar binary gender biases (defined again by the original word embeddings) while preserving the original geometry as much as possible.
Revised: (2020) propose RAN-Debias which attempts to disperse words in the embedding space that share similar binary gender biases (defined again by the original word embeddings) while preserving the original geometry as much as possible.

Original: Lauscher et al.
Revised: Lauscher et al.

Original: (2020) describe multiple bias mitigation methods: the standard projection method, averaging original word vectors with an orthogonal transformation that attempts to swap the bias attribute, and a neural method that uses a loss function to group together words exhibiting a bias attribute away from neutral words.
Revised: (2020) describe multiple bias mitigation methods: the standard projection method, averaging original word vectors with an orthogonal transformation that attempts to swap the bias attribute, and a neural method that uses a loss function to group together words exhibiting a bias attribute away from neutral words.

Original: These methods, similarly to the bias measures of Gonen &
Revised: These methods, similarly to the bias measures of Gonen &

Original: Goldberg (2019), focus on the clustering and dispersion of words in relation to the bias attribute.
Revised: Goldberg (2019), focus on the clustering and dispersion of words in relation to the bias attribute.

Original: In current state-of-the-art models, word embeddings have largely been replaced by contextualized embeddings from transformer models such as BERT (Devlin et al., 2018) and GPT (Radford et al., 2018).
Revised: In current state-of-the-art models, word embeddings have largely been replaced by contextualized embeddings from transformer models such as BERT (Devlin et al., 2018) and GPT (Radford et al., 2018).

Original: However, word embeddings remain a popular object of study when quantifying bias in NLP, in part due to their simplicity and theoretical results that make them easier to reason about.
Revised: However, word embeddings remain a popular object of study when quantifying bias in NLP, in part due to their simplicity and theoretical results that make them easier to reason about.

Original: As advances in the understanding of bias and stereotypes in word embeddings have been adapted for these newer models (Liang et al., 2020; May et al., 2019), novel techniques to measure and mitigate bias in word embeddings remain relevant.
Revised: As advances in the understanding of bias and stereotypes in word embeddings have been adapted for these newer models (Liang et al., 2020; May et al., 2019), novel techniques to measure and mitigate bias in word embeddings remain relevant.

Original:  B ACKGROUND 2. W ORD EMBEDDINGS
Revised:  B ACKGROUND 2. W ORD EMBEDDINGS

Original: Many word embedding algorithms use the empirical probability that two given words appear near each other in the corpus (Levy & Goldberg, 2014; Pennington et al., 2014).
Revised: Many word embedding algorithms use the empirical probability that two given words appear near each other in the corpus (Levy & Goldberg, 2014; Pennington et al., 2014).

Original: This empirical probability is computed by counting how many times one word appears in the context of another as a word–context pair.
Revised: This empirical probability is computed by counting how many times one word appears in the context of another as a word–context pair.

Original: A word–context pair is defined as a pair of words from the corpus that appear within a certain fixed distance from each other, the window size , and within the same sentence.
Revised: A word–context pair is defined as a pair of words from the corpus that appear within a certain fixed distance from each other, the window size , and within the same sentence.

Original: A word–context pair designates one word as appearing in the context of another; in this paper, we will refer to a word–context pair as ( a, b ) where a is a the word appearing in the context of the word b .
Revised: A word–context pair designates one word as appearing in the context of another; in this paper, we will refer to a word–context pair as ( a, b ) where a is a the word appearing in the context of the word b .

Original: Contexts can be unidirectional or bidirectional.
Revised: Contexts can be unidirectional or bidirectional.

Original: In the unidirectional case, in a word–context pair ( a, b ) , a always occurs before b in the corpus (or alternatively, always after).
Revised: In the unidirectional case, in a word–context pair ( a, b ) , a always occurs before b in the corpus (or alternatively, always after).

Original: In the bidirectional case, for any pair of nearby words a and b , there are two word–contexts pairs: ( a, b ) and ( b, a ) .
Revised: In the bidirectional case, for any pair of nearby words a and b , there are two word–contexts pairs: ( a, b ) and ( b, a ) .

Original: From these corpus statistics, word embedding algorithms learn vectors for each word in a way that the word co-occurrence statistics can be derived from the geometry of the vectors.
Revised: From these corpus statistics, word embedding algorithms learn vectors for each word in a way that the word co-occurrence statistics can be derived from the geometry of the vectors.

Original: The exact details for how this is done is dependent on the exact word embedding algorithm used and is not important for this work.
Revised: The exact details for how this is done is dependent on the exact word embedding algorithm used and is not important for this work.

Original: 2. W ORD E MBEDDING A SSOCIATION T EST
Revised: 2. W ORD E MBEDDING A SSOCIATION T EST

Original: The Word Embedding Association Test (WEAT), introduced by Caliskan et al.
Revised: The Word Embedding Association Test (WEAT), introduced by Caliskan et al.

Original: (2017), is a common test used to quantify the presence of specific stereotypes in word embeddings.
Revised: (2017), is a common test used to quantify the presence of specific stereotypes in word embeddings.

Original: Given two sets of target words X and Y of equal size and two sets of attribute words A and B , WEAT measures the association between the targets and the attributes.
Revised: Given two sets of target words X and Y of equal size and two sets of attribute words A and B , WEAT measures the association between the targets and the attributes.

Original: The sets are chosen so that X and A are stereotypically linked with each other, and similarly for Y and B .
Revised: The sets are chosen so that X and A are stereotypically linked with each other, and similarly for Y and B .

Original: The association of a word w ∈ X ∪ Y with the attributes is
Revised: The association of a word w ∈ X ∪ Y with the attributes is

Original: [Equation]
Revised: [Equation]

Original: where ⃗u denotes the word vector corresponding to the word u and cos( ⃗u,⃗v )
Revised: where ⃗u denotes the word vector corresponding to the word u and cos( ⃗u,⃗v )

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ⃗v
Revised: ⃗v

Original: [Equation]
Revised: [Equation]

Original: ∥ is the cosine similarity between ⃗u and ⃗v .
Revised: ∥ is the cosine similarity between ⃗u and ⃗v .

Original: The outputs of WEAT are the test statistic, effect size, and p -value (of a permutation test), defined by the following equations respectively:
Revised: The outputs of WEAT are the test statistic, effect size, and p -value (of a permutation test), defined by the following equations respectively:

Original: [Equation]
Revised: [Equation]

Original: EffectSize
Revised: EffectSize

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: X | (cid:80) x
Revised: X | (cid:80) x

Original: ∈ X s ( x, A , B ) − 1 | Y | (cid:80)
Revised: ∈ X s ( x, A , B ) − 1 | Y | (cid:80)

Original: y ∈
Revised: y ∈

Original: Y s ( x, A , B ) stddev w ∈ X ∪ Y
Revised: Y s ( x, A , B ) stddev w ∈ X ∪ Y

Original: [Equation]
Revised: [Equation]

Original: where ( X i , Y i ) is a random partition of X ∪ Y into two sets of equal size.
Revised: where ( X i , Y i ) is a random partition of X ∪ Y into two sets of equal size.

Original: Of these three measures, the effect size is most commonly used in the literature.
Revised: Of these three measures, the effect size is most commonly used in the literature.

Original: In this work, we report the effect size along with the p -value.
Revised: In this work, we report the effect size along with the p -value.

Original: The effect size retains its original meaning from Caliskan et al.
Revised: The effect size retains its original meaning from Caliskan et al.

Original: (2017) of measuring the strength and direction of the tested stereotype.
Revised: (2017) of measuring the strength and direction of the tested stereotype.

Original: We interpret the pvalue not as a measure of statistical significance, as originally conceived.
Revised: We interpret the pvalue not as a measure of statistical significance, as originally conceived.

Original: Instead, we interpret it as another measure of the tested stereotype, indicating how easily words in X can be separated from words in Y according to their association with A and B : p -values of 0, 0.5, and 1 correspond to perfect separation according to the stereotype, no separation, and perfect separation according to the opposite stereotype, respectively.
Revised: Instead, we interpret it as another measure of the tested stereotype, indicating how easily words in X can be separated from words in Y according to their association with A and B : p -values of 0, 0.5, and 1 correspond to perfect separation according to the stereotype, no separation, and perfect separation according to the opposite stereotype, respectively.

Original: We will interpret these as relative measures: a WEAT effect size closer to zero and a WEAT p -value closer to 0.5 both suggest reduced presence of the stereotype in the embeddings.
Revised: We will interpret these as relative measures: a WEAT effect size closer to zero and a WEAT p -value closer to 0.5 both suggest reduced presence of the stereotype in the embeddings.

Original:  I NDIRECT S TEREOTYPES
Revised:  I NDIRECT S TEREOTYPES

Original: Previous work on measuring stereotypes has typically focused on measuring associations between stereotyped words and explicit markers of the stereotype, such as names 5 and semantically gendered words.
Revised: Previous work on measuring stereotypes has typically focused on measuring associations between stereotyped words and explicit markers of the stereotype, such as names 5 and semantically gendered words.

Original: In turn, previous work on mitigating bias have used these same markers as input for bias mitigation methods.
Revised: In turn, previous work on mitigating bias have used these same markers as input for bias mitigation methods.

Original: By looking at the same sets of words for quantifying and mitigating bias, it is easy to overestimate the effect of bias mitigation.
Revised: By looking at the same sets of words for quantifying and mitigating bias, it is easy to overestimate the effect of bias mitigation.

Original: This is what led Gonen & Goldberg (2019) to propose their new bias measures.
Revised: This is what led Gonen & Goldberg (2019) to propose their new bias measures.

Original: These are based around measuring how clustered previously biased words are, but it is unclear what result should be desired from these measures.
Revised: These are based around measuring how clustered previously biased words are, but it is unclear what result should be desired from these measures.

Original: To address this issue, we demonstrate that we can capture some forms of the remaining bias as stereotypes, which we refer to as indirect stereotypes .
Revised: To address this issue, we demonstrate that we can capture some forms of the remaining bias as stereotypes, which we refer to as indirect stereotypes .

Original: [Figure]
Revised: [Figure]

Original: An example of an indirect stereotype is shown in Figure 1.
Revised: An example of an indirect stereotype is shown in Figure 1.

Original: There are three words here exhibiting a stereotype in regards to binary gender: “handsome” and “engineer,” which are both masculine and not feminine, and “sentimental,” which is feminine and not masculine.
Revised: There are three words here exhibiting a stereotype in regards to binary gender: “handsome” and “engineer,” which are both masculine and not feminine, and “sentimental,” which is feminine and not masculine.

Original: In a corpus with binary gender stereotypes, we would expect a quantifiably stronger association between “handsome” and “engineer” than between “sensitive” and “engineer” to exist purely because they are both stereotypically used to refer to men, 6 even if there is not a stereotype that engineers are more handsome than they are sensitive.
Revised: In a corpus with binary gender stereotypes, we would expect a quantifiably stronger association between “handsome” and “engineer” than between “sensitive” and “engineer” to exist purely because they are both stereotypically used to refer to men, 6 even if there is not a stereotype that engineers are more handsome than they are sensitive.

Original: These stereotypes can come about as a result of sentences that exhibit multiple stereotypes at the same time.
Revised: These stereotypes can come about as a result of sentences that exhibit multiple stereotypes at the same time.

Original: For example, the sentence “he is an a handsome engineer” exhibits the masculine stereotypes for “handsome” and “engineer”.
Revised: For example, the sentence “he is an a handsome engineer” exhibits the masculine stereotypes for “handsome” and “engineer”.

Original: In a corpus that exhibits both of these stereotypes, this sentence would be more likely to occur than the following related sentences:
Revised: In a corpus that exhibits both of these stereotypes, this sentence would be more likely to occur than the following related sentences:

Original: She is a handsome engineer.
Revised: She is a handsome engineer.

Original: He is a sensitive engineer.
Revised: He is a sensitive engineer.

Original: She is a sensitive engineer.
Revised: She is a sensitive engineer.

Original: This will result in “handsome” occurring with “engineer” more frequently than “sensitive” does.
Revised: This will result in “handsome” occurring with “engineer” more frequently than “sensitive” does.

Original: However, when word embeddings are “debiased” with respect to binary gender, the goal in previous work is typically to obtain word vectors that would predict each pair of the previous examples that differ only in choice of pronoun as equally likely to occur.
Revised: However, when word embeddings are “debiased” with respect to binary gender, the goal in previous work is typically to obtain word vectors that would predict each pair of the previous examples that differ only in choice of pronoun as equally likely to occur.

Original: This is not enough to correct the associations between “handsome,” “engineer,” and “sensitive.”
Revised: This is not enough to correct the associations between “handsome,” “engineer,” and “sensitive.”

Original: Furthermore, the association between “engineer” and “handsome” should exist even in sentences without explicit reference to binary gender, as in those cases “engineer” is more likely to refer to a man than a woman.
Revised: Furthermore, the association between “engineer” and “handsome” should exist even in sentences without explicit reference to binary gender, as in those cases “engineer” is more likely to refer to a man than a woman.

Original: More details on this analysis can be found in Appendix A.
Revised: More details on this analysis can be found in Appendix A.

Original: To test indirect stereotypes, we use WEAT with the categories that are stereotyped according to binary gender (these are science/art, math/art, and career/family), as well a test set of professions and adjectives that are stereotypically masculine and feminine.
Revised: To test indirect stereotypes, we use WEAT with the categories that are stereotyped according to binary gender (these are science/art, math/art, and career/family), as well a test set of professions and adjectives that are stereotypically masculine and feminine.

Original: These new test sets and the process used to generate them are detailed in Appendix B. 
Revised: These new test sets and the process used to generate them are detailed in Appendix B. While we will use WEAT as our primary test

Revised: NEW for stereotypes in word embeddings, subsequent work has called attention to shortcomings of the measure.

Revised: We give a more thorough discussion with our analysis, at the end of Section 5.

Original:  B IAS M ITIGATION M ETHOD
Revised:  B IAS M ITIGATION M ETHOD

Original: Instead of only looking at the co-occurrence probability of two words, we consider the probability a word co-occurs with another and one of the two also occurs near a word marking the bias attribute.
Revised: Instead of only looking at the co-occurrence probability of two words, we consider the probability a word co-occurs with another and one of the two also occurs near a word marking the bias attribute.

Original: By using the word marking a bias attribute as a proxy for the bias attribute itself, we can determine how the relationship between the two words varies as the bias attribute varies.
Revised: By using the word marking a bias attribute as a proxy for the bias attribute itself, we can determine how the relationship between the two words varies as the bias attribute varies.

Original: Then, we can approximate what the relationship between two words would be if one of them had no association with the bias attribute.
Revised: Then, we can approximate what the relationship between two words would be if one of them had no association with the bias attribute.

Original: As an example, suppose we would like to mitigate one of the stereotypes in Figure 1, perhaps that “engineer” and “handsome” are more closely associated than they should be based on nonstereotypical semantics.
Revised: FIX Example 1 Suppose we would like to mitigate one of the stereotypes in Figure 1, perhaps that “engineer” and “handsome” are more closely associated than they should be based on non-stereotypical semantics.

Original: In the corpus, it might be that an engineer is described as handsome three times as often when “engineer” refers to a man as opposed to when it refers to a woman.
Revised: In the corpus, it might be that an engineer is described as handsome three times as often when “engineer” refers to a man as opposed to when it refers to a woman.

Original: It might also be the case that “engineer” refers to a man twice as often as it refers to a woman.
Revised: It might also be the case that “engineer” refers to a man twice as often as it refers to a woman.

Original: If “engineer” is adjusted so that it refers to men just as often as women, then the probability that an engineer is referred to as handsome should also be adjusted to be
Revised: If “engineer” is adjusted so that it refers to men just as often as women, then the probability that an engineer is referred to as handsome should also be adjusted to be

Original: [Equation] 
Revised: [Equation] [Equation]

Original: of what it was before, at least when just considering occurrences where “engineer” refers to a man or woman.
Revised: of what it was before, at least when just considering occurrences where “engineer” refers to a man or woman.

Original: Whether “engineer” refers to a man or woman is not known to us a priori, but can be estimated based on whether “engineer” is near a word marking binary gender.
Revised: Whether “engineer” refers to a man or woman is not known to us a priori, but can be estimated based on whether “engineer” is near a word marking binary gender.

Original: Occurrences where “engineer” is not near such a word could be left as is or also be adjusted with this factor based on the reasoning in Section 3.
Revised: Occurrences where “engineer” is not near such a word could be left as is or also be adjusted with this factor based on the reasoning in Section 3.

Original: In this work, we take the latter approach.
Revised: In this work, we take the latter approach.

Original: We note that word embeddings do not attempt to quantify whether or not “handsome” describes an engineer, but rather if “handsome” occurs near “engineer”.
Revised: We note that word embeddings do not attempt to quantify whether or not “handsome” describes an engineer, but rather if “handsome” occurs near “engineer”.

Original: However, the same principle still applies.
Revised: However, the same principle still applies.

Original: The general method is as follows.
Revised: The general method is as follows.

Original: Suppose a word–context pair in the corpus is selected uniformly at random.
Revised: Suppose a word–context pair in the corpus is selected uniformly at random.

Original: Consider two words a and b where at least one of a or b is a word to be neutralized with respect to this bias.
Revised: Consider two words a and b where at least one of a or b is a word to be neutralized with respect to this bias.

Original: Additionally, let X and Y be two sets of words that are similar in usage but differ in terms of the bias attribute that is being mitigated.
Revised: Additionally, let X and Y be two sets of words that are similar in usage but differ in terms of the bias attribute that is being mitigated.

Original: For example, in the case of binary gender, X could be the set { “he” , “man” , . . . }
Revised: For example, in the case of binary gender, X could be the set { “he” , “man” , . . . }

Original: and Y could be the set { “she” , “woman” , . . . } .
Revised: and Y could be the set { “she” , “woman” , . . . } .

Original: Define the following events for a random word–context pair ( c, d ) : 
Revised: Define the following events for a random word–context pair ( c, d ) : [Equation] word [Equation]

Revised: X

Original: [Equation]
Revised: [Equation]

Original: pairappears
Revised: pair

Original: [Equation]
Revised: [Equation]

Original: a word in Y
Revised: word in Y

Original: [Equation]
Revised: [Equation]

Original: A word–context pair is defined as being near a word in X or in Y if this word is within a certain distance from c or from d .
Revised: A word–context pair is defined as being near a word in X or in Y if this word is within a certain distance from c or from d .

Original: This distance can be taken to be the window size used for determining word–context pairs, for simplicity.
Revised: This distance can be taken to be the window size used for determining word–context pairs, for simplicity.

Original: In this work, we weight occurrences where a word in X or Y appears near just one of c or d half as much as if the word appeared near both of c and d , but we could just as easily weight these occurrences the same.
Revised: In this work, we weight occurrences where a word in X or Y appears near just one of c or d half as much as if the word appeared near both of c and d , but we could just as easily weight these occurrences the same.

Original: The probability the word–context pair ( c, d ) is ( a, b ) can be decomposed as[Equation] A
Revised: The probability the word–context pair ( c, d ) is ( a, b ) can be decomposed as

Original: [Equation]
Revised: [Equation]

Original: using the definition of conditional probability and inclusion-exclusion.
Revised: using the definition of conditional probability and inclusion-exclusion.

Original: This representation isolates the contribution of the bias attribute on P ( A ∩ B ) .
Revised: This representation isolates the contribution of the bias attribute on P ( A ∩ B ) .

Original: The terms P ( B ∩ X ) and P ( B ∩ Y ) indicate the bias b has, if any, while the terms P ( A | B ∩ X ) and P ( A | B ∩ Y ) describe how the relationship between a and b is influenced by the bias attribute.
Revised: The terms P ( B ∩ X ) and P ( B ∩ Y ) indicate the bias b has, if any, while the terms P ( A | B ∩ X ) and P ( A | B ∩ Y ) describe how the relationship between a and b is influenced by the bias attribute.

Original: Now if b is replaced with a hypothetical neutral word b ′ that has the same meaning except with no association to the bias attribute, the probability of b ′ occurring with a can be approximated.
Revised: Now if b is replaced with a hypothetical neutral word b ′ that has the same meaning except with no association to the bias attribute, the probability of b ′ occurring with a can be approximated.

Original: Denote by B ′ the event that a randomly selected word–context pair ( c, d ) satisfies d = b ′ .
Revised: Denote by B ′ the event that a randomly selected word–context pair ( c, d ) satisfies d = b ′ .

Original: The following exact equalities hold in this setting:
Revised: The following exact equalities hold in this setting:

Original: [Equation]
Revised: [Equation]

Original: ) = P ( Y ∩ B ′ ) . The first two identities hold because b and b ′ have the same meaning except for the bias.
Revised: The first two identities hold because b and b ′ have the same meaning except for the bias.

Original: The last identity holds because b ′ has no bias.
Revised: The last identity holds because b ′ has no bias.

Original: In addition, the following relations are approximately true:
Revised: In addition, the following relations are approximately true:

Original: [Equation]
Revised: [Equation]

Original: In the first two lines, the explicit presence of the bias attribute of interest is likely to overpower the bias b has.
Revised: In the first two lines, the explicit presence of the bias attribute of interest is likely to overpower the bias b has.

Original: In the last two lines, the approximate relationship is likely to hold as all the probabilities are small: the events are the intersection of three or more events and are subsets of X ∩ Y.
Revised: In the last two lines, the approximate relationship is likely to hold as all the probabilities are small: the events are the intersection of three or more events and are subsets of X ∩ Y.

Original: These relations imply
Revised: These relations imply

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: P

Original: [Equation]
Revised: [Equation]

Original: A
Revised: Reconsider Example 1. The factor in front is the original co-occurrence probability divided by NEW the denominator of equation 1, with semantics replaced by word co-occurrence. In the example,

Original: [Equation] B 
Revised: P ( A ∩ B ∩ X ∩ Y ) = 0 and ( P ( B ∩ X ) + P ( B ∩ Y ))

Original: [Equation]
Revised: = 1 . With these substitutions, the remaining terms are the numerator of equation 1. With the switch from semantics to co-occurrence, this is now an approximation. A further approximation can be obtained by looking at a .

Original: Since b ′ is neutral, if a is replaced with a neutral word a ′ and A replaced with a corresponding event A ′ , it holds that P ( A ′ ∩ B ′ )
Revised: Since b ′ is neutral, if a is replaced with a neutral word a ′ and A replaced with a corresponding event A ′ , it holds that P ( A ′ ∩ B ′ )

Original: ≈ P ( A ∩ B ′ ) .
Revised: ≈ P ( A ∩ B ′ ) .

Original: A similar argument shows equation 1 holds when the role of a and b are switched.
Revised: A similar argument shows equation 2 holds when the role of a and b are switched.

Original: These two approximations can be averaged to yield a better approximation for P ( A ′ ∩ B ′ ) :
Revised: These two approximations can be averaged to yield a better approximation for P ( A ′ ∩ B ′ ) :

Original: [Equation] [Equation] ≈
Revised: P (

Original: [Equation]
Revised: [Equation]

Original: This approximation is the basis of our proposed bias mitigation method.
Revised: This approximation is the basis of our proposed bias mitigation method.

Original: We propose the right hand side of equation 2 can replace the probability of a word–context pair being ( a, b ) whenever this quantity would be used, where a or b is a word that is wanted to be neutralized.
Revised: We propose the right hand side of equation 3 can replace the probability of a word–context pair being ( a, b ) whenever this quantity would be used, where a or b is a word that is wanted to be neutralized.

Original: This can be done easily with word embedding methods that directly use the probability or counts of a specific word– context pair occurring, such as GloVe or PPMI factoring, but in principle could be adapted even to methods that only indirectly use these probabilities.
Revised: This can be done easily with word embedding methods that directly use the probability or counts of a specific word– context pair occurring, such as GloVe or PPMI factoring, but in principle could be adapted even to methods that only indirectly use these probabilities.

Original: For all the terms in equation 2 to be defined, at least one occurrence each of A ∩ X, A ∩ Y, B ∩ X, and B ∩ Y must be present in the corpus.
Revised: For all the terms in equation 3 to be defined, at least one occurrence each of A ∩ X, A ∩ Y, B ∩ X, and B ∩ Y must be present in the corpus.

Original: Therefore,  the vocabulary must  be limited so that all words  in the corpusoccur in a word–context pair satisfying X and in a pair satisfying Y.
Revised: Therefore, modified entries in the co-occurrence matrix must NEW be limited to only those where both words appear in the corpus in a word–context pair satisfying X and in a pair satisfying Y.

Original: While this may  be undesirable, inpractice this is not too strong of a restriction on the vocabulary.
Revised: In practice this may not be a strong restriction. In a test case with binary gender, although only 34% of the words can have co-occurrences modified in this way, this includes most frequently occurring words. The vast majority of words that cannot be modified with this method have less than 100 occurrences in the corpus and would therefore have noisier embeddings. This is shown in Figure 2.

Original: R ESULTS
Revised: Thus, words that cannot be applied equation 3 to can be dropped, yielding more pronounced bias mitigation. This is possible even with only eight words each in X and Y . Of course, binary gender in English is particularly well-suited for this method, as words that mark binary gender are rather common (e.g., “he”, “she”, “man”, “woman”, etc.). For other types of biases, we would likely have to use less common words, such as names, resulting in fewer words we can apply equation 3 to.

Revised: The entries of the original co-occurrence matrix with these words can be kept, although there will likely be less bias mitigation as a result.

Original:  We conduct experiments by training GloVe embeddings on the UMBC webbase corpus (Han et al., 2013) and then attempting to mitigate the presence of bias from binary gender.
Revised: [Figure]  R ESULTS We conduct experiments by training GloVe embeddings on the UMBC webbase corpus (Han et al., 2013) and then attempting to mitigate the presence of bias from binary gender.

Original: Due to computational constraints, we only train on 30 million sentences from the corpus.

Original: We compare the proposed method from Section 4 with GloVe, with and without Counterfactual Data Subsitution (Maudslay et al., 2019).
Revised: We compare the proposed method from Section 4 with GloVe, with and without Counterfactual Data Subsitution (Maudslay et al., 2019).

Original: These are labeled was “Proposed Method”, “Original”, and “CDS” respectively in the following tables.
Revised: These are labeled as “Proposed”, “Original”, and “CDS” respectively in the following tables.

Original: The parameters we use to  train  the models for these experiments are in Appendix C.
Revised: Due to computational constraints, we only train CDS on 30 million sentences from NEW the corpus. The results with training on the reduced corpus are shown on the left side of each table, while the results for the full corpus are shown on the right side of each table.

Original: In all tables, the best results for each  row are shown in bold. 
Revised: In all tables, the best results for each half row are shown in bold. FIX

Revised: For all methods, we train GloVe with an embedding dimension of 300.

Revised: In determining the cooccurrence matrix, we use a window size of 15 and skip over words with less than 5 occurrences in the corpus.

Original: As a first test, we evaluate all three word embeddings on a series of common semantic tests (Jastrzebski et al., 2017).
Revised: We also preprocess our corpus by removing capitalization, skipping sentences with less than five words, and using Stanford CoreNLP (Manning et al., 2014) to separate out certain lexemes from words (e.g., n’t and ’s). For the proposed bias mitigation method, we use the following sets of words: X = { “he”, “him”, “his”, “himself”, “man”, “men”, “boy”, “boys” } Y =

Revised: { “she”, “her”, “hers”, “herself”, “woman”, “women”, “girl”, “girls” } .

Revised: All results are obtained by training word vectors multiple times and reporting the median of the respective measures.

Revised: Five sets each of word vectors are trained for the reduced corpus, while three sets are trained for the full corpus.

Revised: As a first test, we evaluate all three word embeddings on a series of common semantic tests (Jastrzebski et al., 2017).

Original: The relative change in the  results of these tests after the two tested bias  mitigation methods are shown in Table 1.
Revised: The relative change in the median results of these tests after the two tested bias NEW mitigation methods are shown in Table 1.

Original: For all tests, a higher score indicates a better representation of the[Table] [Table] tested semantics in the word embeddings.
Revised: For all tests, a higher score indicates a better representation of the tested semantics in the word embeddings.

Original: The tests are grouped into three categories; from top to bottom, these are word similarity tasks, categorization tasks [Table] and analogy tasks.
Revised: The tests are grouped into three categories; from top to bottom, these are word similarity tasks, categorization tasks, and analogy tasks.

Original: Overall, our proposed method typically results in worse degradation of semantics than CDS does.
Revised: Overall, our proposed method typically results in worse degradation of semantics than CDS does.

Original: This is most pronounced in the analogy tasks
Revised: This is most pronounced in the analogy tasks, where there is an approximately 15% decrease in accuracy in

Original: [Table]
Revised: FIX the smaller corpus. With the full corpus, there is less degradation in semantics. The analogy tasks

Original: where there is an approximately 20% decrease in accuracy.
Revised: NEW still see an approximately 10% decrease in accuracy, but there is surprisingly a substantial increase in performance in the word similarity tasks and mixed results in classification. NEW

Original: As another test for semantics, we compare all the word embeddings on two tests for WEAT that do not capture stereotypes related to binary gender.
Revised: The tasks that suffer with largest degradation in the full corpus (AP, BLESS, Battig, Google, MSR, and SemEval2012 2) are also the ones that see the largest improvement for standard GloVe when comparing the samller corpus to the larger. The degradation resulting from using GloVe trained on the smaller corpus for these tasks are − 6 . 48% , − 8 . 23% , − 10 .

Revised: 63% , − 12 .

Revised: 32% , − 9 .

Revised: 67% , and − 6 .

Revised: This performance is comparable to the performance of the bias mitigation method on the full corpus seen in Table 1.

Revised: This suggests that this bias mitigation yield word vectors with semantic quality equivalent to training with a smaller corpus and therefore is most applicable when the training corpus is large enough that this is an acceptable trade-off.

Revised: [Table]

Revised: [Table]

Revised: As another test for semantics, we compare all the word embeddings on two tests for WEAT that do not capture stereotypes related to binary gender.

Original: These tests exhibit the bias that music instruments are more pleasant than weapons, and flowers are more pleasant than insects.
Revised: These tests exhibit the bias that music instruments are more pleasant than weapons, and flowers are more pleasant than insects.

Original: The results are shown in Tables 2and 3 for  the p -value and effect sizes, respectively.
Revised: The results are shown in Table 2 for bot the p -values and effect sizes.

Original: Since these biases encode information that is not being attempted to remove, a small p -values and a large effect sizes are desirable.
Revised: Since these biases encode information that is not being attempted to remove, a small p -values and a large effect sizes are desirable.

Original: We see that all three word embeddings exhibit strong representations of these biases with only minor differences between them.
Revised: We see that all three word embeddings exhibit strong representations of these biases with only minor differences between them.

Original: To test the standard direct stereotypes towards binary gender, we again use WEAT with three binary gender stereotypes used in the original paper.
Revised: To test the standard direct stereotypes towards binary gender, we again use WEAT with three binary gender stereotypes used in the original paper.

Original: The p -value and effect sizes for these tests are in 
Revised: The p -values and effect sizes for these tests are in Table 3.

Revised: All three of these stereotypes are present in the original GloVe embedding, with

Original: Tables 4 and 5.
Revised: Table and effect

Original: All three of these stereotypes are present in the original GloVe embedding, with the association between traditionally masculine and feminine names with career and home words being the strongest.
Revised: ) for for [Table] Math/Art–Masc./Fem.

Original: The presence of all three of these stereotypes are reduced as a result of both bias mitigation methods, although the career/home stereotype is not substantially reduced by either method.
Revised: Words 0 .0 .0 . [Table] [Table] Science/Art–Masc./Fem.

Revised: Words 0 .0 .0 .0 .0 .

Revised: Masc./Fem.

Revised: Names–Career/Home 7 . 8e −1 .

Revised: 55 e −2 .

Revised: 33e −7 .

Revised: 8e −6 .

Revised: 22 e −

Revised: Math/Art–Masc./Fem.

Revised: Words 0 .0 .− 0 .0 .0 .

Revised: Science/Art–Masc./Fem.

Revised: Words 0 .− 0 .0 .0 .−

Revised: Masc./Fem.

Revised: Names

Original: [Table]
Revised: [Table]

Revised: Career

Original: [Table]
Revised: [Table]

Revised: Home

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: the association between traditionally masculine and feminine names with career and home words being the strongest.

Revised: The presence of all three of these stereotypes are reduced as a result of both bias mitigation methods

Original: [Table]
Revised: [Table]

Revised: although the career/home stereotype is not substantially reduced by either method.

Revised: Lastly

Original: [Table]
Revised: [Table]

Original: Lastly, we investigate the presence of indirect stereotypes as described in Section 3.
Revised: we investigate the presence of indirect stereotypes as described in Section 3.

Original: We use a list of stereotypically masculine and feminine adjectives  and look at its association with stereotypically masculine and feminine professions, as well as the same math/art, science/art, and career/home categories from the standard WEAT tests. 
Revised: We use a list of stereotypically masculine and feminine adjectives from Hosoda & Stone (2000) and look at its association with stereotypically masculine and feminine professions from Caliskan et al. (2017), as well as the same math/art, science/art, and career/home categories from the standard WEAT tests. The full word lists for our new tests are shown in

Original:  The results of these experiments are in Tables 6 and 7.
Revised: Appendix B. The results of these experiments are in Table 4.

Original: From this, it can be seen that all these stereotypes do exist in the original word embeddings.
Revised: From this, it can be seen that all these stereotypes do exist in the original word embeddings.

Original: Counterfactual Data Substitution does not succeed at significantly reducing any of these stereotypes, although it does have modest success at reducing the stereotype with math and art.
Revised: Counterfactual Data Substitution does not succeed at significantly reducing any of these stereotypes.

Original: Our proposed method reduces the presence of all these stereotypes to a greater extent than CDS, although it is not uniformly successful.
Revised: Our proposed method reduces the presence of all these stereotypes to a greater extent than CDS, although it is not uniformly successful.

Original: In particular, it has modest success in  reducing the presence of the indirect stereotype with professions and  is able to significantly reduce the presence of the math and art indirect stereotype.
Revised: In particular, it has modest success in FIX reducing the presence of the indirect stereotypes with professions and with science/art and is able to significantly reduce the presence of the math and art indirect stereotype.

Original: These results are modest but suggestive.
Revised: These results are modest but suggestive.

Original: Even by our own measures, we are not able to substantively mitigate all tested stereotypes present in the word embeddings. 
Revised: Even by our own measures, we are not able to fully mitigate all tested stereotypes present in the word embeddings. These measures themselves are not perfect. NEW

Revised: As shown by Ethayarajh et al.

Original: Furthermore, with  recent results showing that intrinsic bias measures (including WEAT) do not necessarily generalize to downstream tasks (Goldfarb-Tarrant et al., 2021; Orgad & Belinkov, 2022),  the performed tests alone cannot guarantee word embeddings lackany bias.
Revised: (2019); Schr¨oder et al. (2021), the WEAT scores may incorrectly measure the presence of stereotypes. We prefer WEAT for our tests because it emphasizes aggregate stereotype associations over individual ones, but also include experiments with one alternative, Relational Inner Product Association (RIPA), in Appendix C. Another important consideration is that recent results have shown that intrinsic bias measures (including WEAT and RIPA) do not necessarily generalize to downstream tasks (Goldfarb-Tarrant et al., 2021; Orgad & Belinkov, 2022), so the performed tests alone cannot guarantee word embeddings lack bias.

Original: However, by considering stereotypes as a measure of indirect bias, these tests can be extended to downstream bias measures such as WinoBias (Zhao et al., 2018a) and WinoGender (Rudinger et al., 2018) more readily than the previously used dispersion measures.
Revised: However, by considering stereotypes as a measure of indirect bias, these tests can be extended to downstream bias measures such as WinoBias (Zhao et al., 2018a) and WinoGender (Rudinger et al., 2018) more readily than the previously used dispersion measures.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this paper, we discuss how indirect bias in word embeddings can manifest as stereotypes.
Revised: In this paper, we discuss how indirect bias in word embeddings can manifest as stereotypes.

Original: Using the standard Word Embedding Association Test with additional test sets, we demonstrate that these “indirect stereotypes” have a substantial presence in word embeddings and are not removed by current debiasing methods.
Revised: Using the standard Word Embedding Association Test with additional test sets, we demonstrate that these “indirect stereotypes” have a substantial presence in word embeddings and are not removed by current debiasing methods.

Original: Furthermore, we propose a new method that attempts to directly mitigate these indirect stereotypes and demonstrate that this method can have some success in practice, albeit with trade-offs on the semantic quality of the resulting word vectors. 
Revised: Furthermore, we propose a new method that attempts to directly mitigate these indirect stereotypes and demonstrate that this method can have some success in practice, albeit with trade-offs on the semantic quality of the resulting word vectors. This method is a first step towards a more thorough removal on indirect stereotypes. In particular, we demonstrate that indirect stereotypes can be mitigated even with only using direct markers of the bias they come from. NEW

Original: A A NALYSIS OF I NDIRECT S TEREOTYPE O CCURRENCE
Revised: A A NALYSIS OF I NDIRECT S TEREOTYPE O CCURRENCE

Original: Consider the following list of sentences:
Revised: Consider the following list of sentences:

Original: He is a handsome engineer. She is a handsome engineer.
Revised: He is a handsome engineer. She is a handsome engineer.

Original: He is a sensitive engineer. She is a sensitive engineer.
Revised: He is a sensitive engineer. She is a sensitive engineer.

Original: The first sentence exhibits the indirect stereotype found in Figure 1 and is the most likely sentence to occur in a corpus that treats both “engineer” and “handsome” as masculine and non-feminine words and “sensitive” as a feminine and non-masculine word.
Revised: The first sentence exhibits the indirect stereotype found in Figure 1 and is the most likely sentence to occur in a corpus that treats both “engineer” and “handsome” as masculine and non-feminine words and “sensitive” as a feminine and non-masculine word.

Original: Let P ( A ) for a list A of words be the probability that a sentence in the corpus contains the words in A , given that it is one of the four example sentences above.
Revised: Let P ( A ) for a list A of words be the probability that a sentence in the corpus contains the words in A , given that it is one of the four example sentences above.

Original: As long as P ( “he” ) > 12 and
Revised: As long as P ( “he” ) > 12 and

Original: P ( “handsome” | “he” ) > P ( “handsome” | “she” ) , then 
Revised: P ( “handsome” | “he” ) > P ( “handsome” | “she” ) , then P ( “handsome” ) = P ( “handsome” | “he” [Equation] P ( “he”

Original:  P 
Revised: [Equation] P ( “handsome” | “she” ) P [Equation] “she” ) >

Original: [Equation]
Revised: [Equation]

Original:  The expression on the last line is what we would expect the probability of a sentence with “handsome” appearing would be if “engineer” had no bias towards binary gender.
Revised: P ( “handsome” | “he” [Equation] ( “handsome” | “she” ) . The expression on the last line is what we would expect the probability of a sentence with “handsome” appearing would be if “engineer” had no bias towards binary gender.

Original: B I NDIRECT S TEREOTYPE W ORD L ISTS
Revised: B I NDIRECT S TEREOTYPE W ORD L ISTS

Original: For the list of adjectives, we use the “key” feminine and masculine traits from Hosoda & Stone (2000).
Revised: For the list of adjectives, we use the “key” feminine and masculine traits from Hosoda & Stone (2000).

Original: We remove the words “feminine” and “masculine” from these lists as our goal is to quantify indirect stereotypes.
Revised: We remove the words “feminine” and “masculine” from these lists as our goal is to quantify indirect stereotypes.

Original: We also remove “hard-headed” from the masculine list as it it outside of the model’s vocabulary.
Revised: We also remove “hard-headed” from the masculine list as it it outside of the model’s vocabulary.

Original: Feminine adjectives: affectionate, sensitive, appreciative, sentimental, sympathetic, nagging, fussy, emotional
Revised: Feminine adjectives: affectionate, sensitive, appreciative, sentimental, sympathetic, nagging, fussy, emotional

Original: Masculine adjectives: handsome, aggressive, tough, courageous, strong, forceful, arrogant, egotistical, boastful, dominant
Revised: Masculine adjectives: handsome, aggressive, tough, courageous, strong, forceful, arrogant, egotistical, boastful, dominant

Original: For the list of professions, we use the list from Caliskan et al.
Revised: For the list of professions, we use the list from Caliskan et al.

Original: and take the ten jobs with the highest proportion of women and of men according to the data that they derive from the Bureau of
Revised: and take the ten jobs with the highest proportion of women and of men according to the data that they derive from the Bureau of

Original: Labor Statistics
Revised: Labor Statistics.

Original: Feminine professions: therapist, planner, librarian, paralegal, nurse, receptionist, hairdresser, nutritionist, hygienist, pathologist Masculine professions: plumber, mechanic, carpenter, electrician, machinist, engineer, programmer, architect, officer, paramedic 
Revised: Feminine professions: therapist, planner, librarian, paralegal, nurse, receptionist, hairdresser, nutritionist, hygienist, pathologist Masculine professions: plumber, mechanic, carpenter, electrician, machinist, engineer, programmer, architect, officer, paramedic C RIPA T ESTS NEW

Original: C E XPERIMENT D ETAILS{+To test the bias of individual words, we use Relational Inner Product Association (RIPA) from Ethayarajh et al. We test the same words use in the WEAT tests for direct biases from Section 5, [Table] − 1 .− 1 .− 2 .− 
+}
Revised: {+To test the bias of individual words, we use Relational Inner Product Association (RIPA) from Ethayarajh et al. We test the same words use in the WEAT tests for direct biases from Section 5, [Table] − 1 .− 1 .− 2 .− 
+}

Original: For all methods, we train GloVe with an embedding dimension of 300.{+.chemistry − 0 .0 .− 1 .− 2 .− 
 .nasa − 3 .0 .− 
 .− 3 .0 .experiment − 1 .0 .−+}
Revised: {+.chemistry − 0 .0 .− 1 .− 2 .− 
 .nasa − 3 .0 .− 
 .− 3 .0 .experiment − 1 .0 .−+}

Revised: 1 .− 2 .0 .astronomy − 2 .− 1 .0 .−

Revised: 3 .−

Revised: 1 .math 1 .− 1 .− 0 .2 .−

Original: {+.algebra − 1 .− 0 .1 .− 
+}
Revised: {+.algebra − 1 .− 0 .1 .− 
+}

Revised: .− 1 .geometry − 2 .1 .1 .−

Revised: 2 .3 .calculus − 4 .− 2 .0 .−

Original: In determining the cooccurrence matrix, we use a window size of 15 and skip over words with less than 5 occurrences in the corpus.{+.− 1 .equations − 2 .− 0 .0 .− 2 .− 
 .computation − 2 .− 0 .− 0 .−+}
Revised: 5 {+.− 1 .equations − 2 .− 0 .0 .− 2 .− 
 .computation − 2 .− 0 .− 0 .−+}

Original: {+2 .− 
+}
Revised: {+2 .− 
+}

Revised: .numbers − 2 .−

Revised: 1 .− 1 .− 3 .− 2 .addition − 1 .− 0 .−

Revised: 1 .− 1 .−

Revised: .dance

Revised: 4 .− 0 .−

Revised: .literature − 0 .−

Revised: 2 .− 0 .0 .−

Original: {+1 .poetry 1 .1 .− 0 .1 .
+}
Revised: {+1 .poetry 1 .1 .− 0 .1 .
+}

Revised: .novel − 0 .− 0 .0 .−

Revised: 0 .−

Revised: 0 .symphony 0 .2 .0 .−

Revised: 3 .1 .art

Original: {+− 3 .− 1 .− 2 .− 1 .− 0 .sculpture 1 .0 .0 .0 .
+}
Revised: {+− 3 .− 1 .− 2 .− 1 .− 0 .sculpture 1 .0 .0 .0 .
+}

Original: We also preprocess our corpus by removing capitalization, skipping sentences with less than five words, and using Stanford CoreNLP (Manning et al., 2014) to separate out certain lexemes from words (e.g., n’t and ’s).
Revised: .drama 0 . [Table] except the words “Einstein” and “Shakespeare” as a gender association for these words would not be stereotypical. For each word, we average the RIPA score for all pairs of feminine and masculine words tested by WEAT to report a single score.

Revised: A positive score means a feminine association while a negative score means a masculine association.

Revised: The results are shown in Table 5.

Revised: Neither our proposed method nor CDS consistently outperform the other at producing words with no binary gender bias, although it appears that CDS produces the least gendered word vectors slightly more often.

Revised: The new bias mitigation method is able to consistently reduce the gender association of the tested words and rarely produces word with a stronger association with binary gender than the original vectors did.

Revised: We do not report RIPA scores for indirect bias.

Original: For theproposed bias mitigation method, we use the following sets of words:
Revised: This is for two reasons. First, RIPA requires words marking the bias attribute to be paired. For direct binary gender stereotypes, there is usually a natural choice for this pairing. For the indirect stereotypes we test, there is not.

Original: X = { “he”, “him”, “his”, “himself”, “man”, “men”, “boy”, “boys” } Y = { “she”, “her”, “hers”, “herself”, “woman”, “women”, “girl”, “girls” } .
Revised: Since the choice of pairing can have significant effects on the scores, this limits the ability of RIPA to measure stereotypes in this situation. Furthermore, RIPA emphasizes the individual stereotype of words. While this is important in the case of direct stereotypes, for indirect stereotypes the relationship is only undesirable in aggregate. As an example, if the word vector for “carpenter” is more closely related with the vector for “strong” than “emotional”, this isn’t necessarily because of binary gender stereotypes.