Robustness and Reliability of Gender Bias Assessment in Word Embeddings: The Role of Base Pairs (2020)

Abstract: It has been shown that word embeddings can exhibit gender bias, and various methods have been proposed to quantify this. However, the extent to which the methods are capturing social stereotypes inherited from the data has been debated. Bias is a complex concept and there exist multiple ways to define it. Previous work has leveraged gender word pairs to measure bias and extract biased analogies. We show that the reliance on these gendered pairs has strong limitations: bias measures based off of them are not robust and cannot identify common types of real-world bias, whilst analogies utilising them are unsuitable indicators of bias. In particular, the well-known analogy “man is to computer-programmer as woman is to homemaker” is due to word similarity rather than bias. This has important implications for work on measuring bias in embeddings and related work debiasing embeddings.

## introduction word embeddings, distributed representations of words in a low-dimensional vector space, are used in many downstream nlp tasks @xcite @xcite . recent work has shown they can contain harmful bias and proposed techniques to quantify it @xcite @xcite . these techniques leverage cosine similarity to a base pair of gender words, such as (man, woman). they include bias measures, which return a magnitude of bias for a given word, and analogies. a well-known example of the latter is "man is to computer programmer as woman is to homemaker" @xcite , which has been widely interpreted as demonstrating bias. there have also been related attempts to debias * denotes equal contribution. however, to remove bias effectively, an accurate method of identifying it is first required. this is a complex task, not least because the concept of "bias" has multiple interpretations: @xcite identify 23 types of bias that can occur in machine learning applications, including historic (pre-existing in society), algorithmic (introduced by the algorithm) and evaluation (occurs during model evaluation). in the case of word embeddings, it remains an open question if bias identifying techniques reflect social stereotypes in the training data, an artifact of the embedding process or noise. while it is often assumed the first is true, and thus that bias in embeddings can perpetuate harmful stereotypes @xcite , this has not been conclusively established @xcite @xcite . to further complicate matters, multiple methods of quantifying bias have been proposed, often in response to one another's limitations (see section 2.1). it is unclear how they compare and which are more reliable. this work shows that the use of gender base pairs in bias identifying techniques has serious limitations. we propose three criteria to evaluate the performance of gender bias measures using base pairs and systematically compare four popular measures, showing both that they not robust, and that they do not accurately reflect common types of societal bias. in addition, we demonstrate that the types of analogies proposed in @xcite are unsuitable indicators of bias; what is ascribed to social bias in analogies is actually an artifact of high cosine similarity in the base pair, which is arguably positive. our argument is not that embeddings are free of bias; rather it is that bias is a complex problem and current bias measures do not completely solve it. this has important implications for future work on bias in embeddings and debiasing techniques. the primary contributions of this work are to: (1) demonstrate the output of gender bias measures is heavily dependant on a chosen gendered base pair (e.g. (she, he)) and on the form of a word considered (e.g. singular versus plural); (2) show the measures cannot accurately predict either the socially stereotyped gender of human traits or the correct gender of words when this is encoded linguistically (e.g. lioness); (3) show that analogies generated by gender base pairs (e.g. (she, he)) are flawed indicators of bias and the widely-known example "man is to computer programmer as woman is to homemaker" is not due to gender bias and (4) highlight the complexities of identifying bias in word embeddings, and the limitations of these measures. ## approach our aim is to examine the extent to which bias identifying techniques are reliabily capturing societal gender bias. bias is a highly complex concept, and although the four bias measures (db, wa, nbm and ripa) may detect certain kinds of bias, there is no theoretical guarantee they will detect all forms, that the "bias" they find will be accurate or that different choices of base pair will behave similarly. we therefore explore whether the bias measures are robust in detecting the bias they appear to detect and if there are forms of bias they are not sensitive to. we propose three conditions to test this: 1) base pair stability: if bias measures captured real-world information in a reliable way, it would be expected that reasonable changes of the base pair, such as (she, he) to (woman, man) or (she, he) to (she, he), would not frequently cause a significant change in bias. of course, due to noise and the problem of implicit bias, these three conditions may not always be true. however, if they do not hold the majority of the time, it must be questioned if the measures are reliably identifying social bias. ## data to allow for fair comparisons, we use the same datasets as previous work where possible: embeddings: 300-dimensional google news word2vec @xcite . professions: a list of 320 professions @xcite , often used to analyse bias measures. base pairs: a standard list of 10 gender base pairs, including (she, he) @xcite . gender neutral: for nbm, we use the set of 26,145 gender neutral words defined in @xcite . in addition, we construct two new test sets, both listed in appendix a: bsri: to assess whether word embeddings contain undesirable gender stereotypes, we utilise the bem sex role inventory (bsri) which developed a list of 20 traits for men and 20 for women that are considered to be socially desirable, such as "assertive" and "compassionate" respectively @xcite . 2 although derived in the 1970s, this work remains one of the most influential and widely accepted measures of socially constructed gender roles within the social sciences, e.g. @xcite @xcite . of particular relevance to nlp applications, @xcite use bsri to identify gender-biased language in job advertisements and demonstrate this language can contribute to workplace gender inequality. bsri traits not in the embedding vocabulary (e.g. "willing to take risks") were removed. for each remaining trait, we queried merriam webster for other forms of that word (for example, "assertiveness" is a form of "assertive"), resulting in a list of 58 characteristics (27 male and 31 female). animals: some words, including the names of certain animals, encode gender linguistically (e.g. "lioness"). wikipedia provides a table of male and female versions of animal names. 3 this table was downloaded, and duplicates, rare words and terms whose animal usage is uncommon (for example, a "cob" is a male swan) were removed. this resulted a set of 26 terms consisting 13 female-male pairs such as (hen, rooster). ## evaluation evaluating gender bias measures is a complex task as there is no inherent ground truth interpretation of the measure's results. for example, it is unclear when a bias score is problematic. we choose to evaluate the four bias measures (db,wa, nbm and ripa) in two ways, first by considering whether a word is assigned a male or female bias, and second what the magnitude of that score is. the bias direction (male or female) assigned by a measure to a word is determined by the sign of the score (whether a positive score denotes male or female bias depends on the ordering of the base pair words). the assignment of bias direction is viewed an annotation task in which a bias measure (with a specified base pair) is considered an "annotator" making assignments. consistency between annotators (i.e. versions of bias measures) can be computed using cohen's kappa to determine pairwise agreement @xcite and fleiss' kappa @xcite for multiple annotators. we follow a widely used interpretation of kappa scores (landis and @xcite . the second method of evaluation is an analysis of the magnitude of bias assigned. previous work in this area does not define what constitutes a "significant" change of the magnitude of a bias score. therefore, we estimate the mean bias in the embedding space as follows: the 50,000 most frequent words in the embedding vocabulary were selected and, following @xcite , all words containing digits, punctuation or that were more than 20 characters long were removed. for each of the remaining 48,088 words, their bias score was calculated with respect to each of the 10 base pairs (so for each measure, there are 480,880 scores). an examination of these scores revealed them to appear approximately normally distributed and so their mean and standard deviation are used as an approximation of the population mean and standard deviation (see base pair stability: the first experiment explored the robustness of the four measures (db,wa, ripa and nbm) to changing the base pair. for example, figure 1 illustrates the effects of changing the base pair on the bias score of the word "professor." more comprehensively, for each bias measure we computed the bias assigned to each profession for each base pair, and then calculated the agreement between the 10 base pairs via fleiss' kappa coefficient. the changes in the bias magnitude of a word between base pairs were also computed. results are shown in ## conclusions there has been a recent focus in the nlp community on identifying bias in word embeddings. while we strongly support the aim of such work, this paper highlights the complexity of trying to quantify bias in embeddings. we showed the reliance of popular gender bias measures on gender base pairs has strong limitations. none of the measures are robust enough to reliably capture social bias in embeddings, or to be leveraged in debiasing methods. in addition, we showed the use of gender base pairs to generate "biased" analogies is flawed. our analysis can contribute to future work designing robust bias measures and effective debiasing methods. although this paper focused on gender bias, it is relevant to work examining other forms of bias, such as racial stereotyping, in embeddings. code to replicate our experiments can be found at: https://github.com/alisonsneyd/gender_ bias_word_embeddings