Bad Seeds: Evaluating Lexical Methods for Bias Measurement (2021)

Abstract: A common factor in bias measurement methods is the use of hand-curated seed lexicons, but there remains little guidance for their selection. We gather seeds used in prior work, documenting their common sources and rationales, and in case studies of three English-language corpora, we enumerate the different types of social biases and linguistic features that, once encoded in the seeds, can affect subsequent bias measurements. Seeds developed in one context are often re-used in other contexts, but documentation and evaluation remain necessary precursors to relying on seeds for sensitive measurements.

## introduction there has been increasing concern in the nlp community over bias and stereotypes contained in models and how these biases can trickle downstream to practical applications, such as serving job advertisements. in particular, there has been much recent scrutiny of word representations, with many studies finding harmful associations encoded in embedding models. combating such biases requires measuring the bias encoded in a model so that researchers can establish improvements, and many variants of embedding-based measurement techniques have been proposed @xcite @xcite . these measurements have had the additional upstream benefit of providing computational social science and digital humanities scholars with a new means of quantifying bias in datasets of social, political, or literary interest. researchers increasingly use embeddings @xcite @xcite and other lexicon-based methods @xcite @xcite to provide quantitative answers to otherwise elusive political and social ## data description training corpora. our dataset choices are guided by our focus on the upstream use case, where embeddings are trained on relatively small, special-purpose collections to answer social and humanist questions about the training corpus. the scope of these datasets fits the use case of a social scientist interested in measuring bias during a small time window, across specific genres, or in a particular set of authors. our datasets include: new york times articles from april 15th-june 30th, 2016; high quality wikitext articles, using the full wikitext-103 training set @xcite ; and goodreads book reviews for the romance and history and biography genres, sampled from the ucsd book graph @xcite . for added validity, we also replicate existing studies, using a pre-trained model on a large google news corpus @xcite . for each dataset, we lowercase all text, parse and obtain pos tags using spacy @xcite , tokenize the text into unigrams, and filter words that occur fewer than 10 times in the training dataset. lowercasing controls for the varying levels of capitalization used in the gathered seeds. we leave analysis of bigram seeds to future work and rely on unigrams as a simplifying assumption. ## bias measurement algorithms in the upstream use case, locally trained word embeddings remain state of the art because finetuned pre-trained contextual models might introduce extrinsic information, and it is not feasible to pre-train contemporary contextual embeddings on such small collections. here, we focus on two popular seed-based methods to detect bias in word embeddings. @xcite and @xcite both introduce embeddingbased methods for bias detection that rely on sets of seed words. each of these methods requires two sets of seed words, x and y, and one additionally requires matched pairs of seed words {(x 1 , y 1 ), (x 2 , y 2 ), ...}. ## quantifying variation from seeds to quantify how large an effect seed features can have on bias measurements, we calculate a set of metrics for both pca and weat methods that summarizes how well the bias subspace represents the target seeds. for each dataset, we use the popular skip-gram with negative sampling (sgns) algorithm to train a word2vec model. we use the gensim package for training @xcite . we use a window size of 5, a minimum word count of 10, and a vector size of 100 for all experiments. we repeat this process across 20 bootstrapped samples of each dataset. for pca, we calculate the difference vector between the embedding vectors for each pair of words in the two seed sets. for each set of paired seed sets, we run pca and plot the percent of variance explained by each component. for the gathered seeds, we only use pairings documented in prior work. we perform a manual confirmation that the first component g indeed represents the bias subspace by ranking all the words in the vocabulary by their cosine similarity to g. for weat, we hold the attribute terms constant, where @xmath0, while our generated seed sets take the place of the target terms x and y. holding the attribute terms constant is a simplifying assumption; our goal is not to test all possible attribute terms but to show that significant variation is possible. we then calculate the weat test statistic and significance. coherence. in addition to the pca explained variance and weat test statistic, we also measure the coherence of each pairing of seed sets after being mapped to the bias subspace. ideally, when we project all the words in the vocabulary onto the subspace, the two sets would be drawn as far apart as possible. we rank all words by cosine similarity figure 2 : bias measurements depend on seeds. we calculate the cosine similarities between different female seed sets and an averaged upleasantness vector from two embedding models. results are consistent across seeds for romance review embeddings, but vary widely between sets for history and biography. we find similar variation even for a pretrained google news model. to the bias subspace, and we measure the absolute difference in mean rank of the paired seed sets: ## seed choice affects bias measurement before moving to specific seed features, we present some general results showing the instability of measurements using seeds. figure 2 shows a motivating example, in which we imagine a digital humanities scholar interested in measuring whether women are portrayed more negatively in different genres of book reviews. as in the weat test, each seed is plotted according to its cosine similarity to an averaged unpleasantness vector @xcite . for some sets, no significant difference is visible, while for other sets, there are much larger differences, causing the researcher to draw different conclusions when comparing biases across datasets. ## factors causing instability sometimes seeds can reflect the curator's (or crowd's) personal biases. instabilities can also arise from the organization of the seeds and seemingly innocuous linguistic features. we describe a series of distinct sources of instability that can be encoded in seed sets and discuss the implications of each. we rely on a combination of literature review, qualitative close reading of example seeds, and quantitative tests of seed features. we iterated through the seeds, flagging problematic sets, and then manually clustered and labeled the factors that could cause instability. our identified factors can be categorized as definitional factors (reductive definitions, inclusion of confounding concepts), lexical factors (frequency, pos of individual seeds), and set factors (number and order of seeds, similarity of seed sets). reductive definitions. the seeds can be reductive and essentializing, codifying life experiences into traditional categories. using names as placeholders for concepts like race @xcite or reducing gender to a binary with two extremes @xcite can create a distorted view of the source data. sometimes these are simplifying assumptions, made in an effort to measure biases that would otherwise go unexamined. however, these decisions run the risk of further entrenching these category definitions-e.g., see discussions the first pca component dominates for ordered gender pairs but not for shuffled gender pairs (a), while shuffling can produce a component that explains more variance for class (b) and pleasantness (c) pairs. we find similar instabilities using the pretrained model used in @xcite . error bars show standard deviation over the 20 bootstrapped models. seeds pairs are listed in the appendix. 0.36 0.34 0.34 0.31 0.30 -0.39 -0.41 -0.42 -0.45 -0.59 (b) random pairs outcomes son father mother aunt potentially male hood garcia md 0.26 0.26 0.26 0.26 0.25 -0.19 -0.19 -0.29 -0.29 -0.39 (c) shuffled gender pairs figure 4: ranking word vectors by cosine similarity with the top principle component vector for the original gender seed pairs (a) appears to identify female and male gendered words much better than random (b). but shuffling the pairing of seed words (c) maintains correlation with gender but to a less clear degree. results are shown for the nyt corpus with a frequency threshold of 100 and bootstrap resampling. in @xcite ; @xcite for the mistakes and harms that can be caused by mapping names to genders-and these trade-offs should be evaluated and documented. more broadly, recent work has critiqued nlp and ml bias research for not successfully connecting with the literature in sociology and critical race studies @xcite . engaging with this literature would provide a better foundation for decisionmaking about seed sets and provide context for future researchers. ## conclusion: biases all the way down almost all recent work on bias measurement relies on sets of seed terms to ground cultural concepts in language. if we do not pay attention to the seeds, these methods will lack foundation and the claims they support will be left open to criticism and dismissal. seeds and their rationales need to be tested and documented, rather than hidden in code or copied without examination. some of the risks discussed in this paper may seem obvious in retrospect, but our literature survey suggests there are widely varying levels of evaluation and documentation. rationales for picking sources or seeds are not always explained, or the reader is left to assume that prior work has adequately validated the seeds. tests for frequency, semantic similarity, and other features are rare or non-existent, and clear definitions and discussion of limitations are often missing. permutation tests are sometimes used, but these do not account for seeds outside of those already selected. significantly different results can be found using alternative seeds sets for the same target concept, and fine-grained comparisons require validation on multiple sets. we faced a number of challenges in gathering 178 seed sets from prior work. sometimes seeds are shared online at an undocumented location and sometimes hard-coded into code repositories; this can significantly obscure the seeds from public view, which is troubling for tools intended for wide use on sensitive topics. documentation is often scattered across locations, and in more than one case, we found contradictions between different sources for a single project. in one case, we were unable to find the full list of seeds used in the paper, and in several cases, it was unclear which seed sets were used for which experiments. while some authors went to commendable lengths to document their materials, there is a need for more consistent and transparent documentation. we recommend that researchers carefully trace the origins of seed sets, with attention to the risks associated with the origin type. we also recommend that researchers examine seed features. pos, frequency, semantic similarity, and pairing order can significantly affect the results of bias measurements. seeds should be both examined manually and tested as shown in §8; importantly, they should be compared to alternative seeds with different attributes, as in §7. to assist this we release a compilation of 178 seed sets from prior work. these tests are particularly important when comparing biases across datasets. finally, researchers should document all seeds and the rationales underlying their design, including concept definitions. we add to recent calls for better documentation and problem specification in machine learning @xcite @xcite and in studies of social biases in technology @xcite . specifically, when the seeds intentionally encode harmful stereotypes or slurs, it can be beneficial to include a trigger warning or not highlight the seeds in the paper; however, full seed lists should always be accessible, not hard-coded, with unique labels matched to experiments. ultimately, our goal is not to eliminate a problem but to illuminate it: 2 to help practitioners think through the potential risks posed by seed sets used for bias detection. we encourage thoughtful, critical studies, but we observe a trend in which seed sets are used in new research and applications simply because they have been used in prior published work, without additional vetting. research precedents can take on a life of their own and we have a responsibility to explore and document possible sources of error. we believe that seed sets can be useful and are probably unavoidable, but that no technical tool can absolve researchers from the duty to choose seeds carefully and intentionally.