Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers (2021)

Abstract: The ease of access to pre-trained transformers has enabled developers to leverage large-scale language models to build exciting applications for their users. While such pre-trained models offer convenient starting points for researchers and developers, there is little consideration for the societal biases captured within these model risking perpetuation of racial, gender, and other harmful biases when these models are deployed at scale. In this paper, we investigate gender and racial bias across ubiquitous pre-trained language models, including GPT-2, XLNet, BERT, RoBERTa, ALBERT and DistilBERT. We evaluate bias within pre-trained transformers using three metrics: WEAT, sequence likelihood, and pronoun ranking. We conclude with an experiment demonstrating the ineffectiveness of word-embedding techniques, such as WEAT, signaling the need for more robust bias testing in transformers.

## introduction transformer models represent the state-of-the-art for many natural language processing (nlp) tasks, such as question-answering @xcite , dialogue @xcite , search results @xcite , and more. popular pre-trained models, such as those available from hugging face @xcite , allow developers without extensive computation power to benefit from these models. however, it is important to fully understand the latent societal biases within these black-box transformer models. without appropriately considering inherent biases, development on top of pre-trained transformers risks exacerbating and propagating racial, gender, and other biases writ large. before transformers, word embedding models such as word2vec @xcite and glove @xcite were shown to exhibit systematic sexist @xcite ) and * these authors contributed equally to this work racist @xcite biases. initial investigations into bias for transformers @xcite @xcite have found that these new language models are similarly biased. as transformers are increasingly commonplace, a more complete view of the inequalities, biases, or under-representations within pre-trained transformers becomes increasingly important. yet, discovering bias in transformer models has proven to be more nuanced than bias-discovery in word embedding models @xcite . prior work on bias in modern transformer models has used only a single test or metric at a time, which we show in this paper provides an incomplete view of the problem. furthermore, we find evidence that certain tests are ill-suited to understanding bias in transformer architectures, supported by prior work @xcite . moreover, we show that employing multiple tests is necessary for a full picture of the issue as no single test is currently sufficient. in the context of our work, "bias" refers specifically to the preference of a model for one gender or race in the presence of an otherwise neutral context. as an example, consider the sequence " @xcite wept upon arriving to the scene." with no additional information, an equitable system would exhibit no preference for female over male, or african-american over european-american names; however, our results indicate that there is often a statistically significant preference (@xmath0.0001) for associating female and african-american identifiers with being more "emotional." we provide two key contributions to understanding and mitigating bias in contextual language models. first, we conduct a comprehensive, comparative evaluation of gender and racial bias using multiple tests for widely-used pretrained models. second, we construct a novel experiment for debiasing a contextual language model on a downstream task @xcite . our experiment refutes the validity of weat for contextual models, signaling a need for new bias metrics. ## related work after the seminal work of @xcite , bias has been found ubiquitous in word embedding models @xcite @xcite @xcite . researchers have applied association tests between word embeddings to look for inappropriate correlations. @xcite introduce the word embedding associate test (weat) to estimate implicit biases in word embeddings by measuring average cosine similarities of target and attribute sets. the weat has been extended into a sequence test @xcite , though the efficacy of both tests remains in question for transformers @xcite . prior work has also devised methods to measure contextual bias. @xcite introduce the equity evaluation corpus (eec), which includes templated sequences such as " target feels attribute ," where gendered or racial tokens are the "targets" and emotional words are the "attributes." the average of the difference in likelihoods for target sets constitutes the bias score. we leverage this in our work as the sequence ranking test (seq). @xcite and @xcite devise a pronoun-ranking test for bert by comparing relative likelihoods of target words. rather than sequence likelihood, the authors instead measure contextual likelihood, which helps to control for a model's overarching bias. we extend this work, applying the pronoun-ranking test (p n ) to score the most commonly used transformer models and contextualizing the results with seq scores. investigations of biases in contextual language models, e.g. transformers, have yielded mixed results. @xcite found that bert and gpt exhibit a reduced bias-dimension relative to word embedding models, whereas @xcite found that bert is biased and that conventional tests, e.g. weat, are inappropriate. recent work has also looked to identify bias by crowdsourcing a sterotype dataset @xcite @xcite . these approaches develop a bias analysis metric by empirically computing a pretrained model's preference towards stereotyped sentences. however, such work is specifically focused on showcasing the effectiveness of these specific datasets for identifying bias. our results paint a more complete picture, providing insight into specific aspects of gender and racial bias and unifying disparate viewpoints of prior work. furthermore, we present a targeted investigation into the relevance of the weat for transformers. ## approach and results we apply three tests (i.e. the weat (w ), sequence likelihood (seq), and pronoun ranking (p n )) to popular pre-trained transformers from hugging face @xcite , including the cased and uncased 1 bert and distilbert models, the uncased albert models, and the cased roberta, distilroberta, gpt-2, and xlnet models. for gender, we compare the weat tests for career (w c ), math (w m ), and science (w s ), against the sequence likelihood and pronoun ranking tests for anger (seq a and p n a ), fear (seq f and p n f ), sadness (seq s and p n s ), and joy (seq j and p n j ) evaluated between male and female target words. for race, we use the only weat available for race (w r ) as well as the same seq and p n tests evaluated between african-american and european-american targets. the results of our weat, sequence likelihood, and pronoun ranking bias tests are presented in tables 1 and 2. the quantity listed for each model/test pair is the effect size for that two-sided t-test test under they hypothesis that there is a significant difference between the mean likelihoods across the two groups. using multiple tests is important; many models exhibit systematic preference for one target according to seq, while the p n reveals contex-1 casing is a design decision affecting the tokenization for a model. for all models, we test every size available. model size and bias -examining the seq and p n results for distilled models distilbert and distilroberta, we see that these models almost always exhibit statistically significant bias and that the effect sizes for these biases are often much stronger than the original models from which they were distilled (bert and roberta). this finding is in line with contemporary work by @xcite , who show that distillation in vision models disproportionately harms underrepresented groups. we show that the same is true for transformers. the opposite is not true: increasing model capacity does not remove bias. while prior work @xcite has reported increasing model size correlates with decreasing bias, we find that this is not always the case (see gpt2-base vs. gpt2-large), as supported by @xcite in stereotype-likelihood tests. tokenization matters -we consider four architectures that come in cased and uncased versions, differing only in tokenization bert-base, bert-large, bert-largem, and distilbert. across race and gender, the uncased models exhibit less bias and greater diversity for names and pronouns. the effects of tokenization may also play a role in weat's underperformance, as the meanembeddings used to estimate a weat effect do not accurately reflect the expected words for the test. for example, under the albert tokenizer, "nichelle" becomes "niche" and "lle", two subwords which may not average out to a name. weat is inconsistent -we find that weat is a poor predictor of contextual bias and an internallyinconsistent metric. the weat for math (w m ) and science (w s ) use words which are very similar and, at times, even overlapping. as such, we would expect the w m and w s scores to indicate bias in the same direction for every model. instead, we see that the weat results show differing magnitudes and occasionally point in different directions. given the inconsistency of weat and its poor correlation with seq and p n effects, we propose a debiasing scheme using the weat effect. if neutralizing the weat effect also neutralizes seq and p n bias, then the weat remains a useful test for transformers. however, if neutralizing the weat has no effect on the seq and p n scores, we can conclude that the weat is simply not appropriate for contextual models. ## debiasing transformers with weat we now employ weat scores as a loss regularizer to "de-bias" a roberta model being trained on the situations with adversarial generations (swag) dataset, a commonsense inference dataset in which each sample is a sentence with four possible endings @xcite . the swag training objective is to minimize the model's cross-entropy loss, l m c , for choosing the correct ending. in addition to this loss, we incorporate weat scores as a regularizer, as shown in equation 1. here, λ w is a hyper-parameter, and w m , w r , w c , w s are the weat scores for each category. we hypothesize that, even if a model is able to minimize weat effects, the model will remain significantly biased. ## conclusion we systematically quantify bias in commonly used pre-trained transformers, presenting a unified view of bias in the form of gender and racial likelihoods across a range of popular pre-trained transformers. we analyze factors influencing bias in transformers using three tests, seq, p n , and w eat , and demonstrate the inadequacies of word-embedding neutralization for contextual models. we call for future work to develop robust bias tests and carefully consider the ramifications of design choices.