Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models (2021)

Abstract: This paper proposes two intuitive metrics, skew and stereotype, that quantify and analyse the gender bias present in contextual language models when tackling the WinoBias pronoun resolution task. We find evidence that gender stereotype correlates approximately negatively with gender skew in out-of-the-box models, suggesting that there is a trade-off between these two forms of bias. We investigate two methods to mitigate bias. The first approach is an online method which is effective at removing skew at the expense of stereotype. The second, inspired by previous work on ELMo, involves the fine-tuning of BERT using an augmented gender-balanced dataset. We show that this reduces both skew and stereotype relative to its unaugmented fine-tuned counterpart. However, we find that existing gender bias benchmarks do not fully probe professional bias as pronoun resolution may be obfuscated by cross-correlations from other manifestations of gender prejudice.

## introduction transformer-based transfer learning models for nlp -referred to henceforth as tbtl models for brevity -such as bert @xcite , roberta @xcite , and albert @xcite perform well on a variety of nlp tasks with minimal fine-tuning. however, prior to finetuning, tbtl models require a vast amount of data to train @xcite . this training is only performed once, with users downloading and fine-tuning such language models to their specific task. in doing so, we are trusting large tech companies to train the base model responsibly since we have no control over this. this seems inherently undemocratic. we ideally want these models to be † equal contribution. as tbtl models become increasingly prevalent in our everyday lives, we want to avoid such prejudices influencing decision making. examples where this is important include automatic resume filtering @xcite and criminal sentencing recommendations @xcite . in this paper we focus on the specific problem of gender bias, and analyse the extent to which it persists in modern tbtl models. we build upon @xcite , in which quantification and mitigation of bias in elmo was centre stage. in addressing this problem for more recent models, we aim to answer three main questions: i) how can we quantify bias in pre-trained language models? ii) how do different models compare in terms of bias? iii) how to mitigate bias in these models? we believe that current gender bias metrics in the existing literature do not offer sufficient granularity to properly analyse this problem. indeed, they mostly focus on measuring the assignment of stereotypical pronouns to professions @xcite . by focusing solely on this, they fail to address a model's overall preference for predicting male pronouns. an alternative bias which models can demonstrate is unequal preference towards male and female pronoun resolution across stereotypical and anti-stereotypical professions. we refer to these two forms of bias as skew and stereotype, respectively. in section 3, we propose a new scheme to capture and quantify the important distinction between the two. when comparing different tbtl models, we find evidence that gender skew and gender stereotype correlate approximately negatively with each other in out-of-the-box models, suggesting that a tradeoff between these two forms of bias may exist. to mitigate bias in these models, we use the method proposed by @xcite to show that fine-tuning with augmented data, which references male and female entities with equal frequencies, can reduce professional gender stereotype and skew compared to fine-tuning on the original dataset. however, we show that gender prejudice may persist in forms other than professional bias, and these are ineffectively probed by current nlp benchmarks. statistical tests such as word/sentence embedding association tests (weat/seat) have been developed to measure bias in static word embeddings using the cosine similarity of specific target words @xcite . however, when it comes to contextual embeddings, these traditional metrics have been shown to be ineffective at quantifying bias. in particular, @xcite demonstrate that while weat tests are unable to identify any statistically significant bias in bert, probing the underlying language model with a gender pronoun resolution (gpr) task does reveal strong evidence that these non-static models also encode gender bias. indeed, since contextual word embedding models such as bert are optimised to capture the statistical properties of training data, they tend to pick up and amplify any social stereotypes that may be present @xcite . having established gpr as a downstream task suitable for detecting gender bias, @xcite introduced a new benchmark, winobias, to measure bias in coreference pronoun resolution. the dataset consists of two files, test set 1 and test set 2 (hereafter t1 and t2), representing two different gender pronoun resolution tasks. each file consists of wino-grad schema pairs of sentences involving a variety of occupations, differing only in one or two words and with a pronoun ambiguity that is resolved in opposite directions across the two sentences, giving both a pro-and anti-stereotypical resolution @xcite . example sentences are shown in fig. 1 and fig. 2 . @xcite consider a coreference resolution system unbiased on the winobias test if it achieves similar f1 scores for gender pronoun resolution on both the pro-and anti-stereotypical datasets whilst maintaining strong gpr performance. one of the main findings in @xcite is that three different coreference resolution architectures (rule based, feature-rich and neural-net based) built on top of static word embeddings all display significant disparity in f1 scores across the two datasets, with the f1 score for the pro-stereotypical dataset being on average 21.1 higher. this alarming observation was also discovered by @xcite and was attributed to the inherent bias of the underlying word embeddings @xcite , as well as the training of these coreference resolution pipelines on the ontonotes 5.0 dataset @xcite which is known to suffer from severe gender imbalance @xcite . more recently, @xcite investigated the existence of gender bias in the elmo contextual embedding. specifically, they note elmo is trained on the billion word corpus @xcite which, just like ontonotes 5.0, shows substantial imbalance in the counts of male vs. female pronouns. training on this, elmo then learns a language representation that reflects this gender inequality. to expose this, @xcite analyse the behaviour of a coreference resolution system proposed by @xcite with elmo contextual weights on the winobias benchmark, revealing a significant disparity in performance on the proand anti-stereotypical datasets. in fact, this disparity is 30% higher than a similar result based only on glove embeddings @xcite . this is particularly worrying; as commented earlier, contextual embeddings may, by construction, be amplifying the doctor hired the receptionist because he was overwhelmed with clients. the doctor hired the receptionist because she was overwhelmed with clients. the doctor hired the receptionist because she was highly recommended. the doctor hired the receptionist because he was highly recommended. these take the form [entity1] [interacts with] [entity2] [conjunction] [pronoun] [circumstances]. solid blue boxes indicate male entities, dashed orange boxes indicate female entities, solid purple lines indicate pro-stereotypical scenarios and dashed purple lines indicate anti-stereotypical scenarios. such stereotypes are determined according to u.s. bureau of labor statistics (2017). the receptionist called the doctor and told him about a new patient. the receptionist called the doctor and told her about a new patient. the doctor called the receptionist and told her to cancel the appointment. the doctor called the receptionist and told him to cancel the appointment. undesirable statistical artefacts of the dataset more than their static counterparts. therefore, it is of the utmost importance to perform a similar analysis on recent tbtl models. @xcite explicitly state that bert is not trained on the billion word corpus, since this only provides examples of isolated sentences and the authors preferred to use a document-level corpus to get contiguous training data, allowing richer contexts to be learnt. specifically, bert is trained using the book-corpus dataset @xcite as well as english wikipedia. however, the bookcorpus data has since been shown to suffer from similar gender imbalance problems (tan and celis, 2019) as has english wikipedia where, for example, only 15.5% of the biographies are of women @xcite . we believe that this imbalance is the principle cause of skew in the model. ## conclusion and future work quantifying gender bias in coreference resolution is challenging, since co-referencing performance and bias manifestation are closely linked. we have proposed skew and stereotype as new measures of gender bias, allowing us to better probe model prejudice. we have shown that there is an approximate trade-off between the skew and stereotype of outof-the-box models. distilbert and bert models have high skew and low stereotype whilst roberta and albert-xxlarge have reduced skew at the cost of higher stereotype. two methods have been proposed to mitigate bias: online skewness mitigation and data augmentation. the online approach has been shown to be effective at mitigating stereotype at the expense of skew, demonstrating the opposite effect to what it was designed for. we took the data augmentation method proposed by @xcite for debiasing elmo and extended it to bert, demonstrating that it reduces both forms of gender bias compared to unaugmented fine-tuned models. however, the reduction of explicit professional gender skew and stereotype reveal the model's underlying bias towards gender competency. we successfully expose these using winobias gpr sentence probes labelled for competency. since contextual language models consider the full sentence contents when assigning a pronoun, we believe that the winobias data used in this paper does not purely measure professional biases. a second popular dataset taken from the superglue benchmarks, winogender @xcite , is increasingly used for evaluating a model's gender bias. however, its limited size relative to winobias makes it less robust and hence it was not used in this paper. we observed that language models may also consider other stereotyped gender characteristics in the sentence when classifying pronouns. given the above considerations, we believe that a more comprehensive set of gender bias benchmarks should be developed which can better isolate specific biases within models. @xcite have shown that both race and gender bias are prevalent in a large proportion of state-of-the-art language models. recently, a number of other datasets have appeared for detecting these and other kinds of bias such as age and religion @xcite . it would be interesting to see if competency bias obscures analyses on these datasets similarly. future research is recommended on how data augmentation affects other models from the bert family. additionally, it will be valuable to explore whether data augmentation could be applied to larger corpora to train a new contextual model from scratch. lastly, in contrast to static embeddings, it is notoriously hard, if not impossible, to define bias in contextual embeddings @xcite . it is likely that without extensive research and transparent communication, the field of nlp will be further scrutinised as more applications are found to exhibit undesired biases. discussions, both within and outside the community, are required to determine what separates bias from semantic assumptions, allowing bias disclaimers and guidelines to be provided to downstream developers.