Measuring Fairness with Biased Rulers: A Comparative Study on Bias Metrics for Pre-trained Language Models (2022)

Abstract: An increasing awareness of biased patterns in natural language processing resources such as BERT has motivated many metrics to quantify ‘bias’ and ‘fairness’ in these resources. However, comparing the results of different metrics and the works that evaluate with such metrics remains difficult, if not outright impossible. We survey the literature on fairness metrics for pre-trained language models and experimentally evaluate compatibility, including both biases in language models and in their downstream tasks. We do this by combining traditional literature survey, correlation analysis and empirical evaluations. We find that many metrics are not compatible with each other and highly depend on (i) templates, (ii) attribute and target seeds and (iii) the choice of embeddings. We also see no tangible evidence of intrinsic bias relating to extrinsic bias. These results indicate that fairness or bias evaluation remains challenging for contextualized language models, among other reasons because these choices remain subjective. To improve future comparisons and fairness evaluations, we recommend to avoid embedding-based metrics and focus on fairness evaluations in downstream tasks.

## introduction with the popularization of word embeddings by works such as word2vec @xcite , glove @xcite and, more recently, contextualized variants such as elmo @xcite and bert @xcite , natural language processing (nlp) has seen significant growth and advancement. word embeddings and later language models have been adopted by many applications. many of these embeddings have been probed by researchers for biases such as gender stereotypes. word embeddings are generally trained on realworld data such that they model statistical properties from the training data. hence, they pick up biases and stereotypes that are typically present in the data @xcite . although @xcite and @xcite opine that this can pose significant challenges in downstream applications, this view has been questioned, especially for non-contextualized word embeddings @xcite . early works such as @xcite ; @xcite ; @xcite widely explored fairness in non-contextualized embedding methods. in non-contextualized embeddings such as word2vec and glove embeddings, models are trained to generate vectors that map directly to dictionary words and hence are independent of the context in which the word is used. in contrast, contextualized word embeddings take polysemy (words could have multiple meanings, e.g. 'a stick' vs 'let's stick to') into consideration. thus different embeddings are generated for a given word depending on the context in which it appears. because of such differences between the two approaches, popular techniques for detecting and measuring bias in non-contextualized word embeddings, such as weat @xcite , do not apply naturally to contextualized variants. many techniques have been proposed to measure bias in contextualized word embeddings, either as a standalone method @xcite or as an additional contribution to evaluate fairness interventions @xcite @xcite . this broad selection of methods makes it difficult for nlp practitioners to select an appropriate and reliable set of metrics to quantify bias and to compare results. this is further exacerbated as these quantifying techniques also involve different choices for attribute and target words, commonly jointly referred to as seed words, templates for context, and different methods for measuring similarity. in this paper, we combine literature survey and experimental comparisons to compare fairness met-rics for contextualized language models. we are guided by the following research questions: ## background static word embeddings have typically been used with recurrent neural networks (rnns), optionally with an attention mechanism @xcite . the transformer architecture @xcite introduced a new paradigm relying only on attention, which proved faster and more accurate than rnns and did not rely on static word embeddings. the transformer consists of two stacks of attention layers, the encoder and the decoder, with each layer consisting of multiple parallel attention heads. bert @xcite is based on the encoder from this transformer and obtained state-of-the-art results for multiple nlp tasks using transfer learning with a pre-training step and a second finetuning step. the pre-training task is to reconstruct missing words in a sentence, called masked language modeling (mlm), which helps capture interesting semantics. the training objective for a model with parameters θ is to predict the the original token on the position of a randomly masked token x m based on the positional-dependent context with 1 x @xmath0. after training, the language model can infer the probability that a token occurs on the masked position. as an illustration with the original bert model, the sentence '[mask]is a doctor.' is filled in with the token 'he' (62%), followed by 'she' (32%). because the mlm task relies on co-occurrences, this example illustrates how this task captures stereotypes that are present in pre-training datasets, which is referred to as intrinsic bias. pre-trained model e.g. bert pretraining corpora e.g. oscar, wiki, ... ## on the compatibility of measures in this section, our goal is to objectively investigate the consistency in indicating bias between various techniques used by previous works. as mentioned earlier, besides the metric choice, three primary factors are important when measuring intrinsic bias in an embedding model: (i) choice of seed words, (ii) choice of templates and (iii) how representations for seed words are generated. recent works investigating bias in language models have found issues with inconsistencies between seed words @xcite , unvoiced assumptions and data quality issues in stereoset and crows-pairs templates @xcite , and issues with semantically bleached templates (tan and celis, 2019). these issues raise some questions for the remaining two factors, for example whether or not the choice of template and technique for selecting embeddings to represent seed words matters in measuring bias? and are "semantically bleached" templates really semantically bleached? meaning, do they not affect bias measurements? or in the extreme, can bias in embedding model stay hidden by picking the "wrong" templates or representations? these are questions we seek to answer with a series of experimental analysis where we measure correlations between various approaches to test if these templates and representations measure the same bias. 2 when using two different embedding approaches, namely the [cls] (figure 2a ) and the pooled target token embeddings (figure 2b ). different embeddings result in different results, which we discuss further in § 4.3. the pearson correlation coefficients in bold are significant at the α = 0.05 level. ## code we make the source code available and also publish a package to bundle fairness metrics at https:// github.com/ipieter/biased-rulers. ## discussion and ethical considerations we mostly compare one of the most frequently studied settings, namely binary gender biases with a focus on professions. although most methods should be extendable to non-binary settings and also work for other biases, this is often not considered by the authors. furthermore, different works also consider different notions of gender and conflate multiple notions (cao and @xcite . both issues should be addressed in future works. we also observed that crows-pairs correlates less with other extrinsic measures, which could be caused by data issues @xcite . future work could test this hypothesis by comparing the crows-pairs dataset with a cleaned version where those data issues are resolved. however, such a version does currently not exist. related to this, is the design of the templates. we observed excessive variation between templates, similar to the differences between few-shot prompts that are used with autoregressive models like gpt-2 @xcite . future work could also focus on template designing and refine the concept of semantically bleached templates. with the availability of fairness metrics, we also risk that such metrics are used as proof or as insurance that the models are unbiased, although most metrics can only be considered indicators of bias at most @xcite . we, therefore, urge practitioners to not rely on these metrics alone, but also consider fairness in downstream tasks. we also did not draw much attention to many other negative impacts of language models that practitioners should consider, e.g. high energy usage or not including all stakeholders when training a language model @xcite . ## conclusion in this paper, we presented an overview of fairness metrics for contextualized language models and we focused on which templates, embeddings and measures these metrics used. we evaluated how these metrics correlate with each other, as well as how parts of these metrics correlate. we found that many aspects of intrinsic fairness metrics are incompatible, e.g. choosing different templates, embeddings, or even metrics. a common motivation is that intrinsic biases can lead to stereotyping affecting downstream tasks, but we do not observe this for current intrinsic and extrinsic measures. our advice is to use a mix of some intrinsic measures of fairness that don't use embeddings directly and eliminate one source of variance, for example disco or lpbs, in addition to a measure like tan and celis (2019) that seems to correlate well with at least some notion of extrinsic bias. however, we also recommend to perform extrinsic fairness evaluations on downstream tasks, since this is where actual resource allocations happen and where intrinsic and extrinsic biases collude.