DiFair: A Benchmark for Disentangled Assessment of Gender Knowledge and Bias (2023)

Abstract: Numerous debiasing techniques have been proposed to mitigate the gender bias that is prevalent in pretrained language models. These are often evaluated on datasets that check the extent to which the model is gender-neutral in its predictions. Importantly, this evaluation protocol overlooks the possible adverse impact of bias mitigation on useful gender knowledge. To fill this gap, we propose **DiFair**, a manually curated dataset based on masked language modeling objectives. **DiFair** allows us to introduce a unified metric, *gender invariance score*, that not only quantifies a model’s biased behavior, but also checks if useful gender knowledge is preserved. We use **DiFair** as a benchmark for a number of widely-used pretained language models and debiasing techniques. Experimental results corroborate previous findings on the existing gender biases, while also demonstrating that although debiasing techniques ameliorate the issue of gender bias, this improvement usually comes at the price of lowering useful gender knowledge of the model.

## introduction it is widely acknowledged that pre-trained language models may demonstrate biased behavior against underrepresented demographic groups, such as women @xcite or racial minorities @xcite . given the broad adoption of these models across various use cases, it is imperative for social good to understand these biases and strive to mitigate them while retaining factual gender information that is required to make meaningful gender-based predictions. in recent years, numerous studies have attempted to address the biased behaviour of language models, either by manipulating the training data @xcite , altering the training objective @xcite , or by modifying the architecture @xcite . although figure 1 : ideally, a language model is expected not to favor a gender in a sentence that does not explicitly specify one (top example), while it should prefer genderspecific words when the gender is explicitly specified in the sentence (bottom example). cf. section 2.1 for task formulation. in this study, we try to fill this evaluation gap by presenting difair, a benchmark for evaluating gender bias in language models via a masked lan-guage modeling (mlm) objective, while also considering the preservation of relevant gender data. we test several widely-used pretrained models on this dataset, along with recent debiasing techniques. our findings echo prior research, indicating that these models struggle to discern when to differentiate between genders. this is emphasized by their performance lag compared to human upper bound on our dataset. we also note that while debiasing techniques enhance gender fairness, they typically compromise the model's ability to retain factual gender information. our contributions are as follows: (i) we introduce difair, a human curated language modeling dataset that aims at simultaneously measuring fairness and performance on gendered instances in pretrained language models; (ii) we propose gender invariance score, a novel metric that takes into account the dual objectives of a model's fairness and its ability to comprehend a given sentence with respect to its gender information; (iii) we test a number of transformer based models on the difair dataset, finding that their overall score is significantly worse than human performance. however, we additionally find that larger language models perform better; and (iv) we observe that bias mitigation methods, while being capable of improving a model's gender fairness, damage the factual gender information encoded in their representations, reducing their usefulness in cases where actual gender information is required. foot_0 2 the task preliminaries. gender knowledge implies the correct assignment of gender to words or entities in context. it involves a language model's aptitude to accurately detect and apply gender-related data such as gender-specific pronouns, names, historical and biological references, and coreference links. a well-performing model should comprehend gender cues and allocate the appropriate gendered tokens to fill the @xcite , guided by the sentence's context. gender bias in language models refers to the manifestation of prejudiced or unfair treatment towards a particular gender when filling the [mask] token, resulting from prior information and societal biases embedded in the training data. it arises when the model consistently favors one gender over another, irrespective of the contextual gender cues. ## dataset construction difair is mainly constructed based on samples drawn from the english wikipedia. 2 to increase the diversity of the dataset and to ensure that our findings are not limited to wikipedia-based instances, we also added samples from reddit (popular subreddits such as askreddit and relationship advice). these samples make up about 23% of the total 3,293 instances in the dataset. we demonstrate in to reduce unrelated sentences, we sampled only sentences with specific gendered pronouns like he or she or gendered names like waiter or waitress. the collected data was then labeled based on the following criteria: we expect a well performing model to correctly assign a gender to these instances. this categorization is necessary as the genderneutral sentences reveal the degree to which a model favors one gender over another, whereas the gender-specific sentences verify the ability of a model in identifying the gender of a sentence, and thus measuring a model's capability in retaining factual gender information. we then asked a trained annotator to manually mask a single gendered pronoun or gendered word using the following guideline: in order to mitigate the influence of the model's memory on its decision-making process and enhance anonymity in instances sourced from reddit, we incorporate a diverse range of specialized tokens. these tokens are designed to replace specific elements such as names and dates. we then replace every name with a random, relevant name. in case of gender-specific instances where the name plays a role in the final response, the names are replaced with a random name based on the context of the sentence, meaning that deciding to whether completely randomize the name, or preserve its gender is dependent on the context. in gender-neutral instances where the name does not play a role in the final response, the names are replaced with a randomly selected name representing a person (view the complete list of names in the appendix b). finally, we replace every date with a random relevant date. to this end, years are substituted with a randomly generated year, and every other date is also replaced with a date of the same format. more specifically, we replace all dates with random dates ranging from 100 years ago to the present. in section 4.2 we show that models can potentially exhibit sensitiveness with respect to dates in their gender prediction. we additionally show that balancing of the dataset with respect to dates is necessary in order to get reliable results. ## experiments and results we used difair as a benchmark for evaluating various widely-used pre-trained language models. specifically, we experimented with bidirectional models from five different families: bert @xcite , roberta @xcite , xlnet @xcite , albert @xcite , and distilled versions of bert and roberta @xcite . in the ideal case, a model should not favor one gender over another when predicting masked words in gender-neutral sentences, but should have preference for one gender in gender-specific instances. such a model attains the maximum gis of 1.0. a random baseline would not perform better than 0 in terms of gis as it does not have any meaningful gender preference (gss of 0). ## related work numerous methods have been developed in recent years for quantifying bias in language models. these methods can roughly be divided into two main categories: intrinsic and extrinsic bias metrics @xcite . intrinsic bias metrics assess the bias in language models directly based on model's representations and the language modeling task, whereas extrinsic metrics gauge bias through a model's performance on a downstream task. these downstream tasks encompass coreference resolution @xcite , question answering @xcite , and occupation prediction @xcite . while extrinsic metrics can provide insights into bias manifestation in specific task settings, discerning whether the bias originates from the task-specific training data or the pretrained representations remains a challenge. on the other hand, intrinsic bias metrics offer a more holistic view by quantifying the bias inherently encoded within models, thus addressing a crucial aspect of bias measurement not covered by extrinsic metrics. in essence, while extrinsic metrics analyze the propagation of encoded bias to specific downstream tasks, intrinsic metrics provide a broader perspective on the bias encoded within models. this distinction is significant, especially as nlp models evolve to assist practitioners in a more generalized manner. in the rapidly advancing era of nlp, where models are increasingly utilized in a general form employing in-context learning and prompt engineering, it becomes imperative to measure bias in their foundational language modeling form. hence, intrinsic bias metrics serve as indis-pensable tools for a comprehensive understanding and mitigation of bias in contemporary nlp models. as an intrinsic metric, difair evaluates bias using masked language modeling. therefore, we now review significant intrinsic bias metrics and datasets proposed to date. the first studies on bias in word embeddings were conducted by @xcite and @xcite . they demonstrated that word embeddings display similar biases to those of humans. @xcite utilized a word analogy test to illustrate some stereotypical word analogies made by a model, showcasing the bias against women (e.g. @xmath0:: @xmath1. in the same year, @xcite proposed weat, a statistical test for the measurement of bias in static word embeddings based on iat (implicitassociation test). using their metric, they showed that the nlp models behave similarly to humans in their associations, indicating the presence of a human-like bias in their knowledge. with the advent of contextualized word embeddings, and in particular, transformer-based language models @xcite , a number of studies have attempted to adapt previous work to be used with these new architectures. seat @xcite is a weat extension that attempts to evaluate the bias in sentence embeddings generated by bert @xcite and elmo @xcite models by incorporating the gendered words used in weat into sentence templates. as for the datasets, the most relevant to our work are crows-pairs @xcite and stere-oset @xcite . @xcite have recently introduced crows-pairs, which measures the model's preference for biased instances as opposed to anti-biased sentences. crows-pairs includes tuples of sentences, containing one stereotypical and one anti-stereotypical instance, and measures a model's tendency to rank one above the other. similarly, @xcite have proposed stereoset, which consists of two tasks to quantify bias, one of which is similar to the crows-pairs task in that it computes the model's bias in an intersentential manner, with the distinction that it provides an unrelated option for each sample. furthermore, they also compute a model's bias in an intrasentential manner by creating a fill-inthe-blank task and providing the model with three levels of attributes with respect to stereotype. in addition to measuring the bias encoded in the representations of a language model, they computed, the proportion of instances for each task in which the model selected the unrelated option as a proxy for the language modeling capability of a model. the existing evaluation approach suffers from the limitation of providing only one unrelated token per sample, which increases the likelihood of another unrelated token having a higher probability. in contrast, our proposed metrics assess both the model's gender bias and its encoded gender knowledge, offering a more accurate measure of its language modeling capability. ## conclusion in this paper, we proposed difair, a novel benchmark for the measurement of gender bias in bidirectional pretrained language models. the benchmark makes use of a new unified metric, called gender invariance score (gis), that can assess how biased language models are in gender-neutral contexts while taking into account the factual gender information that is retained, which is required to make correct predictions in sentences where gender is explicitly specified. the dataset comprises 2,506 carefully-curated gender-specific and genderneutral sentences. we demonstrate that even the model with the highest performance falls far short of human performance in this task. in addition, we show that current debiasing methods, as well as distilled models, despite being effective with regard to other bias metrics, severely damage gender factual data presented in the models' representations, resulting in subpar performance for instances in which gender is explicitly specified or required.