S tereo S et: Measuring stereotypical bias in pretrained language models (2021)

Abstract: A stereotype is an over-generalized belief about a particular group of people, e.g., Asians are good at math or African Americans are athletic. Such beliefs (biases) are known to hurt target groups. Since pretrained language models are trained on large real-world data, they are known to capture stereotypical biases. It is important to quantify to what extent these biases are present in them. Although this is a rapidly growing area of research, existing literature lacks in two important aspects: 1) they mainly evaluate bias of pretrained language models on a small set of artificial sentences, even though these models are trained on natural data 2) current evaluations focus on measuring bias without considering the language modeling ability of a model, which could lead to misleading trust on a model even if it is a poor language model. We address both these problems. We present StereoSet, a large-scale natural English dataset to measure stereotypical biases in four domains: gender, profession, race, and religion. We contrast both stereotypical bias and language modeling ability of popular models like BERT, GPT-2, RoBERTa, and XLnet. We show that these models exhibit strong stereotypical biases. Our data and code are available at https://stereoset.mit.edu .

## introduction a key idea behind the current success of neural network models for language is pretrained representations such as word embeddings @xcite and pretrained language models @xcite @xcite @xcite . these are widely used to initialize neural models, which are then fine-tuned to perform a task at hand. typically, these are learned from massive text cor- choose the appropriate word: domain: gender target: girl context: girls tend to be more ____ than boys option 1: soft (stereotype) option 2: determined (anti-stereotype) option 3: fish (meaningless) (a) the intrasentence context association test choose the appropriate sentence: domain: race target: arab context: he is an arab from the middle east. option 1: he is probably a terrorist with bombs. (stereotype) option 2: he is a pacifist. (anti-stereotype) option 3: my dog wants a walk. (meaningless) (b) the intersentence context association test pora using variants of language modeling objective (i.e., predicting a word given its surrounding context). in the recent years, these representations empowered neural models to attain unprecedented levels of performance gains on multiple language tasks. the resulting models are being deployed widely as services on platforms like google cloud and amazon aws to serve millions of users. while this growth is commendable, there are concerns about the fairness of these models. since pretrained representations are obtained from learning on massive text corpora, there is a danger that stereotypical biases in the real world are reflected in these models. for example, gpt2 @xcite , a pretrained language model, has shown to generate unpleasant stereotypical text when prompted with context containing certain races such as african-americans @xcite . in this work, we assess the stereotypical biases of popular pretrained language models. the seminal works of @xcite and @xcite show that word embeddings such as word2vec @xcite and glove @xcite contain stereotypical biases using diagnostic methods like word analogies and association tests. for example, @xcite show that male names are more likely to be associated with career terms than female names where the association is measured using embedding similarity. recently, studies have attempted to evaluate bias in contextual word embeddings where a word is provided with artificial context @xcite , e.g., the contextual embedding of man is obtained from the embedding of man in the sentence this is a man. however, these have limitations. first, the context does not reflect the natural usage of a word. second, they require stereotypical attribute terms to be predefined (e.g., pleasant and unpleasant terms). third, they focus on single word terms and ignore multiword terms like construction worker. lastly, they study bias of a model independent of its language modeling ability which could lead to undeserved trust in a model if it is a poor language model. in this work, we propose methods to evaluate stereotypical bias of pretrained language models. these methods do not have the aforementioned limitations. specifically, we design two different association tests, one for measuring bias at sentence level (intrasentence), and the other at discourse level (intersentence) as shown in figure 1 .. in these tests, each target term (e.g., arab) is provided with a natural context in which it appears, along with three possible associative contexts. the associative contexts help us to evaluate the biases of the model, as well as measure its language modeling performance. we crowdsource stere-oset, a dataset for associative contexts in english containing 4 target domains, 321 target terms and 16,995 test instances (triplets). ## related work our work is inspired from related attempts that aim to measure bias in pretrained representations such as word embeddings and language models. ## dataset creation in stereoset, we select four domains as the target domains of interest for measuring bias: gender, profession, race and religion. for each domain, we select terms (e.g., asian) that represent a social group. for collecting target term contexts and their associative contexts, we employ crowdworkers via amazon mechanical turk. 1 we restrict ourselves to crowdworkers in usa since stereotypes could change based on the country. ## experimental setup in this section, we describe the data splits, evaluation metrics and the baselines. ## main experiments in this section, we evaluate pretrained models such as bert @xcite , roberta @xcite , xlnet @xcite and gpt2 @xcite on stereoset. ## results and discussion relation between lms and ss all models exhibit a strong correlation between lms and ss (spearman rank correlation ρ of 0.87). as the language model becomes stronger, its stereotypical bias (ss) does too. we build the strongest language model, ensemble, using a linear weighted combination of bert-large, gpt2-medium, and gpt2-large, which is also found to be the most biased model (@xmath0.5). the correlation between lms and ss is unfortunate and perhaps unavoidable as long as we rely on the real world distribution of corpora to train language models since these corpora are likely to reflect stereotypes. amongst the models, gpt2 exhibits more unbiased behavior than other models (icat score of 73.0). however, this metric is not intended as the sole criterion for model selection. further research is required in designing better metrics. impact of model size for a given architecture, all of its pretrained models are trained on the same corpora but with different number of parameters. for example, both bert-base and bert-large are trained on wikipedia and bookcorpus @xcite with 110m and 340m parameters respectively. as the model size increases, we see that its language modeling ability (lms) increases, and correspondingly its stereotypical score. ## conclusions in this work, we develop the context association test (cat) to measure the stereotypical biases of pretrained language models in contrast with their language modeling ability. we crowdsource stere-oset, a dataset containing 16,995 cats to test biases in four domains: gender, profession, race and religion. we show that current pretrained language models exhibit strong stereotypical biases. we also find that language modeling ability correlates with the degree of stereotypical bias. this dependence has to be broken if we are to achieve unbiased language models. we hope that stereoset will spur further research in evaluating and mitigating bias in language models. we also note that achieving an ideal performance on stereoset does not guarantee that a model is unbiased since bias can manifest in many ways @xcite .