Who is better at math, Jenny or Jingzhen? Uncovering Stereotypes in Large Language Models (2024)

Abstract: Large language models (LLMs) have been shown to propagate and amplify harmful stereotypes, particularly those that disproportionately affect marginalised communities. To understand the effect of these stereotypes more comprehensively, we introduce GlobalBias, a dataset of 876k sentences incorporating 40 distinct gender-by-ethnicity groups alongside descriptors typically used in bias literature, which enables us to study a broad set of stereotypes from around the world. We use GlobalBias to directly probe a suite of LMs via perplexity, which we use as a proxy to determine how certain stereotypes are represented in the model’s internal representations. Following this, we generate character profiles based on given names and evaluate the prevalence of stereotypes in model outputs. We find that the demographic groups associated with various stereotypes remain consistent across model likelihoods and model outputs. Furthermore, larger models consistently display higher levels of stereotypical outputs, even when explicitly instructed not to.

## introduction llms are increasingly used for tasks that span areas of concern for bias and fairness @xcite , such as user discrimination in recommendations @xcite . despite the obvious need for ethical frameworks around these models, these are mostly lacking or incomplete, and make research into fairness and bias essential for supporting public confidence in the use of generative ai. while bias is often defined in ambiguous and conflicting ways @xcite , in this paper we focus on representational harms, defined by crawford (2017) as harms that "occur when systems reinforce the subordination of some groups along the lines of identity," specifically harms caused by stereotyping. attribute details name kazuyo age 45 personality traits calm, wise, observant negative traits perfectionist, indecisive, shy hobbies bonsai gardening, origami, tea ceremonies occupation librarian height 5.2 ft hair colour black eye colour brown skin colour light build petite socioeconomic status middle class sexual orientation asexual religion shinto in sections 4 and 5, we utilise templates involving stereotypes for 40 groups, defined by both an ethnicity and a gender, e.g. english female or chinese female, along with a descriptor (e.g. 'good at math') to explore which descriptors are more likely to appear in a sentence with certain given names across different llms. considering the limitations of using a fixed set of stereotypes and the fact that likelihoods do not always correspond to model outputs @xcite , in section 6, we take a lexicon-free approach that utilizes the given names in our dataset in a generation task. an example output can be seen in ## the dataset we propose a new dataset foot_0 named globalbias for studying harmful stereotypes, which consists of sets of 10 proper names spanning 40 groups. a summary of key statistics for the dataset can be found in ## adjusted perplexity across descriptors (apx) for masked language models (mlm) such as roberta @xcite , pseudo-perplexity @xcite is used instead, which replaces the likelihood p in equation 1 by p mask (x i |x ¬i ), the pseudo-likelihood to predict the masked token x i @xcite . for encoder-decoder lms such as flan-ul2 @xcite , we compute p lm on the decoder, which is conditioned by the encoder. ## stereotypes via apx we propose a statistically robust methodology to identify the demographic groups associated with the 730 descriptors in globalbias. we calculate the apx for the 876,000 sentences in the dataset. as described in the previous section, we compute the average of the 10 names per group for each template, and take the normalised average of the three templates to obtain a bias score for each gender-byethnicity group for each descriptor. once we have the bias scores for each of the 40 groups, we identify any groups with a 1% one-tailed significance level, as shown in figure 1 . our methodology can be applied to any descriptor and extended to additional gender-by-ethnicity groups and demographic axes in future. ## stereotypes via generation the above experiment sheds light on the plausibility assigned to sentences by llms containing combinations of proper nouns and descriptors. we complement this experiment by directly looking at models' generations, which has advantages such as potentially higher correlation with downstream performance @xcite . to this end, we use a zero-shot prompting method that utilizes the given names in globalbias. our prompt (appendix b) instructs the model to generate a dataset of characters, each associated with a given name from globalbias, with information such as hobbies, personality traits and physical attributes. an example can be found in with widespread usage: claude 3 opus, llama 3 70b instruct, and openai's gpt 3.5 and gpt 4o. 2 the rationale for using an open-ended generation setting was two-fold: (1) the likelihoods studied in the previous section do not always correspond to model outputs @xcite , and (2) taking a lexicon-free approach allows us to capture stereotypes that we had not thought of a priori. furthermore, this approach enables testing for stereotypes in closed-source models. ## conclusion in this work, we present the globalbias dataset, which allows us to undertake a comprehensive study of intersectional stereotypes. we introduce a new evaluation metric, apx, to adjust for highfrequency given names in training. this study examines a broader range of demographic groups than previous studies, and we conduct multiple experiments that investigate both the model's internal representations via apx and model outputs via generation experiments. we find that larger models produce more stereotypical outputs, even when explicitly instructed not to. we also show using the example of religion that bias stays consistent across model's internal representation and outputs. our work reveals the prevalence and impact of stereotypes across a diverse range of ethnic and gender groups through the introduction of the global-bias dataset. we highlight the importance of a comprehensive and intersectional approach to studying bias in lms, which is essential for ensuring ethical, fair, and effective use of lms in real-world scenarios, ultimately fostering trust and inclusivity in technology.