Investigating Subtler Biases in LLM s: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models (2024)

Abstract: LLMs are increasingly powerful and widely used to assist users in a variety of tasks. This use risks introducing LLM biases into consequential decisions such as job hiring, human performance evaluation, and criminal sentencing. Bias in NLP systems along the lines of gender and ethnicity has been widely studied, especially for specific stereotypes (e.g., Asians are good at math). In this paper, we investigate bias along less-studied but still consequential, dimensions, such as age and beauty, measuring subtler correlated decisions that LLMs make between social groups and unrelated positive and negative attributes. Although these subtler biases are understudied they follow people as much as gender and ethnicity do. So, we want to see whether they also follow one with LLMs.We introduce a template-generated dataset of sentence completion tasks that asks the model to select the most appropriate attribute to complete an evaluative statement about a person described as a member of a specific social group. We also reverse the completion task to select the social group based on an attribute. We report the correlations that we find for 4 cutting-edge LLMs. This dataset can be used as a benchmark to evaluate progress in more generalized biases and the templating technique can be used to expand the benchmark with minimal additional human annotation.

## introduction alongside the impressive new capabilities of recent language generation models such as chat-gpt @xcite , @xcite , and llama-2 @xcite , these systems are increasingly involved in consequential decisions made in the real world. this includes job hiring and performance reviews, with tips for hiring 1 our dataset and code is available at https://github.com/kamruzzaman15/ identifying-subtler-biases-in-llms . managers appearing across the internet. even before these recent advancements, ai has been used in the criminal justice system leading to the amplification of social inequities @xcite . in order to manage these biases prior research has investigated the most salient dimensions of bias in word embeddings and llms, such as gender and ethnicity @xcite @xcite . prior work also focuses particularly on whether these ai systems produce specific stereotypes of underrepresented minorities, such as associating middle eastern people with perfumes or jewish people with greed @xcite . social scientists found that human biases extend beyond simple stereotypes and can lead to general associations of positive attributes to members holding (or perceived to hold) certain key characteristics. for example, @xcite found that people are more likely to infer a plethora of other desirable characteristics to people that are judged more attractive-a result that has been confirmed and elaborated upon for the present day by more recent studies @xcite @xcite . in this paper, we extend the evaluation of bias in llms in the following key ways: we investigate whether llms make general associations between stereotyped categories and unrelated positive, negative, and neutral attributes rather than specific stereotype inferences. in addition, we investigate dimensions of bias that have been largely overlooked: age, beauty, academic institution, and nationality. although understudied, these dimensions of bias follow people as much as gender and ethnicity do. furthermore, the few existing studies regarding these biases in llms all study them in terms of specific stereotypes. figure 1 shows an example of how we formulate the completion task for llms in the bias domain of academic institutions. the contributions of this paper are the following. ## motivating studies from social science many studies demonstrate the pervasive impact of social biases in various spheres of life. @xcite found that people who are considered attractive are more likely to be believed to hold socially desirable traits and higher occupational status. this attractiveness bias also occurs inversely, as shown by @xcite , where known desirable qualities influence our perceptions of an individual's beauty. these two papers inspired us to set up our experimental design. more recent psychological research has confirm using modern scientific standards that such biases extend in specific real-world settings like employee termination @xcite , hiring practices @xcite , interview callbacks @xcite , and even in victim blaming scenarios @xcite . an opposite bias effect for beauty exists for people in particular contexts, in which a highly attractive person is dispreferred @xcite . these effects are subject to interactions with sexual motive, social competition, culture, and the nature of the interaction making them more difficult to study @xcite . generally, this opposing beauty bias effect appears when a preference for beauty would introduce a social threat (e.g., hiring a same-sex person, an embarrassing purchase from an opposite-sex person, etc.). in this paper we focus on the most general beauty bias where beauty is unduly associated with positive attributes leaving the opposite bias effect for future work. age-related stereotypes similarly impact societal perceptions and actions, with studies by perdue and @xcite , @xcite indicating a general propensity to infer negative traits based on age. this aligns with the foundational work on ageism by @xcite and has been found to have implications in professional settings @xcite . organizations and institutions have been found to be associated with specific personality traits, both within the us @xcite and internationally @xcite . @xcite found that universities leverage specific personality traits when branding and @xcite student personality traits play an important role in student choice of university and alternatives (e.g., vocational education). such bidirectional effects of personality traits result in hiring biases that prioritize students from academic institutions with particular reputations @xcite . nationality bias has found to affect student interactions in multicultural online learning environments @xcite , consumer perception towards products @xcite , academic philosophy @xcite , and peer evaluation @xcite . @xcite identified the economic development of a person's country of origin as an important factor within this bias, overshadowing individual qualities when one evaluates their peers. as llms are trained on data created by humans, we hypothesize that they are prone to similar biases to those identified in people. if present, we must measure the degree to which such biases are present so that we can appropriately account for them when llms are used for consequential de-cisions, such as recruiting and hiring. in order to do this, we measure our bias in terms of "representational harms" (using the terminology from blodgett et al. ( 2020 )), that is, an llm is biased if it makes general associations between stereotyped categories and unrelated positive, negative, and neutral attributes. following the guidance of @xcite , our objective is to define and measure these biases, particularly how they manifest and potentially perpetuate existing social hierarchies. ## related work moving on to related work in nlp specifically, bias in models have been studied for word embeddings using cosine similarity (e.g., the word embedding association test) and sentence embeddings using templates such as "this will [target]" (e.g., the sentence encoder association test.) @xcite @xcite . @xcite created crowdsourced stereotype pairs (crows-pairs), a dataset that studied nine different types of social biases (e.g., age, nationality, physical appearance, etc.) in masked language models. this work is the closest to our own in its study of ageism, beauty, and nationality and the use of intrasentence biases. however, it differs in both the model type (autoregressive vs. masked) and in the generality of the associations that are studied. @xcite introduced a dataset called stereoset to measure stereotypical bias in both masked and autoregressive pretrained language models. nationality bias has been studied before in gpt-2 by @xcite . they generated stories and analyzed how other factors (e.g., number of internet users, economic condition, etc.) affect nationality bias. @xcite explored seven social biases in three roberta-based models, including age, gender, and nationality. they used gdp to categorize countries for nationality bias. @xcite systematic study on pretrained language model-based evaluation metrics revealed that they exhibit greater bias than traditional metrics on attributes such race and gender. zhang et al. (2021) designed cloze-style example sentences to explore performance differences across demographic groups (age, race, etc.). smith et al. (2022) created holisticbias, a dataset for measuring bias across various demographic identities in language models. ## task definition unlike previous studies which have focused on identifying bias in a single direction, we take a more general measurement by considering two directions of bias. for example, @xcite measured stereotypes of llms by giving the race description (e.g., hispanic, ghanaian etc.) or a profession (e.g., physicist, tailor, etc.) and asked llms to choose between stereotypicallty associated attributes (e.g., poor, creative, etc.), but they did not measure the bias in reverse context (e.g., by giving attributes and asking llms to choose the associated race or profession). in our study, we study both directions, inspired by @xcite and @xcite work showing that people demonstrate beauty bias in both directions. in our experiment, we use fill-in-the-blank style sentences which evoke the biases in the provided text using a description of a person based on the bias category we are studying. we will refer to this description as simply the stimulus. for example, in figure 1 (a), the stimulus is "hinds community college" for institutional bias. ## dataset creation we consider four domains of bias in creating our dataset: age, beauty, academic institutions, and nations. we refer to academic institutions as institutions throughout our writing. 2 figure 2 : a high-level of diagram of our dataset sample creation procedure in both sai and asa directions. in the sai direction, if we pick "unattractive" as a stimulus, then the examples will be "he looks unattractive, he is blank", "she looks unattractive, she is blank", "they look unattractive, they are blank"; then we give 3 attributes as options to fill the blank. we also do this for all other stimuli for additional examples. in the asa direction, if we pick "smart" as an attribute, then the examples will be "he looks blank, he is smart", "she looks blank, she is smart", "they look blank, they are smart"; then we give 3 stimuli as options to fill the blank. we also do this for all others attributes. ## experimental setup we use gpt-4, palm-2, llama-2-13b, and mistral-7b in our experiments. 16,17 for model details, see appendix b. exact experimental details are in our supplementary materials, including scripts for replication. we examine how llms respond to positive and negative attributes and stimuli. specifically, we calculate the conditional likelihood of a model selecting positive, negative, and neutral attributes in response to stereotypically positive and negative stimuli. we will refer to these as [stimulus]to-[attribute] likelihoods. for example, we call the likelihood of the model to select positive attributes (e.g., friendly, motivated, creative, etc.) in response to stereotypically negative stimuli (e.g., unattractive, hinds community college, south sudan, 65 years old, etc.) the negative-to-positive likelihood (npl). our shorthand uses p for positive, n for negative, and nu for neutral. we consider a system to be biased if the conditional likelihood of positive, negative, or neutral completions change when the polarity of the prompting text changes. that is, an unbiased system will have ∆pl, ∆nl, and ∆nul values (defined in figure 3 ) of 0. for example, in figure 3 , we find that ∆nl is systematically negative, so the negative predictions of the models are biased in the opposite direction of the prompt text changes. we further report correlations and statistical significance using kendall's τ test @xcite . 18 in the sai direction, we calculate kendall's τ statistic between the binary positive and negative stimulus variable and the ternary positive, negative, and neutral attribute variable. we reverse everything in the asa direction, e.g., calculating the likelihood of selecting positive, negative, and neutral stimuli in response to positive and negative attributes. figure 3 shows the high-level trends in model predictions for each of the categories in both directions for gpt-4 and mistral. the prediction rates are summarized in terms of the change of rates when ## results and discussion we next break down the results by bias category, where the τ -test results are presented in here we focus on the broad trends of our results. complete results and additional discussions are available in appendix c. ## conclusion we examine the behavior of four common llms across several less-studied domains of bias, looking at general positive and negative polarity associations rather than precise stereotypes. our findings indicate that all four models exhibit statistically significant biases. our dataset and experimental design draw upon prior literature on both social science and computer science. as the use of llms continues to grow and these models are increasingly employed in various tools, it becomes crucial to be vigilant about even subtle forms of bias. while much work has been done to investigate overt biases such as those related to race, gender, and religion, less attention has been paid to subtler biases such as ageism, beauty, and institution. through the introduction of our dataset, we encourage the consideration of these overlooked biases when using llms. we hope that this dataset will help to further research and mitigate these types of biases. future research is needed to extend our findings to other models and further biases that have been identified by social scientists.