Uncovering Stereotypes in Large Language Models: A Task Complexity-based Approach (2024)

Abstract: Recent Large Language Models (LLMs) have unlocked unprecedented applications of AI. As these models continue to transform human life, there are growing socio-ethical concerns around their inherent stereotypes that can lead to bias in their applications. There is an urgent need for holistic bias evaluation of these LLMs. Few such benchmarks exist today and evaluation techniques that do exist are either non-holistic or may provide a false sense of security as LLMs become better at hiding their biases on simpler tasks. We address these issues with an extensible benchmark - LLM Stereotype Index (LSI). LSI is grounded on Social Progress Index, a holistic social benchmark. We also test the breadth and depth of bias protection provided by LLMs via a variety of tasks with varying complexities. Our findings show that both ChatGPT and GPT-4 have strong inherent prejudice with respect to nationality, gender, race, and religion. The exhibition of such issues becomes increasingly apparent as we increase task complexity. Furthermore, GPT-4 is better at hiding the biases, but when displayed it is more significant. Our findings highlight the harms and divide that these LLMs can bring to society if we do not take very diligent care in their use.

## introduction large language models (llms) are now considered a foundational breakthrough with applications across various aspects of life, including but not limited to sectors critical to society such as governance, education, and healthcare @xcite . with @xcite we can already observe traces of artificial general intelligence (agi) that can match and surpass human intelligence @xcite . while llms' potential for good is immense, there is a commensurate potential for socio-ethical harms as outlined in the risk landscape presented by @xcite . given the broadness of the risks posed, there is a need to make collaborative efforts towards a deeper and a more diverse understanding of these. language has historically been at the forefront of perpetuating stereotypes and prejudice, and these harms carry over to the ai models of today that are predominantly language-based @xcite @xcite . on top of this, ai such as llms are also being used as decisionmakers in critical applications such as creditworthiness, crime recidivism, and human resourcing where these biases lead to material impact on people's lives @xcite @xcite . for example, @xcite highlight how compas -which is an ai-based tool used to decide criminal detention and releases in the united stateswas found to be biased against african americans leading to stricter detentions for this demographic. another example is pointed out by @xcite , on how amazon's ai-based hiring tool was found to be discriminating against female candidates. the above examples showcase that these harms are not only deeply-rooted in ai models, but are also becoming more ubiquitous in the society. thus, especially with increasing popularity of llms, it is paramount that these be measured across demographic categories as well as various social dimensions. there have been attempts to improve the coverage of various stereotypes and demographic groups in the bias evaluation literature @xcite . and, also to leverage from the fields of psychology and social science to evaluate bias across more social dimensions @xcite , but these are based on pre-trained word embeddings. although these have alleviated some of the issues, they do not allow us to measure these harms in a continuous and comprehensive way, specifically for new llm technology. in particular, we have concerns in the following three areas that might inhibit such a measurement of llm technology: in this work we introduce a novel benchmark, llm stereotype index (lsi), for evaluating stereotypes and the resulting bias in llms. lsi addresses the aforementioned three issues and is designed to be extensible: we then use lsi to evaluate chatgpt and gpt-4 for bias across four demographics: nationality, gender, race, and religion. we share some critical insights from our large-scale experiments with 157k generations that we believe are quite concerning. as there are many more research insights to be drawn from these experiments and data, we are releasing all the code and data publicly. foot_0 ## related work bias is front and center in works pertaining to risks of llms @xcite @xcite . however, literature @xcite in this area tends to have its own challenges in terms of not covering enough demographics, dimensions, or limited bias identification techniques. moreover, the studies that are broad enough rely on static datasets often oriented towards western countries @xcite . recent works like @xcite are more global in nature, but they are restricted in the demography categories considered or the evaluation methodology deployed. in this work, our focus is on detecting bias in llm generations. thus, in particular, we focus on works that analyze bias using llmgenerated texts and not using other methodologies like word embeddings. we divide these broadly into two categories. ## need for complex bias identification techniques llms have long been known to encode and perpetuate bias, including stereotypes @xcite . llm mirror their training data that reflects the historical injustice and under-representation present in most data sources @xcite . to address these issues, a significant portion of crowd intelligence (a.k.a. rlhf (christiano et al., 2017a)) used in llm training is allocated to steering them toward safe generations (openai, 2023). llms are trained to mirror certain ethical, moral, and social values. the most common methods achieve this by using reinforcement learning to align agents using rewards based on human feedback @xcite @xcite . however, these methods are not reliable, mainly due to the alignment problem @xcite @xcite and the fact that these values are instilled by a team of experts not representative of global diversity, that also introduces bias @xcite . even though these methods are not perfect, they do provide a layer of protection that makes it difficult to uncover the inherent bias of these models. to test this protection we evaluated newer gpt models using the text completion method described in @xcite . 2 while this may paint a rosy picture that new llms have reduced inherent bias, sadly that is not the case. this is evident as the bias betrays these protections as we move to more complex tasks or beyond the english language @xcite . figure 1 shows that when the language is not directly harmful, llms are still stunningly prone to generate biased content. talat et al. ( 2022) also raise similar concerns that existing bias benchmarks can actually obscure the problem if we are not diligent. to address this concern, we define a framework based on task complexity that provides a way to keep up with protections being put in new llms and still be able to detect the inherent bias present in the system. task prompt size (# task components) variety (diversity in terms of dissimilar task components) relationship (interdependencie s between task components) action complexity (inherent cognitive requirements of the task components) ## stereotypes based on social progress dimensions lsi is inspired by the social progress index (spi) @xcite . spi is a framework that evaluates and ranks countries by using a holistic set of social dimensions deemed critical for societal progress. stereotypes arise due to societal generalization of a people based on certain ground realities (past or present) @xcite . as an example, consider the social progress dimension of sanitation. india ranks low in this dimension compared to germany. while this is based on real facts and data @xcite , it is biased to assume that any individual from india is dirty. thus, we model lsi around these same social dimensions but converted to the form of social stereotypes. this allows a way to understand llm bias on the most critical of factors for society. for this work, we focus on stereotypes the spi dimensions along with the associated lsi stereotypes are presented in figure 3 . ## methodology lsi relies on four pivots {demography (e.g. religion), demography group (e.g. judaism), lsi stereotype pair (e.g. homeless and settled person) and task id} to create an llm task prompt. these are presented in the configuration in figure 4 . after an llm is prompted, we label the generation into three categories: whether the llm refused to make a choice (this is the ideal label), or whether it chose either of the positive/negative stereotypes. given the large scale of data, we use gpt-4 for labeling using a simple choice detection prompt. 7 this process is repeated n times for each possible configuration to account for non-determinism and statistical significance. examples of llm generations across various pivots are presented in appendix d. after these generations and labels are generated, we analyze them to identify bias in two layers: ideally, crp should be 100%, because in all these tasks any choice made would be assigning a label to a person based solely on their race, gender, religion, or nationality. this is harmful and considered stereotyping. for the cases where a choice is indeed made, we would expect similar stereotype polarity for all the different groups of the demographic. for example, blacks having lower sp than whites for the safety dimension will reinforce existing stereotypes around criminality. we understand that not all harms are equal @xcite -blacks being stereotyped as criminals could be more harmful than asians being stereotyped as math geniuses if the llm application under consideration is crime recidivism. since the impact of these harms is application-dependent, we consider uniform weight for lsi in this work. but as with other aspects -lsi can be easily extended with different weight distributions to account for various power dynamics and societal contexts. ## experimental setup in our experiments, we compared two openai models, chatgpt (gpt-3.5-turbo) and gpt-4, alongside an open-source model llama2-7b @xcite for evaluation. unfortunately, llama2-7b could not complete many tasks, hindering a fair comparison. 8 we examined twelve pairs of lsi stereotypes as in figure 3 , seven tasks, and four demographics (nationality, gender, race and religion) each with a different number of groups @xcite . each configuration was repeated n times foot_8 leading to a total of 157k generations (cf. appendix c for detailed calculation). to allow for creativity in some tasks, we used a temperature greater than 0.5 and a maximum token length of 300. ## results & insights based on the generations and their labels, we wanted to find answers to three questions in the following sections. ## conclusion llms like chatgpt and gpt-4 have immense potential to improve human life across the board. but, there is also a significant risk of systemic harms being ingrained deeper due to their use (fol-institute, 2023). the most urgent need is to understand the issues and measure their degrees in a comprehensive manner. this will allow us to gauge the potential impact on society and prioritize future development that limits the harm caused. our work is a step in that direction for the potential harm of bias. through our work, we want to highlight three key observations that are worrisome about the continued use of llms now and in the future, especially in scenarios with inherent complexity and nuance: on the flip side, we also noticed continuous improvements. a lot of techniques developed in the recent past like rlhf, and significant investment into using these for improving the safety of these models have paid off. most common use cases of these models (non-complex requests) are protected much better. our hope is to inspire more such developments and investments to address the identified areas of concern.