Intersectional Stereotypes in Large Language Models: Dataset and Analysis (2023)

Abstract: Despite many stereotypes targeting intersectional demographic groups, prior studies on stereotypes within Large Language Models (LLMs) primarily focus on broader, individual categories. This research bridges this gap by introducing a novel dataset of intersectional stereotypes, curated with the assistance of the ChatGPT model and manually validated. Moreover, this paper offers a comprehensive analysis of intersectional stereotype propagation in three contemporary LLMs by leveraging this dataset. The findings underscore the urgency of focusing on intersectional biases in ongoing efforts to reduce stereotype prevalence in LLMs.

## introduction the current body of research concerning the propagation of stereotypes by large language models (llms) predominantly focuses on single-group stereotypes, such as racial bias against african americans or gender bias against women @xcite @xcite @xcite . nevertheless, it is crucial to acknowledge that numerous stereotypes are directed toward intersectional groups (e.g., bias against african american women), which do not fit into broad single-group classifications. existing studies on intersectional stereotypes @xcite often adopt a reductionist approach, primarily focusing on intersectional groups comprising just two demographic attributes. such research also tends to limit the analysis to word-level, neglecting the possibility of more covert, context-dependent stereotypes. furthermore, the exploration of stereotypes is often constrained to a few aspects, like appearances or illegal behavior. to address these limitations, we have curated an intersectional stereotype dataset with the aid of the chatgpt model foot_0 . for constructing the intersectional groups, we remove all the constraints and enable any combination of 14 demographic features across six categories, namely, race (white, black, and asian), age (young and old), religion (nonreligious, christian, and muslim), gender (men and women), political leanings (conservative and progressive), and disability status (with disability and without). this approach allows us to assess a wide range of stereotypes targeted at diverse group combinations, as generated by chatgpt. our results show that chatgpt effectively discerns our objectives and generates common stereotypes for up to four intersecting demographic groups. the quality of the stereotypes generated was also substantiated by human validation. however, as the demographic traits exceed four, the groups become exceedingly specific, leading chat-gpt to make overly broad generalizations. by incorporating rigorous post-generation validation using both chatgpt and human validation, we successfully mitigated this overgeneralization, thereby enhancing the quality of the data points. this shows the strength of chatgpt (and potentially other llms) for helping with stereotype-related research. section 2 discusses the complete dataset construction process. leveraging this newly created dataset, we probed the presence of stereotypes within two contemporary llms, gpt-3 @xcite and chatgpt. following a methodology similar to @xcite , we interrogated the llms and analyzed their responses. however, we expanded the scope of inquiry by designing questions that spanned 16 different categories of stereotypes. our findings revealed that all the models studied produced stereotypical responses to certain intersectional groups. this observation underscores that stereotypes persist in even the most modern llms, despite the moderation measures enforced during their training stage @xcite . we argue that future de-biasing efforts should prioritize mitigating intersectional and implicit stereotypes. section 3 discusses stereotype examination in more details. ## dataset construction understanding intersectional stereotypes can pose a significant challenge, particularly for non-experts, due to their complexity and overlap with more general group-based stereotypes. to address this, we have curated the dataset leveraging chatgpt and have ensured its integrity through validation by both the model and human validators. the objective of our dataset is to facilitate the expansion of intersectional stereotype research to include a wider array of demographic groups, going beyond the scope of past investigations, with llms. ## conclusion & future work in this paper, we introduce an intersectional stereotype dataset and evaluate the prevalence of stereotypes in three contemporary language learning models (llms) across 106 intersectional groups. the dataset is automatically created and filtered using chatgpt, and it undergoes manual validation to ensure it encompasses the common stereotypes specifically targeting these demographic groups. furthermore, we classify the stereotypes in this dataset into 16 categories and formulate categoryspecific questions to assess the stereotypical behaviors of llms. the findings from our stereotype examination underscore the necessity for additional investigation and mitigation of stereotypes in llms, particularly the more complex intersectional stereotypes, especially when these models are made publicly available. our dataset serves as a valuable resource that can be employed and expanded upon to attain a broader understanding of intersectional stereotypes and to work toward the reduction of harmful stereotypes in llms.