S ee GULL : A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models (2023)

Abstract: Stereotype benchmark datasets are crucial to detect and mitigate social stereotypes about groups of people in NLP models. However, existing datasets are limited in size and coverage, and are largely restricted to stereotypes prevalent in the Western society. This is especially problematic as language technologies gain hold across the globe. To address this gap, we present SeeGULL, a broad-coverage stereotype dataset, built by utilizing generative capabilities of large language models such as PaLM, and GPT-3, and leveraging a globally diverse rater pool to validate the prevalence of those stereotypes in society. SeeGULL is in English, and contains stereotypes about identity groups spanning 178 countries across 8 different geo-political regions across 6 continents, as well as state-level identities within the US and India. We also include fine-grained offensiveness scores for different stereotypes and demonstrate their global disparities. Furthermore, we include comparative annotations about the same groups by annotators living in the region vs. those that are based in North America, and demonstrate that within-region stereotypes about groups differ from those prevalent in North America.

## introduction language technologies have recently seen impressive gains in their capabilities and potential downstream applications, mostly aided by advancements in large language models (llms) trained on web data @xcite . however, there is also increasing evidence that these technologies may reflect and propagate undesirable societal biases and stereotypes @xcite ; sheng * work done while at google research et al., 2019; @xcite @xcite . stereotypes are generalized beliefs about categories of people, 1 and are often reflected in data as statistical associations, which the language models rely on to associate concepts. for instance, @xcite demonstrate that llmbased question-answer models rely on stereotypes to answer questions in under-informative contexts. not all statistical associations learned from data about a subgroup are stereotypes; for instance, data may associate women with both breast cancer and nursing as a profession, but only the latter association is a commonly held stereotype @xcite . recent work has built stereotype benchmark datasets (e.g., stereoset @xcite , crows-pairs @xcite ) aimed to detect such stereotypes in nlp model predictions. while these datasets have been instrumental in demonstrating that language models may reinforce stereotypes, they have several key limitations. first, they are limited in their size and coverage, especially for subgroups across the globe. second, they are curated exclusively with manual effort, and are thus limited by the world-view of the data creators and miss out stereotypes they might not be aware of. third, they do not qualify the stereotypes with any associated harms or offense @xcite . finally, they assume a single ground truth on whether a certain association is a stereotype or not, whereas stereotypes often vary from place to place. these limitations greatly reduce their utility in preventing stereotype harms in language technologies in the global landscape. in this paper, we show that we can leverage the few-shot learning and generative capabilities of llms to obtain a broad coverage set of stereotype candidates. while prior studies demonstrating that llms reproduce social stereotypes were in the interest of evaluating them, we are instead tapping into it as a capability of llms to generate a larger and broader-coverage set of potential stereotypes. we demonstrate that this approach works at a global scale (i.e., across 178 countries) as well as within local contexts (i.e., state-level identities within the us and india). we then employ a globally diverse pool of annotators to obtain richer socially situated validation of the generated stereotype candidates. our contributions are five-fold: • a novel llm-human partnership approach to create large-scale broad-coverage eval datasets. ## related work stereotypes are beliefs and generalizations made about the identity of a person such as their race, gender, and nationality. categorizing people into groups with associated social stereotypes is a reoccurring cognitive process in our everyday lives @xcite . decades of social scientific studies have led to developing several frameworks for understanding dimensions of social stereotyping @xcite @xcite . however, nuances of social stereotypes manifested in realworld data cannot be uniquely explored through any single framework @xcite . most classic studies of stereotypes rely on theory-driven scales and checklists. recent data-driven, bottomup approaches capture dynamic, context-dependent dimensions of stereotyping. for instance, @xcite propose an nlp-driven approach for capturing spontaneous social stereotypes. with the advances in nlp, specifically with significant development of llms in recent years, a large body of work has focused on understanding and evaluating their potential risks and harms @xcite @xcite . language models such as bert and gpt-2 have been shown to exhibit societal biases @xcite ; and roberta @xcite , and de-berta @xcite have been shown to rely on stereotypes to answer questions @xcite , to cite a few examples. to address this issue, there has been significant work on building evaluation datasets for stereotypes, using combinations of crowd-sourcing and web-text scraping. some notable work in english language include stereoset @xcite , that has stereotypes across 4 different dimensions -race, gender, religion, and profession; crows-pairs @xcite , which is a crowd-sourced dataset that contains sentences covering 9 dimensions such as race, gender, and nationality. @xcite introduce french crows-pairs containing stereotypical and antistereotypical sentence-pairs in french. @xcite cover stereotypes in the indian context. additionally, there are studies that have collected stereotypes for different sub-groups as part of social psychological research @xcite @xcite . while they add immense value to measuring stereotyping harms, the above datasets are limited in that they contain stereotypes only widely known in one specific region (such as the united states, or india), are small in size with limited coverage of stereotypes, and reflect limited world views. (such as the western context). alternately, for scalable downstream evaluations of fairness of models, artificially constructed datasets @xcite @xcite that test for preferential association of descriptive terms with specific identity group in tasks such as question answering and natural language inference, have been used. while they typically target stereotypical associations, they lack ground knowledge to differentiate them from spurious correlations, leading to vague measurements of 'bias @xcite . building resources with broad coverage of both identities of persons, and social stereotypes about them is pivotal towards holistic estimation of a model's safety when deployed. we demonstrate a way to achieve this coverage at scale by simulating a free-response, open-ended approach for capturing social stereotypes in a novel setting with llms. ## seegull: benchmark creation large language models (llms) are pre-trained on a subset of the real-world data @xcite @xcite which contains both implicit and explicit stereotypes @xcite . this makes llms a good candidate for generating stereotypes about geographical identity groups that exist around the globe. however, since generative models also gen-eralize well beyond the training data, they can generate statistical associations that look like stereotypes but are instead statistical noise. to filter out such stereotypical-looking noisy associations, we leverage a globally diverse rater-pool to validate the prevalence of the generated stereotype candidates in the society. we use a novel llm-human partnership to create a broad-coverage stereotype benchmark, seegull: stereotypes generated using llms in the loop, that captures a subset of the real-world stereotypes. our focus in this paper is on broad geocultural coverage of stereotype evaluation in english nlp for two primary reasons. first, english nlp sees disproportionately more research/resources/benchmarks, and is increasingly being deployed in products across the globe. hence there is an immediate need for making evaluation resources (including stereotype benchmarks) in english itself that have global/cross-cultural coverage. secondly, this is in line with recent calls @xcite @xcite to look beyond cross-lingual nlp and build cross-cultural competence in ai/nlp. our work is a first step towards this goal w.r.t. stereotype evaluations, and we envision future work expanding it to multilingual coverage.there are two main steps in creating seegull: (i) stereotype generation using llms, and (ii) human validation of the generated associations. figure 2 presents an overview of the overall approach. ## seegull: characteristics and utility in this section we discuss the characteristics, coverage, and utility of the resource created. seegull also captures the regional sensitivity (rs) of stereotype perceptions by situating them in different societal contexts (described in section 5.1), unlike existing benchmarks that present stereotypes only in a singular context. addition- ## conclusion we employ a novel llm-human partnership based approach to create a unique stereotype benchmark, seegull, that covers a geo-culturally broad range of stereotypes about 179 identity groups spanning 8 different regions and 6 continents. in addition to stereotypes at a global level for nationality, the dataset also contains state-level stereotypes for 50 us states, and 31 indian states and union territories. we leverage the few-shot capabilities of llms such as palm, gpt-3, and t0 and get a salience score that demonstrates the uniqueness of the associations as generated by llms. we also get annotations from a geographically diverse rater pool and demonstrate the contextual nature and the regional sensitivity of these stereotypes. further, we investigate the offensiveness of the stereotypes collected in the dataset. the scale and coverage of the dataset enable development of different fairness evaluation paradigms that are contextual, decentralized from a western focus to a global perspective, thus enabling better representation of global stereotypes in measurements of harm in language technologies.