Building Stereotype Repositories with Complementary Approaches for Scale and Depth (2023)

Abstract: Measurements of fairness in NLP have been critiqued for lacking concrete definitions of biases or harms measured, and for perpetuating a singular, Western narrative of fairness globally. To combat some of these pivotal issues, methods for curating datasets and benchmarks that target specific harms are rapidly emerging. However, these methods still face the significant challenge of achieving coverage over global cultures and perspectives at scale. To address this, in this paper, we highlight the utility and importance of complementary approaches that leverage both community engagement as well as large generative models, in these curation strategies. We specifically target the harm of stereotyping and demonstrate a pathway to build a benchmark that covers stereotypes about diverse, and intersectional identities. We discuss the two approaches, their advantages and constraints, the characteristics of the data they produce, and finally, their potential to be used complementarily for better evaluation of stereotyping harms.

## introduction generative language models are widely used in diverse global settings across applications such as writing assistants @xcite , search tools, 1 and more @xcite . recent years have seen immense progress in the development of such large language models @xcite @xcite , accompanied by detailed analysis of their abilities @xcite . recent work has demonstrated the need for assessing their potential risks and harms to be contextually situated within the specific global 1 https://openai.com/blog/chatgpt/ socio-cultural settings they are deployed in @xcite . this need in turn highlights the gaps in current evaluation paradigms, within which a vast majority of resources are in english language, and/or is limited to a western perspective of fairness and harms @xcite . this is especially troubling for evaluation benchmarks that require socially situated resources, for instance, to assess stereotyping harms that vary across cultures. addressing this growing need for evaluation strategies to be more globally relevant has its own challenges. first, the scale of operation becomes massive, given how diverse different languages and cultures are. every region has its own unique axes of identities and with varying granularity of inspection, a large possible number of unique and intersectional identities and associated harms need to be examined. second, stereotypes can be locally situated; some stereotypes are prevalent only within a region and can be about people residing in it or outside it. hence, a lack of involvement of some communities can result in major gaps in evaluations, leading to disparately increased risks to those communities. this is interlinked with the third challenge, of ensuring that our resources and evaluations are not dominated by a western perspective of what unfairness or stereotypes look like. in this paper, we first discuss the challenges and limitations of current paradigms of stereotype resource collection, which are rooted in the enormity of global scale, and differential prevalence of stereotypes in different contexts. we then propose and demonstrate using exemplar methods, how complementary investigations of stereotypes which target scale and depth can achieve greater coverage and address aforementioned challengesour first approach involves generation of candidate stereotypes using large language models (llms), followed by human annotations to verify which associations are stereotypical; the second approach involves reaching out to communities to directly collect the stereotypes known to them. ## complementary approaches to build stereotype resources stereotypes are generalizations about groups of people defined by their identity such as their gender, race, sexuality, age, etc. stereotyping when propagated though language technologies can lead to many harmful outcomes including misrepresentation, targeted hateful speech generation, disparate access to resources, and opportunities @xcite @xcite . there have been several efforts to build resources which document stereotypes in society @xcite , how they percolate into language technologies @xcite @xcite , and cause unfair model behavior @xcite . while existing stereotype resources are rich and enable model evaluations, most of them were collected by employing methods that rely on human annotations about statements describing a potential stereotype. however, stereotypes are not absolute, in that they vary by societies, communities, and individual experiences of people. any individual annotator will not be aware of all stereotypes present globally and can only confirm stereotypes they individually know of. as a result, annotations from sets of people or even stereotypical statements or text written down by people will still present a limited view of all stereotypes across the world. also, the statements or text that is annotated for presence of stereotypes is typically human generated, which is an additional challenge towards both scale and coverage of global identities and stereotypes. for broader coverage, llms can be imagined as a lens on the society, since they are trained over copious amounts of naturally occurring, humangenerated text that reflect the underlying societal context including social stereotypes. their generations attempt to mimic human knowledge and predispositions, and has been shown to reproduce stereotypes @xcite @xcite . consequently, they can, inexpensively create generalizations that are diverse and representative of a wide range of identities across the globe @xcite . so we can tap into the generalizing capabilities of llms to create a broad-coverage candidate set for stereotypes. however, llm generations are not always grounded factually, and reflect spurious correlations, and noise @xcite . hence, for usage as a stereotype resource, associations generated by llms about groups of people need to be validated for social presence of such stereotypes by human raters familiar with the corresponding socio-cultural contexts. on the other hand, llms may not capture all social stereotypes globally. while they are trained on large amounts of data, there are still gaps in global representativeness in such data @xcite , which will also carry over to stereotype resources built using llms. furthermore, since most state-of-the-art llms are trained on online data that has a western lens @xcite , the stereotypes we get through llms may also reflect this western gaze, and miss the nuances of stereotypes in local cultural contexts @xcite . hence, it is important to complement the llm-based approach with community engagements to build richer resources. methods that rely on community engagement are expensive and time consuming but help collect socially situated perspectives. when used in targeted ways to understand one specific culture or society, annotations, surveys, and free form data collection can provide depth and nuance to the collected stereotype resource. figure 1 imagines this juxtaposition of challenges and complementarity of community engaged and llm generation based approaches. if our goal is to uncover the universal set of all stereotypes in the world, different strategies are warranted. ideally, the results of community engagements, when deployed globally would overlap a 100% with this set. however, that would be expensive both cost and time wise to completely attain. meanwhile, if we consider a second set consisting of all associations llms generate with the identities of people, only a certain fraction of it would be socially present stereotypes. however, llm generations, combined with human annotations would give us a list of stereotypes which is represented in figure 1 as the intersection of these two sets. ## case study in this section, we summarize insights from two separate studies that take these complimentary approaches towards stereotype resource building, and outline their strengths and limitations. one approach crowd-sources stereotypes by engaging with communities, and the other uses generative models in conjunction with human annotations to scale coverage. 2 these complementary approaches can be extended globally, to different harms such as hateful speech, toxic language and so on, which are also geo-culturally and socially situated. ## discussion in the paper, we presented two approaches to expand the coverage of stereotype resources used to evaluate language technologies. while we demonstrated the advantages of each individual method, it is also important to note how the complementary usage of the methods can lead to broad, and granular coverage of stereotype harms globally. each method uncovered different kinds of stereotypes that were not found using the other. additionally, the output of one method can serve as the seed for the other; the stereotypes recovered from engaging with communities can be used as prompts in subsequent usage of the generative approach using llms. meanwhile, the generative approach highlights prevalence of associations and can help understand which communities to engage with for uncovering finer-grained stereotypes. further, the collection of non-overlapping, complementary sets of stereotypes enhances coverage both in terms of global communities covered as well as fine-grained identities present in different regions. measurements of harm in language tasks like question answering @xcite and natural language inference @xcite which are built on preferential associations with identities can leverage this more comprehensive list to make more holistic estimations.