Theory-Grounded Measurement of U . S . Social Stereotypes in E nglish Language Models (2022)

Abstract: NLP models trained on text have been shown to reproduce human stereotypes, which can magnify harms to marginalized groups when systems are deployed at scale. We adapt the Agency-Belief-Communion (ABC) stereotype model of Koch et al. (2016) from social psychology as a framework for the systematic study and discovery of stereotypic group-trait associations in language models (LMs). We introduce the sensitivity test (SeT) for measuring stereotypical associations from language models. To evaluate SeT and other measures using the ABC model, we collect group-trait judgments from U.S.-based subjects to compare with English LM stereotypes. Finally, we extend this framework to measure LM stereotyping of intersectional identities.

## introduction stereotypes are abstract and over-generalized pictures in people's minds that capture attributes about groups of people in the complex social world @xcite . they influence people's thoughts and behaviors, and allow people to make predictions beyond their personal experience or information given @xcite . stereotypes are also entwined with the production of prejudice, discrimination, and in-group favoritism @xcite . a long line of research in social psychology has established models of generic dimensions that estimate people's stereotypes of social groups @xcite . we build on the agency beliefs communion (abc) model, which measures stereotypes toward a social group with respect to 16 traits in three dimensions: agency/socioeconomic success, conservative-progressive beliefs, and com- * equal contribution. figure 1 : crowdsourced analysis of the social group "man" under the abc model @xcite . colors: @xmath0, @xmath1, @xmath2. munion ( §2); an analysis of the group "man" across 32 traits (16 opposing dyads) is shown in figure 1 . pre-trained language models (lms) encode correlations between social groups and traits, like associating the group "muslim" with the trait threatening, or "man" with confident (e.g., @xcite @xcite . we conduct a systematic study of social stereotypes in contextualized english masked lms, grounded in group-trait associations from the abc model. to capture the group-trait associations in the lm, we first assess two previously proposed word association tests and also propose a new measurement: the sensitivity test (set) ( §3). to evaluate the degree to which two lms-bert @xcite and roberta @xcite -align with human stereotype judgments, we design a human study for collecting group-trait judgments ( §4). we show that our measure, set, best aligns with human judgements on group-trait associations and find that, in general, the association from language models have moderate alignment with human judgements. finally, with the best-aligned association measurement, we extend the abc approach to study agency powerless ↔ powerful ## background and related work people's impressions of the world and the actions they take are guided by their stereotypes. to systematize this observation, the field of social psychology has proposed models of stereotypes, including traits that can coordinate social behaviors to serve as fundamental dimensions of stereotyping. some models are designed to focus on social evaluation towards individual persons @xcite , ingroup members (ellemers, 2017; yzerbyt, 2018), or a small set of outgroups @xcite ; the agency beliefs communion (abc) model-whose traits are designed to distinguish groups-is suited for a larger set of u.s. social groups @xcite . the abc model takes a data-driven strategy to select a set of traits by eliminating those that are less effective in capturing stereotypes. the list contains 16 pairs, where each pair represents two polarities (see ours is far from the first work to assess stereotypes in language models, and has both advantages and disadvantages compared to previous approaches (see ## measuring stereotypes in lms our goal is to measure stereotypes in (masked) lms, and compare them to stereotypes elicited from people. 2 in §4 we describe our approach for eliciting human judgments of group-trait affinities; https://github.com/tristacao/u.s _ stereotypes . ## human study in the previous section, we describe how we compute associations between groups and traits in lan-guage models. 5 in this section, we assess stereotypes of social groups through groups-trait association, like in figure 1 . we adopt this approach because it is widely used to evaluate group stereotypes in social psychology field @xcite . it also aligns with @xcite 's theory of stereotypes that they are abstract pictures in people's head. we broadly follow procedures from previous social psychology papers to collect human evaluation on social groups. ## results in this section we present results on correlations between human and model stereotypes for individual groups, comparing across different measurements, including our proposed measurement, set ( §5.1). next, we analyze how model scores change for intersectional social groups. we consider several possible factors that may influence the score changes such as identity order, some domain domination, and consider emergent traits ( §5.2). ## limitations and ethical considerations there are several limitations to our work, which should be taken into account in the interpretation of our results. first, our results are likely affected by reporting bias and by a defaulting effect where, when people annotate traits for "men", they may actually have in their head "cis straight white men", because the defaults go unremarked. this goes both for the human scores (how does a participant conceptualize "men"?) and language model scores (what do sentences containing the word "man" assume given that most language a langauge model has been trained on likely exhibits defaulting?). second, our work only focus on assessing stereotypes within language models and not in any deployed system. though stereotypes from language models may impact the outputs of downstream systems which are built upon these language models, it is not clear how exactly the stereotypes transfer @xcite . additionally, our work is limited to english and u.s. social stereotypes. third, although we followed and built on best practices from social psychology in developing the human study, it nevertheless has some shortcomings. in particular, even after many iterations on wording, it was difficult to phrase the survey questions to encourage people to reporting their true impressions. there is tension between asking a participant what they think-which risks a counfounding potential social desirability bias @xcite ) (people's tendency to respond in socially acceptable ways)-and asking what they think others think-which led to comments from a few participants that they felt unqualified to speak for others. asking these questions of participants and collecting the data also raises the possibility of this work inadvertantly reinforcing stereotypes. finally, aggregating human judgements into a single number by averaging (or any other statistic) to compare to model predictions risks collapsing a significant amount of information down to a single number. this number cannot distinguish between a weakly held but common stereotype and a strongly held but rare one. nor can it distinguish between traits where half of annotators say 0 and the other half say 100, from traits where all annotators say 50. these average judgments should be interpreted as not what any single person would say, but an average over people. this limitation is exacerbated by the defaulting effect, where some people may imagine a different prototype for a given group, and other people may imagine another. ## conclusion in this paper, we measured language model (lm) stereotypes by adopting the abc stereotype model from social psychology. comparing to previous work on detecting lm stereotypes, our approach is easy to extend to previously unconsidered groups, grounded in traits proven effective by social psychology, and exhaustively covering the space of possible stereotypes, at the cost of being more abstract than in other nlp work. this yields a different set of trade-offs than previous approaches to measuring stereotypes in lms. with the abc model and data regarding human stereotypes from our human study, we assessed lm stereotypes using three different association measurements, including set, a metric we proposed. we showed that lm group-trait stereotypes in general have moderate correlation with human judgements, and that set provides correlations that better align with human's. based on these results, we extended our analysis to intersectional groups. we found that the lm may be able to capture intersectional stereotypes but is not particularly good on identifying emergent intersectional stereotypes. our results also show that that, in general, age and political stance are dominant domains in language models, whereas race and nationality are dominated domains. we hope that our work provides insights for future works on measuring and mitigating stereotypes in natural language processing systems, and that the grounding in theories from social psychology has benefits beyond just studying stereotypes. grant no. 2131508. the authors are also grateful to all the reviewers who have provided helpful suggestions to improve this work, and thank members of the clip lab at the university of maryland for the support on this project. we are grateful to all those who participated in our human study, without whom this research would not have been possible.