Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you? (2021)

Abstract: In this paper, we investigate what types of stereotypical information are captured by pretrained language models. We present the first dataset comprising stereotypical attributes of a range of social groups and propose a method to elicit stereotypes encoded by pretrained language models in an unsupervised fashion. Moreover, we link the emergent stereotypes to their manifestation as basic emotions as a means to study their emotional effects in a more generalized manner. To demonstrate how our methods can be used to analyze emotion and stereotype shifts due to linguistic experience, we use fine-tuning on news sources as a case study. Our experiments expose how attitudes towards different social groups vary across models and how quickly emotions and stereotypes can shift at the fine-tuning stage.

## introduction pretraining strategies for large-scale language models (lms) require unsupervised training on large amounts of human generated text data. while highly successful, these methods come at the cost of interpretability as it has become increasingly unclear what relationships they capture. yet, as their presence in society increases, so does the importance of recognising the role they play in perpetuating social biases. in this regard, @xcite first discovered that contextualized word representations reflect gender biases captured in the training data. what followed was a suite of studies that aimed to quantify and mitigate the effect of harmful social biases in word @xcite and sentence encoders @xcite . despite these studies, it has remained difficult to define what constitutes "bias", with most work focusing on "gender bias" @xcite or "racial bias" @xcite . more broadly, biases in the models can comprise a wide range of harmful behaviors that may affect different social groups for various reasons @xcite . in this work, we take a different focus and study stereotypes that emerge within pretrained lms instead. while bias is a personal preference that can be harmful when the tendency interferes with the ability to be impartial, stereotypes can be defined as a preconceived idea that (incorrectly) attributes general characteristics to all members of a group. while the two concepts are closely related i.e., stereotypes can evoke new biases or reinforce existing ones, stereotypical thinking appears to be a crucial part of human cognition that often emerges implicitly @xcite . @xcite argued that implicit stereotypical associations are established through bayesian principles, where the experience of their prevalence in the world of the perceiver causes the association. thus, as stereotypical associations are not solely reflections of cognitive bias but also stem from real data, we suspect that our models, like human individuals, pick up on these associations. this is particularly true given that their knowledge is largely considered to be a reflection of the data they are trained on. yet, while we consider stereotypical thinking to be a natural sideeffect of learning, it is still important to be aware of the stereotypes that models encode. psychology studies show that beliefs about social groups are transmitted and shaped through language @xcite . thus, specific lexical choices in downstream applications not only reflect the model's attitude towards groups but may also influence the audience's reaction to it, thereby inadvertently propagating the stereotypes they capture @xcite . studies focused on measuring stereotypes in pretrained models have thus far taken supervised approaches, relying on human knowledge of common stereotypes about (a smaller set of) social groups @xcite . this, however, bears a few disadvantages: (1) due to the implicit nature of stereotypes, human defined examples can only expose a subset of popular stereotypes, but will omit those that human annotators are unaware of (e.g. models might encode stereotypes that are not as prevalent in the real world); we present the first dataset of stereotypical attributes of a wide range of social groups, comprising ∼ 2k attributes in total. furthermore, we propose a stereotype elicitation method that enables the retrieval of salient attributes of social groups encoded by state-of-the-art lms in an unsupervised manner. we use this method to test the extent to which models encode the human stereotypes captured in our dataset. moreover, we are the first to demonstrate how training data at the finetuning stage can directly affect stereotypical associations within the models. in addition, we propose a complementary method to study stereotypes in a more generalized way through the use of emotion profiles, and systematically compare the emerging emotion profiles for different social groups across models. we find that all models vary considerably in the information they encode, with some models being overall more negatively biased while others are mostly positive instead. yet, in contrast to previous work, this study is not meant to advocate the need for debiasing. instead, it is meant to expose varying implicit stereotypes that different models incorporate and to bring awareness to how quickly attitudes towards groups change based on contextual differences in the training data used both at the pretraining and fine-tuning stage. to the best of our knowledge, nadeem et al. (2020); nangia et al. (2020) and manela et al. (2021) are the first to explicitly study stereotypes in pretrained sentence encoders. while manela et al. (2021) focus on gender stereotypes using the wino-bias dataset @xcite , the other works propose new crowdsourced datasets (i.e. stereoset and crowspair) with stereotypes that cover a wide range of social groups. all datasets, however, have a similar set-up: they contain pairs of sentences of which one is more stereotypical than the other. working in the language modeling framework, they evaluated whether the model "prefers" the stereotypical sentence over the anti-stereotypical one. in contrast, we propose a different experimental setup and introduce a new dataset that leverages search engines' autocomplete suggestions for the acquisition of explicit stereotypical attributes. instead of indirectly uncovering stereotypes through comparison, our elicitation method directly retrieves salient attributes encoded in the models. our technique is inspired by @xcite , but while they measure the lm probability for completing sentences with the pronouns she and he specifically, we study the top k salient attributes without posing any restrictions on what these could be. moreover, we are the first to include both monolingual and multilingual models in our analysis. ## stereotypes from search engines retrieving human stereotypes in an implicit manner can be useful as people are likely to give more politically correct answers when asked for stereotypes explicitly. questions we ask to search engines are often done in the comfort of our own homes, making them likely to reflect true stereotypes that are out there in the real world @xcite . when feeding search engines queries that already imply a stereotype about a specific social group, the suggested completions are often salient (i.e. stereotypical) attributes of that particular group e.g., search query: 'why are asian parents so' → completion: 'strict'. as the autocomplete suggestions are based on popular human generated queries, we exploit this property for the acquisition of stereotypical attributes. social groups we use 382 social groups which we divide into 9 categories: age (11), gender (26), profession (115), race (99), country (73), religion (10), political (9), sexuality (12) and lifestyle (19). see appendix c for the full lists of social groups per category. the social groups include, but are not limited to, the ones used by nadeem et al. (2020) who collected wikidata relation triples to automatically retrieve target terms. templates to increase coverage, we design 5 slightly varying templates to retrieve autocomplete suggestions for. for groups pertaining to people we use the templates: • 'why are [tgt] so [attr] ?' • 'what makes [tgt] so [attr]?' • 'how come [tgt] are so [attr]?' • 'why are [tgt] always so [attr]?' • 'why are all [tgt] so [attr]?' for countries we use: • 'why is [tgt] so [attr] ?' • 'what makes [tgt] so [attr]?' • 'how come [tgt] is so [attr]?' • 'why is [tgt] always so [attr]?' • 'why are all people in [tgt] so [attr]?' where [tgt] are social groups for which we search stereotypes and [attr] search engines due to google's hate speech filtering system the autocompletion feature is disabled for frequently targeted groups e.g. black people, jewish people and members of the lgbtq+ community. thus, we retrieve autocomplete suggestions from 3 search engines: google, yahoo and duckduckgo. in many cases, identical completions were given by multiple search engines. we sort these duplicate samples under the category 'multiple engines'. we find that most negative (offensive) stereotypes are retrieved from yahoo. pre-processing we clean up the dataset manually, using the following procedure: the final dataset contains ∼2k stereotypes about 274 social groups. the stereotypes are distributed across categories as follows -profession: 713, race: 412, country: 396, gender: 198, age: 171, lifestyle: 123, political: 50, religion: 36. none of the search engines produce stereotypical autocomplete suggestions for members of the lgbtq+ community. in ## correlating human stereotypes with salient attributes in pretrained models to test for human stereotypes, we propose a stereotype elicitation method that is inspired by cloze testing, a technique that stems from psycholinguistics. using our method we retrieve salient attributes from the model in an unsupervised manner and compute recall scores over the stereotypes captured in our search engine dataset. pretrained models we study different types of pretrained lms of which 3 are monolingual and 2 multilingual: bert @xcite uncased trained on the bookscorpus dataset @xcite and english wikipedia; roberta @xcite , the optimized version of bert that is in addition trained on data from commoncrawl news @xcite , openwebtextcorpus @xcite and stories (trinh and le, 2018); bart, a denoising autoencoder @xcite ) that while using a different architecture and pretraining strategy from roberta, uses the same training data. moreover, we use mbert, that apart from being trained on wikipedia in multiple languages, is identical to bert. we use the uncased version that supports 102 languages. similarly, xlm-r is the multilingual variant of roberta @xcite ) that is trained on cleaned commoncrawl data @xcite ) and 1 although incompatible with our set-up, we do not remove them from the dataset as they can be valuable in future studies. results figure 1 shows the recall@k scores per model separated by category, showcasing the ability to directly retrieve stereotypical attributes of social groups using our elicitation method. while models capture the human stereotypes to similar extents, results vary when comparing across categories with most models obtaining the highest recall for country stereotypes. multilingual models obtain relatively low scores when recalling stereotypical attributes pertaining to age, gender and political groups. yet, xlmr-l is scoring relatively high on stereotypical profession and race attributes. the suboptimal performance of multilingual models could be explained in different ways. for instance, as multilingual models are known to suffer from negative interference @xcite , their quality on individual languages is lower compared to monolingual models, due to limited model capacity. this could result in a loss of stereotypical information. alternatively, multilingual models are trained on more culturally diverse data, thus conflicting information could counteract within the model with stereotypes from different languages dampening each other's effect. cultural differences might also be more pronounced when it comes to e.g. age and gender, whilst profession and race stereotypes might be established more universally. ## quantifying emotion towards different social groups to study stereotypes through emotion, we draw inspiration from psychology studies showing that stereotypes evoke distinct emotions based on different types of perceived threats @xcite or perceived social status and competitiveness of the targeted group @xcite . for instance, @xcite show that both feminists and african americans elicit anger, but while the former group is perceived as a threat to social values, the latter is perceived as a threat to property instead. thus, the stereotypes that underlie the emotion are likely different. whilst strong emotions are not evidence of stereotypes per se, they do suggest the powerful effects of subtle biases captured in the model. thus, the study into emotion profiles provides us with a good starting point to identify which stereotypes associated with the social groups evoke those emotions. to this end, we (1) build emotion profiles for social groups in the models and ( 2 ) retrieve stereotypes about the groups that most strongly elicit emotions. ## stereotype shifts during fine-tuning many debiasing studies intervene at the data level e.g., by augmenting imbalanced datasets @xcite @xcite or reducing annotator bias @xcite . these methods are, however, dependent on the dataset, domain, or task, making new mitigation needed when transferring to a new setup @xcite . this raises the question of how emotion profiles and stereotypes are established through language use, and how they might shift due to new linguistic experience at the fine-tuning stage. we take u.s. news sources from across the political spectrum as a case study, as media outlets are known to be biased @xcite . by revealing stereotypes learned as an effect of fine-tuning on a specific source, we can trace the newly learned stereotypes back to the respective source. we rely on the political bias categorisation of news sources from the allsides foot_0 media bias rating website. these ratings are retrieved using multiple 100% 50% 25% 10% proportion of data 0.6 0.5 0.4 0.3 0.2 0.1 0.0 spearman correlation source new yorker guardian reuters fox news breitbart figure 5: decrease in spearman correlation (∆ρ) after fine-tuning the pretrained models compared to no finetuning (∆ρ = 1) (no correlation left:∆ρ = -1). we show results for models trained on varying proportions of the data. results are averaged over categories and standard deviations are indicated by error bars. results we find that fine-tuning on news sources can directly alter the encoded stereotypes. rugby are lost. moreover, from fine-tuning on both breitbart and fox news the association emerges that black women are violent, while this is not the case when fine-tuning on the other sources. in fact, guardian and breitbart are the only news sources that result in the encoding of the salient attribute racist for white americans. we find that such shifts are already visible after training on as little as 25% of the original data (∼ 1k articles). when comparing to human stereotypes, we find that fine-tuning on reuters decreases the overall recall scores (see figure 4 ). although new yorker exhibits a similar trend, fine-tuning on the other sources have little effect on the number of stereotypes recalled from the dataset. as reuters has a center bias rating i.e., it does not predictably favor either end of the political spectrum, we speculate that large amounts of more nuanced data helps transmit fewer stereotypes. figure 5 shows the decrease in correlation between the emotion profiles from pretrained bert-b and bert-b fine-tuned on different proportions of the data. interestingly, fine-tuning on less articles does not automatically result in smaller changes to the models. in fact, in many cases, the amount of relative change in emotion profiles is heavily dependent on the social category as indicated by the error bars. this is not unexpected as news sources might propagate stronger opinions about specific categories. moreover, we find that emotions towards different social categories cannot always be distinguished by the political bias of the source. figure 3 , shows how news sources compare to each other w.r.t. different social categories, exposing that e.g. guardian and fox news show lower correlation on gender than on age. computing correlation between all pretrained and fine-tuned models, we find that emotion profiles are prone to change irrespective of model or news source (see appendix e). in figure 6 , we showcase the effect of fine-tuning from the model that exhibits the lowest change in correlation, i.e. roberta-b, to highlight how quickly emotions shift. we find that while reuters results in weaker emotional responses, guardian elicits stronger negative emotions than fox news e.g. towards conservatives and academics. yet, while both sources result in anger towards similar groups, for fox news anger is more often accompanied with fear while for guardian this seems to more strongly stems from disgust (e.g. see christians and iraq). lastly, figure 7 shows specific stereotype shifts found on the top 15 predictions per template. we illustrate the salient attributes that are removed, added and remained constant after fine-tuning. for instance, the role of news media in shaping public opinion about police has received much attention in the wake of the growing polarization over highprofile incidents @xcite . we find clear evidence of this polarization as fine-tuning on new yorker results in attributes such as cold, unreliable, deadly and inept, yet, finetuning on fox news yields positive associations such as polite, loyal, cautious and exceptional. in addition, we find evidence for other stark contrasts such as the model picking up on sexist (e.g. women are not interesting and equal but late, insecure and entitled) and racist stereotypes (e.g. black people are not misunderstood and powerful, but bitter, rude and stubborn) after fine-tuning on fox news. ## conclusion we present the first dataset containing stereotypical attributes of a range of social groups. importantly, our data acquisition technique enables the inexpensive retrieval of similar datasets in the future, enabling comparative analysis on stereotype shifts over time. additionally, our proposed methods could inspire future work on analyzing the effect of training data content, and simultaneously contribute to the field of social psychology by providing a testbed for studies on how stereotypes emerge from linguistic experience. to this end, we have shown that our methods can be used to identify stereotypes evoked during fine-tuning by taking news sources as a case study. moreover, we have exposed how quickly stereotypes and emotions shift based on training data content, and linked stereotypes to their manifestations as emotions to quantify and compare attitudes towards groups within lms. we plan to extent our approach to more languages in future work to collect different, more culturally dependent, stereotypes as well.